Files

T

2026-09-16 10:41:17 +07:00

61 KiB

Raw Blame History

Specification: msghandler

Version: 1.4.0
Date: 2026-09-16
Status: Active
Ground Truth: src/msghandler.jl
Specification Format: JSON Schema + AsyncAPI
ASG Framework Alignment: Eight Pillars - Requirements → Solution Design → Walkthrough → Specification → UI Specification → Validation → Implementation Plan → Runbook Specification Item IDs: SP-XXX (see Specification Traceability Matrix); every spec item cites its Requirement ID(s) (FR-XXX/NFR-XXX) and Solution Design reference(s) (SD-XXX).

1. Technical Contract Overview

This document defines the technical contract for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between Julia, JavaScript, Python, Rust, and MicroPython applications using a message broker as the transport layer.

This specification serves as the single source of truth for:

Inputs: What data structures are accepted by smartpack()
Outputs: What data structures are returned by smartunpack()
Data Shapes: Exact field names, types, and constraints
Error Codes: Standardized error responses for failure scenarios

1.1 Requirements Traceability

Per the ASG Framework, every technical contract below carries a unique SP-XXX Specification Item ID and cites its Requirement ID(s) (the what) and Solution Design reference(s) (the how/why). The canonical cross-reference is the Specification Traceability Matrix.

| Specification Section | Spec ID (SP) | Requirement ID(s) | Solution Design Ref(s) | Description | |----------------------|-------------------|------------------------|-------------| | Section 2 (Message Envelope) | SP-001 | FR-012, FR-013, NFR-101, NFR-102 | SD-008 | Message envelope structure and validation | | Section 3 (Payload Schema) | SP-002 | FR-001, FR-002, FR-003, FR-004, NFR-101, NFR-102 | SD-001, SD-005 | Payload structure and field definitions | | Section 4 (Payload Format) | SP-003 | FR-006, FR-007 | SD-004 | Tuple format for smartpack() | | Section 5 (Enumerations) | SP-004..SP-007 | FR-003, FR-004, FR-006, NFR-101 | SD-001, SD-002, SD-005 | Enumerations for transport and encoding | | Section 6 (Transport Protocols) | SP-008, SP-009 | FR-003, FR-004, NFR-104, NFR-105 | SD-001, SD-002 | Direct and link transport protocols | | Section 7 (Size Thresholds) | SP-010 | FR-004, FR-005, NFR-104, NFR-105 | SD-002 | Size thresholds for transport selection | | Section 8 (Topic Convention) | SP-011 | FR-013, FR-014 | SD-006 | Topic/subject naming patterns | | Section 9 (Error Handling) | SP-012..SP-014 | FR-010, FR-011, NFR-201, NFR-202, NFR-203 | SD-003, SD-007 | Error codes and exception handling | | Section 10 (Serialization Rules) | SP-015 | FR-001, FR-002, FR-003, FR-012, NFR-101, NFR-102 | SD-005 | Serialization and encoding rules | | Section 11 (API Contract) | SP-016, SP-017 | FR-001 through FR-014, NFR-101 through NFR-107 | SD-001 through SD-008 | Function signatures for all platforms | | Section 12 (File Server Interface) | SP-018, SP-019 | FR-008, FR-009, FR-010 | SD-003, SD-007 | Upload and download handler contracts | | Section 13 (Platform-Specific Constraints) | SP-020 | FR-005, FR-006, NFR-106, NFR-107 | SD-004, SD-006 | Platform-specific feature support | | Section 14 (Implementation Files) | SP-021 | FR-001 through FR-007 | - | Implementation file mapping | | Section 15 (Message Flow) | SP-022 | FR-001 through FR-014, NFR-101 through NFR-107 | SD-001 through SD-008 | Mermaid diagrams for send/receive flows | | Section 16 (Validation Rules) | SP-023 | FR-001 through FR-014, NFR-101 through NFR-405 | SD-001 through SD-008 | Envelope and payload validation rules | | Section 17 (Test Contracts) | SP-024 | FR-001 through FR-014, NFR-101 through NFR-107 | SD-001 through SD-008 | Unit and integration test scenarios | | Section 18 (Dependencies) | SP-025 | FR-001 through FR-014, NFR-101 through NFR-107 | SD-001 through SD-008 | Platform-specific dependencies | | Section 19 (Change Log) | — | — | — | Version history and changes |

Specification Traceability Matrix

Canonical mapping of each SP-XXX Specification Item ID to its Requirement IDs and Solution Design references, per the ASG Framework. Every SP-XXX below is implemented by at least one file under src/ (ground truth: src/msghandler.jl).

SP ID	Specification Item	Requirement ID(s)	Solution Design Ref	Implementation File
SP-001	Message Envelope Schema (`envelope_to_json`)	FR-012, FR-013, NFR-101, NFR-102	SD-008	msghandler.jl
SP-002	Payload Schema	FR-001, FR-002, FR-003, FR-004, NFR-101, NFR-102	SD-001, SD-005	msghandler.jl
SP-003	Payload Tuple Format `("name", data, "type")`	FR-006, FR-007	SD-004	msghandler.jl
SP-004	`msg_purpose` Enum (ACK/NACK/updateStatus/shutdown/chat/command/event)	FR-003, FR-004, FR-006, NFR-101	SD-001	msghandler.jl
SP-005	`payload_type` Enum	FR-003, FR-004, FR-006, NFR-101	SD-002	msghandler.jl
SP-006	`transport` Enum (direct/link)	FR-003, FR-004, NFR-104, NFR-105	SD-001, SD-002	msghandler.jl
SP-007	`encoding` Enum (none/base64/json/arrow-ipc)	FR-001, FR-002, FR-003, FR-012	SD-005	msghandler.jl
SP-008	Direct Transport Protocol	FR-003, FR-004, NFR-104, NFR-105	SD-001, SD-002	msghandler.jl
SP-009	Link Transport Protocol (Claim-Check / file server)	FR-003, FR-004, NFR-104, NFR-105	SD-001, SD-002	msghandler.jl
SP-010	Size Thresholds (10,000 bytes desktop; 100,000 MicroPython)	FR-004, FR-005, NFR-104, NFR-105	SD-002	msghandler.jl
SP-011	Topic/Subject Convention `<scope>/<service>/<version>/<operation>`	FR-013, FR-014	SD-006	msghandler.jl
SP-012	Error Response Format	FR-010, FR-011, NFR-201, NFR-202, NFR-203	SD-003, SD-007	msghandler.jl
SP-013	Error Codes	FR-010, FR-011, NFR-201, NFR-202, NFR-203	SD-003, SD-007	msghandler.jl
SP-014	Exception Handling / Retry Policy	FR-010, FR-011, NFR-201, NFR-202, NFR-203	SD-003, SD-007	msghandler.jl
SP-015	Serialization Rules	FR-001, FR-002, FR-003, FR-012, NFR-101, NFR-102	SD-005	msghandler.jl
SP-016	`smartpack()` API Contract	FR-001..FR-014, NFR-101..NFR-107	SD-001..SD-008	msghandler.jl
SP-017	`smartunpack()` API Contract	FR-001..FR-014, NFR-101..NFR-107	SD-001..SD-008	msghandler.jl
SP-018	Upload Handler Contract (`plik_oneshot_upload`)	FR-008, FR-009, FR-010	SD-003, SD-007	msghandler.jl
SP-019	Download Handler Contract (`_fetch_with_backoff`)	FR-010, FR-011, NFR-202	SD-003, SD-007	msghandler.jl
SP-020	Platform-Specific Constraints	FR-005, FR-006, NFR-106, NFR-107	SD-004, SD-006	msghandler.jl, msghandler-csr.js, msghandler.py, msghandler.rs, msghandler_mpy.py
SP-021	Implementation Files	FR-001..FR-007	—	msghandler.jl, msghandler-csr.js, msghandler.py, msghandler.rs, msghandler_mpy.py
SP-022	Message Flow (send/receive)	FR-001..FR-014, NFR-101..NFR-107	SD-001..SD-008	msghandler.jl
SP-023	Validation Rules	FR-001..FR-014, NFR-101..NFR-405	SD-001..SD-008	msghandler.jl
SP-024	Test Contracts	FR-001..FR-014, NFR-101..NFR-107	SD-001..SD-008	msghandler.jl
SP-025	Dependencies	FR-001..FR-014, NFR-101..NFR-107	SD-001..SD-008	msghandler.jl

Walkthrough → Specification Trace

Maps each WT-XXX walkthrough step (from 3-walkthrough.md) to the SP-XXX specification item(s) it exercises, closing the walkthrough↔specification gap-check. Derived from the walkthrough's WT-XXX → SD-XXX → FR-XXX trace combined with the SD-XXX → SP-XXX mapping above.

WT ID	Participant	Walkthrough Step	SP ID(s)
WT-001	JavaScript	Session start / transport connect	SP-011, SP-016
WT-002	JavaScript	Send mixed payloads (text + image)	SP-003, SP-015, SP-016
WT-002.1	JavaScript	Per-payload transport selection	SP-010, SP-008, SP-016
WT-003	JavaScript	Publish JSON via transport	SP-011, SP-016
WT-004	JavaScript	Envelope shape crosses the wire	SP-001
WT-005	JavaScript	Receive reply	SP-017
WT-006	JavaScript	Large file upload fails (link transport)	SP-009, SP-018, SP-013
WT-007	JavaScript	Transport connection lost (caller retry)	SP-011, SP-016
WT-010	Julia	Session start / subscribe	SP-011, SP-017
WT-011	Julia	Receive & unpack chat	SP-017, SP-015, SP-002
WT-011.1	Julia	Direct path base64 decode / deserialize	SP-015, SP-008, SP-002
WT-012	Julia	Send reply	SP-016, SP-001
WT-013	Julia	Receive large file (link transport)	SP-009, SP-019, SP-010
WT-014	Julia	Download fails after max_retries	SP-014, SP-019, SP-013
WT-015	Julia	Unknown payload type throws	SP-005, SP-013
WT-020	Python	Session start / connect	SP-011, SP-017
WT-021	Python	Send arrowtable	SP-002, SP-015, SP-016
WT-021.1	Python	Arrow IPC serialization (arrow-ipc)	SP-015, SP-010, SP-008
WT-022	Python	Receive results	SP-017, SP-015
WT-023	Python	Large arrowtable upload fails	SP-009, SP-018, SP-013
WT-024	Python	pyarrow not available	SP-005, SP-013, SP-020
WT-030	Rust	Session start / subscribe	SP-011, SP-016
WT-031	Rust	Receive & dispatch by payload_type	SP-017, SP-002, SP-005
WT-032	Rust	Send type-safe results	SP-016, SP-002
WT-033	Rust	Large file via link transport	SP-009, SP-018, SP-010
WT-034	Rust	Download exceeds retries	SP-014, SP-019, SP-013
WT-040	MicroPython	Session start / publish sensor reading	SP-011, SP-016
WT-041	MicroPython	Send dictionary (direct transport)	SP-002, SP-015, SP-008, SP-016
WT-041.1	MicroPython	JSON serialization, direct transport	SP-015, SP-010, SP-008
WT-042	MicroPython	Receive ACK	SP-017
WT-043	MicroPython	Threshold exceeded (direct-only)	SP-010, SP-009, SP-020
WT-044	MicroPython	Non-serializable dictionary	SP-005, SP-013, SP-020
WT-050	—	Cross-participant chat broadcast	SP-011, SP-022, SP-016

2. Message Envelope Schema

Specification Versioning

Component	Version	Notes
Specification	1.0.0	Initial release
Protocol	v1	Message envelope protocol version

Message Envelope Schema

Envelope Structure (JSON)

{
  "correlation_id": "string (UUID)",
  "msg_id": "string (UUID)",
  "timestamp": "string (ISO 8601 UTC)",
  "send_to": "string",
  "msg_purpose": "string",
  "sender_name": "string",
  "sender_id": "string (UUID)",
  "receiver_name": "string",
  "receiver_id": "string (UUID)",
  "reply_to": "string",
  "reply_to_msg_id": "string",
  "broker_url": "string",
  "metadata": "object",
  "payloads": [
    {
      "id": "string (UUID)",
      "dataname": "string",
      "payload_type": "string",
      "transport": "string",
      "encoding": "string",
      "size": "integer",
      "data": "string or URL",
      "metadata": "object"
    }
  ]
}

Field Definitions (SP-001)

SP ID	Field	Type	Required	Validation	Description	Requirement ID	Solution Design Ref
SP-001	`correlation_id`	`string`	Yes	UUID v4 format	Track message flow across distributed systems	FR-011, NFR-401	SD-008
SP-001	`msg_id`	`string`	Yes	UUID v4 format	Unique identifier for this specific message	FR-012, NFR-401	SD-001
SP-001	`timestamp`	`string`	Yes	ISO 8601 UTC	Message publication timestamp	FR-012, NFR-401	SD-001
SP-001	`send_to`	`string`	Yes	Non-empty string	Topic/subject to publish the message to	FR-013	SD-006
SP-001	`msg_purpose`	`string`	Yes	Enum	Purpose of the message	FR-013	SD-006
SP-001	`sender_name`	`string`	Yes	Non-empty string	Name of the sender application	FR-013	SD-006
SP-001	`receiver_name`	`string`	Yes	Any string	Name of the receiver (empty = broadcast)	FR-013	SD-006
SP-001	`receiver_id`	`string`	Yes	Any string	UUID of the receiver (empty = broadcast)	FR-013	SD-006
SP-001	`reply_to`	`string`	Yes	Any string	Topic where receiver should reply	FR-013	SD-006
SP-001	`reply_to_msg_id`	`string`	Yes	Any string	Message ID this message is replying to	FR-013	SD-006
SP-001	`broker_url`	`string`	Yes	Valid URL	Broker URL for the transport layer	FR-013	SD-006
SP-001	`metadata`	`object`	No	Any JSON object	Message-level metadata	NFR-401	SD-001, SD-008
SP-001	`payloads`	`array`	Yes	Non-empty array	List of payload objects	FR-012, FR-013	SD-001, SD-005

Payload Schema

Payload Structure (JSON)

{
  "id": "string (UUID)",
  "dataname": "string",
  "payload_type": "string",
  "transport": "string",
  "encoding": "string",
  "size": "integer",
  "data": "string or URL",
  "metadata": "object"
}

Payload Field Definitions (SP-002)

SP ID	Field	Type	Required	Validation	Description	Requirement ID	Solution Design Ref
SP-002	`id`	`string`	Yes	UUID v4 format	Unique identifier for this payload	FR-012	SD-001
SP-002	`dataname`	`string`	Yes	Non-empty string	Name of the payload (e.g., `login_image`, `user_data`)	FR-001, FR-002, FR-003	SD-004
SP-002	`payload_type`	`string`	Yes	Enum	Type of payload (see `payload_type` enum)	FR-001, FR-002, FR-003, FR-006	SD-004, SD-005
SP-002	`transport`	`string`	Yes	Enum	Transport method: `direct` or `link`	FR-003, FR-004, NFR-104, NFR-105	SD-001, SD-002
SP-002	`encoding`	`string`	Yes	Enum	Encoding method (see `encoding` enum)	FR-001, FR-002, FR-003, FR-012	SD-005
SP-002	`size`	`integer`	Yes	Positive integer	Size of the payload in bytes	FR-003, FR-004, NFR-104, NFR-105	SD-002
SP-002	`data`	`string` or `URL`	Yes	Base64 string or URL	Payload data (base64 for direct, URL for link)	FR-003, FR-004, FR-008, FR-009, FR-010	SD-001, SD-003
SP-002	`metadata`	`object`	No	Any JSON object	Payload-level metadata	NFR-401	SD-008

Payload Format (SP-003)

Tuple Format for `smartpack()`

The smartpack() function accepts data as an array of tuples with the format:

("data_name", data, "data_type")

Position	Type	Description	Example
1	`string`	Data name - identifier for the payload	`"msg"`, `"login_image"`, `"user_data"`
2	`any`	Actual data - content to be serialized	`"Hello"`, `{"key": "value"}`, `DataFrame(...)`
3	`string`	Data type - must be in `payload_type` enum	`"text"`, `"dictionary"`, `"arrowtable"`

Single Payload Example

# Julia
smartpack("/chat/user/v1/message", [("msg", "Hello World", "text")])

# Python
await smartpack("/chat/user/v1/message", [("msg", "Hello World", "text")])

// JavaScript
await smartpack("/chat/user/v1/message", [["msg", "Hello World", "text"]]);

Multiple Payloads Example

# Julia - Mixed text and binary data
data = [
    ("msg", "Hello", "text"),
    ("img", binary_data, "image")
]
smartpack("/agent/v1/process", data)

# Python - Mixed types
data = [
    ("msg", "Hello", "text"),
    ("img", binary_data, "image")
]
await smartpack("/agent/v1/process", data)

Data Type Mapping

Platform	Input Type	Data Type String
All	`String`	`"text"`
All	`Dict`/`Object`	`"dictionary"`
Desktop	`DataFrame`	`"arrowtable"` or `"jsontable"`
Browser	`Array` of objects	`"jsontable"` (only table type)
All	`Array` of objects	`"jsontable"`
All	`Uint8Array`/`Buffer`/`bytes`	`"binary"`
Desktop	`Arrow.Table`	`"arrowtable"`
All	Image/Audio/Video binary	`"image"`, `"audio"`, `"video"`

Enumerations (SP-004..SP-007)

`msg_purpose` Enum

Value	Description
`ACK`	Acknowledgment of successful message processing
`NACK`	Negative acknowledgment of message processing failure
`updateStatus`	Status update message
`shutdown`	Graceful shutdown request
`chat`	Chat/message payload
`command`	Command payload
`event`	Event payload

`payload_type` Enum

Value	Description	Supported Platforms	Encoding Options
`text`	Plain text string	All	`base64`
`dictionary`	JSON object/dictionary	All	`base64`, `json`
`arrowtable`	Apache Arrow IPC table	Desktop (Julia/Python)	`base64`, `arrow-ipc`
`jsontable`	JSON array of objects	All (including Browser)	`base64`, `json`
`image`	Binary image data	All	`base64`
`audio`	Binary audio data	All	`base64`
`video`	Binary video data	All	`base64`
`binary`	Generic binary data	All	`base64`

`transport` Enum

Value	Description	Data Format	Use Case
`direct`	Payload sent directly via the transport layer	Base64-encoded string	Payloads < size_threshold
`link`	Payload uploaded to file server	HTTP URL	Payloads ≥ size_threshold

`encoding` Enum

Value	Description	Payload Types
`none`	No additional encoding	Link transport URLs
`base64`	Base64 encoding	Text, binary, image, audio, video
`json`	JSON encoding	Dictionary, jsontable
`arrow-ipc`	Apache Arrow IPC format	Arrowtable

Transport Protocols (SP-008, SP-009)

Direct Transport Protocol (SP-008)

When transport = "direct", the data field contains a Base64-encoded string of the serialized payload.

Flow:

Serialize payload according to payload_type
Encode serialized bytes as Base64
Include Base64 string in data field

Example:

{
  "transport": "direct",
  "encoding": "base64",
  "size": 11,
  "data": "SGVsbG8gV29ybGQ="
}

Link Transport Protocol (SP-009)

When transport = "link", the data field contains a URL pointing to the uploaded payload.

Flow:

Serialize payload according to payload_type
Upload to HTTP file server (e.g., Plik)
Include returned URL in data field

Example:

{
  "transport": "link",
  "encoding": "none",
  "size": 1000000,
  "data": "http://localhost:8080/file/3F62E/4AgGT/data.zip"
}

Size Thresholds (SP-010)

Desktop Platforms (Julia/JS/Python)

Platform	Size Threshold	Notes	Requirement ID	Solution Design Ref
Desktop	10,000 bytes (10KB)	Default threshold	FR-004, FR-005, NFR-104, NFR-105	SD-002

MicroPython Platform

Platform	Size Threshold	Maximum Payload	Notes	Requirement ID	Solution Design Ref
MicroPython	100,000 bytes (100KB)	50,000 bytes	Hard limit due to memory constraints	FR-005, NFR-106	SD-002

Rate Limiting & Quotas (SP-010.1)

Per the ASG Framework content guidelines, this section defines rate-limit and quota behavior. msghandler is a transport-agnostic bridge and does not implement rate limiting or quotas itself; those are governed by the underlying transport layer (broker) and the file server.

Level	Rate Limit / Quota	Owner	Notes	SP ID
Transport (broker)	Publish/subscribe rate limits defined by the broker	Message broker (NATS / MQTT / WebSocket)	Enforced by the broker, not by msghandler	SP-010.1
File server	Per-file and per-session upload/download size limits	File server (Plik or custom upload/download handler)	Enforced by the upload/download handler	SP-010.1
Payload	10,000-byte default threshold (direct vs link)	msghandler (`size_threshold`, configurable)	Applied per payload in `smartpack()`	SP-010
MicroPython	50,000-byte hard per-payload limit	msghandler	Hard limit due to memory constraints	SP-010

Topic Convention (SP-011)

Subject Naming Pattern

<scope>/<service>/<version>/<operation>

Examples:

/agent/wine/api/v1/prompt - AI agent prompt endpoint | FR-013 | SD-006
/chat/user/v1/message - User chat message | FR-013 | SD-006
/system/worker/v1/status - Worker status update | FR-013 | SD-006

Subject Wildcards

Wildcard	Description	Example
`*`	Single-level wildcard	`/chat/user/v1/*` matches `/chat/user/v1/message`
`>`	Multi-level wildcard	`/chat/user/v1/>` matches all `/chat/user/v1/*` subjects

Error Handling (SP-012..SP-014)

Error Response Format (SP-012)

{
  "error": {
    "code": "string",
    "message": "string",
    "details": "object"
  }
}

Error Codes (SP-013)

Code	HTTP Status	Description	Recovery	Requirement ID
`INVALID_ENVELOPE`	400	Message envelope validation failed	Fix envelope structure	FR-012, FR-013, FR-014
`INVALID_PAYLOAD_TYPE`	400	Unsupported payload type	Use supported payload_type	FR-001, FR-002, FR-003, FR-006
`INVALID_TRANSPORT`	400	Unsupported transport type	Use `direct` or `link`	FR-003, FR-004, FR-006
`UPLOAD_FAILED`	500	File server upload failed	Retry or use direct transport	FR-008, FR-009
`DOWNLOAD_FAILED`	503	File server download failed	Retry with exponential backoff	FR-010, FR-011, NFR-201, NFR-202
`TRANSPORT_CONNECTION_FAILED`	503	Transport connection failed	Check broker/server availability	FR-013, FR-014, NFR-201, NFR-203
`DESERIALIZATION_ERROR`	500	Payload deserialization failed	Check payload_type matches data	FR-001, FR-002, FR-003, FR-012
`SIZE_EXCEEDED`	413	Payload exceeds maximum size	Split payload or use link transport	FR-003, FR-004, FR-005, NFR-104, NFR-105

Exception Handling (SP-014)

Scenario	Handler	Retry Policy	Requirement ID
File server unavailable	Retry up to 5 times	Exponential backoff (100ms → 5000ms)	FR-010, NFR-202
Transport publish failure	Handled by caller	Caller's retry logic	FR-013, FR-014, NFR-201, NFR-203
Deserialization error	Log correlation ID and throw	No retry (data corruption)	FR-001, FR-002, FR-003, FR-012, NFR-401
Memory overflow (MicroPython)	Reject payloads >50KB	No retry (client-side check)	FR-005, NFR-106

Serialization Rules (SP-015)

Text Serialization

Platform	Input Type	Serialization	Encoding
All	`String`	UTF-8 bytes	Base64

Dictionary Serialization

Platform	Input Type	Serialization	Encoding
All	`Object`/`Dict`	JSON string	Base64 or direct JSON

Arrow Table Serialization

Platform	Input Type	Serialization	Encoding
Desktop	`DataFrame`	Arrow IPC stream	Base64 or arrow-ipc
Desktop	`Arrow.Table`	Arrow IPC stream	Base64 or arrow-ipc
MicroPython	❌	Not supported	N/A

JSON Table Serialization

Platform	Input Type	Serialization	Encoding
All	`Vector{Dict}`/`Array<Object>`	JSON array	Base64 or direct JSON
Desktop	`pandas.DataFrame`	JSON array	Base64 or direct JSON

Binary Serialization

Platform	Input Type	Serialization	Encoding
All	`Uint8Array`/`Buffer`/`bytes`	Raw bytes	Base64

API Contract (SP-016, SP-017)

Platform	Function	Requirements	Solution Design Ref
All	`smartpack()`	FR-001 through FR-014, NFR-101 through NFR-107	SD-001 through SD-008
All	`smartunpack()`	FR-001 through FR-014, NFR-101 through NFR-107	SD-001 through SD-008

`smartpack` Function Signature (SP-016)

Julia

function smartpack(
    subject::String,
    data::AbstractArray{Tuple{String, T1, String}, 1};
    broker_url::String = DEFAULT_BROKER_URL,
    fileserver_url::String = DEFAULT_FILESERVER_URL,
    fileserver_upload_handler::Function = plik_oneshot_upload,
    size_threshold::Int = DEFAULT_SIZE_THRESHOLD,
    correlation_id::String = string(uuid4()),
    msg_purpose::String = "chat",
    sender_name::String = "msghandler",
    receiver_name::String = "",
    receiver_id::String = "",
    reply_to::String = "",
    reply_to_msg_id::String = "",
    msg_id::String = string(uuid4()),
    sender_id::String = string(uuid4())
)::Tuple{msg_envelope_v1, String} where {T1<:Any}

Note: Publishing via the transport layer is the caller's responsibility. Returns (env::msg_envelope_v1, env_json_str::String).

Python

async def smartpack(
    subject: str,
    data: List[Tuple[str, Any, str]],
    broker_url: str = DEFAULT_BROKER_URL,
    fileserver_url: str = "http://localhost:8080",
    fileserver_upload_handler: Callable = plik_oneshot_upload,
    size_threshold: int = 10_000,
    correlation_id: str = None,
    msg_purpose: str = "chat",
    sender_name: str = "msghandler",
    receiver_name: str = "",
    receiver_id: str = "",
    reply_to: str = "",
    reply_to_msg_id: str = "",
    msg_id: str = None,
    sender_id: str = None
) -> Tuple[Dict, str]:

Note: Publishing via the transport layer is the caller's responsibility.

JavaScript (Browser)

async function smartpack(
    subject: string,
    data: Array<[string, any, string]>,
    options?: {
        broker_url?: string;
        fileserver_url?: string;
        fileserver_upload_handler?: Function;
        size_threshold?: number;
        correlation_id?: string;
        msg_purpose?: string;
        sender_name?: string;
        receiver_name?: string;
        receiver_id?: string;
        reply_to?: string;
        reply_to_msg_id?: string;
        msg_id?: string;
        sender_id?: string;
    }
): Promise<[Object, string]>;

Note: Publishing via the transport layer is the caller's responsibility.

MicroPython

def smartpack(
    subject: str,
    data: List[Tuple[str, Any, str]],
    size_threshold: int = 100_000,  # Lower threshold for memory constraints
    **kwargs
) -> Tuple[Dict, str]:

Note: Publishing via the transport layer is the caller's responsibility.

Rust

pub async fn smartpack(
    subject: &str,
    data: &[(String, Payload, String)],
    options: &smartpackOptions,
) -> Result<(MsgEnvelopeV1, String), msghandlerError>

// smartpackOptions struct
pub struct smartpackOptions {
    pub broker_url: String,
    pub fileserver_url: String,
    pub fileserver_upload_handler: Option<UploadHandler>,
    pub size_threshold: usize,
    pub correlation_id: String,
    pub msg_purpose: String,
    pub sender_name: String,
    pub receiver_name: String,
    pub receiver_id: String,
    pub reply_to: String,
    pub reply_to_msg_id: String,
    pub msg_id: String,
    pub sender_id: String,
}

// Payload enum for type-safe data handling
#[derive(Serialize, Deserialize, Clone)]
pub enum Payload {
    Text(String),
    Dictionary(serde_json::Value),
    ArrowTable(Vec<u8>),
    JsonTable(serde_json::Value),
    Image(Vec<u8>),
    Audio(Vec<u8>),
    Video(Vec<u8>),
    Binary(Vec<u8>),
}

// MsgEnvelopeV1 struct (serde-serializable)
#[derive(Serialize, Deserialize, Clone)]
pub struct MsgEnvelopeV1 {
    pub correlation_id: String,
    pub msg_id: String,
    pub timestamp: String,
    pub send_to: String,
    pub msg_purpose: String,
    pub sender_name: String,
    pub sender_id: String,
    pub receiver_name: String,
    pub receiver_id: String,
    pub reply_to: String,
    pub reply_to_msg_id: String,
    pub broker_url: String,
    pub metadata: serde_json::Value,
    pub payloads: Vec<MsgPayloadV1>,
}

Note: Publishing via the transport layer is the caller's responsibility. Returns Result<(MsgEnvelopeV1, String), msghandlerError>. Uses serde for JSON serialization.

`smartunpack` Function Signature (SP-017)

Note: Input is the JSON string payload from the transport subscription. Returns (envelope::msg_envelope_v1, payloads::Array{Tuple{String, Any, String}, 1}).

Traceability: FR-001 through FR-014, NFR-101 through NFR-107 | SD-001 through SD-008

Julia

function smartunpack(
    msg_json_str::String;  # Pass payload from transport subscription
    fileserver_download_handler::Function = _fetch_with_backoff,
    max_retries::Int = 5,
    base_delay::Int = 100,
    max_delay::Int = 5000
)::JSON.Object{String, Any}

Note: Input is the JSON string payload from the transport subscription, not the transport message object directly.

Python

async def smartunpack(
    msg_json_str: str,  # JSON string from transport message payload
    fileserver_download_handler: Callable = fetch_with_backoff,
    max_retries: int = 5,
    base_delay: int = 100,
    max_delay: int = 5000
) -> Dict[str, Any]:

Note: Input is the JSON string payload from the transport message.

JavaScript (Browser)

async function smartunpack(
    msg_json_str: string,  // JSON string from transport message payload
    options?: {
        fileserver_download_handler?: Function;
        max_retries?: number;
        base_delay?: number;
        max_delay?: number;
    }
): Promise<Object>;

Note: Input is the JSON string payload from the transport message.

MicroPython

def smartunpack(msg_json_str: str, **kwargs) -> Dict[str, Any]:

Note: Input is the JSON string payload from the transport message.

Rust

pub async fn smartunpack(
    msg_json_str: &str,  // JSON string from transport message payload
    options: &smartunpackOptions,
) -> Result<MsgEnvelopeV1, msghandlerError>

// smartunpackOptions struct
pub struct smartunpackOptions {
    pub fileserver_download_handler: Option<DownloadHandler>,
    pub max_retries: u32,
    pub base_delay: u64,
    pub max_delay: u64,
}

Note: Input is the JSON string payload from the transport message. Returns Result<MsgEnvelopeV1, msghandlerError>.

File Server Interface (SP-018, SP-019)

Traceability: FR-008, FR-009, FR-010, NFR-202 | SD-003, SD-007

Upload Handler Contract (SP-018)

Function Signature:

function fileserver_upload_handler(
    file_server_url::String,
    dataname::String,
    data::Vector{UInt8}
)::Dict{String, Any}

# Overload: Upload file from disk
function fileserver_upload_handler(
    file_server_url::String,
    filepath::String
)::Dict{String, Any}

Return Format:

{
  "status": 200,
  "uploadid": "string",
  "fileid": "string",
  "url": "string"
}

Required Keys:

Key	Type	Description	Requirement ID	Solution Design Ref
`status`	`integer`	HTTP response status code	FR-008, FR-009	SD-003
`uploadid`	`string`	Upload session identifier	FR-008	SD-003
`fileid`	`string`	File identifier within session	FR-008	SD-003
`url`	`string`	Full download URL	FR-008, FR-009, FR-010	SD-003, SD-007

Download Handler Contract (SP-019)

Function Signature:

Traceability: FR-010, NFR-202 | SD-003, SD-007

Function Signature:

function fileserver_download_handler(
    url::String,
    max_retries::Int,
    base_delay::Int,
    max_delay::Int,
    correlation_id::String
)::Vector{UInt8}

Retry Policy:

Initial delay: base_delay milliseconds
Maximum delay: max_delay milliseconds
Multiplier: 2x per retry
Maximum retries: max_retries

Platform-Specific Constraints (SP-020)

Traceability: FR-005, FR-006, NFR-106, NFR-107 | SD-002, SD-004

Desktop (Julia/Python)

Feature	Status	Notes	Requirement ID	Solution Design Ref
Arrow IPC	✅ Supported	Requires Arrow.jl/pyarrow	FR-002, FR-012	SD-004, SD-005
JSON table	✅ Supported	Human-readable format	FR-002, FR-006	SD-004, SD-005
File server upload	✅ Supported	HTTP/HTTPS	FR-008, FR-009	SD-003
File server download	✅ Supported	HTTP/HTTPS	FR-010, NFR-202	SD-003, SD-007
Size threshold	10,000 bytes (10KB)	Configurable	FR-004, FR-005, NFR-104, NFR-105	SD-002

Browser (JavaScript)

Feature	Status	Notes	Requirement ID	Solution Design Ref
Arrow IPC	❌ Not supported	Apache Arrow not browser-compatible	FR-002	SD-004, SD-005
JSON table	✅ Supported	Only table type available in browser	FR-002, FR-006	SD-004, SD-005
File server upload	✅ Supported	HTTP/HTTPS	FR-008, FR-009	SD-003
File server download	✅ Supported	HTTP/HTTPS	FR-010, NFR-202	SD-003, SD-007
Size threshold	10,000 bytes (10KB)	Configurable	FR-004, FR-005, NFR-104, NFR-105	SD-002

Rust

Feature	Status	Notes	Requirement ID	Solution Design Ref
Arrow IPC	✅ Supported	Requires `arrow2` crate	FR-002, FR-012	SD-004, SD-005
JSON table	✅ Supported	Uses `serde_json`	FR-002, FR-006	SD-004, SD-005
File server upload	✅ Supported	HTTP/HTTPS via `reqwest`	FR-008, FR-009	SD-003
File server download	✅ Supported	HTTP/HTTPS via `reqwest` with retry	FR-010, NFR-202	SD-003, SD-007
Size threshold	10,000 bytes (10KB)	Configurable	FR-004, FR-005, NFR-104, NFR-105	SD-002
Async runtime	✅ Supported	Uses `tokio` for async I/O	FR-013, FR-014	SD-006
Type safety	✅ Supported	Compile-time type checking via Rust enums	FR-006, FR-007	SD-004

MicroPython

Feature	Status	Notes	Requirement ID	Solution Design Ref
Arrow IPC	❌ Not supported	Memory constraints	FR-005	SD-002
JSON table	⚠️ Limited	Only direct transport	FR-005, FR-006	SD-002, SD-004
File server upload	❌ Not implemented	Placeholder only	FR-005	SD-002
File server download	❌ Not implemented	Placeholder only	FR-005	SD-002
Size threshold	100KB	Hard limit enforced	FR-005, NFR-106	SD-002
Max payload	50KB	Hard limit enforced	FR-005	SD-002

Implementation Files (SP-021)

Traceability: FR-001 through FR-007 | SD-001 through SD-008

File	Platform	Features	Notes	Requirement ID	Solution Design Ref
`src/msghandler.jl`	Julia	Full feature set, Arrow IPC, multiple dispatch	Ground truth implementation	FR-001 through FR-014, NFR-101 through NFR-107	SD-001 through SD-008
`src/msghandler_csr.js`	Browser	JSON table only	Client-side rendering	FR-001 through FR-014, NFR-101 through NFR-107	SD-001 through SD-008
`src/msghandler.py`	Python	Arrow IPC, async/await	Desktop Python	FR-001 through FR-014, NFR-101 through NFR-107	SD-001 through SD-008
`src/msghandler.rs`	Rust	Full feature set, Arrow IPC, async/await, type-safe	Uses tokio + serde + arrow2	FR-001 through FR-014, NFR-101 through NFR-107	SD-001 through SD-008
`src/msghandler_mpy.py`	MicroPython	Limited to direct transport	Memory-constrained	FR-005, FR-006	SD-002, SD-004

Browser Implementation Notes

Traceability: FR-002, FR-006 | SD-004, SD-005

The browser implementation (src/msghandler_csr.js) has the following constraints:

Constraint	Reason	Workaround	Requirement ID	Solution Design Ref
No Apache Arrow IPC	Browser-incompatible dependency	Use `jsontable` for tabular data	FR-002	SD-004, SD-005
WebSocket only	Browser cannot use TCP directly	Use `ws://` or `wss://` broker URLs	FR-013, FR-014	SD-006
Fetch API for HTTP	Browser fetch() API only	Compatible with Plik and other HTTP servers	FR-008, FR-009	SD-003

Payload Type Availability by Platform

Payload Type	Julia	Browser	Python	Rust	MicroPython
`text`	✅	✅	✅	✅	✅
`dictionary`	✅	✅	✅	✅	✅
`arrowtable`	✅	❌	✅	✅	❌
`jsontable`	✅	✅	✅	✅	⚠️
`image`	✅	✅	✅	✅	✅
`audio`	✅	✅	✅	✅	✅
`video`	✅	✅	✅	✅	✅
`binary`	✅	✅	✅	✅	✅

Message Flow (SP-022)

Sending Flow

flowchart TD
    A[User calls smartpack subject data] --> B[Serialize payload according to payload_type]
    B --> C{Calculate serialized size}
    C -->|Size < Threshold| D[Direct Transport: Encode as Base64]
    C -->|Size >= Threshold| E[Link Transport: Upload to file server]
    D --> F[Build envelope with metadata]
    E --> F
    F --> G[Convert envelope to JSON string]
    G --> H[Publish to topic via transport]
    H --> I[Return envelope and JSON string to caller]
    
    style A fill:#f9f9f9,stroke:#333
    style I fill:#e0e7ff,stroke:#3b82f6
    style D fill:#d1fae5,stroke:#10b981
    style E fill:#fef3c7,stroke:#f59e0b

Receiving Flow

flowchart TD
  A[Transport message arrives] --> B[Parse JSON envelope]
    B --> C[For each payload: Check transport type]
    C -->|transport == direct| D[Direct Transport: Extract Base64]
    C -->|transport == link| E[Link Transport: Fetch from URL]
    D --> F[Decode Base64]
    E --> G[Fetch with exponential backoff]
    F --> H[Deserialize based on payload_type]
    G --> H
    H --> I[Build payloads array]
    I --> J[Replace payloads array with deserialized tuples]
    J --> K[Return envelope with processed payloads]
    
    style A fill:#f9f9f9,stroke:#333
    style K fill:#e0e7ff,stroke:#3b82f6
    style D fill:#d1fae5,stroke:#10b981
    style E fill:#fef3c7,stroke:#f59e0b

Validation Rules (SP-023)

Traceability: FR-001 through FR-014, NFR-101 through NFR-405 | SD-001 through SD-008

Envelope Validation

Rule	Condition	Error Code	Requirement ID	Solution Design Ref
Required fields present	`correlation_id`, `msg_id`, `timestamp`, `send_to`, `payloads`	`INVALID_ENVELOPE`	FR-012, FR-013	SD-001, SD-006
Valid UUID format	`correlation_id`, `msg_id`, `sender_id`, `receiver_id`	`INVALID_ENVELOPE`	FR-011, FR-012, NFR-401	SD-001, SD-008
Valid timestamp format	ISO 8601 UTC	`INVALID_ENVELOPE`	FR-012, NFR-401	SD-001, SD-008
Non-empty payloads array	`length(payloads) > 0`	`INVALID_ENVELOPE`	FR-012, FR-013	SD-001, SD-005

Payload Validation

Rule	Condition	Error Code	Requirement ID	Solution Design Ref
Valid payload_type	Must be in `payload_type` enum	`INVALID_PAYLOAD_TYPE`	FR-001, FR-002, FR-003, FR-006	SD-004, SD-005
Valid transport	Must be `direct` or `link`	`INVALID_TRANSPORT`	FR-003, FR-004, FR-006	SD-001, SD-002
Valid encoding	Must match payload_type and transport	`INVALID_TRANSPORT`	FR-001, FR-002, FR-003, FR-012	SD-004, SD-005
Positive size	`size > 0`	`INVALID_PAYLOAD`	FR-003, FR-004, NFR-104, NFR-105	SD-001, SD-002
Valid Base64 for direct	`data` matches Base64 pattern	`DESERIALIZATION_ERROR`	FR-001, FR-002, FR-003, FR-012	SD-004, SD-005
Valid URL for link	`data` matches HTTP(S) URL pattern	`DOWNLOAD_FAILED`	FR-008, FR-009, FR-010	SD-003, SD-007

Test Contracts (SP-024)

Traceability: FR-001 through FR-014, NFR-101 through NFR-107 | SD-001 through SD-008

Unit Test Validation

Test	Input	Expected Output	Notes	Requirement ID	Solution Design Ref
Text round-trip	`("msg", "Hello", "text")`	`("msg", "Hello", "text")`	String serialization	FR-001, FR-012, NFR-101, NFR-102	SD-004, SD-005
Dictionary round-trip	`("data", {"key": "value"}, "dictionary")`	`("data", {"key": "value"}, "dictionary")`	JSON object round-trip	FR-002, FR-012, NFR-101, NFR-102	SD-004, SD-005
Arrow table round-trip	`("table", arrow_table_data, "arrowtable")`	`("table", arrow_table_data, "arrowtable")`	Arrow IPC round-trip	FR-002, FR-012, NFR-101, NFR-102	SD-004, SD-005
JSON table round-trip	`("table", [{"a":1},{"b":2}], "jsontable")`	`("table", [{"a":1},{"b":2}], "jsontable")`	JSON array of objects	FR-001, FR-002, FR-006, FR-012	SD-004, SD-005
Mixed payloads	`[("msg", "Hello", "text"), ("imgname", bytes, "binary")]`	`[("msg", "Hello", "text"), ("imgname", bytes, "binary")]`	Multiple payload types	FR-006, FR-007	SD-004
Large payload	`("data", rand(10_000_000), "arrowtable")`	`("data", URL, "arrowtable")` with link transport	File server upload	FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105	SD-001, SD-002, SD-003

Platform-Specific Notes:

Julia: Use Dict, Vector{Dict}, or convert DataFrame to dictionary for testing
Python: Use dict, list[dict], or convert pandas.DataFrame to dictionary for testing
JavaScript: Use plain objects {} and arrays []
MicroPython: Use plain dict and list (limited to JSON table and text types)

Integration Test Scenarios

Scenario	Platforms	Payloads	Size Mix	Transport	Expected Result	Requirement ID	Solution Design Ref
Single text (small)	All	`text`	Small	direct	Round-trip successful	FR-001, FR-012, NFR-101, NFR-102	SD-004, SD-005
Single dictionary (small)	All	`dictionary`	Small	direct	Round-trip successful	FR-002, FR-012, NFR-101, NFR-102	SD-004, SD-005
Single arrow table (small)	Julia/JS/Python	`arrowtable`	Small	direct	Arrow IPC round-trip	FR-002, FR-012, NFR-101, NFR-102	SD-004, SD-005
Single JSON table (small)	All	`jsontable`	Small	direct	Dictionary array round-trip	FR-001, FR-002, FR-006, FR-012	SD-004, SD-005
Single image (small)	All	`image`	Small	direct	Binary round-trip	FR-001, FR-006, FR-012	SD-004, SD-005
Single audio (small)	All	`audio`	Small	direct	Binary round-trip	FR-001, FR-006, FR-012	SD-004, SD-005
Single video (small)	All	`video`	Small	direct	Binary round-trip	FR-001, FR-006, FR-012	SD-004, SD-005
Single binary (small)	All	`binary`	Small	direct	Binary round-trip	FR-001, FR-006, FR-012	SD-004, SD-005
Single text (large)	All	`text`	Large	link	File server upload/download	FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105	SD-001, SD-002, SD-003
Single JSON table (large)	All	`jsontable`	Large	link	File server upload/download	FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105	SD-001, SD-002, SD-003
Single image (large)	All	`image`	Large	link	File server upload/download	FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105	SD-001, SD-002, SD-003
Ultimate Test	Julia/JS/Python	All types	Mixed	direct/link	All payloads preserved with correct transport	FR-001 through FR-014, NFR-101 through NFR-107	SD-001 through SD-008
Ultimate Test	MicroPython	`text`/`dictionary`	Mixed	direct	Limited to text/dictionary with direct transport only	FR-005, FR-006, FR-012	SD-002, SD-004
Cross-platform JSON table	All	`jsontable`	Small	direct	Dictionary array round-trip	FR-001, FR-002, FR-006, FR-012	SD-004, SD-005
MicroPython ↔ Desktop	MicroPython ↔ Desktop	`text`/`dictionary`	Small	direct	Limited payload types	FR-005, FR-006, FR-012	SD-002, SD-004
Desktop ↔ Desktop (all combos)	Julia↔JS↔Python	All types	Small/Large	direct/link	Full compatibility	FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014

Dependencies (SP-025)

Required Dependencies by Platform

Platform	Package	Version	Purpose	Requirement ID	Solution Design Ref
Julia	JSON.jl	Latest	JSON serialization	FR-012, NFR-101, NFR-102	SD-005
Julia	Arrow.jl	Latest	Arrow IPC support	FR-002, FR-012	SD-004, SD-005
Julia	HTTP.jl	Latest	HTTP file server	FR-008, FR-009	SD-003
Julia	UUIDs.jl	Latest	UUID generation	FR-011, NFR-401	SD-008
Browser	-	-	Transport-agnostic (caller provides)	FR-013, FR-014	SD-006
Python	aiohttp	Latest	HTTP file server	FR-008, FR-009	SD-003
Python	pyarrow	Latest	Arrow IPC support	FR-002, FR-012	SD-004, SD-005
Rust	serde	Latest	JSON serialization	FR-012, NFR-101, NFR-102	SD-005
Rust	serde_json	Latest	JSON handling	FR-012, NFR-101, NFR-102	SD-005
Rust	tokio	Latest	Async runtime	FR-013, FR-014	SD-006
Rust	reqwest	Latest	HTTP file server	FR-008, FR-009	SD-003
Rust	uuid	Latest	UUID generation	FR-011, NFR-401	SD-008
Rust	arrow2	Latest	Arrow IPC support	FR-002, FR-012	SD-004, SD-005
MicroPython	builtin	N/A	Limited implementation	FR-005, FR-006	SD-002

Optional Dependencies

Platform	Package	Purpose
Julia	DataFrames.jl	DataFrame support
Python	pandas	DataFrame support

Change Log

Date	Version	Changes
2026-09-16	1.4.0	Aligned to ASG Framework eight pillars; added SP-XXX Specification Item IDs, Specification Traceability Matrix, section-level SP references, and Rate Limiting & Quotas (SP-010.1)
2026-05-15	1.3.0	Made transport layer agnostic
-	-	Removed all NATS-specific dependencies (NATS.jl, nats, nats-py, nats.ws)
-	-	Updated docs to reference generic message broker/transport
-	-	broker_url is now metadata only, not used for active connections
2026-03-15	1.1.0	Browser connection management
-	-	Added NATSClient class with keepAlive support
-	-	Added NATSConnectionPool for connection reuse
-	-	Added publishMessage function with closeConnection option
-	-	Added nats.ws to browser dependencies
2026-03-13	1.0.0	Initial specification
-	-	Message envelope schema defined
-	-	Payload schema with transport modes
-	-	Enumerations for payload_type, transport, encoding
-	-	Size thresholds for desktop/MicroPython
-	-	Error codes and validation rules
-	-	API contracts for all platforms

References

ASG Framework Reference: This specification follows the ASG Framework v8 pillars. Each specification item must cite:

Requirement ID(s) from requirements.md (e.g., FR-001, NFR-201) — defines what the system must do
Solution Design reference(s) from solution-design.md (e.g., SD-2.1, SD-3.3) — defines how and why this technical approach was chosen

20.1 Documentation Artifacts

Document	Purpose	Requirements Traceability	Solution Design Traceability
`docs/requirements.md`	Business requirements and user stories	FR-001 through FR-014, NFR-101 through NFR-405	SD-001 through SD-008
`docs/solution-design.md`	Technical solution approach	FR-001 through FR-014, NFR-101 through NFR-405	SD-001 through SD-008
`docs/specification.md`	Technical contract for msghandler	FR-001 through FR-014, NFR-101 through NFR-405	SD-001 through SD-008
`docs/ui-specification.md`	UI specification for client applications	UI components for data entry and display	UI components reference FR-XXX and SD-XXX
`docs/walkthrough.md`	End-to-end system flow	Traceability from user journey to technical implementation	Full flow validation against SD-XXX
`docs/architecture.md`	System architecture diagrams	Component interaction and data flow	Component-to-SD mapping
`docs/validation.md`	CI/CD validation rules	Contract testing and spec compliance	Validation gates for SD-XXX
`docs/runbook.md`	Operational runbook	Deployment, scaling, and troubleshooting	Operation-to-SD mapping

20.2 Implementation Files

File	Platform	Features	Requirements Traceability	Solution Design Traceability
`src/msghandler.jl`	Julia	Full feature set, Arrow IPC, multiple dispatch	FR-001 through FR-014, NFR-101 through NFR-405	SD-001 through SD-008
`src/msghandler_csr.js`	Browser	JSON table only	FR-001 through FR-014, NFR-101 through NFR-405	SD-001 through SD-008
`src/msghandler.py`	Python	Arrow IPC, async/await	FR-001 through FR-014, NFR-101 through NFR-405	SD-001 through SD-008
`src/msghandler.rs`	Rust	Full feature set, Arrow IPC, async/await, type-safe	FR-001 through FR-014, NFR-101 through NFR-405	SD-001 through SD-008
`src/msghandler_mpy.py`	MicroPython	Limited to direct transport	FR-005, FR-006, FR-012	SD-002, SD-004

20.3 External Dependencies

Platform	Package	Version	Purpose	Requirements Traceability	Solution Design Traceability
Julia	JSON.jl	Latest	JSON serialization	FR-012, NFR-101, NFR-102	SD-005
Julia	Arrow.jl	Latest	Arrow IPC support	FR-002, FR-012	SD-004, SD-005
Julia	HTTP.jl	Latest	HTTP file server	FR-008, FR-009	SD-003
Julia	UUIDs.jl	Latest	UUID generation	FR-011, NFR-401	SD-008
Browser	-	-	Transport-agnostic (caller provides)	FR-013, FR-014	SD-006
Python	aiohttp	Latest	HTTP file server	FR-008, FR-009	SD-003
Python	pyarrow	Latest	Arrow IPC support	FR-002, FR-012	SD-004, SD-005
Rust	serde	Latest	JSON serialization	FR-012, NFR-101, NFR-102	SD-005
Rust	serde_json	Latest	JSON handling	FR-012, NFR-101, NFR-102	SD-005
Rust	tokio	Latest	Async runtime	FR-013, FR-014	SD-006
Rust	reqwest	Latest	HTTP file server	FR-008, FR-009	SD-003
Rust	uuid	Latest	UUID generation	FR-011, NFR-401	SD-008
Rust	arrow2	Latest	Arrow IPC support	FR-002, FR-012	SD-004, SD-005
MicroPython	builtin	N/A	Limited implementation	FR-005, FR-006	SD-002

Change Log

Date	Version	Changes	Requirement ID(s)	Solution Design Ref(s)
2026-09-16	1.4.0	Aligned to ASG Framework eight pillars; assigned SP-XXX IDs to all spec items; added Specification Traceability Matrix, Rate Limiting & Quotas (SP-010.1); fixed desktop size threshold to 10,000 bytes per ground truth	All	All SD-XXX
2026-05-22	1.3.0	Updated to ASG Framework v8 pillars	All	All SD-XXX
-	-	Added Solution Design references to all specification items	All	All SD-XXX
-	-	Updated version to 1.3.0 to match requirements and solution-design	All	-
-	-	Added ASG Framework alignment section	All	-
2026-05-15	1.3.0	Made transport layer agnostic	All	SD-006
-	-	Removed all NATS-specific dependencies (NATS.jl, nats, nats-py, nats.ws)	All	SD-006
-	-	Updated docs to reference generic message broker/transport	All	SD-006
-	-	broker_url is now metadata only, not used for active connections	All	SD-006
2026-03-15	1.1.0	Browser connection management	FR-001 through FR-014	SD-006
-	-	Added NATSClient class with keepAlive support	FR-013, FR-014	SD-006
-	-	Added NATSConnectionPool for connection reuse	FR-013, FR-014	SD-006
-	-	Added publishMessage function with closeConnection option	FR-013, FR-014	SD-006
-	-	Added nats.ws to browser dependencies	FR-013, FR-014	SD-006
2026-03-13	1.0.0	Initial specification	FR-001 through FR-014, NFR-101 through NFR-405	SD-001 through SD-008
-	-	Message envelope schema defined	FR-012, FR-013	SD-001, SD-006
-	-	Payload schema with transport modes	FR-001, FR-002, FR-003, FR-004	SD-001, SD-002
-	-	Enumerations for payload_type, transport, encoding	FR-001, FR-002, FR-003, FR-006	SD-004, SD-005
-	-	Size thresholds for desktop/MicroPython	FR-004, FR-005	SD-002
-	-	Error codes and validation rules	FR-001 through FR-010	SD-001 through SD-007
-	-	API contracts for all platforms	FR-001 through FR-014	SD-001 through SD-008

Appendix

A. Complete JSON Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "msghandler Envelope",
  "type": "object",
  "properties": {
    "correlation_id": {
      "type": "string",
      "pattern": "^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$",
      "description": "UUID v4 format for tracking message flow"
    },
    "msg_id": {
      "type": "string",
      "pattern": "^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$",
      "description": "Unique message identifier"
    },
    "timestamp": {
      "type": "string",
      "pattern": "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3}Z$",
      "description": "ISO 8601 UTC timestamp"
    },
    "send_to": {
      "type": "string",
      "minLength": 1,
      "description": "Topic/subject to publish to"
    },
    "msg_purpose": {
      "type": "string",
      "enum": ["ACK", "NACK", "updateStatus", "shutdown", "chat", "command", "event"],
      "description": "Purpose of the message"
    },
    "sender_name": {
      "type": "string",
      "minLength": 1,
      "description": "Sender application name"
    },
    "sender_id": {
      "type": "string",
      "pattern": "^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$",
      "description": "Sender UUID"
    },
    "receiver_name": {
      "type": "string",
      "description": "Receiver name (empty = broadcast)"
    },
    "receiver_id": {
      "type": "string",
      "pattern": "^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$|^$",
      "description": "Receiver UUID (empty = broadcast)"
    },
    "reply_to": {
      "type": "string",
      "description": "Topic for reply messages"
    },
    "reply_to_msg_id": {
      "type": "string",
      "description": "Message ID being replied to"
    },
    "broker_url": {
      "type": "string",
      "description": "Broker URL for the transport layer"
    },
    "metadata": {
      "type": "object",
      "description": "Message-level metadata"
    },
    "payloads": {
      "type": "array",
      "minItems": 1,
      "items": {
        "$ref": "#/definitions/Payload"
      }
    }
  },
  "required": ["correlation_id", "msg_id", "timestamp", "send_to", "msg_purpose", "sender_name", "sender_id", "receiver_name", "receiver_id", "reply_to", "reply_to_msg_id", "broker_url", "payloads"],
  "definitions": {
    "Payload": {
      "type": "object",
      "properties": {
        "id": {
          "type": "string",
          "pattern": "^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$"
        },
        "dataname": {
          "type": "string",
          "minLength": 1
        },
        "payload_type": {
          "type": "string",
          "enum": ["text", "dictionary", "arrowtable", "jsontable", "image", "audio", "video", "binary"]
        },
        "transport": {
          "type": "string",
          "enum": ["direct", "link"]
        },
        "encoding": {
          "type": "string",
          "enum": ["none", "base64", "json", "arrow-ipc"]
        },
        "size": {
          "type": "integer",
          "minimum": 1
        },
        "data": {
          "anyOf": [
            {
              "type": "string",
              "pattern": "^(https?://[^\\s]+)$"
            },
            {
              "type": "string",
              "pattern": "^[A-Za-z0-9+/]+=*$"
            }
          ]
        },
        "metadata": {
          "type": "object"
        }
      },
      "required": ["id", "dataname", "payload_type", "transport", "encoding", "size", "data"]
    }
  }
}

B. AsyncAPI Specification

asyncapi: '2.6.0'
info:
  title: msghandler API
  version: '1.0.0'
  description: Cross-platform bi-directional data bridge using a message broker
  contact:
    name: msghandler Team
    url: https://github.com/your-org/msghandler
  license:
    name: MIT
    url: https://opensource.org/licenses/MIT
channels:
  /agent/{service}/api/v{version}/{operation}:
    address: /agent/{service}/api/v{version}/{operation}
    parameters:
      service:
        schema:
          type: string
      version:
        schema:
          type: string
          enum: ['v1']
      operation:
        schema:
          type: string
    publish:
      summary: Publish message to transport
      operationId: publishMessage
      message:
        $ref: '#/components/message'
    subscribe:
      summary: Subscribe to messages from transport
      operationId: subscribeMessage
      message:
        $ref: '#/components/message'
components:
  message:
    payload:
      $ref: '#/components/schemas/Envelope'
  schemas:
    Envelope:
      type: object
      properties:
        correlation_id:
          type: string
          format: uuid
        msg_id:
          type: string
          format: uuid
        timestamp:
          type: string
          format: date-time
        send_to:
          type: string
        msg_purpose:
          type: string
          enum: [ACK, NACK, updateStatus, shutdown, chat, command, event]
        sender_name:
          type: string
        sender_id:
          type: string
          format: uuid
        receiver_name:
          type: string
        receiver_id:
          type: string
          format: uuid
        reply_to:
          type: string
        reply_to_msg_id:
          type: string
        broker_url:
          type: string
        metadata:
          type: object
        payloads:
          type: array
          items:
            $ref: '#/components/schemas/Payload'
      required:
        - correlation_id
        - msg_id
        - timestamp
        - send_to
        - msg_purpose
        - sender_name
        - sender_id
        - receiver_name
        - receiver_id
        - reply_to
        - reply_to_msg_id
        - broker_url
        - payloads
    Payload:
      type: object
      properties:
        id:
          type: string
          format: uuid
        dataname:
          type: string
        payload_type:
          type: string
          enum: [text, dictionary, arrowtable, jsontable, image, audio, video, binary]
        transport:
          type: string
          enum: [direct, link]
        encoding:
          type: string
          enum: [none, base64, json, arrow-ipc]
        size:
          type: integer
          minimum: 1
        data:
          type: string
        metadata:
          type: object
      required:
        - id
        - dataname
        - payload_type
        - transport
        - encoding
        - size
        - data

This specification is versioned and maintained in git alongside the codebase. All implementations must adhere to this specification.

61 KiB Raw Blame History

Specification: msghandler

1. Technical Contract Overview

1.1 Requirements Traceability

Specification Traceability Matrix

Walkthrough → Specification Trace

2. Message Envelope Schema

Specification Versioning

Message Envelope Schema

Envelope Structure (JSON)

Field Definitions (SP-001)

Payload Schema

Payload Structure (JSON)

Payload Field Definitions (SP-002)

Payload Format (SP-003)

Tuple Format for smartpack()

Single Payload Example

Multiple Payloads Example

Data Type Mapping

Enumerations (SP-004..SP-007)

msg_purpose Enum

payload_type Enum

transport Enum

encoding Enum

Transport Protocols (SP-008, SP-009)

Direct Transport Protocol (SP-008)

Link Transport Protocol (SP-009)

Size Thresholds (SP-010)

Desktop Platforms (Julia/JS/Python)

MicroPython Platform

Rate Limiting & Quotas (SP-010.1)

Topic Convention (SP-011)

Subject Naming Pattern

Subject Wildcards

Error Handling (SP-012..SP-014)

Error Response Format (SP-012)

Error Codes (SP-013)

Exception Handling (SP-014)

Serialization Rules (SP-015)

Text Serialization

Dictionary Serialization

Arrow Table Serialization

JSON Table Serialization

Binary Serialization

API Contract (SP-016, SP-017)

smartpack Function Signature (SP-016)

Julia

Python

JavaScript (Browser)

MicroPython

Rust

smartunpack Function Signature (SP-017)

Julia

Python

JavaScript (Browser)

MicroPython

Rust

File Server Interface (SP-018, SP-019)

Upload Handler Contract (SP-018)

Download Handler Contract (SP-019)

Platform-Specific Constraints (SP-020)

Desktop (Julia/Python)

Browser (JavaScript)

Rust

MicroPython

Implementation Files (SP-021)

Browser Implementation Notes

Payload Type Availability by Platform

Message Flow (SP-022)

Sending Flow

Receiving Flow

Validation Rules (SP-023)

Envelope Validation

Payload Validation

Test Contracts (SP-024)

Unit Test Validation

Integration Test Scenarios

Dependencies (SP-025)

Required Dependencies by Platform

Optional Dependencies

61 KiB

Raw Blame History

Tuple Format for `smartpack()`

`msg_purpose` Enum

`payload_type` Enum

`transport` Enum

`encoding` Enum

`smartpack` Function Signature (SP-016)

`smartunpack` Function Signature (SP-017)