ton/msghandler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
d0910ccc3f5cff6e7b826e46a4eb60f88ffdd2ae
msghandler/docs/specification.md
narawat d0910ccc3f update docs
2026-05-15 13:25:48 +07:00

50 KiB
Raw Blame History

Specification: msghandler

Version: 1.2.0
Date: 2026-05-13
Status: Active
Ground Truth: src/msghandler.jl
Specification Format: JSON Schema + AsyncAPI

1. Technical Contract Overview

This document defines the technical contract for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between Julia, JavaScript, Python, Dart, Rust, and MicroPython applications using a message broker as the transport layer.

This specification serves as the single source of truth for:

  • Inputs: What data structures are accepted by smartsend()
  • Outputs: What data structures are returned by smartreceive()
  • Data Shapes: Exact field names, types, and constraints
  • Error Codes: Standardized error responses for failure scenarios

1.1 Requirements Traceability

Specification Section Requirement ID(s) Description
Section 2 (Message Envelope) FR-012, FR-013, NFR-101, NFR-102 Message envelope structure and validation
Section 3 (Payload Schema) FR-001, FR-002, FR-003, FR-004, NFR-101, NFR-102 Payload structure and field definitions
Section 4 (Payload Format) FR-006, FR-007 Tuple format for smartsend()
Section 5 (Enumerations) FR-003, FR-004, FR-006, NFR-101 Enumerations for transport and encoding
Section 6 (Transport Protocols) FR-003, FR-004, NFR-104, NFR-105 Direct and link transport protocols
Section 7 (Size Thresholds) FR-004, FR-005, NFR-104, NFR-105 Size thresholds for transport selection
Section 8 (Topic Convention) FR-013, FR-014 Topic/subject naming patterns
Section 9 (Error Handling) FR-010, FR-011, NFR-201, NFR-202, NFR-203 Error codes and exception handling
Section 10 (Serialization Rules) FR-001, FR-002, FR-003, FR-012, NFR-101, NFR-102 Serialization and encoding rules
Section 11 (API Contract) FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 Function signatures for all platforms
Section 12 (File Server Interface) FR-008, FR-009, FR-010 Upload and download handler contracts
Section 13 (Platform-Specific Constraints) FR-005, FR-006, NFR-106, NFR-107 Platform-specific feature support
Section 14 (Implementation Files) FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007 Implementation file mapping
Section 15 (Message Flow) FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 Mermaid diagrams for send/receive flows
Section 16 (Validation Rules) FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 Envelope and payload validation rules
Section 17 (Test Contracts) FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 Unit and integration test scenarios
Section 18 (Dependencies) FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 Platform-specific dependencies
Section 19 (Change Log) N/A Version history and changes

2. Message Envelope Schema

Specification Versioning

Component Version Notes
Specification 1.0.0 Initial release
Protocol v1 Message envelope protocol version

Message Envelope Schema

Envelope Structure (JSON)

{
  "correlation_id": "string (UUID)",
  "msg_id": "string (UUID)",
  "timestamp": "string (ISO 8601 UTC)",
  "send_to": "string",
  "msg_purpose": "string",
  "sender_name": "string",
  "sender_id": "string (UUID)",
  "receiver_name": "string",
  "receiver_id": "string (UUID)",
  "reply_to": "string",
  "reply_to_msg_id": "string",
  "broker_url": "string",
  "metadata": "object",
  "payloads": [
    {
      "id": "string (UUID)",
      "dataname": "string",
      "payload_type": "string",
      "transport": "string",
      "encoding": "string",
      "size": "integer",
      "data": "string or URL",
      "metadata": "object"
    }
  ]
}

Field Definitions

Field Type Required Validation Description Requirement ID
correlation_id string Yes UUID v4 format Track message flow across distributed systems FR-011, NFR-401
msg_id string Yes UUID v4 format Unique identifier for this specific message FR-012, NFR-401
timestamp string Yes ISO 8601 UTC Message publication timestamp FR-012, NFR-401
send_to string Yes Non-empty string Topic/subject to publish the message to FR-013
msg_purpose string Yes Enum Purpose of the message FR-013
sender_name string Yes Non-empty string Name of the sender application FR-013
sender_id string Yes UUID v4 format Unique identifier for the sender FR-013
receiver_name string Yes Any string Name of the receiver (empty = broadcast) FR-013
receiver_id string Yes Any string UUID of the receiver (empty = broadcast) FR-013
reply_to string Yes Any string Topic where receiver should reply FR-013
reply_to_msg_id string Yes Any string Message ID this message is replying to FR-013
broker_url string Yes Valid URL Broker URL for the transport layer FR-013
metadata object No Any JSON object Message-level metadata NFR-401
payloads array Yes Non-empty array List of payload objects FR-012, FR-013

Payload Schema

Payload Structure (JSON)

{
  "id": "string (UUID)",
  "dataname": "string",
  "payload_type": "string",
  "transport": "string",
  "encoding": "string",
  "size": "integer",
  "data": "string or URL",
  "metadata": "object"
}

Payload Field Definitions

Field Type Required Validation Description Requirement ID
id string Yes UUID v4 format Unique identifier for this payload FR-012
dataname string Yes Non-empty string Name of the payload (e.g., login_image, user_data) FR-001, FR-002, FR-003
payload_type string Yes Enum Type of payload (see payload_type enum) FR-001, FR-002, FR-003, FR-006
transport string Yes Enum Transport method: direct or link FR-003, FR-004, NFR-104, NFR-105
encoding string Yes Enum Encoding method (see encoding enum) FR-001, FR-002, FR-003, FR-012
size integer Yes Positive integer Size of the payload in bytes FR-003, FR-004, NFR-104, NFR-105
data string or URL Yes Base64 string or URL Payload data (base64 for direct, URL for link) FR-003, FR-004, FR-008, FR-009, FR-010
metadata object No Any JSON object Payload-level metadata NFR-401

Payload Format

Tuple Format for smartsend()

The smartsend() function accepts data as an array of tuples with the format:

("data_name", data, "data_type")
Position Type Description Example
1 string Data name - identifier for the payload "msg", "login_image", "user_data"
2 any Actual data - content to be serialized "Hello", {"key": "value"}, DataFrame(...)
3 string Data type - must be in payload_type enum "text", "dictionary", "arrowtable"

Single Payload Example

# Julia
smartsend("/chat/user/v1/message", [("msg", "Hello World", "text")])

# Python
await smartsend("/chat/user/v1/message", [("msg", "Hello World", "text")])

// JavaScript
await smartsend("/chat/user/v1/message", [["msg", "Hello World", "text"]]);

Multiple Payloads Example

# Julia - Mixed text and binary data
data = [
    ("msg", "Hello", "text"),
    ("img", binary_data, "image")
]
smartsend("/agent/v1/process", data)

# Python - Mixed types
data = [
    ("msg", "Hello", "text"),
    ("img", binary_data, "image")
]
await smartsend("/agent/v1/process", data)

Data Type Mapping

Platform Input Type Data Type String
All String "text"
All Dict/Object "dictionary"
Desktop DataFrame "arrowtable" or "jsontable"
Browser Array of objects "jsontable" (only table type)
All Array of objects "jsontable"
All Uint8Array/Buffer/bytes "binary"
Desktop Arrow.Table "arrowtable"
All Image/Audio/Video binary "image", "audio", "video"

Enumerations

msg_purpose Enum

Value Description
ACK Acknowledgment of successful message processing
NACK Negative acknowledgment of message processing failure
updateStatus Status update message
shutdown Graceful shutdown request
chat Chat/message payload
command Command payload
event Event payload

payload_type Enum

Value Description Supported Platforms Encoding Options
text Plain text string All base64
dictionary JSON object/dictionary All base64, json
arrowtable Apache Arrow IPC table Desktop (Julia/Python/Node.js/Dart) base64, arrow-ipc
jsontable JSON array of objects All (including Browser/Dart Web) base64, json
image Binary image data All base64
audio Binary audio data All base64
video Binary video data All base64
binary Generic binary data All base64

transport Enum

Value Description Data Format Use Case
direct Payload sent directly via the transport layer Base64-encoded string Payloads < size_threshold
link Payload uploaded to file server HTTP URL Payloads ≥ size_threshold

encoding Enum

Value Description Payload Types
none No additional encoding Link transport URLs
base64 Base64 encoding Text, binary, image, audio, video
json JSON encoding Dictionary, jsontable
arrow-ipc Apache Arrow IPC format Arrowtable

Transport Protocols

Direct Transport Protocol

When transport = "direct", the data field contains a Base64-encoded string of the serialized payload.

Flow:

  1. Serialize payload according to payload_type
  2. Encode serialized bytes as Base64
  3. Include Base64 string in data field

Example:

{
  "transport": "direct",
  "encoding": "base64",
  "size": 11,
  "data": "SGVsbG8gV29ybGQ="
}

Link Transport Protocol

When transport = "link", the data field contains a URL pointing to the uploaded payload.

Flow:

  1. Serialize payload according to payload_type
  2. Upload to HTTP file server (e.g., Plik)
  3. Include returned URL in data field

Example:

{
  "transport": "link",
  "encoding": "none",
  "size": 1000000,
  "data": "http://localhost:8080/file/3F62E/4AgGT/data.zip"
}

Size Thresholds

Desktop Platforms (Julia/JS/Python)

Platform Size Threshold Notes
Desktop 500,000 bytes (0.5MB) Default threshold

MicroPython Platform

Platform Size Threshold Maximum Payload Notes
MicroPython 100,000 bytes (100KB) 50,000 bytes Hard limit due to memory constraints

Topic Convention

Subject Naming Pattern

<scope>/<service>/<version>/<operation>

Examples:

  • /agent/wine/api/v1/prompt - AI agent prompt endpoint
  • /chat/user/v1/message - User chat message
  • /system/worker/v1/status - Worker status update

Subject Wildcards

Wildcard Description Example
* Single-level wildcard /chat/user/v1/* matches /chat/user/v1/message
> Multi-level wildcard /chat/user/v1/> matches all /chat/user/v1/* subjects

Error Handling

Error Response Format

{
  "error": {
    "code": "string",
    "message": "string",
    "details": "object"
  }
}

Error Codes

Code HTTP Status Description Recovery Requirement ID
INVALID_ENVELOPE 400 Message envelope validation failed Fix envelope structure FR-012, FR-013, FR-014
INVALID_PAYLOAD_TYPE 400 Unsupported payload type Use supported payload_type FR-001, FR-002, FR-003, FR-006
INVALID_TRANSPORT 400 Unsupported transport type Use direct or link FR-003, FR-004, FR-006
UPLOAD_FAILED 500 File server upload failed Retry or use direct transport FR-008, FR-009
DOWNLOAD_FAILED 503 File server download failed Retry with exponential backoff FR-010, FR-011, NFR-201, NFR-202
TRANSPORT_CONNECTION_FAILED 503 Transport connection failed Check broker/server availability FR-013, FR-014, NFR-201, NFR-203
DESERIALIZATION_ERROR 500 Payload deserialization failed Check payload_type matches data FR-001, FR-002, FR-003, FR-012
SIZE_EXCEEDED 413 Payload exceeds maximum size Split payload or use link transport FR-003, FR-004, FR-005, NFR-104, NFR-105

Exception Handling

Scenario Handler Retry Policy Requirement ID
File server unavailable Retry up to 5 times Exponential backoff (100ms → 5000ms) FR-010, NFR-202
Transport publish failure Handled by caller Caller's retry logic FR-013, FR-014, NFR-201, NFR-203
Deserialization error Log correlation ID and throw No retry (data corruption) FR-001, FR-002, FR-003, FR-012, NFR-401
Memory overflow (MicroPython) Reject payloads >50KB No retry (client-side check) FR-005, NFR-106

Serialization Rules

Text Serialization

Platform Input Type Serialization Encoding
All String UTF-8 bytes Base64

Dictionary Serialization

Platform Input Type Serialization Encoding
All Object/Dict JSON string Base64 or direct JSON

Arrow Table Serialization

Platform Input Type Serialization Encoding
Desktop DataFrame Arrow IPC stream Base64 or arrow-ipc
Desktop Arrow.Table Arrow IPC stream Base64 or arrow-ipc
MicroPython ❌ Not supported N/A

JSON Table Serialization

Platform Input Type Serialization Encoding
All Vector{Dict}/Array<Object> JSON array Base64 or direct JSON
Desktop pandas.DataFrame JSON array Base64 or direct JSON

Binary Serialization

Platform Input Type Serialization Encoding
All Uint8Array/Buffer/bytes Raw bytes Base64

API Contract

smartsend Function Signature

Julia

function smartsend(
    subject::String,
    data::AbstractArray{Tuple{String, T1, String}, 1};
    broker_url::String = DEFAULT_BROKER_URL,
    fileserver_url::String = DEFAULT_FILESERVER_URL,
    fileserver_upload_handler::Function = plik_oneshot_upload,
    size_threshold::Int = DEFAULT_SIZE_THRESHOLD,
    correlation_id::String = string(uuid4()),
    msg_purpose::String = "chat",
    sender_name::String = "msghandler",
    receiver_name::String = "",
    receiver_id::String = "",
    reply_to::String = "",
    reply_to_msg_id::String = "",
    msg_id::String = string(uuid4()),
    sender_id::String = string(uuid4())
)::Tuple{msg_envelope_v1, String} where {T1<:Any}

Note: Publishing via the transport layer is the caller's responsibility. Returns (env::msg_envelope_v1, env_json_str::String).

Python

async def smartsend(
    subject: str,
    data: List[Tuple[str, Any, str]],
    broker_url: str = DEFAULT_BROKER_URL,
    fileserver_url: str = "http://localhost:8080",
    fileserver_upload_handler: Callable = plik_oneshot_upload,
    size_threshold: int = 500_000,
    correlation_id: str = None,
    msg_purpose: str = "chat",
    sender_name: str = "msghandler",
    receiver_name: str = "",
    receiver_id: str = "",
    reply_to: str = "",
    reply_to_msg_id: str = "",
    msg_id: str = None,
    sender_id: str = None
) -> Tuple[Dict, str]:

Note: Publishing via the transport layer is the caller's responsibility.

JavaScript (Node.js)

async function smartsend(
    subject: string,
    data: Array<[string, any, string]>,
    options?: {
        broker_url?: string;
        fileserver_url?: string;
        fileserver_upload_handler?: Function;
        size_threshold?: number;
        correlation_id?: string;
        msg_purpose?: string;
        sender_name?: string;
        receiver_name?: string;
        receiver_id?: string;
        reply_to?: string;
        reply_to_msg_id?: string;
        msg_id?: string;
        sender_id?: string;
    }
): Promise<[Object, string]>;

Note: Publishing via the transport layer is the caller's responsibility.

JavaScript (Browser)

async function smartsend(
    subject: string,
    data: Array<[string, any, string]>,
    options?: {
        broker_url?: string;
        fileserver_url?: string;
        fileserver_upload_handler?: Function;
        size_threshold?: number;
        correlation_id?: string;
        msg_purpose?: string;
        sender_name?: string;
        receiver_name?: string;
        receiver_id?: string;
        reply_to?: string;
        reply_to_msg_id?: string;
        msg_id?: string;
        sender_id?: string;
    }
): Promise<[Object, string]>;

Note: Publishing via the transport layer is the caller's responsibility.

MicroPython

def smartsend(
    subject: str,
    data: List[Tuple[str, Any, str]],
    size_threshold: int = 100_000,  # Lower threshold for memory constraints
    **kwargs
) -> Tuple[Dict, str]:

Note: Publishing via the transport layer is the caller's responsibility.

Dart (Desktop/Flutter)

Future<[Map<String, dynamic>, String]> smartsend(
  String subject,
  List<List<dynamic>> data, {
 String brokerUrl = DEFAULT_BROKER_URL,
  String fileserverUrl = DEFAULT_FILESERVER_URL,
  Function? fileserverUploadHandler,
  int sizeThreshold = DEFAULT_SIZE_THRESHOLD,
  String? correlationId,
  String msgPurpose = 'chat',
  String senderName = 'msghandler',
  String receiverName = '',
  String receiverId = '',
  String replyTo = '',
  String replyToMsgId = '',
  String? msgId,
  String? senderId,
}) async {
  // Returns [envelope, jsonString]
  // Publishing via transport layer is caller's responsibility
}

#### Dart Web

```dart
Future<[Map<String, dynamic>, String]> smartsend(
  String subject,
  List<List<dynamic>> data, {
  String brokerUrl = DEFAULT_BROKER_URL,
  String fileserverUrl = 'http://localhost:8080',
  Function? fileserverUploadHandler,
  int sizeThreshold = 500000,
  String? correlationId,
  String msgPurpose = 'chat',
  String senderName = 'msghandler',
  String receiverName = '',
  String receiverId = '',
  String replyTo = '',
  String replyToMsgId = '',
  String? msgId,
  String? senderId,
}) async {
  // Returns [envelope, jsonString]
  // Publishing via transport layer is caller's responsibility
}

Rust

pub async fn smartsend(
    subject: &str,
    data: &[(String, Payload, String)],
    options: &SmartsendOptions,
) -> Result<(MsgEnvelopeV1, String), msghandlerError>

// SmartsendOptions struct
pub struct SmartsendOptions {
    pub broker_url: String,
    pub fileserver_url: String,
    pub fileserver_upload_handler: Option<UploadHandler>,
    pub size_threshold: usize,
    pub correlation_id: String,
    pub msg_purpose: String,
    pub sender_name: String,
    pub receiver_name: String,
    pub receiver_id: String,
    pub reply_to: String,
    pub reply_to_msg_id: String,
    pub msg_id: String,
    pub sender_id: String,
}

// Payload enum for type-safe data handling
#[derive(Serialize, Deserialize, Clone)]
pub enum Payload {
    Text(String),
    Dictionary(serde_json::Value),
    ArrowTable(Vec<u8>),
    JsonTable(serde_json::Value),
    Image(Vec<u8>),
    Audio(Vec<u8>),
    Video(Vec<u8>),
    Binary(Vec<u8>),
}

// MsgEnvelopeV1 struct (serde-serializable)
#[derive(Serialize, Deserialize, Clone)]
pub struct MsgEnvelopeV1 {
    pub correlation_id: String,
    pub msg_id: String,
    pub timestamp: String,
    pub send_to: String,
    pub msg_purpose: String,
    pub sender_name: String,
    pub sender_id: String,
    pub receiver_name: String,
    pub receiver_id: String,
    pub reply_to: String,
    pub reply_to_msg_id: String,
    pub broker_url: String,
    pub metadata: serde_json::Value,
    pub payloads: Vec<MsgPayloadV1>,
}

Note: Publishing via the transport layer is the caller's responsibility. Returns Result<(MsgEnvelopeV1, String), msghandlerError>. Uses serde for JSON serialization.

smartreceive Function Signature

Julia

function smartreceive(
    msg_json_str::String;  # Pass payload from transport subscription
    fileserver_download_handler::Function = _fetch_with_backoff,
    max_retries::Int = 5,
    base_delay::Int = 100,
    max_delay::Int = 5000
)::JSON.Object{String, Any}

Note: Input is the JSON string payload from the transport subscription, not the transport message object directly.

Python

async def smartreceive(
    msg_json_str: str,  # JSON string from transport message payload
    fileserver_download_handler: Callable = fetch_with_backoff,
    max_retries: int = 5,
    base_delay: int = 100,
    max_delay: int = 5000
) -> Dict[str, Any]:

Note: Input is the JSON string payload from the transport message.

JavaScript (Node.js)

async function smartreceive(
    msg_json_str: string,  // JSON string from transport message payload
    options?: {
        fileserver_download_handler?: Function;
        max_retries?: number;
        base_delay?: number;
        max_delay?: number;
    }
): Promise<Object>;

JavaScript (Browser)

async function smartreceive(
    msg_json_str: string,  // JSON string from transport message payload
    options?: {
        fileserver_download_handler?: Function;
        max_retries?: number;
        base_delay?: number;
        max_delay?: number;
    }
): Promise<Object>;

Note: Input is the JSON string payload from the transport message.

MicroPython

def smartreceive(msg_json_str: str, **kwargs) -> Dict[str, Any]:

Note: Input is the JSON string payload from the transport message.

Dart (Desktop/Flutter)

Future<Map<String, dynamic>> smartreceive(
  Map<String, dynamic> msg_json_str,  // JSON object from transport message payload
  {
  Function? fileserverDownloadHandler,
  int maxRetries = 5,
  int baseDelay = 100,
  int maxDelay = 5000,
}) async {
  // Returns envelope with processed payloads
}

Dart Web

Future<Map<String, dynamic>> smartreceive(
  Map<String, dynamic> msg_json_str,  // JSON object from transport message payload
  {
  Function? fileserverDownloadHandler,
  int maxRetries = 5,
  int baseDelay = 100,
  int maxDelay = 5000,
}) async {
  // Returns envelope with processed payloads
}

Rust

pub async fn smartreceive(
    msg_json_str: &str,  // JSON string from transport message payload
    options: &SmartreceiveOptions,
) -> Result<MsgEnvelopeV1, msghandlerError>

// SmartreceiveOptions struct
pub struct SmartreceiveOptions {
    pub fileserver_download_handler: Option<DownloadHandler>,
    pub max_retries: u32,
    pub base_delay: u64,
    pub max_delay: u64,
}

Note: Input is the JSON string payload from the transport message. Returns Result<MsgEnvelopeV1, msghandlerError>.

File Server Interface

Upload Handler Contract

Function Signature:

function fileserver_upload_handler(
    file_server_url::String,
    dataname::String,
    data::Vector{UInt8}
)::Dict{String, Any}

# Overload: Upload file from disk
function fileserver_upload_handler(
    file_server_url::String,
    filepath::String
)::Dict{String, Any}

Return Format:

{
  "status": 200,
  "uploadid": "string",
  "fileid": "string",
  "url": "string"
}

Required Keys:

Key Type Description
status integer HTTP response status code
uploadid string Upload session identifier
fileid string File identifier within session
url string Full download URL

Download Handler Contract

Function Signature:

function fileserver_download_handler(
    url::String,
    max_retries::Int,
    base_delay::Int,
    max_delay::Int,
    correlation_id::String
)::Vector{UInt8}

Retry Policy:

  • Initial delay: base_delay milliseconds
  • Maximum delay: max_delay milliseconds
  • Multiplier: 2x per retry
  • Maximum retries: max_retries

Platform-Specific Constraints

Desktop (Julia/Python/Node.js/Dart)

Feature Status Notes
Arrow IPC ✅ Supported Requires Arrow.jl/pyarrow/dart-arrow
JSON table ✅ Supported Human-readable format
File server upload ✅ Supported HTTP/HTTPS
File server download ✅ Supported HTTP/HTTPS
Size threshold 500KB Configurable

Browser (JavaScript)

Feature Status Notes
Arrow IPC ❌ Not supported Apache Arrow not browser-compatible
JSON table ✅ Supported Only table type available in browser
File server upload ✅ Supported HTTP/HTTPS
File server download ✅ Supported HTTP/HTTPS
Size threshold 500KB Configurable

Dart Desktop (Dart SDK)

Feature Status Notes
Arrow IPC ✅ Supported Requires dart-arrow package
JSON table ✅ Supported Human-readable format
File server upload ✅ Supported HTTP/HTTPS
File server download ✅ Supported HTTP/HTTPS
Size threshold 500KB Configurable

Dart Flutter (Dart SDK)

Feature Status Notes
Arrow IPC ✅ Supported Requires dart-arrow package
JSON table ✅ Supported Human-readable format
File server upload ✅ Supported HTTP/HTTPS
File server download ✅ Supported HTTP/HTTPS
Size threshold 500KB Configurable

Dart Web (Dart SDK)

Feature Status Notes
Arrow IPC ❌ Not supported Apache Arrow not browser-compatible
JSON table ✅ Supported Only table type available in browser
File server upload ✅ Supported HTTP/HTTPS
File server download ✅ Supported HTTP/HTTPS
Size threshold 500KB Configurable

Rust

Feature Status Notes
Arrow IPC ✅ Supported Requires arrow2 crate
JSON table ✅ Supported Uses serde_json
File server upload ✅ Supported HTTP/HTTPS via reqwest
File server download ✅ Supported HTTP/HTTPS via reqwest with retry
Size threshold 500KB Configurable
Async runtime ✅ Supported Uses tokio for async I/O
Type safety ✅ Supported Compile-time type checking via Rust enums

MicroPython

Feature Status Notes
Arrow IPC ❌ Not supported Memory constraints
JSON table ⚠️ Limited Only direct transport
File server upload ❌ Not implemented Placeholder only
File server download ❌ Not implemented Placeholder only
Size threshold 100KB Hard limit enforced
Max payload 50KB Hard limit enforced

Implementation Files

File Platform Features Notes
src/msghandler.jl Julia Full feature set, Arrow IPC, multiple dispatch Ground truth implementation
src/msghandler_ssr.js Node.js Arrow IPC, async/await Server-side JavaScript
src/msghandler_csr.js Browser JSON table only Client-side rendering
src/msghandler.py Python Arrow IPC, async/await Desktop Python
src/msghandler.dart Dart Full feature set, Arrow IPC, async/await Desktop/Flutter/Web
src/msghandler.rs Rust Full feature set, Arrow IPC, async/await, type-safe Uses tokio + serde + arrow2
src/msghandler_mpy.py MicroPython Limited to direct transport Memory-constrained

Browser Implementation Notes

The browser implementation (src/msghandler_csr.js) has the following constraints:

Constraint Reason Workaround
No Apache Arrow IPC Browser-incompatible dependency Use jsontable for tabular data
WebSocket only Browser cannot use TCP directly Use ws:// or wss:// broker URLs
Fetch API for HTTP Browser fetch() API only Compatible with Plik and other HTTP servers

Payload Type Availability by Platform

Payload Type Julia Node.js Browser Python Dart Rust MicroPython
text ✅ ✅ ✅ ✅ ✅ ✅ ✅
dictionary ✅ ✅ ✅ ✅ ✅ ✅ ✅
arrowtable ✅ ✅ ❌ ✅ ✅ ✅ ❌
jsontable ✅ ✅ ✅ ✅ ✅ ✅ ⚠️
image ✅ ✅ ✅ ✅ ✅ ✅ ✅
audio ✅ ✅ ✅ ✅ ✅ ✅ ✅
video ✅ ✅ ✅ ✅ ✅ ✅ ✅
binary ✅ ✅ ✅ ✅ ✅ ✅ ✅

Message Flow

Sending Flow

flowchart TD
    A[User calls smartsend subject data] --> B[Serialize payload according to payload_type]
    B --> C{Calculate serialized size}
    C -->|Size < Threshold| D[Direct Transport: Encode as Base64]
    C -->|Size >= Threshold| E[Link Transport: Upload to file server]
    D --> F[Build envelope with metadata]
    E --> F
    F --> G[Convert envelope to JSON string]
    G --> H[Publish to topic via transport]
    H --> I[Return envelope and JSON string to caller]
    
    style A fill:#f9f9f9,stroke:#333
    style I fill:#e0e7ff,stroke:#3b82f6
    style D fill:#d1fae5,stroke:#10b981
    style E fill:#fef3c7,stroke:#f59e0b

Receiving Flow

flowchart TD
  A[Transport message arrives] --> B[Parse JSON envelope]
    B --> C[For each payload: Check transport type]
    C -->|transport == direct| D[Direct Transport: Extract Base64]
    C -->|transport == link| E[Link Transport: Fetch from URL]
    D --> F[Decode Base64]
    E --> G[Fetch with exponential backoff]
    F --> H[Deserialize based on payload_type]
    G --> H
    H --> I[Build payloads array]
    I --> J[Replace payloads array with deserialized tuples]
    J --> K[Return envelope with processed payloads]
    
    style A fill:#f9f9f9,stroke:#333
    style K fill:#e0e7ff,stroke:#3b82f6
    style D fill:#d1fae5,stroke:#10b981
    style E fill:#fef3c7,stroke:#f59e0b

Validation Rules

Envelope Validation

Rule Condition Error Code Requirement ID
Required fields present correlation_id, msg_id, timestamp, send_to, payloads INVALID_ENVELOPE FR-012, FR-013
Valid UUID format correlation_id, msg_id, sender_id, receiver_id INVALID_ENVELOPE FR-011, FR-012, NFR-401
Valid timestamp format ISO 8601 UTC INVALID_ENVELOPE FR-012, NFR-401
Non-empty payloads array length(payloads) > 0 INVALID_ENVELOPE FR-012, FR-013

Payload Validation

Rule Condition Error Code Requirement ID
Valid payload_type Must be in payload_type enum INVALID_PAYLOAD_TYPE FR-001, FR-002, FR-003, FR-006
Valid transport Must be direct or link INVALID_TRANSPORT FR-003, FR-004, FR-006
Valid encoding Must match payload_type and transport INVALID_TRANSPORT FR-001, FR-002, FR-003, FR-012
Positive size size > 0 INVALID_PAYLOAD FR-003, FR-004, NFR-104, NFR-105
Valid Base64 for direct data matches Base64 pattern DESERIALIZATION_ERROR FR-001, FR-002, FR-003, FR-012
Valid URL for link data matches HTTP(S) URL pattern DOWNLOAD_FAILED FR-008, FR-009, FR-010

Test Contracts

Unit Test Validation

Test Input Expected Output Notes Requirement ID
Text round-trip ("msg", "Hello", "text") ("msg", "Hello", "text") String serialization FR-001, FR-012, NFR-101, NFR-102
Dictionary round-trip ("data", {"key": "value"}, "dictionary") ("data", {"key": "value"}, "dictionary") JSON object round-trip FR-002, FR-012, NFR-101, NFR-102
Arrow table round-trip ("table", arrow_table_data, "arrowtable") ("table", arrow_table_data, "arrowtable") Arrow IPC round-trip FR-002, FR-012, NFR-101, NFR-102
JSON table round-trip ("table", [{"a":1},{"b":2}], "jsontable") ("table", [{"a":1},{"b":2}], "jsontable") JSON array of objects FR-001, FR-002, FR-006, FR-012
Mixed payloads [("msg", "Hello", "text"), ("imgname", bytes, "binary")] [("msg", "Hello", "text"), ("imgname", bytes, "binary")] Multiple payload types FR-006, FR-007
Large payload ("data", rand(10_000_000), "arrowtable") ("data", URL, "arrowtable") with link transport File server upload FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105

Platform-Specific Notes:

  • Julia: Use Dict, Vector{Dict}, or convert DataFrame to dictionary for testing
  • Python: Use dict, list[dict], or convert pandas.DataFrame to dictionary for testing
  • JavaScript: Use plain objects {} and arrays []
  • MicroPython: Use plain dict and list (limited to JSON table and text types)

Integration Test Scenarios

Scenario Platforms Payloads Size Mix Transport Expected Result Requirement ID
Single text (small) All text Small direct Round-trip successful FR-001, FR-012, NFR-101, NFR-102
Single dictionary (small) All dictionary Small direct Round-trip successful FR-002, FR-012, NFR-101, NFR-102
Single arrow table (small) Julia/JS/Python arrowtable Small direct Arrow IPC round-trip FR-002, FR-012, NFR-101, NFR-102
Single JSON table (small) All jsontable Small direct Dictionary array round-trip FR-001, FR-002, FR-006, FR-012
Single image (small) All image Small direct Binary round-trip FR-001, FR-006, FR-012
Single audio (small) All audio Small direct Binary round-trip FR-001, FR-006, FR-012
Single video (small) All video Small direct Binary round-trip FR-001, FR-006, FR-012
Single binary (small) All binary Small direct Binary round-trip FR-001, FR-006, FR-012
Single text (large) All text Large link File server upload/download FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105
Single JSON table (large) All jsontable Large link File server upload/download FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105
Single image (large) All image Large link File server upload/download FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105
Ultimate Test Julia/JS/Python text (small) + dictionary (small) + arrowtable (small) + jsontable (small) + image (small) + audio (small) + video (small) + binary (small) + text (large) + dictionary (large) + arrowtable (large) + jsontable (large) + image (large) Mixed direct/link All payloads preserved with correct transport FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014
Ultimate Test MicroPython text (small) + dictionary (small) + text (large) + dictionary (large) Mixed direct Limited to text/dictionary with direct transport only FR-005, FR-006, FR-012
Cross-platform JSON table All jsontable Small direct Dictionary array round-trip FR-001, FR-002, FR-006, FR-012
MicroPython ↔ Desktop MicroPython ↔ Desktop text/dictionary Small direct Limited payload types FR-005, FR-006, FR-012
Desktop ↔ Desktop (all combos) Julia↔JS↔Python All types Small/Large direct/link Full compatibility FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014

Dependencies

Required Dependencies by Platform

Platform Package Version Purpose
Julia JSON.jl Latest JSON serialization
Julia Arrow.jl Latest Arrow IPC support
Julia HTTP.jl Latest HTTP file server
Julia UUIDs.jl Latest UUID generation
Node.js node-fetch Latest HTTP file server
Browser - - Transport-agnostic (caller provides)
Python aiohttp Latest HTTP file server
Python pyarrow Latest Arrow IPC support
Dart http Latest HTTP file server
Dart uuid Latest UUID generation
Dart dart-arrow Latest Arrow IPC support (Desktop/Flutter)
Rust serde Latest JSON serialization
Rust serde_json Latest JSON handling
Rust tokio Latest Async runtime
Rust reqwest Latest HTTP file server
Rust uuid Latest UUID generation
Rust arrow2 Latest Arrow IPC support
MicroPython builtin N/A Limited implementation

Optional Dependencies

Platform Package Purpose
Julia DataFrames.jl DataFrame support
Python pandas DataFrame support

Change Log

Date Version Changes
2026-05-15 1.3.0 Made transport layer agnostic
- - Removed all NATS-specific dependencies (NATS.jl, nats, nats-py, nats.ws)
- - Updated docs to reference generic message broker/transport
- - broker_url is now metadata only, not used for active connections
2026-03-15 1.1.0 Browser connection management
- - Added NATSClient class with keepAlive support
- - Added NATSConnectionPool for connection reuse
- - Added publishMessage function with closeConnection option
- - Added nats.ws to browser dependencies
2026-03-13 1.0.0 Initial specification
- - Message envelope schema defined
- - Payload schema with transport modes
- - Enumerations for payload_type, transport, encoding
- - Size thresholds for desktop/MicroPython
- - Error codes and validation rules
- - API contracts for all platforms

References

20.1 Documentation Artifacts

Document Purpose Requirements Traceability
docs/requirements.md Business requirements and user stories FR-001 through FR-014, NFR-101 through NFR-405
docs/specification.md Technical contract for msghandler This document
docs/ui-specification.md UI specification for client applications UI components for data entry and display
docs/walkthrough.md End-to-end system flow Traceability from user journey to technical implementation
docs/architecture.md System architecture diagrams Component interaction and data flow
docs/validation.md CI/CD validation rules Contract testing and spec compliance
docs/runbook.md Operational runbook Deployment, scaling, and troubleshooting

20.2 Implementation Files

File Platform Features Requirements Traceability
src/msghandler.jl Julia Full feature set, Arrow IPC, multiple dispatch FR-001 through FR-014, NFR-101 through NFR-405
src/msghandler_ssr.js Node.js Arrow IPC, async/await FR-001 through FR-014, NFR-101 through NFR-405
src/msghandler_csr.js Browser JSON table only FR-001 through FR-014, NFR-101 through NFR-405
src/msghandler.py Python Arrow IPC, async/await FR-001 through FR-014, NFR-101 through NFR-405
src/msghandler.dart Dart Full feature set, Arrow IPC, async/await FR-001 through FR-014, NFR-101 through NFR-405
src/msghandler.rs Rust Full feature set, Arrow IPC, async/await, type-safe FR-001 through FR-014, NFR-101 through NFR-405
src/msghandler_mpy.py MicroPython Limited to direct transport FR-005, FR-006, FR-012

20.3 External Dependencies

Platform Package Version Purpose Requirements Traceability
Julia JSON.jl Latest JSON serialization FR-012, NFR-101, NFR-102
Julia Arrow.jl Latest Arrow IPC support FR-002, FR-012
Julia HTTP.jl Latest HTTP file server FR-008, FR-009
Julia UUIDs.jl Latest UUID generation FR-011, NFR-401
Node.js node-fetch Latest HTTP file server FR-008, FR-009
Browser - - Transport-agnostic (caller provides) FR-013, FR-014
Python aiohttp Latest HTTP file server FR-008, FR-009
Python pyarrow Latest Arrow IPC support FR-002, FR-012
Dart http Latest HTTP file server FR-008, FR-009
Dart uuid Latest UUID generation FR-011, NFR-401
Dart dart-arrow Latest Arrow IPC support FR-002, FR-012
Rust serde Latest JSON serialization FR-012, NFR-101, NFR-102
Rust serde_json Latest JSON handling FR-012, NFR-101, NFR-102
Rust tokio Latest Async runtime FR-013, FR-014
Rust reqwest Latest HTTP file server FR-008, FR-009
Rust uuid Latest UUID generation FR-011, NFR-401
Rust arrow2 Latest Arrow IPC support FR-002, FR-012
MicroPython builtin N/A Limited implementation FR-005, FR-006

21. Change Log

Date Version Changes Requirement ID(s)
2026-05-15 1.3.0 Made transport layer agnostic All
- - Removed NATS-specific dependencies and references from all docs All
- - Updated all NATS references to generic "transport layer"/"message broker" All
- - Removed NATS client packages from dependencies tables All
2026-05-13 1.2.0 Aligned with ground truth implementation (src/msghandler.jl) All
- - Updated smartsend signatures: removed is_publish, nats_connection; added sender_name FR-001 through FR-014
- - Updated smartreceive signatures: takes msg_json_str::String instead of msg FR-001 through FR-014
- - Removed publishMessage function and NATSClient/NATSConnectionPool classes from browser section FR-013, FR-014
- - Added plik_oneshot_upload(filepath) overload to file server interface FR-008, FR-009
- - Fixed SIZE_THRESHOLD default to 500,000 bytes FR-003, FR-004
2026-03-23 1.1.0 Updated to ASG Framework specification guidelines All
2026-03-15 1.1.0 Browser connection management FR-001 through FR-014
2026-03-13 1.0.0 Initial specification FR-001 through FR-014, NFR-101 through NFR-405

Appendix

A. Complete JSON Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "msghandler Envelope",
  "type": "object",
  "properties": {
    "correlation_id": {
      "type": "string",
      "pattern": "^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$",
      "description": "UUID v4 format for tracking message flow"
    },
    "msg_id": {
      "type": "string",
      "pattern": "^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$",
      "description": "Unique message identifier"
    },
    "timestamp": {
      "type": "string",
      "pattern": "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3}Z$",
      "description": "ISO 8601 UTC timestamp"
    },
    "send_to": {
      "type": "string",
      "minLength": 1,
      "description": "Topic/subject to publish to"
    },
    "msg_purpose": {
      "type": "string",
      "enum": ["ACK", "NACK", "updateStatus", "shutdown", "chat", "command", "event"],
      "description": "Purpose of the message"
    },
    "sender_name": {
      "type": "string",
      "minLength": 1,
      "description": "Sender application name"
    },
    "sender_id": {
      "type": "string",
      "pattern": "^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$",
      "description": "Sender UUID"
    },
    "receiver_name": {
      "type": "string",
      "description": "Receiver name (empty = broadcast)"
    },
    "receiver_id": {
      "type": "string",
      "pattern": "^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$|^$",
      "description": "Receiver UUID (empty = broadcast)"
    },
    "reply_to": {
      "type": "string",
      "description": "Topic for reply messages"
    },
    "reply_to_msg_id": {
      "type": "string",
      "description": "Message ID being replied to"
    },
    "broker_url": {
      "type": "string",
      "description": "Broker URL for the transport layer"
    },
    "metadata": {
      "type": "object",
      "description": "Message-level metadata"
    },
    "payloads": {
      "type": "array",
      "minItems": 1,
      "items": {
        "$ref": "#/definitions/Payload"
      }
    }
  },
  "required": ["correlation_id", "msg_id", "timestamp", "send_to", "msg_purpose", "sender_name", "sender_id", "receiver_name", "receiver_id", "reply_to", "reply_to_msg_id", "broker_url", "payloads"],
  "definitions": {
    "Payload": {
      "type": "object",
      "properties": {
        "id": {
          "type": "string",
          "pattern": "^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$"
        },
        "dataname": {
          "type": "string",
          "minLength": 1
        },
        "payload_type": {
          "type": "string",
          "enum": ["text", "dictionary", "arrowtable", "jsontable", "image", "audio", "video", "binary"]
        },
        "transport": {
          "type": "string",
          "enum": ["direct", "link"]
        },
        "encoding": {
          "type": "string",
          "enum": ["none", "base64", "json", "arrow-ipc"]
        },
        "size": {
          "type": "integer",
          "minimum": 1
        },
        "data": {
          "anyOf": [
            {
              "type": "string",
              "pattern": "^(https?://[^\\s]+)$"
            },
            {
              "type": "string",
              "pattern": "^[A-Za-z0-9+/]+=*$"
            }
          ]
        },
        "metadata": {
          "type": "object"
        }
      },
      "required": ["id", "dataname", "payload_type", "transport", "encoding", "size", "data"]
    }
  }
}

B. AsyncAPI Specification

asyncapi: '2.6.0'
info:
  title: msghandler API
  version: '1.0.0'
  description: Cross-platform bi-directional data bridge using a message broker
  contact:
    name: msghandler Team
    url: https://github.com/your-org/msghandler
  license:
    name: MIT
    url: https://opensource.org/licenses/MIT
channels:
  /agent/{service}/api/v{version}/{operation}:
    address: /agent/{service}/api/v{version}/{operation}
    parameters:
      service:
        schema:
          type: string
      version:
        schema:
          type: string
          enum: ['v1']
      operation:
        schema:
          type: string
    publish:
      summary: Publish message to transport
      operationId: publishMessage
      message:
        $ref: '#/components/message'
    subscribe:
      summary: Subscribe to messages from transport
      operationId: subscribeMessage
      message:
        $ref: '#/components/message'
components:
  message:
    payload:
      $ref: '#/components/schemas/Envelope'
  schemas:
    Envelope:
      type: object
      properties:
        correlation_id:
          type: string
          format: uuid
        msg_id:
          type: string
          format: uuid
        timestamp:
          type: string
          format: date-time
        send_to:
          type: string
        msg_purpose:
          type: string
          enum: [ACK, NACK, updateStatus, shutdown, chat, command, event]
        sender_name:
          type: string
        sender_id:
          type: string
          format: uuid
        receiver_name:
          type: string
        receiver_id:
          type: string
          format: uuid
        reply_to:
          type: string
        reply_to_msg_id:
          type: string
        broker_url:
          type: string
        metadata:
          type: object
        payloads:
          type: array
          items:
            $ref: '#/components/schemas/Payload'
      required:
        - correlation_id
        - msg_id
        - timestamp
        - send_to
        - msg_purpose
        - sender_name
        - sender_id
        - receiver_name
        - receiver_id
        - reply_to
        - reply_to_msg_id
        - broker_url
        - payloads
    Payload:
      type: object
      properties:
        id:
          type: string
          format: uuid
        dataname:
          type: string
        payload_type:
          type: string
          enum: [text, dictionary, arrowtable, jsontable, image, audio, video, binary]
        transport:
          type: string
          enum: [direct, link]
        encoding:
          type: string
          enum: [none, base64, json, arrow-ipc]
        size:
          type: integer
          minimum: 1
        data:
          type: string
        metadata:
          type: object
      required:
        - id
        - dataname
        - payload_type
        - transport
        - encoding
        - size
        - data

This specification is versioned and maintained in git alongside the codebase. All implementations must adhere to this specification.

Reference in New Issue View Git Blame Copy Permalink