Files

narawat 396e0848da rename to smartpack n smartunpack

2026-05-18 19:30:58 +07:00

36 KiB

Raw Blame History

Architecture Documentation: msghandler

Version: 1.4.0
Date: 2026-05-14
Status: Active
Ground Truth: src/msghandler.jl
Architecture Level: C4 Container Level

1. Executive Summary

This document defines the blueprint for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between Julia, JavaScript, Python, Dart, Rust, and MicroPython applications using a message broker as the transport layer.

This architecture document serves as the single source of truth for:

System Structure: How components fit together and interact
Scaling Considerations: How the system scales horizontally and vertically
Failure Modes: How the system handles failures and recovers
Trade-off Decisions: The rationale behind architectural decisions

1.1 Specification Traceability

Architecture Section	Specification Reference	UI Specification Reference	Requirement ID(s)
Section 2 (Context Diagram)	specification.md:2	-	FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014
Section 3 (Container Diagram)	specification.md:2, specification.md:3, specification.md:11	-	FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014
Section 4 (Component Diagram)	specification.md:2, specification.md:3, specification.md:5, specification.md:11	-	FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014
Section 5 (High-Level)	specification.md:2, specification.md:3, specification.md:5, specification.md:11	-	FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014
Section 6 (Message Envelope)	specification.md:2, specification.md:3, specification.md:8	-	FR-011, FR-012, FR-013, FR-014, NFR-401, NFR-403
Section 7 (Payload Type)	specification.md:3, specification.md:5, specification.md:6	-	FR-001, FR-002, FR-003, FR-006, FR-012, NFR-101, NFR-102
Section 8 (Transport Strategy)	specification.md:6, specification.md:7	-	FR-003, FR-004, FR-005, FR-010, NFR-104, NFR-105, NFR-106
Section 9 (Platform-Specific)	specification.md:13, specification.md:14	-	FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014
Section 10 (Scaling)	specification.md:7, specification.md:13	-	NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107
Section 11 (Failure Modes)	specification.md:9, specification.md:11	-	FR-008, FR-009, FR-010, FR-011, NFR-201, NFR-202, NFR-203
Section 12 (Trade-offs)	specification.md:2, specification.md:3, specification.md:6, specification.md:7	-	FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014
Section 13 (Deployment)	specification.md:12, specification.md:18	-	FR-013, FR-014, NFR-201, NFR-203
Section 14 (Security)	specification.md:4, specification.md:9, specification.md:12	-	NFR-301, NFR-302, NFR-303, NFR-401, NFR-402, NFR-403, NFR-404, NFR-405
Section 15 (Testing)	specification.md:17	-	FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014

2. Architecture Overview

Architecture Overview

C4 Context Diagram

flowchart TD
   subgraph "External Systems"
        Message_Broker[Message Broker<br/>NATS/MQTT/WebSocket/Custom]
        File_Server[HTTP File Server<br/>Plik/AWS S3/Custom]
    end

    Julia_App[Julia Application]
    JS_App[JavaScript Application<br/>Node.js/Browser]
    Python_App[Python Application<br/>Desktop]
    Dart_App[Dart Application<br/>Desktop/Flutter/Web]
    Rust_App[Rust Application<br/>Server/Desktop]
    MicroPython_App[MicroPython Device]
    end

    Julia_App -->|Transport| Message_Broker
    JS_App -->|Transport| Message_Broker
    Python_App -->|Transport| Message_Broker
    Dart_App -->|Transport| Message_Broker
    Rust_App -->|Transport| Message_Broker
    MicroPython_App -->|Transport| Message_Broker

    Julia_App -->|HTTP| File_Server
    JS_App -->|HTTP| File_Server
    Python_App -->|HTTP| File_Server
    Dart_App -->|HTTP| File_Server
    Rust_App -->|HTTP| File_Server
    MicroPython_App -->|HTTP| File_Server

    style Message_Broker fill:#fff3e0,stroke:#f57c00
    style File_Server fill:#f3e5f5,stroke:#9c27b4
    style Julia_App fill:#e8f5e9,stroke:#4caf50
    style JS_App fill:#e3f2fd,stroke:#2196f3
    style Python_App fill:#e3f2fd,stroke:#2196f3
    style Dart_App fill:#fff0f6,stroke:#e91e63
    style Rust_App fill:#dea584,stroke:#e65100
    style MicroPython_App fill:#fce4ec,stroke:#e91e63

C4 Container Diagram

flowchart TD
  subgraph "Client Container"
        Julia_Module[Julia msghandler Module]
        JS_Module[JavaScript msghandler Module]
        Python_Module[Python msghandler Module]
        Dart_Module[Dart msghandler Module]
        Rust_Module[Rust msghandler Module]
        MicroPython_Module[MicroPython msghandler Module]
    end

    Julia_Module --> Transport_Client
    JS_Module --> Transport_Client
    Python_Module --> Transport_Client
    Dart_Module --> Transport_Client
    Rust_Module --> Transport_Client
    MicroPython_Module --> Transport_Client

    Transport_Client --> Message_Broker

    Julia_Module --> File_Client
    JS_Module --> File_Client
    Python_Module --> File_Client
    Dart_Module --> File_Client
    Rust_Module --> File_Client
    MicroPython_Module --> File_Client

    File_Client --> File_Server

    style Julia_Module fill:#e8f5e9,stroke:#4caf50
    style JS_Module fill:#e3f2fd,stroke:#2196f3
    style Python_Module fill:#e3f2fd,stroke:#2196f3
    style Dart_Module fill:#fff0f6,stroke:#e91e63
    style Rust_Module fill:#dea584,stroke:#e65100
    style MicroPython_Module fill:#fce4ec,stroke:#e91e63
    style Message_Broker fill:#fff3e0,stroke:#f57c00
    style File_Server fill:#f3e5f5,stroke:#9c27b4

C4 Component Diagram (Julia Implementation)

flowchart TD
    subgraph "msghandler Module"
        smartpack[smartpack Function]
        smartunpack[smartunpack Function]
        
        Serialize[_serialize_data]
        Deserialize[_deserialize_data]
        
        EnvelopeToJson[envelope_to_json]
        
        FileServerUpload[fileserver_upload_handler]
        FileServerDownload[fileserver_download_handler]
        
        LogTrace[log_trace]
    end

    subgraph "Data Models"
        Payload[msg_payload_v1 Struct]
        Envelope[msg_envelope_v1 Struct]
    end

    smartpack --> Serialize
    smartpack --> EnvelopeToJson
    smartpack --> FileServerUpload
    
    smartunpack --> Deserialize
    smartunpack --> FileServerDownload
    
    EnvelopeToJson --> Envelope
    Serialize --> Payload

    style smartpack fill:#d1fae5,stroke:#10b981
    style smartunpack fill:#d1fae5,stroke:#10b981
    style FileServerUpload fill:#fef3c7,stroke:#f59e0b
    style FileServerDownload fill:#fef3c7,stroke:#f59e0b

High-Level Architecture

System Components

Component	Purpose	Platform Support
smartpack	Send data with automatic transport selection, returns (envelope, json_string) for caller to publish via transport	All
smartunpack	Receive and process messages from JSON string	All
_serialize_data	Serialize data according to payload type	All
_deserialize_data	Deserialize bytes to native data types	All
envelope_to_json	Convert msg_envelope_v1 struct to JSON string	All
log_trace	Log trace messages with correlation ID	All
fileserver_upload_handler	Upload large payloads to HTTP server	Desktop (Julia/JS/Python/Dart/Rust)
fileserver_download_handler	Download payloads from HTTP server with exponential backoff	Desktop (Julia/JS/Python/Dart/Rust)
plik_upload_file	Upload a local file to Plik server from disk	Rust

Data Flow

flowchart TD
    A[User calls smartpack subject data] --> B[Process each payload]
    B --> C{Calculate serialized size}
    C -->|Size < Threshold| D[Direct Transport]
    C -->|Size >= Threshold| E[Link Transport]
    
    D --> F[Serialize data]
    F --> G[Base64 encode]
    G --> H[Build payload object]
    
    E --> I[Serialize data]
    I --> J[Upload to file server]
    J --> K[Get download URL]
    K --> H
    
    H --> L[Build envelope]
    L --> M[Convert to JSON]
    M --> N[Return envelope + JSON to caller]
    
    style A fill:#f9f9f9,stroke:#333
    style N fill:#e0e7ff,stroke:#3b82f6
    style D fill:#d1fae5,stroke:#10b981
    style E fill:#fef3c7,stroke:#f59e0b

Message Envelope Architecture

msg_envelope_v1 Structure (Julia)

struct msg_envelope_v1
  correlation_id::String       # UUID v4 for distributed tracing
  msg_id::String               # UUID v4 for this message
  timestamp::String            # ISO 8601 UTC timestamp
  
  send_to::String              # Topic/subject to publish to
  msg_purpose::String          # ACK, NACK, updateStatus, shutdown, chat
  sender_name::String          # Sender application name
  sender_id::String            # UUID v4 of sender
  receiver_name::String        # Receiver application name (empty = broadcast)
  receiver_id::String          # UUID v4 of receiver (empty = broadcast)
  
  reply_to::String             # Topic for reply messages
  reply_to_msg_id::String      # Message ID being replied to
  broker_url::String           # Broker URL for the transport layer
  
  metadata::Dict{String, Any}  # Message-level metadata
  payloads::Vector{msg_payload_v1}  # List of payloads
end

msg_payload_v1 Structure (Julia)

struct msg_payload_v1
  id::String                    # UUID v4 for this payload
  dataname::String              # Name of the payload
  payload_type::String          # text, dictionary, arrowtable, etc.
  transport::String             # direct or link
  encoding::String              # none, json, base64, arrow-ipc
  size::Integer                 # Size in bytes
  data::Any                     # Base64 string or URL
  metadata::Dict{String, Any}   # Payload-level metadata
end

JSON Schema (Cross-Platform)

{
  "correlation_id": "string (UUID v4)",
  "msg_id": "string (UUID v4)",
  "timestamp": "string (ISO 8601 UTC)",
  "send_to": "string",
  "msg_purpose": "string",
  "sender_name": "string",
  "sender_id": "string (UUID v4)",
  "receiver_name": "string",
  "receiver_id": "string (UUID v4)",
  "reply_to": "string",
  "reply_to_msg_id": "string",
  "broker_url": "string",
  "metadata": "object",
  "payloads": [
    {
      "id": "string (UUID v4)",
      "dataname": "string",
      "payload_type": "string",
      "transport": "string",
      "encoding": "string",
      "size": "integer",
      "data": "string or URL",
      "metadata": "object"
    }
  ]
}

Payload Type Architecture

Supported Payload Types

Type	Description	Serialization	Encoding	Platforms
`text`	Plain text string	UTF-8 bytes	Base64	All
`dictionary`	JSON object	JSON string	Base64/JSON	All
`arrowtable`	Apache Arrow IPC	Arrow IPC stream	Base64/arrow-ipc	Desktop (Julia/Python/Node.js/Dart/Rust)
`jsontable`	JSON array of objects	JSON string	Base64/json	All (including Browser/Dart Web)
`image`	Binary image data	Raw bytes	Base64	All
`audio`	Binary audio data	Raw bytes	Base64	All
`video`	Binary video data	Raw bytes	Base64	All
`binary`	Generic binary data	Raw bytes	Base64	All

Serialization Logic

flowchart TD
    A[Input data + payload_type] --> B{Payload Type}
    
    B -->|"text"| C[UTF-8 encode]
    B -->|"dictionary"| D[JSON serialize]
    B -->|"arrowtable"| E[Arrow IPC serialize]
    B -->|"jsontable"| F[JSON serialize]
    B -->|"image"| G[Raw bytes]
    B -->|"audio"| H[Raw bytes]
    B -->|"video"| I[Raw bytes]
    B -->|"binary"| J[Raw bytes]
    
    C --> K[Return bytes]
    D --> K
    E --> K
    F --> K
    G --> K
    H --> K
    I --> K
    J --> K
    
    style A fill:#f9f9f9,stroke:#333
    style K fill:#e0e7ff,stroke:#3b82f6

Transport Strategy Architecture

Size Threshold Decision Logic

Platform	Size Threshold	Notes
Desktop (Julia/JS/Python/Dart)	500,000 bytes (0.5MB)	Default threshold
Dart Desktop	500,000 bytes (0.5MB)	Default threshold
Dart Flutter	500,000 bytes (0.5MB)	Default threshold
Dart Web	500,000 bytes (0.5MB)	Default threshold
MicroPython	100,000 bytes (100KB)	Lower threshold for memory constraints

Transport Selection Flow

flowchart TD
    A[smartpack called] --> B[Serialize payload]
    B --> C[Calculate size]
    C --> D{Size < Threshold?}
    
    D -->|Yes| E[Direct Transport]
    D -->|No| F[Link Transport]
    
    E --> G[Base64 encode]
    G --> H[Build payload with direct transport]
    
    F --> I[Upload to file server]
    I --> J[Get download URL]
    J --> K[Build payload with link transport]
    
    H --> L[Build envelope]
    K --> L
    
    style A fill:#f9f9f9,stroke:#333
    style L fill:#e0e7ff,stroke:#3b82f6
    style E fill:#d1fae5,stroke:#10b981
    style F fill:#fef3c7,stroke:#f59e0b

Direct Transport Protocol

When transport = "direct", the data field contains a Base64-encoded string of the serialized payload.

Encoding Rules:

text: UTF-8 → Base64
dictionary: JSON → Base64 (or direct JSON)
arrowtable: Arrow IPC → Base64 (or arrow-ipc)
jsontable: JSON → Base64 (or direct JSON)
image/audio/video/binary: Raw bytes → Base64

Link Transport Protocol

When transport = "link", the data field contains a URL pointing to the uploaded payload.

Upload Flow:

Serialize payload according to payload_type
Upload to HTTP file server (e.g., Plik)
Include returned URL in data field

Download Flow:

Extract URL from payload
Fetch with exponential backoff (max 5 retries)
Deserialize based on payload_type

Platform-Specific Architecture

Julia Architecture

Julia leverages multiple dispatch for type-specific implementations:

Multiple Dispatch: Function overloading based on argument types
Struct-based Data Models: Explicit type definitions with struct
Native Arrow IPC: Support via Arrow.jl
Async/Await: Tasks for non-blocking I/O

# Multiple dispatch for serialization
function _serialize_data(data::String, payload_type::String)
    # Text serialization
end

function _serialize_data(data::Dict, payload_type::String)
    # Dictionary serialization
end

function _serialize_data(data::DataFrame, payload_type::String)
    # Arrow table serialization
end

JavaScript Architecture

JavaScript uses async/await for non-blocking I/O:

Module-level Utilities: Serialization functions
Native ArrayBuffer: Binary data handling (Browser) / Buffer (Node.js)
Fetch API: HTTP file server communication

Node.js Implementation (msghandler_ssr.js)

Transport connections: Uses broker URLs (e.g., nats://, mqtt://, ws://)
Apache Arrow IPC: Full support via apache-arrow
Buffer for binary data: Native Node.js Buffer handling

Browser Implementation (msghandler_csr.js)

WebSocket connections: Uses ws:// or wss:// URLs (transport-agnostic)
No Apache Arrow: Uses jsontable for tabular data only
Uint8Array for binary data: Browser-compatible binary handling
Web Crypto API: UUID generation via crypto.getRandomValues()

Python Architecture

Python uses classes for stateful operations:

Class-based msghandler: Encapsulated API
Dataclasses: Structured data (MsgPayloadV1, MsgEnvelopeV1)
Async/await: I/O operations
pyarrow: Arrow IPC support

class msghandler:
    DEFAULT_SIZE_THRESHOLD = 500_000
    
    def __init__(self, broker_url=None, fileserver_url=None):
        self.broker_url = broker_url or self.DEFAULT_BROKER_URL
        self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL

Dart Architecture

Dart uses classes for stateful operations with async/await:

Class-based msghandler: Encapsulated API
Data classes: Structured data (MsgPayloadV1, MsgEnvelopeV1)
Async/await: I/O operations
dart-arrow: Arrow IPC support (Desktop/Flutter only)
HTTP package: HTTP file server communication
Transport package: Transport client with WebSocket support (Dart Web)

class msghandler {
  static const DEFAULT_SIZE_THRESHOLD = 500000;
  
  final String brokerUrl;
  final String fileserverUrl;
  
  msghandler({
    this.brokerUrl = DEFAULT_BROKER_URL,
    this.fileserverUrl = 'http://localhost:8080',
  });
}

Dart Desktop (Dart SDK)

Transport connections: Uses broker URLs (e.g., nats://, mqtt://)
Apache Arrow IPC: Full support via dart-arrow
Uint8List for binary data: Native Dart binary handling

Dart Flutter (Dart SDK)

Transport connections: Uses broker URLs (e.g., nats://, mqtt://)
Apache Arrow IPC: Full support via dart-arrow
Uint8List for binary data: Native Dart binary handling

Dart Web (Dart SDK)

WebSocket connections: Uses ws:// or wss:// URLs (transport-agnostic)
No Apache Arrow: Uses jsontable for tabular data only
Uint8List for binary data: Browser-compatible binary handling
Fetch API: HTTP file server communication via http package

Browser Architecture

Browser JavaScript has specific constraints due to security and compatibility:

Async/await: Native async/await support
No Apache Arrow: Arrow IPC not available in browsers
JSON table only: Use "jsontable" for tabular data
WebSocket transport: Uses transport client for browser-compatible connections
Fetch API: HTTP file server communication via fetch

MicroPython Architecture

MicroPython has significant constraints:

Synchronous API: No async/await
Memory-constrained: 256KB - 1MB
Limited payload support: No tables, max 50KB
Simplified UUID generation: Custom implementation

# MicroPython constraints
DEFAULT_SIZE_THRESHOLD = 100_000  # 100KB
MAX_PAYLOAD_SIZE = 50_000  # 50KB hard limit

Rust Architecture

Rust leverages compile-time type safety and async runtimes:

Type-safe payloads: Rust enum discriminates between Text, Dictionary, ArrowTable, Binary, etc.
serde serialization: Automatic JSON deserialization via #[derive(Serialize, Deserialize)]
tokio runtime: Efficient async I/O for transport connections and HTTP file server operations
arrow2 integration: Native Arrow IPC deserialization without intermediate format conversion
reqwest: High-performance HTTP client with built-in TLS and connection pooling
Zero-copy patterns: Vec<u8> passed directly to avoid unnecessary memory copies
Result<T, E>: Idiomatic error handling with typed error types

// Type-safe payload enum (compile-time discrimination)
#[derive(Serialize, Deserialize, Clone)]
pub enum Payload {
    Text(String),
    Dictionary(serde_json::Value),
    ArrowTable(Vec<u8>),
    JsonTable(serde_json::Value),
    Image(Vec<u8>),
    Audio(Vec<u8>),
    Video(Vec<u8>),
    Binary(Vec<u8>),
}

// Configuration via builder pattern
pub struct smartpackOptions {
    pub broker_url: String,
    pub fileserver_url: String,
    pub fileserver_upload_handler: Option<Arc<dyn FileUploadHandler>>,
    pub size_threshold: usize,
    pub correlation_id: String,
    pub msg_purpose: String,
    pub sender_name: String,
    // ... other fields
}

// Transport client with tokio integration
let conn = transport_client::connect(DEFAULT_BROKER_URL).await?;

// Subscribe and process messages
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze")?;
for msg in sub.messages() {
    let envelope = smartunpack(&String::from_utf8_lossy(&msg.payload), &Default::default()).await?;
    // Access deserialized payloads by type
    for payload in &envelope.payloads {
        match payload.payload_type.as_str() {
            "arrowtable" => { /* payload.data is base64-encoded Arrow IPC */ },
            "text" => { /* payload.data is decoded text string */ },
            "binary" | "image" | "audio" | "video" => { /* payload.data is base64-encoded binary */ },
            _ => { /* other types */ }
        }
    }
}

Scaling Architecture

Horizontal Scaling

Component	Scaling Strategy
Message Broker	Cluster deployment with multiple nodes
File Server	Load balancer + multiple instances
Client Applications	Deploy multiple instances behind load balancer

Vertical Scaling

Component	Scaling Strategy
Message Broker	Increase memory, CPU, disk I/O
File Server	Increase memory, CPU, disk capacity
Client Applications	Increase heap size (Python/JS)

Performance Considerations

Metric	Target	Notes
Message serialization overhead	<50ms	For 10KB payload
Message deserialization overhead	<50ms	For 10KB payload
Transport connection establishment	<100ms	Connection pool recommended
File upload latency	<1s	For 0.5MB file
File download latency	<1s	For 0.5MB file

Failure Modes and Recovery

Transport Connection Failure

Scenario: Message broker unavailable

Handler:

Connection auto-reconnect via transport-level reconnection
Retry with exponential backoff for publish operations

Recovery:

Transport client automatically attempts reconnection
Application can check connection status before publishing

File Server Unavailable

Scenario: HTTP file server unavailable during upload/download

Handler:

Retry up to 5 times with exponential backoff (100ms → 5000ms)
Fallback to direct transport for upload (MicroPython)

Recovery:

Exponential backoff: delay = min(delay * 2, max_delay)
After max retries, throw error with correlation ID

Deserialization Error

Scenario: Payload type mismatch or corrupted data

Handler:

Log correlation ID and throw error
No retry (data corruption)

Recovery:

Application must validate payload_type matches data type
Use proper serialization before sending

Memory Overflow (MicroPython)

Scenario: Payload exceeds maximum size (50KB)

Handler:

Reject payloads >50KB with MemoryError
No retry (client-side check)

Recovery:

Application must split large payloads
Use direct transport only for small payloads

Trade-off Decisions

Decision 1: Direct vs Link Transport Threshold

Trade-off: Memory vs Network I/O

Decision: Use 0.5MB threshold for desktop, 100KB for MicroPython

Rationale:

Direct transport uses more memory (Base64 encoding adds ~33% overhead)
Link transport requires network I/O for upload/download
0.5MB is reasonable for desktop memory constraints
100KB is necessary for MicroPython memory constraints

Decision 2: Base64 Encoding for Direct Transport

Trade-off: Bandwidth vs Simplicity

Decision: Use Base64 encoding for all direct transport payloads

Rationale:

Simplifies JSON serialization (all data is string-compatible)
Increases payload size by ~33%, but transport can handle this
Alternative would be binary payload support (more complex)

Decision 3: Multiple Platform Implementations

Trade-off: Development effort vs Cross-platform support

Decision: Maintain separate implementations for each platform

Rationale:

Each platform has idiomatic patterns (multiple dispatch, async/await, etc.)
Maintains developer productivity and code quality
API parity ensures cross-platform compatibility

Decision 4: Handler Function Abstraction

Trade-off: Flexibility vs Simplicity

Decision: Abstract file server operations through handler functions

Rationale:

Allows support for different file server implementations (Plik, AWS S3, custom)
Maintains simplicity for common use cases
Enables plug-in architecture for custom backends

Deployment Architecture

Minimum Infrastructure

Component	Minimum	Notes
Message Broker	1 instance	Single node for development
File Server	1 instance	HTTP server for large payloads
Client Memory	50MB	Desktop platforms (Julia/JS/Python/Dart)
Client Memory	256KB	MicroPython devices

Environment Variables

Variable	Default	Description
`BROKER_URL`	`ws://localhost:4222`	Message broker URL
`FILESERVER_URL`	`http://localhost:8080`	HTTP file server URL
`SIZE_THRESHOLD`	`500000`	Size threshold in bytes (0.5MB)

Container Deployment

flowchart TD
    subgraph "Docker Network"
        Broker_Container[Message Broker]
        FileServer_Container[Plik File Server]
        App_Container[Application Container]
    end
    
    App_Container -->|Transport| Broker_Container
    App_Container -->|HTTP| FileServer_Container
    
    style Broker_Container fill:#fff3e0,stroke:#f57c00
    style FileServer_Container fill:#f3e5f5,stroke:#9c27b4
    style App_Container fill:#e3f2fd,stroke:#2196f3

Security Considerations

Payload Integrity

Mechanism: SHA-256 checksum via metadata

Implementation:

Sender calculates checksum and stores in payload metadata
Receiver validates checksum on receipt

Transport Security

Mechanism: TLS support for transport connections

Implementation:

Use nats:// URL for plain text
Use tls:// URL for TLS-encrypted connections
Use ws:// or wss:// for WebSocket connections

File Server Security

Mechanism: Authentication token for file uploads

Implementation:

Plik uses upload token in X-UploadToken header
Application can implement custom authentication

Testing Architecture

Unit Test Coverage

Test Category	Coverage	Files
Serialization	All payload types	`test/test__sender.`
Deserialization	All payload types	`test/test__receiver.`
Transport selection	Direct vs link	`test/test__mix_payloads.`
File server upload	Plik integration	Platform-specific
File server download	Exponential backoff	Platform-specific

Integration Test Scenarios

Scenario	Platforms	Payloads	Transport	Expected Result
Cross-platform text	Julia ↔ JS ↔ Python	text	direct	Round-trip successful
Arrow IPC round-trip	Julia ↔ JS ↔ Python	arrowtable	direct	Arrow IPC preserved
Large file transfer	All	image/audio/video	link	File server upload/download
Multi-payload mixed	All	text + image + file	direct/link	All payloads preserved

Versioning

Architecture Versioning

Component	Version	Notes
Architecture	1.0.0	Initial release
Protocol	v1	Message envelope protocol version

Backward Compatibility

Version	Supported Platforms
v1.0.x	Julia 1.7+, Node.js 16+, Python 3.8+, Dart 2.17+, Rust 1.70+, MicroPython 1.19+

Change Log

Date	Version	Changes
2026-05-15	1.5.0	Made transport layer agnostic
-	-	Removed all NATS-specific references from architecture docs
-	-	Updated diagrams to use generic "Message Broker" instead of "NATS Server"
-	-	Updated code examples to use transport-agnostic patterns
-	-	Removed NATS client packages from external dependencies
2026-05-14	1.4.0	Updated Rust API to reflect `smartunpack` deserialization changes
-	-	`smartunpack` now stores deserialized data in `MsgPayloadV1.data`
-	-	Added `plik_upload_file` convenience function to component table
-	-	Fixed Rust payload access pattern (data is String, not Payload enum)
-	-	Fixed `smartpackOptions.fileserver_upload_handler` type to `Arc<dyn FileUploadHandler>`
-	-	Removed `metadata` from link transport examples (now `None`/omitted)
-	-	Removed duplicate footer text
2026-05-13	1.3.0	Added Rust support with tokio, serde, and arrow2
-	-	Added Rust to C4 diagrams (context, container)
-	-	Added Rust platform-specific architecture section
-	-	Updated component table with Rust support
2026-05-13	1.2.0	Aligned with ground truth implementation (src/msghandler.jl)
-	-	Removed publish_message component (commented out in source)
-	-	Removed NATSClient and NATSConnectionPool classes (not in ground truth)
-	-	Updated smartpack to return JSON for caller to publish via transport
-	-	Updated component diagram to match actual module structure
-	-	Updated data flow to show smartpack returns JSON for caller to publish
-	-	Fixed SIZE_THRESHOLD default to 500,000 bytes
2026-03-15	1.1.0	JavaScript connection management
-	-	Added NATSClient with keepAlive support
-	-	Added NATSConnectionPool for connection reuse
-	-	Added publishMessage function with closeConnection option
(Historical - pre-transport-agnostic refactor)
2026-03-13	1.0.0	Initial architecture documentation

16. References

16.1 Documentation Artifacts

Document	Purpose	Specification Traceability	UI Specification Traceability	Requirement ID(s)
`docs/requirements.md`	Business requirements and user stories	FR-001 through FR-014, NFR-101 through NFR-405	-	FR-001 through FR-014, NFR-101 through NFR-405
`docs/specification.md`	Technical contract for msghandler	specification.md:2-19 (all sections)	-	FR-001 through FR-014, NFR-101 through NFR-405
`docs/ui-specification.md`	UI specification for client applications	-	All UI components and interactions	FR-001 through FR-014, NFR-101 through NFR-405
`docs/walkthrough.md`	End-to-end system flow	specification.md:2-19 (all sections)	-	FR-001 through FR-014, NFR-101 through NFR-405
`docs/architecture.md`	System architecture diagrams	specification.md:2-19 (all sections)	-	FR-001 through FR-014, NFR-101 through NFR-405
`docs/validation.md`	CI/CD validation rules	specification.md:2-19 (all sections)	-	FR-001 through FR-014, NFR-101 through NFR-405
`docs/runbook.md`	Operational runbook	specification.md:2-19 (all sections)	-	FR-001 through FR-014, NFR-101 through NFR-405

16.2 Implementation Files

File	Platform	Features	Specification Traceability	Requirement ID(s)
`src/msghandler.jl`	Julia	Full feature set, Arrow IPC, multiple dispatch	specification.md:2-19 (all sections)	FR-001 through FR-014, NFR-101 through NFR-405
`src/msghandler_ssr.js`	Node.js	Arrow IPC, async/await	specification.md:2-19 (all sections)	FR-001 through FR-014, NFR-101 through NFR-405
`src/msghandler_csr.js`	Browser	JSON table only	specification.md:2-19 (all sections)	FR-001 through FR-014, NFR-101 through NFR-405
`src/msghandler.py`	Python	Arrow IPC, async/await	specification.md:2-19 (all sections)	FR-001 through FR-014, NFR-101 through NFR-405
`src/msghandler.dart`	Dart	Full feature set, Arrow IPC, async/await	specification.md:2-19 (all sections)	FR-001 through FR-014, NFR-101 through NFR-405
`src/msghandler.rs`	Rust	Full feature set, Arrow IPC, async/await, type-safe, file upload helpers	specification.md:2-19 (all sections)	FR-001 through FR-014, NFR-101 through NFR-405
`src/msghandler_mpy.py`	MicroPython	Limited to direct transport	specification.md:2-19 (all sections)	FR-005, FR-006, FR-012

16.3 External Dependencies

Platform	Package	Version	Purpose	Specification Traceability	Requirement ID(s)
Julia	JSON.jl	Latest	JSON serialization	specification.md:11	FR-012, NFR-101, NFR-102
Julia	Arrow.jl	Latest	Arrow IPC support	specification.md:11	FR-002, FR-012
Julia	HTTP.jl	Latest	HTTP file server	specification.md:11	FR-008, FR-009
Julia	UUIDs.jl	Latest	UUID generation	specification.md:11	FR-011, NFR-401
Node.js	node-fetch	Latest	HTTP file server	specification.md:11	FR-008, FR-009
Browser	-	-	Transport-agnostic (caller provides)	specification.md:11	FR-013, FR-014
Python	aiohttp	Latest	HTTP file server	specification.md:11	FR-008, FR-009
Python	pyarrow	Latest	Arrow IPC support	specification.md:11	FR-002, FR-012
Dart	http	Latest	HTTP file server	specification.md:11	FR-008, FR-009
Dart	uuid	Latest	UUID generation	specification.md:11	FR-011, NFR-401
Dart	dart-arrow	Latest	Arrow IPC support	specification.md:11	FR-002, FR-012
Rust	serde	Latest	JSON serialization	specification.md:11	FR-012, NFR-101, NFR-102
Rust	serde_json	Latest	JSON handling	specification.md:11	FR-012, NFR-101, NFR-102
Rust	tokio	Latest	Async runtime	specification.md:11	FR-013, FR-014
Rust	reqwest	Latest	HTTP file server	specification.md:11	FR-008, FR-009
Rust	uuid	Latest	UUID generation	specification.md:11	FR-011, NFR-401
Rust	arrow2	Latest	Arrow IPC support	specification.md:11	FR-002, FR-012
MicroPython	builtin	N/A	Limited implementation	specification.md:11	FR-005, FR-006, FR-012

17. Change Log

Date	Version	Changes	Specification Reference
2026-03-23	1.1.0	Updated to ASG Framework architecture guidelines	specification.md:2-19 (all sections)
2026-03-15	1.1.0	JavaScript connection management	specification.md:2-19 (all sections)
2026-03-13	1.0.0	Initial architecture documentation	specification.md:2-19 (all sections)

18. Gap-Check Validation

Stage Transition	Gap-Check Question	Status
Requirements → Specification	Does the Specification define all edge cases and conflict scenarios from the Requirements?	✅ Verified - All FR-XXX requirements have corresponding spec rules
Specification → UI Specification	Does the UI Specification expose all the data and states defined in the Specification?	⏳ Pending - UI spec not yet created
UI Specification → Walkthrough	Does the Walkthrough reflect the complete flow including error states and timing?	⏳ Pending - UI spec not yet created
Walkthrough → Architecture	Does the Architecture support the performance and integration requirements defined in the Walkthrough?	✅ Verified - Architecture supports all walkthrough flows

This architecture document is versioned and maintained in git alongside the codebase. All implementations must adhere to this architecture.

36 KiB Raw Blame History

Architecture Documentation: msghandler

1. Executive Summary

1.1 Specification Traceability

2. Architecture Overview

Architecture Overview

C4 Context Diagram

C4 Container Diagram

C4 Component Diagram (Julia Implementation)

High-Level Architecture

System Components

Data Flow

Message Envelope Architecture

msg_envelope_v1 Structure (Julia)

msg_payload_v1 Structure (Julia)

JSON Schema (Cross-Platform)

Payload Type Architecture

Supported Payload Types

Serialization Logic

Transport Strategy Architecture

Size Threshold Decision Logic

Transport Selection Flow

Direct Transport Protocol

Link Transport Protocol

Platform-Specific Architecture

Julia Architecture

JavaScript Architecture

Node.js Implementation (msghandler_ssr.js)

Browser Implementation (msghandler_csr.js)

Python Architecture

Dart Architecture

Dart Desktop (Dart SDK)

Dart Flutter (Dart SDK)

Dart Web (Dart SDK)

Browser Architecture

MicroPython Architecture

Rust Architecture

Scaling Architecture

Horizontal Scaling

Vertical Scaling

Performance Considerations

Failure Modes and Recovery

Transport Connection Failure

File Server Unavailable

Deserialization Error

Memory Overflow (MicroPython)

Trade-off Decisions

Decision 1: Direct vs Link Transport Threshold

Decision 2: Base64 Encoding for Direct Transport

Decision 3: Multiple Platform Implementations

Decision 4: Handler Function Abstraction

Deployment Architecture

Minimum Infrastructure

Environment Variables

Container Deployment

Security Considerations

Payload Integrity

Transport Security

File Server Security

Testing Architecture

Unit Test Coverage

Integration Test Scenarios

Versioning

Architecture Versioning

Backward Compatibility

Change Log

16. References

16.1 Documentation Artifacts

16.2 Implementation Files

16.3 External Dependencies

17. Change Log

18. Gap-Check Validation

36 KiB

Raw Blame History