ton/msghandler

Fork 0

Files

narawat 312d14b28f update docs

2026-05-22 06:17:30 +07:00

17 KiB

Raw Blame History

Solution Design: msghandler

Version: 1.3.0
Date: 2026-05-22
Status: Active
Ground Truth: src/msghandler.jl

1. Problem Decomposition

msghandler addresses the challenge of cross-platform data exchange between Julia, JavaScript, Python, Dart, Rust, and MicroPython applications using message brokers as transport layers.

Problem Statement

Developers working across multiple programming languages face significant obstacles when trying to share data:

Problem	Description	User Impact
P-001: Cross-platform data serialization	Different languages have incompatible data types and serialization formats	Developers must write platform-specific conversion code
P-002: Large payload handling	Message brokers have size limits, but large files need to be transferred	Large files either fail or require complex workarounds
P-003: Transport abstraction	Each platform has different message broker libraries and APIs	No unified interface across platforms
P-004: Request-response patterns	Bi-directional communication requires complex correlation tracking	Developers must implement custom message routing

Solution Boundaries

In Scope:

Unified API for smartpack() and smartunpack() across all platforms
Automatic transport selection based on payload size
File server integration using Claim-Check pattern
Multi-payload support with mixed types in single message
Exponential backoff for reliable file downloads

Out of Scope:

Message compression (adds complexity without clear benefit)
Message encryption (application-layer concern)
Advanced message routing (simple topic matching sufficient)
Persistent message queues (transport pattern sufficient)

Decision IDs

Decision ID	Decision	Description
SD-001	Claim-Check Pattern	Large payloads uploaded to HTTP server, small payloads sent directly
SD-002	Automatic Transport Selection	<0.5MB = direct, ≥0.5MB = link based on size threshold
SD-003	Handler Function Abstraction	Pluggable file server implementations via handler functions
SD-004	Unified Tuple Format	Same `(dataname, data, type)` format across all platforms
SD-005	Base64 Encoding	JSON-compatible binary data transport
SD-006	Transport Abstraction	Support multiple broker protocols (NATS/MQTT/WebSocket) transparently

2. Solution Approach

msghandler implements a Claim-Check pattern with intelligent transport selection:

Sender (smartpack)              Transport Layer              Receiver (smartunpack)
┌─────────────────┐             ┌───────────────┐            ┌───────────────────┐
│                 │             │               │            │                   │
│ 1. Data tuples  │────────────>│               │───────────>│ 1. Parse envelope │
│    [(name,      │   JSON      │  Message      │   JSON     │ 2. Check transport│
│     data, type)]│   format    │  Broker       │   format   │ 3. Fetch/Decode   │
│                 │             │  (NATS/MQTT/  │            │ 4. Return tuples  │
└─────────────────┘             │  WebSocket)   │            │                   │
                                │               │            └───────────────────┘
                                └───────────────┘

Key Design Decisions

Decision ID	Decision	Rationale	Alternatives Rejected
SD-001	Claim-Check Pattern	Large payloads (>0.5MB) uploaded to HTTP server, small payloads sent directly via transport	Client-side compression - adds complexity; Server-side compression - not universally supported
SD-002	Automatic Transport Selection	<0.5MB = direct (fast), ≥0.5MB = link (avoid transport limits)	Manual selection - error-prone; Fixed threshold - not adaptive
SD-003	Handler Function Abstraction	Allows pluggable file server implementations (Plik, AWS S3, custom)	Hardcoded Plik - not flexible; Interface-based - too complex for this use case
SD-004	Unified Tuple Format	Same input/output format across all platforms	Platform-native formats - no interoperability; Protocol buffers - too heavy
SD-005	Base64 Encoding	JSON-compatible binary data transport	Raw bytes - not JSON-compatible; Hex encoding - 2x size overhead
SD-006	Transport Abstraction	Support multiple broker protocols (NATS/MQTT/WebSocket) transparently	Platform-specific libraries - no interoperability

Architecture Components

flowchart TB
    subgraph Client["Client Application"]
        direction TB
        APP["Application Code"]
        API["msghandler API"]
        
        APP -->|Data tuples| API
        API -->|JSON envelope| TRANSPORT
    end
    
    subgraph Transport["Transport Layer"]
        direction TB
        BROKER["Message Broker<br/>NATS/MQTT/WebSocket"]
        TOPICS["Topic Subscription"]
        
        API -->|Publish| BROKER
        BROKER -->|Deliver| TOPICS
        TOPICS -->|Subscribe| API
    end
    
    subgraph FileServer["File Server"]
        direction TB
        UPLOAD["Upload Handler"]
        DOWNLOAD["Download Handler"]
        
        API -.->|Upload URL| UPLOAD
        DOWNLOAD -.->|Fetch URL| API
    end
    
    style CLIENT fill:#e1f5fe,stroke:#0288d1,stroke-width:2px
    style Transport fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
    style FileServer fill:#c8e6c9,stroke:#43a047,stroke-width:2px

3. Alternatives Considered

Alternative	Pros	Cons	Decision
gRPC/Protobuf	Strong typing, efficient binary format	No native MicroPython support; Complex schema management	Rejected - not cross-platform enough
MessagePack	Compact binary, good performance	Browser support limited; No standard for tabular data	Rejected - missing Arrow IPC alternative
Protocol Buffers	Type-safe, efficient	No native support for tabular data exchange	Rejected - cannot represent DataFrames natively
REST HTTP Upload	Simple, universal	High latency; No real-time capability	Rejected - not suitable for message broker pattern
Hybrid (direct/link)	Optimal for both small and large payloads	More complex implementation	Accepted - matches user requirements (FR-003, FR-004)
Single transport type	Simpler implementation	Cannot handle large payloads efficiently	Rejected - violates FR-003 requirement
Platform-specific APIs	Native performance	No interoperability; Maintenance burden	Rejected - violates cross-platform goal

4. High-Level Component Diagram

flowchart TD
    subgraph msghandler["msghandler Core Module"]
        direction TB
        
        subgraph Serialization["Serialization Layer"]
            DIR["Direct Transport"]
            LNK["Link Transport"]
            
            DIR -->|Base64| JSON_MSG
            LNK -->|HTTP URL| JSON_MSG
        end
        
        subgraph Envelope["Envelope Builder"]
            HDR["Message Header"]
            PAY["Payload Manager"]
            
            HDR --> PAY
        end
        
        subgraph Handlers["Handler Functions"]
            UPD["Upload Handler"]
            DWN["Download Handler"]
            
            UPD --> LNK
            DWN --> LNK
        end
        
        API["smartpack() / smartunpack()"]
        
        API -->|Input| Serialization
        API -->|Output| Serialization
        API -->|Configure| Handlers
    end
    
    subgraph Transport["Transport Layer"]
        BROKER["NATS / MQTT / WebSocket"]
        API -->|JSON| BROKER
        BROKER -->|JSON| API
    end
    
    subgraph FileServer["File Server"]
        Plik["HTTP Server"]
        UPD -.->|POST| Plik
        Plik -.->|URL| DWN
    end
    
    style msghandler fill:#b3e5fc,stroke:#0288d1,stroke-width:2px
    style Transport fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
    style FileServer fill:#c8e6c9,stroke:#43a047,stroke-width:2px

Component Responsibilities

Component	Responsibilities	Decision IDs	Requirements Addressed
Serialization Layer	Convert data types to transport format (Base64/URL)	SD-005	FR-001, FR-002, FR-012
Envelope Builder	Create standardized message envelope with metadata	SD-001	FR-011, FR-013, FR-014
Handler Functions	Abstract file server operations for pluggability	SD-003	FR-008, FR-009
Transport Adapter	Support multiple broker protocols transparently	SD-006	FR-013, FR-014
Payload Manager	Track payload types, sizes, and encoding	SD-004	FR-006, FR-007

5. Decision Rationale

SD-001: Why Claim-Check Pattern?

Requirement: FR-003 - Large file handling, FR-004 - Direct transport for small payloads

Rationale:

Transport layers (NATS, MQTT) have message size limits (typically 1MB)
Direct transport is faster for small payloads (no file server round-trip)
Link transport avoids transport limits for large payloads
User doesn't need to manually choose - automatic selection based on threshold

SD-002: Why Handler Functions for File Server?

Requirement: FR-008 - Plik integration, FR-009 - Custom file server support

Rationale:

Plik is common open-source solution for file server
Some users need AWS S3 or custom implementation
Handler functions provide clean abstraction without vendor lock-in
Same signature across all platforms (unified API)

SD-003: Why Tuple Format for Payloads?

Requirement: FR-006 - Multi-payload messages, FR-007 - Payload type preservation

Rationale:

(dataname, data, type) tuple is language-agnostic
Simple to understand: name, content, type
Supports mixed payload types in single message
Easy to serialize/deserialize across platforms

SD-004: Why Base64 Encoding?

Requirement: FR-012 - Message serialization, FR-001 - Cross-platform text messaging

Rationale:

JSON is universal - works on all platforms
Base64 converts binary to ASCII for JSON compatibility
Standard format with native support in all languages
No additional dependencies needed

SD-005: Why Automatic Transport Selection?

Requirement: FR-003, FR-004, NFR-104, NFR-105

Rationale:

<0.5MB payloads use direct transport (<1s latency, FR-004 KPI)
≥0.5MB payloads use link transport to avoid transport limits (FR-003 KPI: 99% successful uploads)
User doesn't need to manually choose - automatic selection based on threshold

SD-006: Why Transport Abstraction?

Requirement: FR-013, FR-014, NFR-201

Rationale:

Support multiple broker protocols (NATS, MQTT, WebSocket) transparently
Caller handles actual transport publishing/subscription
Unified API across all platforms
At-least-once delivery semantics via transport layer

6. Risk Assessment

Risk	Impact	Probability	Mitigation
Performance degradation with >500KB payloads	High	Medium	Size threshold detection; Link transport fallback
File server availability issues	Medium	Low	Exponential backoff retry; Graceful degradation
Platform-specific bugs	Medium	Low	Comprehensive test suite per platform; CI validation
Encoding mismatches between platforms	High	Low	Strict specification; Test contracts; Validation rules
Transport layer incompatibility	Medium	Low	Transport-agnostic design; Handler abstraction

7. Requirements Traceability

Solution Component	Decision ID	Requirement ID	Description
smartpack() function	SD-001, SD-002, SD-004, SD-005, SD-006	FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014	Unified API for sending messages across all platforms
smartunpack() function	SD-001, SD-002, SD-004, SD-005, SD-006	FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014	Unified API for receiving messages across all platforms
Direct transport	SD-002	FR-004, NFR-101, NFR-102, NFR-104, NFR-105	Send payloads < threshold directly via transport
Link transport	SD-001, SD-002	FR-003, NFR-104, NFR-105	Upload payloads ≥ threshold to file server
File server handler	SD-003	FR-008, FR-009, FR-010	Pluggable upload/download handlers with retry logic
Payload type preservation	SD-004	FR-006, FR-007	Support text, dictionary, arrowtable, jsontable, image, audio, video, binary
Correlation ID	SD-001	FR-011, NFR-401, NFR-403	Message tracing across distributed systems
Multi-payload support	SD-004	FR-006, FR-007	List of (dataname, data, type) tuples

Non-Functional Requirements Traceability

Solution Component	Decision ID	NFR ID	Description
Serialization optimization	SD-005	NFR-101, NFR-102	<50ms overhead for 10KB payloads
Transport efficiency	SD-006	NFR-103	<100ms connection establishment
File server latency	SD-001, SD-002	NFR-104, NFR-105	<1s upload/download for 0.5MB files
Concurrent connections	SD-006	NFR-106	Support 100+ simultaneous connections
Message throughput	SD-005, SD-006	NFR-107	Handle 1000+ messages/second per instance
At-least-once delivery	SD-006	NFR-201	Transport layer semantics
Graceful degradation	SD-003	NFR-202	File server unavailability handling
Auto-reconnect	SD-006	NFR-203	Transport connection failure recovery
Required logs	SD-001	NFR-401	Correlation ID, msg_id, timestamp, etc.
Critical metrics	SD-001, SD-005	NFR-402	messages_sent_total, file upload/download duration
Tracing	SD-001	NFR-403	Correlation ID propagation

8. Gap-Check Validation

Stage Transition	Gap-Check Question	Status
Requirements → Solution Design	Does the Solution Design clearly explain how the system solves the user problem, not just what it does?	✅ Verified - All user stories mapped to solution components with requirement ID and decision ID references
Solution Design → Specification	Does the Specification define all technical details that the solution approach requires?	⏳ Pending - Specification needs review for completeness
Solution Design → Walkthrough	Does the Walkthrough reflect the complete flow including error states and timing?	⏳ Pending - Walkthrough needs validation against design

Solution Design Validation

Problem: Users need to send mixed payload types (text + image + large file) between Julia, JavaScript, Python, and MicroPython applications.

Solution Components:

SD-001 - smartpack() - Unified API for all platforms
SD-002 - Tuple format - (dataname, data, type) - platform-agnostic
SD-003 - Automatic transport selection - <0.5MB = direct, ≥0.5MB = link
SD-004 - File server handler abstraction - Plik/AWS S3/custom support
SD-005 - Exponential backoff - Reliable file downloads
SD-006 - Correlation ID - Message tracing

Requirement Mapping:

FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 ✅

Gap Check: Does this solution explain how users will actually use the system?

Answer: Yes - the walkthrough provides concrete examples:

JavaScript sends [(msg, "Hello", "text"), (avatar, binary_data, "image")]
smartpack() automatically selects transport based on size (SD-002)
Large file (≥0.5MB) → link transport → file server upload (SD-001)
Small payload (<0.5MB) → direct transport → base64 encoding (SD-005)
Receiver calls smartunpack() → receives same tuple format

This solution design document is versioned and maintained in git alongside the codebase. All implementations must adhere to this design.

Traceability Summary:

All requirements traced to solution components with SD-XXX decision IDs
Each decision ID references the corresponding requirement IDs (FR-XXX, NFR-XXX)
Specification must cite SD-XXX references for each technical detail

17 KiB Raw Blame History

Solution Design: msghandler

1. Problem Decomposition

Problem Statement

Solution Boundaries

Decision IDs

2. Solution Approach

Key Design Decisions

Architecture Components

3. Alternatives Considered

4. High-Level Component Diagram

Component Responsibilities

5. Decision Rationale

SD-001: Why Claim-Check Pattern?

SD-002: Why Handler Functions for File Server?

SD-003: Why Tuple Format for Payloads?

SD-004: Why Base64 Encoding?

SD-005: Why Automatic Transport Selection?

SD-006: Why Transport Abstraction?

6. Risk Assessment

7. Requirements Traceability

Non-Functional Requirements Traceability

8. Gap-Check Validation

Solution Design Validation

17 KiB

Raw Blame History