msghandler/docs/2-solution_design.md

Solution Design: msghandler

Version: 1.4.0
Date: 2026-09-16
Status: Active
Ground Truth: src/msghandler.jl
ASG Framework Alignment: Eight Pillars — Requirements → Solution Design → Walkthrough → Specification → UI Specification → Validation → Implementation Plan → Runbook

---

1. Problem Decomposition

msghandler addresses the challenge of cross-platform data exchange between Julia, JavaScript, Python, Rust, and MicroPython applications using a message broker as the transport layer.

User Problems

Problem	Description	User Impact	Requirement ID
P-001: Cross-platform data serialization	Different languages have incompatible data types and serialization formats	Developers must write platform-specific conversion code	FR-001, FR-002
P-002: Large payload handling	Message brokers have size limits, but large files need to be transferred	Large files either fail or require complex workarounds	FR-003
P-003: Transport abstraction	Each platform has different message broker libraries and APIs	No unified interface across platforms	FR-013, FR-014
P-004: Request-response patterns	Bi-directional communication requires complex correlation tracking	Developers must implement custom message routing	FR-011
P-005: File server reliability	File server may be temporarily unavailable during downloads	Failed downloads without retry mechanism	FR-010
P-006: Payload type preservation	Different platforms have different type systems	Data corruption or misinterpretation on receiving end	FR-006, FR-007

Solution Boundaries

In Scope:

• Unified API for smartpack() and smartunpack() across all platforms
• Automatic transport selection based on payload size
• File server integration using Claim-Check pattern
• Multi-payload support with mixed types in single message
• Exponential backoff for reliable file downloads
• Correlation ID propagation for message tracing

Out of Scope:

• Message compression (adds complexity without clear benefit)
• Message encryption (application-layer concern)
• Advanced message routing (simple topic matching sufficient)
• Persistent message queues (transport pattern sufficient)

Decision IDs

Decision ID	Decision	Description	Requirement IDs	NFR IDs
SD-001	Claim-Check Pattern	Large payloads uploaded to HTTP server, small payloads sent directly	FR-003, FR-004	NFR-104, NFR-105
SD-002	Automatic Transport Selection	<10KB = direct, ≥10KB = link based on size threshold	FR-003, FR-004	NFR-104, NFR-105
SD-003	Handler Function Abstraction	Pluggable file server implementations via handler functions	FR-008, FR-009	NFR-202
SD-004	Unified Tuple Format	Same (dataname, data, type) format across all platforms	FR-006, FR-007	-
SD-005	Base64 Encoding	JSON-compatible binary data transport	FR-012	-
SD-006	Transport Abstraction	Support multiple broker protocols (NATS/MQTT/WebSocket) transparently	FR-013, FR-014	NFR-201
SD-007	Exponential Backoff	Retry failed file downloads with exponential backoff	FR-010	NFR-202
SD-008	Correlation ID Propagation	Propagate correlation IDs through all message processing steps	FR-011	NFR-401, NFR-403

---

2. Solution Approach

What we deliver: two tiny, symmetric functions — smartpack() and smartunpack() — that every platform calls the same way. A developer passes a list of (dataname, data, type) tuples and receives them back on the other side with types intact, without ever choosing how the data moves. The framework decides, per payload, whether to send it inline or hand it to a file server, so callers never write transport, encoding, retry, or routing code themselves.

The high-level flow:

Sender (smartpack)              Transport Layer              Receiver (smartunpack)
┌─────────────────┐             ┌───────────────┐            ┌───────────────────┐
│                 │             │               │            │                   │
│ 1. Data tuples  │────────────>│               │───────────>│ 1. Parse envelope │
│    [(name,      │   JSON      │  Message      │   JSON     │ 2. Check transport│
│     data, type)]│   format    │  Broker       │   format   │ 3. Fetch/Decode   │
│                 │             │  (NATS/MQTT/  │            │ 4. Return tuples  │
└─────────────────┘             │  WebSocket)   │            └───────────────────┘
                                └───────────────┘

Key Principles:

• Unified API — one smartpack() / smartunpack() signature on every platform (SD-004)
• Automatic transport selection — the caller never picks direct vs. link; payload size decides (SD-002)
• Claim-Check for large payloads — only metadata + a download URL cross the transport for big data (SD-001)
• Type preservation — (dataname, data, type) tuples keep types intact end-to-end (SD-004)
• Transport-agnostic — works over NATS, MQTT, or WebSocket with no caller changes (SD-006)

---

3. Decision Rationale

The design reasoning behind each major decision — why the solution was built this way, and how each decision connects to form a cohesive approach that solves the user problem.

Decision ID	Decision	Rationale	Alternatives Rejected
SD-001	Claim-Check Pattern	Large payloads (>10KB) uploaded to HTTP server, small payloads sent directly via transport	Client-side compression - adds complexity; Server-side compression - not universally supported
SD-002	Automatic Transport Selection	<10KB = direct (fast), ≥10KB = link (avoid transport limits)	Manual selection - error-prone; Fixed threshold - not adaptive
SD-003	Handler Function Abstraction	Allows pluggable file server implementations (Plik, AWS S3, custom)	Hardcoded Plik - not flexible; Interface-based - too complex for this use case
SD-004	Unified Tuple Format	Same input/output format across all platforms	Platform-native formats - no interoperability; Protocol buffers - too heavy
SD-005	Base64 Encoding	JSON-compatible binary data transport	Raw bytes - not JSON-compatible; Hex encoding - 2x size overhead
SD-006	Transport Abstraction	Support multiple broker protocols (NATS/MQTT/WebSocket) transparently	Platform-specific libraries - no interoperability
SD-007	Exponential Backoff	Retry failed file downloads with exponential backoff	Simple retry - too aggressive; No retry - poor reliability
SD-008	Correlation ID Propagation	Propagate correlation IDs through all message processing steps	Manual correlation - error-prone; No tracing - debug impossible

How the decisions connect: SD-004 (unified tuple format) gives every platform the same input contract; SD-005 (base64) makes those payloads cross any JSON transport; SD-001/SD-002 decide, per payload, whether it travels inline or via a file-server link; SD-003 makes the file server swappable; SD-006 keeps the whole thing transport-agnostic; SD-007 protects the link path against transient file-server outages; and SD-008 threads a correlation ID through every step so the distributed flow remains observable.

---

4. High-Level Component Diagram

flowchart TD
    subgraph msghandler["msghandler Core Module"]
        direction TB
        
        subgraph Serialization["Serialization Layer"]
            DIR["Direct Transport"]
            LNK["Link Transport"]
            
            DIR -->|Base64| JSON_MSG
            LNK -->|HTTP URL| JSON_MSG
        end
        
        subgraph Envelope["Envelope Builder"]
            HDR["Message Header"]
            PAY["Payload Manager"]
            
            HDR --> PAY
        end
        
        subgraph Handlers["Handler Functions"]
            UPD["Upload Handler"]
            DWN["Download Handler"]
            
            UPD --> LNK
            DWN --> LNK
        end
        
        API["smartpack() / smartunpack()"]
        
        API -->|Input| Serialization
        API -->|Output| Serialization
        API -->|Configure| Handlers
    end
    
    subgraph Transport["Transport Layer"]
        BROKER["NATS / MQTT / WebSocket"]
        API -->|JSON| BROKER
        BROKER -->|JSON| API
    end
    
    subgraph FileServer["File Server"]
        Plik["HTTP Server"]
        UPD -.->|POST| Plik
        Plik -.->|URL| DWN
    end
    
    style msghandler fill:#b3e5fc,stroke:#0288d1,stroke-width:2px
    style Transport fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
    style FileServer fill:#c8e6c9,stroke:#43a047,stroke-width:2px

Component Responsibilities

Component	Responsibilities	Decision IDs	Requirements Addressed
Serialization Layer	Convert data types to transport format (Base64/URL)	SD-005	FR-001, FR-002, FR-012
Envelope Builder	Create standardized message envelope with metadata	SD-001, SD-008	FR-011, FR-013, FR-014
Handler Functions	Abstract file server operations for pluggability	SD-003, SD-007	FR-008, FR-009, FR-010
Transport Adapter	Support multiple broker protocols transparently	SD-006	FR-013, FR-014
Payload Manager	Track payload types, sizes, and encoding	SD-004	FR-006, FR-007

---

5. Alternatives Considered

Alternative	Pros	Cons	Decision
gRPC/Protobuf	Strong typing, efficient binary format	No native MicroPython support; Complex schema management	Rejected - not cross-platform enough
MessagePack	Compact binary, good performance	Browser support limited; No standard for tabular data	Rejected - missing Arrow IPC alternative
Protocol Buffers	Type-safe, efficient	No native support for tabular data	Rejected - cannot represent DataFrames natively
REST HTTP Upload	Simple, universal	High latency; No real-time capability	Rejected - not suitable for message broker pattern
Hybrid (direct/link)	Optimal for both small and large payloads	More complex implementation	Accepted - matches user requirements (FR-003, FR-004)
Single transport type	Simpler implementation	Cannot handle large payloads efficiently	Rejected - violates FR-003 requirement
Platform-specific APIs	Native performance	No interoperability; Maintenance burden	Rejected - violates cross-platform goal

---

6. Risk Assessment

Risk	Impact	Probability	Mitigation	Requirement IDs	NFR IDs
Performance degradation with >500KB payloads	High	Medium	Size threshold detection; Link transport fallback	FR-003, FR-004	NFR-104, NFR-105
File server availability issues	Medium	Low	Exponential backoff retry; Graceful degradation	FR-010	NFR-202
Platform-specific bugs	Medium	Low	Comprehensive test suite per platform; CI validation	FR-001, FR-002, FR-006, FR-007	-
Encoding mismatches between platforms	High	Low	Strict specification; Test contracts; Validation rules	FR-012	NFR-301
Transport layer incompatibility	Medium	Low	Transport-agnostic design; Handler abstraction	FR-013, FR-014	NFR-201
Correlation ID loss in processing	Medium	Low	Centralized trace context management	FR-011	NFR-401, NFR-403

---

7. Requirements Traceability

Solution Component	Decision ID	Requirement ID	Description
smartpack() function	SD-001, SD-002, SD-004, SD-005, SD-006, SD-008	FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014	Unified API for sending messages across all platforms
smartunpack() function	SD-001, SD-002, SD-004, SD-005, SD-006, SD-007, SD-008	FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014	Unified API for receiving messages across all platforms
Direct transport	SD-002	FR-004	Send payloads < threshold directly via transport
Link transport	SD-001, SD-002	FR-003	Upload payloads ≥ threshold to file server
File server handler	SD-003, SD-007	FR-008, FR-009, FR-010	Pluggable upload/download handlers with retry logic
Payload type preservation	SD-004	FR-006, FR-007	Support text, dictionary, arrowtable, jsontable, image, audio, video, binary
Correlation ID propagation	SD-008	FR-011	Message tracing across distributed systems
Multi-payload support	SD-004	FR-006, FR-007	List of (dataname, data, type) tuples

Non-Functional Requirements Traceability

Solution Component	Decision ID	NFR ID	Description
Serialization optimization	SD-005	NFR-101, NFR-102	<50ms overhead for 10KB payloads
Transport efficiency	SD-006	NFR-103	<100ms connection establishment
File server latency	SD-001, SD-002	NFR-104, NFR-105	<1s upload/download for 10KB files
Concurrent connections	SD-006	NFR-106	Support 100+ simultaneous connections
Message throughput	SD-005, SD-006	NFR-107	Handle 1000+ messages/second per instance
At-least-once delivery	SD-006	NFR-201	Transport layer semantics
Graceful degradation	SD-003, SD-007	NFR-202	File server unavailability handling
Auto-reconnect	SD-006	NFR-203	Transport connection failure recovery
Payload integrity	SD-005	NFR-301	100% SHA-256 checksum validation
Transport security	SD-006	NFR-302	100% TLS connections in production
File server security	SD-003	NFR-303	100% authenticated file uploads
Required logs	SD-001, SD-008	NFR-401	Correlation ID, msg_id, timestamp, etc.
Critical metrics	SD-001, SD-005	NFR-402	messages_sent_total, file upload/download duration
Tracing	SD-001, SD-008	NFR-403	Correlation ID propagation
Alerting	SD-007	NFR-404	<5min alert latency for download_retry_exceeded

---

8. Gap-Check Validation

Stage Transition	Gap-Check Question	Status
Requirements → Solution Design	Does the Solution Design clearly explain how the system solves the user problem, not just what it does?	✅ Verified - All user problems (P-001..P-006) mapped to solution components with SD-XXX decision IDs and requirement ID references; the two-function API explains how cross-platform exchange works, not just that it does
Solution Design → Specification	Does every SD-XXX decision appear as a traceable technical rule in the specification?	✅ Verified - SD-001..SD-008 referenced in 4-specification.md
Solution Design → Walkthrough	Does every SD-XXX decision appear in at least one participant's walkthrough step with a WT-XXX ID?	✅ Verified - SD-001..SD-008 traced in 3-walkthrough.md

Solution Design Validation

User Problems (from requirements.md):

• P-001: Cross-platform data serialization (FR-001, FR-002)
• P-002: Large payload handling (FR-003)
• P-003: Transport abstraction (FR-013, FR-014)
• P-004: Request-response patterns (FR-011)
• P-005: File server reliability (FR-010)
• P-006: Payload type preservation (FR-006, FR-007)

Solution Components:

1. SD-001 - Claim-Check pattern - large payloads uploaded to file server, small payloads sent directly
2. SD-002 - Automatic transport selection - transport chosen by size threshold
3. SD-003 - Handler function abstraction - Plik/AWS S3/custom file server support
4. SD-004 - Unified tuple format - (dataname, data, type) - platform-agnostic
5. SD-005 - Base64 encoding - JSON-compatible binary data transport
6. SD-006 - Transport abstraction - support multiple broker protocols transparently
7. SD-007 - Exponential backoff - reliable file downloads with retry logic
8. SD-008 - Correlation ID propagation - message tracing across distributed systems

Requirement Mapping:

• Functional Requirements: FR-001 through FR-014 ✅
• Non-Functional Requirements: NFR-101 through NFR-405 ✅

Gap Check: Does this solution explain how users will actually use the system?

Answer: Yes - the walkthrough provides concrete examples:

1. JavaScript sends [(msg, "Hello", "text"), (avatar, binary_data, "image")]
2. smartpack() automatically selects transport based on size (SD-002)
3. Large file (≥10KB) → link transport → file server upload (SD-001)
4. Small payload (<10KB) → direct transport → base64 encoding (SD-005)
5. Receiver calls smartunpack() → receives same tuple format with preserved types

NFR Traceability:

• Performance: NFR-101 (serialization <50ms), NFR-102 (deserialization <50ms), NFR-103 (connection <100ms) ✅
• Reliability: NFR-201 (at-least-once delivery), NFR-202 (file server <5% failure), NFR-203 (auto-reconnect <30s) ✅
• Security: NFR-301 (SHA-256 checksum), NFR-302 (TLS 100%), NFR-303 (authenticated uploads) ✅
• Observability: NFR-401 (required logs), NFR-402 (metrics), NFR-403 (tracing), NFR-404 (alerting <5min) ✅

---

This solution design document is versioned and maintained in git alongside the codebase. All implementations must adhere to this design.

Traceability Summary:

• All requirements traced to solution components with SD-XXX decision IDs
• Each decision ID references the corresponding requirement IDs (FR-XXX, NFR-XXX)
• Specification must cite SD-XXX references for each technical detail