286 lines
18 KiB
Markdown
286 lines
18 KiB
Markdown
# Solution Design: msghandler
|
|
|
|
**Version**: 1.4.0
|
|
**Date**: 2026-09-16
|
|
**Status**: Active
|
|
**Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl)
|
|
**ASG Framework Alignment**: Eight Pillars — Requirements → Solution Design → Walkthrough → Specification → UI Specification → Validation → Implementation Plan → Runbook
|
|
|
|
---
|
|
|
|
## 1. Problem Decomposition
|
|
|
|
msghandler addresses the challenge of cross-platform data exchange between **Julia**, **JavaScript**, **Python**, **Rust**, and **MicroPython** applications using a message broker as the transport layer.
|
|
|
|
### User Problems
|
|
|
|
| Problem | Description | User Impact | Requirement ID |
|
|
|---------|-------------|-------------|----------------|
|
|
| **P-001**: Cross-platform data serialization | Different languages have incompatible data types and serialization formats | Developers must write platform-specific conversion code | FR-001, FR-002 |
|
|
| **P-002**: Large payload handling | Message brokers have size limits, but large files need to be transferred | Large files either fail or require complex workarounds | FR-003 |
|
|
| **P-003**: Transport abstraction | Each platform has different message broker libraries and APIs | No unified interface across platforms | FR-013, FR-014 |
|
|
| **P-004**: Request-response patterns | Bi-directional communication requires complex correlation tracking | Developers must implement custom message routing | FR-011 |
|
|
| **P-005**: File server reliability | File server may be temporarily unavailable during downloads | Failed downloads without retry mechanism | FR-010 |
|
|
| **P-006**: Payload type preservation | Different platforms have different type systems | Data corruption or misinterpretation on receiving end | FR-006, FR-007 |
|
|
|
|
### Solution Boundaries
|
|
|
|
**In Scope**:
|
|
- Unified API for `smartpack()` and `smartunpack()` across all platforms
|
|
- Automatic transport selection based on payload size
|
|
- File server integration using Claim-Check pattern
|
|
- Multi-payload support with mixed types in single message
|
|
- Exponential backoff for reliable file downloads
|
|
- Correlation ID propagation for message tracing
|
|
|
|
**Out of Scope**:
|
|
- Message compression (adds complexity without clear benefit)
|
|
- Message encryption (application-layer concern)
|
|
- Advanced message routing (simple topic matching sufficient)
|
|
- Persistent message queues (transport pattern sufficient)
|
|
|
|
### Decision IDs
|
|
|
|
| Decision ID | Decision | Description | Requirement IDs | NFR IDs |
|
|
|-------------|----------|-------------|-----------------|---------|
|
|
| SD-001 | Claim-Check Pattern | Large payloads uploaded to HTTP server, small payloads sent directly | FR-003, FR-004 | NFR-104, NFR-105 |
|
|
| SD-002 | Automatic Transport Selection | <10KB = direct, ≥10KB = link based on size threshold | FR-003, FR-004 | NFR-104, NFR-105 |
|
|
| SD-003 | Handler Function Abstraction | Pluggable file server implementations via handler functions | FR-008, FR-009 | NFR-202 |
|
|
| SD-004 | Unified Tuple Format | Same `(dataname, data, type)` format across all platforms | FR-006, FR-007 | - |
|
|
| SD-005 | Base64 Encoding | JSON-compatible binary data transport | FR-012 | - |
|
|
| SD-006 | Transport Abstraction | Support multiple broker protocols (NATS/MQTT/WebSocket) transparently | FR-013, FR-014 | NFR-201 |
|
|
| SD-007 | Exponential Backoff | Retry failed file downloads with exponential backoff | FR-010 | NFR-202 |
|
|
| SD-008 | Correlation ID Propagation | Propagate correlation IDs through all message processing steps | FR-011 | NFR-401, NFR-403 |
|
|
|
|
---
|
|
|
|
## 2. Solution Approach
|
|
|
|
**What we deliver**: two tiny, symmetric functions — `smartpack()` and `smartunpack()` — that every platform calls the same way. A developer passes a list of `(dataname, data, type)` tuples and receives them back on the other side with types intact, without ever choosing how the data moves. The framework decides, per payload, whether to send it inline or hand it to a file server, so callers never write transport, encoding, retry, or routing code themselves.
|
|
|
|
The high-level flow:
|
|
|
|
```
|
|
Sender (smartpack) Transport Layer Receiver (smartunpack)
|
|
┌─────────────────┐ ┌───────────────┐ ┌───────────────────┐
|
|
│ │ │ │ │ │
|
|
│ 1. Data tuples │────────────>│ │───────────>│ 1. Parse envelope │
|
|
│ [(name, │ JSON │ Message │ JSON │ 2. Check transport│
|
|
│ data, type)]│ format │ Broker │ format │ 3. Fetch/Decode │
|
|
│ │ │ (NATS/MQTT/ │ │ 4. Return tuples │
|
|
└─────────────────┘ │ WebSocket) │ └───────────────────┘
|
|
└───────────────┘
|
|
```
|
|
|
|
**Key Principles**:
|
|
- **Unified API** — one `smartpack()` / `smartunpack()` signature on every platform (SD-004)
|
|
- **Automatic transport selection** — the caller never picks direct vs. link; payload size decides (SD-002)
|
|
- **Claim-Check for large payloads** — only metadata + a download URL cross the transport for big data (SD-001)
|
|
- **Type preservation** — `(dataname, data, type)` tuples keep types intact end-to-end (SD-004)
|
|
- **Transport-agnostic** — works over NATS, MQTT, or WebSocket with no caller changes (SD-006)
|
|
|
|
---
|
|
|
|
## 3. Decision Rationale
|
|
|
|
The design reasoning behind each major decision — why the solution was built this way, and how each decision connects to form a cohesive approach that solves the user problem.
|
|
|
|
| Decision ID | Decision | Rationale | Alternatives Rejected |
|
|
|-------------|----------|-----------|----------------------|
|
|
| **SD-001** | Claim-Check Pattern | Large payloads (>10KB) uploaded to HTTP server, small payloads sent directly via transport | Client-side compression - adds complexity; Server-side compression - not universally supported |
|
|
| **SD-002** | Automatic Transport Selection | <10KB = direct (fast), ≥10KB = link (avoid transport limits) | Manual selection - error-prone; Fixed threshold - not adaptive |
|
|
| **SD-003** | Handler Function Abstraction | Allows pluggable file server implementations (Plik, AWS S3, custom) | Hardcoded Plik - not flexible; Interface-based - too complex for this use case |
|
|
| **SD-004** | Unified Tuple Format | Same input/output format across all platforms | Platform-native formats - no interoperability; Protocol buffers - too heavy |
|
|
| **SD-005** | Base64 Encoding | JSON-compatible binary data transport | Raw bytes - not JSON-compatible; Hex encoding - 2x size overhead |
|
|
| **SD-006** | Transport Abstraction | Support multiple broker protocols (NATS/MQTT/WebSocket) transparently | Platform-specific libraries - no interoperability |
|
|
| **SD-007** | Exponential Backoff | Retry failed file downloads with exponential backoff | Simple retry - too aggressive; No retry - poor reliability |
|
|
| **SD-008** | Correlation ID Propagation | Propagate correlation IDs through all message processing steps | Manual correlation - error-prone; No tracing - debug impossible |
|
|
|
|
**How the decisions connect**: SD-004 (unified tuple format) gives every platform the same input contract; SD-005 (base64) makes those payloads cross any JSON transport; SD-001/SD-002 decide, per payload, whether it travels inline or via a file-server link; SD-003 makes the file server swappable; SD-006 keeps the whole thing transport-agnostic; SD-007 protects the link path against transient file-server outages; and SD-008 threads a correlation ID through every step so the distributed flow remains observable.
|
|
|
|
---
|
|
|
|
## 4. High-Level Component Diagram
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
subgraph msghandler["msghandler Core Module"]
|
|
direction TB
|
|
|
|
subgraph Serialization["Serialization Layer"]
|
|
DIR["Direct Transport"]
|
|
LNK["Link Transport"]
|
|
|
|
DIR -->|Base64| JSON_MSG
|
|
LNK -->|HTTP URL| JSON_MSG
|
|
end
|
|
|
|
subgraph Envelope["Envelope Builder"]
|
|
HDR["Message Header"]
|
|
PAY["Payload Manager"]
|
|
|
|
HDR --> PAY
|
|
end
|
|
|
|
subgraph Handlers["Handler Functions"]
|
|
UPD["Upload Handler"]
|
|
DWN["Download Handler"]
|
|
|
|
UPD --> LNK
|
|
DWN --> LNK
|
|
end
|
|
|
|
API["smartpack() / smartunpack()"]
|
|
|
|
API -->|Input| Serialization
|
|
API -->|Output| Serialization
|
|
API -->|Configure| Handlers
|
|
end
|
|
|
|
subgraph Transport["Transport Layer"]
|
|
BROKER["NATS / MQTT / WebSocket"]
|
|
API -->|JSON| BROKER
|
|
BROKER -->|JSON| API
|
|
end
|
|
|
|
subgraph FileServer["File Server"]
|
|
Plik["HTTP Server"]
|
|
UPD -.->|POST| Plik
|
|
Plik -.->|URL| DWN
|
|
end
|
|
|
|
style msghandler fill:#b3e5fc,stroke:#0288d1,stroke-width:2px
|
|
style Transport fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
|
|
style FileServer fill:#c8e6c9,stroke:#43a047,stroke-width:2px
|
|
```
|
|
|
|
### Component Responsibilities
|
|
|
|
| Component | Responsibilities | Decision IDs | Requirements Addressed |
|
|
|-----------|-----------------|--------------|----------------------|
|
|
| **Serialization Layer** | Convert data types to transport format (Base64/URL) | SD-005 | FR-001, FR-002, FR-012 |
|
|
| **Envelope Builder** | Create standardized message envelope with metadata | SD-001, SD-008 | FR-011, FR-013, FR-014 |
|
|
| **Handler Functions** | Abstract file server operations for pluggability | SD-003, SD-007 | FR-008, FR-009, FR-010 |
|
|
| **Transport Adapter** | Support multiple broker protocols transparently | SD-006 | FR-013, FR-014 |
|
|
| **Payload Manager** | Track payload types, sizes, and encoding | SD-004 | FR-006, FR-007 |
|
|
|
|
---
|
|
|
|
## 5. Alternatives Considered
|
|
|
|
| Alternative | Pros | Cons | Decision |
|
|
|-------------|------|------|----------|
|
|
| **gRPC/Protobuf** | Strong typing, efficient binary format | No native MicroPython support; Complex schema management | Rejected - not cross-platform enough |
|
|
| **MessagePack** | Compact binary, good performance | Browser support limited; No standard for tabular data | Rejected - missing Arrow IPC alternative |
|
|
| **Protocol Buffers** | Type-safe, efficient | No native support for tabular data | Rejected - cannot represent DataFrames natively |
|
|
| **REST HTTP Upload** | Simple, universal | High latency; No real-time capability | Rejected - not suitable for message broker pattern |
|
|
| **Hybrid (direct/link)** | Optimal for both small and large payloads | More complex implementation | Accepted - matches user requirements (FR-003, FR-004) |
|
|
| **Single transport type** | Simpler implementation | Cannot handle large payloads efficiently | Rejected - violates FR-003 requirement |
|
|
| **Platform-specific APIs** | Native performance | No interoperability; Maintenance burden | Rejected - violates cross-platform goal |
|
|
|
|
---
|
|
|
|
## 6. Risk Assessment
|
|
|
|
| Risk | Impact | Probability | Mitigation | Requirement IDs | NFR IDs |
|
|
|------|--------|-------------|------------|-----------------|---------|
|
|
| **Performance degradation with >500KB payloads** | High | Medium | Size threshold detection; Link transport fallback | FR-003, FR-004 | NFR-104, NFR-105 |
|
|
| **File server availability issues** | Medium | Low | Exponential backoff retry; Graceful degradation | FR-010 | NFR-202 |
|
|
| **Platform-specific bugs** | Medium | Low | Comprehensive test suite per platform; CI validation | FR-001, FR-002, FR-006, FR-007 | - |
|
|
| **Encoding mismatches between platforms** | High | Low | Strict specification; Test contracts; Validation rules | FR-012 | NFR-301 |
|
|
| **Transport layer incompatibility** | Medium | Low | Transport-agnostic design; Handler abstraction | FR-013, FR-014 | NFR-201 |
|
|
| **Correlation ID loss in processing** | Medium | Low | Centralized trace context management | FR-011 | NFR-401, NFR-403 |
|
|
|
|
---
|
|
|
|
## 7. Requirements Traceability
|
|
|
|
| Solution Component | Decision ID | Requirement ID | Description |
|
|
|-------------------|-------------|----------------|-------------|
|
|
| **smartpack() function** | SD-001, SD-002, SD-004, SD-005, SD-006, SD-008 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Unified API for sending messages across all platforms |
|
|
| **smartunpack() function** | SD-001, SD-002, SD-004, SD-005, SD-006, SD-007, SD-008 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Unified API for receiving messages across all platforms |
|
|
| **Direct transport** | SD-002 | FR-004 | Send payloads < threshold directly via transport |
|
|
| **Link transport** | SD-001, SD-002 | FR-003 | Upload payloads ≥ threshold to file server |
|
|
| **File server handler** | SD-003, SD-007 | FR-008, FR-009, FR-010 | Pluggable upload/download handlers with retry logic |
|
|
| **Payload type preservation** | SD-004 | FR-006, FR-007 | Support text, dictionary, arrowtable, jsontable, image, audio, video, binary |
|
|
| **Correlation ID propagation** | SD-008 | FR-011 | Message tracing across distributed systems |
|
|
| **Multi-payload support** | SD-004 | FR-006, FR-007 | List of (dataname, data, type) tuples |
|
|
|
|
### Non-Functional Requirements Traceability
|
|
|
|
| Solution Component | Decision ID | NFR ID | Description |
|
|
|-------------------|-------------|--------|-------------|
|
|
| **Serialization optimization** | SD-005 | NFR-101, NFR-102 | <50ms overhead for 10KB payloads |
|
|
| **Transport efficiency** | SD-006 | NFR-103 | <100ms connection establishment |
|
|
| **File server latency** | SD-001, SD-002 | NFR-104, NFR-105 | <1s upload/download for 10KB files |
|
|
| **Concurrent connections** | SD-006 | NFR-106 | Support 100+ simultaneous connections |
|
|
| **Message throughput** | SD-005, SD-006 | NFR-107 | Handle 1000+ messages/second per instance |
|
|
| **At-least-once delivery** | SD-006 | NFR-201 | Transport layer semantics |
|
|
| **Graceful degradation** | SD-003, SD-007 | NFR-202 | File server unavailability handling |
|
|
| **Auto-reconnect** | SD-006 | NFR-203 | Transport connection failure recovery |
|
|
| **Payload integrity** | SD-005 | NFR-301 | 100% SHA-256 checksum validation |
|
|
| **Transport security** | SD-006 | NFR-302 | 100% TLS connections in production |
|
|
| **File server security** | SD-003 | NFR-303 | 100% authenticated file uploads |
|
|
| **Required logs** | SD-001, SD-008 | NFR-401 | Correlation ID, msg_id, timestamp, etc. |
|
|
| **Critical metrics** | SD-001, SD-005 | NFR-402 | messages_sent_total, file upload/download duration |
|
|
| **Tracing** | SD-001, SD-008 | NFR-403 | Correlation ID propagation |
|
|
| **Alerting** | SD-007 | NFR-404 | <5min alert latency for `download_retry_exceeded` |
|
|
|
|
---
|
|
|
|
## 8. Gap-Check Validation
|
|
|
|
| Stage Transition | Gap-Check Question | Status |
|
|
|------------------|-------------------|--------|
|
|
| **Requirements → Solution Design** | Does the Solution Design clearly explain how the system solves the user problem, not just what it does? | ✅ Verified - All user problems (P-001..P-006) mapped to solution components with SD-XXX decision IDs and requirement ID references; the two-function API explains *how* cross-platform exchange works, not just that it does |
|
|
| **Solution Design → Specification** | Does every SD-XXX decision appear as a traceable technical rule in the specification? | ✅ Verified - SD-001..SD-008 referenced in [`4-specification.md`](4-specification.md) |
|
|
| **Solution Design → Walkthrough** | Does every SD-XXX decision appear in at least one participant's walkthrough step with a WT-XXX ID? | ✅ Verified - SD-001..SD-008 traced in [`3-walkthrough.md`](3-walkthrough.md) |
|
|
|
|
### Solution Design Validation
|
|
|
|
**User Problems** (from requirements.md):
|
|
- **P-001**: Cross-platform data serialization (FR-001, FR-002)
|
|
- **P-002**: Large payload handling (FR-003)
|
|
- **P-003**: Transport abstraction (FR-013, FR-014)
|
|
- **P-004**: Request-response patterns (FR-011)
|
|
- **P-005**: File server reliability (FR-010)
|
|
- **P-006**: Payload type preservation (FR-006, FR-007)
|
|
|
|
**Solution Components**:
|
|
1. **SD-001** - Claim-Check pattern - large payloads uploaded to file server, small payloads sent directly
|
|
2. **SD-002** - Automatic transport selection - transport chosen by size threshold
|
|
3. **SD-003** - Handler function abstraction - Plik/AWS S3/custom file server support
|
|
4. **SD-004** - Unified tuple format - `(dataname, data, type)` - platform-agnostic
|
|
5. **SD-005** - Base64 encoding - JSON-compatible binary data transport
|
|
6. **SD-006** - Transport abstraction - support multiple broker protocols transparently
|
|
7. **SD-007** - Exponential backoff - reliable file downloads with retry logic
|
|
8. **SD-008** - Correlation ID propagation - message tracing across distributed systems
|
|
|
|
**Requirement Mapping**:
|
|
- **Functional Requirements**: FR-001 through FR-014 ✅
|
|
- **Non-Functional Requirements**: NFR-101 through NFR-405 ✅
|
|
|
|
**Gap Check**: Does this solution explain *how* users will actually use the system?
|
|
|
|
**Answer**: Yes - the walkthrough provides concrete examples:
|
|
1. JavaScript sends `[(msg, "Hello", "text"), (avatar, binary_data, "image")]`
|
|
2. `smartpack()` automatically selects transport based on size (SD-002)
|
|
3. Large file (≥10KB) → link transport → file server upload (SD-001)
|
|
4. Small payload (<10KB) → direct transport → base64 encoding (SD-005)
|
|
5. Receiver calls `smartunpack()` → receives same tuple format with preserved types
|
|
|
|
**NFR Traceability**:
|
|
- **Performance**: NFR-101 (serialization <50ms), NFR-102 (deserialization <50ms), NFR-103 (connection <100ms) ✅
|
|
- **Reliability**: NFR-201 (at-least-once delivery), NFR-202 (file server <5% failure), NFR-203 (auto-reconnect <30s) ✅
|
|
- **Security**: NFR-301 (SHA-256 checksum), NFR-302 (TLS 100%), NFR-303 (authenticated uploads) ✅
|
|
- **Observability**: NFR-401 (required logs), NFR-402 (metrics), NFR-403 (tracing), NFR-404 (alerting <5min) ✅
|
|
|
|
---
|
|
|
|
*This solution design document is versioned and maintained in git alongside the codebase. All implementations must adhere to this design.*
|
|
|
|
**Traceability Summary**:
|
|
- All requirements traced to solution components with SD-XXX decision IDs
|
|
- Each decision ID references the corresponding requirement IDs (FR-XXX, NFR-XXX)
|
|
- Specification must cite SD-XXX references for each technical detail
|