ton/msghandler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

update

Browse Source
This commit is contained in:
narawat
2026-05-22 08:51:47 +07:00
parent 72e0c3be1e
commit 6171923c05
2 changed files with 413 additions and 323 deletions
Download Patch File Download Diff File Expand all files Collapse all files

156
docs/solution-design.md
View File

@@ -4,6 +4,7 @@
**Date**: 2026-05-22
**Status**: Active
**Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl)
**ASG Framework Alignment**: v8 pillars - Requirements → Solution Design → Specification → Walkthrough → Implementation Plan → Validation → Runbook
---
@@ -11,16 +12,16 @@
msghandler addresses the challenge of cross-platform data exchange between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using message brokers as transport layers.
### Problem Statement
### User Problems
Developers working across multiple programming languages face significant obstacles when trying to share data:
| Problem | Description | User Impact |
|---------|-------------|-------------|
| **P-001**: Cross-platform data serialization | Different languages have incompatible data types and serialization formats | Developers must write platform-specific conversion code |
| **P-002**: Large payload handling | Message brokers have size limits, but large files need to be transferred | Large files either fail or require complex workarounds |
| **P-003**: Transport abstraction | Each platform has different message broker libraries and APIs | No unified interface across platforms |
| **P-004**: Request-response patterns | Bi-directional communication requires complex correlation tracking | Developers must implement custom message routing |
| Problem | Description | User Impact | Requirement ID |
|---------|-------------|-------------|----------------|
| **P-001**: Cross-platform data serialization | Different languages have incompatible data types and serialization formats | Developers must write platform-specific conversion code | FR-001, FR-002 |
| **P-002**: Large payload handling | Message brokers have size limits, but large files need to be transferred | Large files either fail or require complex workarounds | FR-003 |
| **P-003**: Transport abstraction | Each platform has different message broker libraries and APIs | No unified interface across platforms | FR-013, FR-014 |
| **P-004**: Request-response patterns | Bi-directional communication requires complex correlation tracking | Developers must implement custom message routing | FR-011 |
| **P-005**: File server reliability | File server may be temporarily unavailable during downloads | Failed downloads without retry mechanism | FR-010 |
| **P-006**: Payload type preservation | Different platforms have different type systems | Data corruption or misinterpretation on receiving end | FR-006, FR-007 |
### Solution Boundaries
@@ -30,6 +31,7 @@ Developers working across multiple programming languages face significant obstac
- File server integration using Claim-Check pattern
- Multi-payload support with mixed types in single message
- Exponential backoff for reliable file downloads
- Correlation ID propagation for message tracing
**Out of Scope**:
- Message compression (adds complexity without clear benefit)
@@ -39,14 +41,16 @@ Developers working across multiple programming languages face significant obstac
### Decision IDs
| Decision ID | Decision | Description |
|-------------|----------|-------------|
| SD-001 | Claim-Check Pattern | Large payloads uploaded to HTTP server, small payloads sent directly |
| SD-002 | Automatic Transport Selection | <0.5MB = direct, ≥0.5MB = link based on size threshold |
| SD-003 | Handler Function Abstraction | Pluggable file server implementations via handler functions |
| SD-004 | Unified Tuple Format | Same `(dataname, data, type)` format across all platforms |
| SD-005 | Base64 Encoding | JSON-compatible binary data transport |
| SD-006 | Transport Abstraction | Support multiple broker protocols (NATS/MQTT/WebSocket) transparently |
| Decision ID | Decision | Description | Requirement IDs | NFR IDs |
|-------------|----------|-------------|-----------------|---------|
| SD-001 | Claim-Check Pattern | Large payloads uploaded to HTTP server, small payloads sent directly | FR-003, FR-004 | NFR-104, NFR-105 |
| SD-002 | Automatic Transport Selection | <0.5MB = direct, ≥0.5MB = link based on size threshold | FR-003, FR-004 | NFR-104, NFR-105 |
| SD-003 | Handler Function Abstraction | Pluggable file server implementations via handler functions | FR-008, FR-009 | NFR-202 |
| SD-004 | Unified Tuple Format | Same `(dataname, data, type)` format across all platforms | FR-006, FR-007 | - |
| SD-005 | Base64 Encoding | JSON-compatible binary data transport | FR-012 | - |
| SD-006 | Transport Abstraction | Support multiple broker protocols (NATS/MQTT/WebSocket) transparently | FR-013, FR-014 | NFR-201 |
| SD-007 | Exponential Backoff | Retry failed file downloads with exponential backoff | FR-010 | NFR-202 |
| SD-008 | Correlation ID Propagation | Propagate correlation IDs through all message processing steps | FR-011 | NFR-401, NFR-403 |
---
@@ -77,6 +81,8 @@ Sender (smartpack) Transport Layer Receiver (smartunpa
| **SD-004** | Unified Tuple Format | Same input/output format across all platforms | Platform-native formats - no interoperability; Protocol buffers - too heavy |
| **SD-005** | Base64 Encoding | JSON-compatible binary data transport | Raw bytes - not JSON-compatible; Hex encoding - 2x size overhead |
| **SD-006** | Transport Abstraction | Support multiple broker protocols (NATS/MQTT/WebSocket) transparently | Platform-specific libraries - no interoperability |
| **SD-007** | Exponential Backoff | Retry failed file downloads with exponential backoff | Simple retry - too aggressive; No retry - poor reliability |
| **SD-008** | Correlation ID Propagation | Propagate correlation IDs through all message processing steps | Manual correlation - error-prone; No tracing - debug impossible |
### Architecture Components
@@ -110,7 +116,7 @@ flowchart TB
DOWNLOAD -.->|Fetch URL| API
end
style CLIENT fill:#e1f5fe,stroke:#0288d1,stroke-width:2px
style Client fill:#e1f5fe,stroke:#0288d1,stroke-width:2px
style Transport fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
style FileServer fill:#c8e6c9,stroke:#43a047,stroke-width:2px
```
@@ -190,8 +196,8 @@ flowchart TD
| Component | Responsibilities | Decision IDs | Requirements Addressed |
|-----------|-----------------|--------------|----------------------|
| **Serialization Layer** | Convert data types to transport format (Base64/URL) | SD-005 | FR-001, FR-002, FR-012 |
| **Envelope Builder** | Create standardized message envelope with metadata | SD-001 | FR-011, FR-013, FR-014 |
| **Handler Functions** | Abstract file server operations for pluggability | SD-003 | FR-008, FR-009 |
| **Envelope Builder** | Create standardized message envelope with metadata | SD-001, SD-008 | FR-011, FR-013, FR-014 |
| **Handler Functions** | Abstract file server operations for pluggability | SD-003, SD-007 | FR-008, FR-009, FR-010 |
| **Transport Adapter** | Support multiple broker protocols transparently | SD-006 | FR-013, FR-014 |
| **Payload Manager** | Track payload types, sizes, and encoding | SD-004 | FR-006, FR-007 |
@@ -201,7 +207,8 @@ flowchart TD
### SD-001: Why Claim-Check Pattern?
**Requirement**: FR-003 - Large file handling, FR-004 - Direct transport for small payloads
**Requirement**: FR-003 (Large file handling), FR-004 (Direct transport for small payloads)
**NFRs**: NFR-104 (File upload latency <1s), NFR-105 (File download latency <1s)
**Rationale**:
- Transport layers (NATS, MQTT) have message size limits (typically 1MB)
@@ -211,7 +218,8 @@ flowchart TD
### SD-002: Why Handler Functions for File Server?
**Requirement**: FR-008 - Plik integration, FR-009 - Custom file server support
**Requirement**: FR-008 (Plik integration), FR-009 (Custom file server support)
**NFR**: NFR-202 (File server availability <5% failure rate)
**Rationale**:
- Plik is common open-source solution for file server
@@ -221,7 +229,7 @@ flowchart TD
### SD-003: Why Tuple Format for Payloads?
**Requirement**: FR-006 - Multi-payload messages, FR-007 - Payload type preservation
**Requirement**: FR-006 (Multi-payload messages), FR-007 (Payload type preservation)
**Rationale**:
- `(dataname, data, type)` tuple is language-agnostic
@@ -231,7 +239,7 @@ flowchart TD
### SD-004: Why Base64 Encoding?
**Requirement**: FR-012 - Message serialization, FR-001 - Cross-platform text messaging
**Requirement**: FR-012 (Message serialization), FR-001 (Cross-platform text messaging)
**Rationale**:
- JSON is universal - works on all platforms
@@ -241,7 +249,8 @@ flowchart TD
### SD-005: Why Automatic Transport Selection?
**Requirement**: FR-003, FR-004, NFR-104, NFR-105
**Requirement**: FR-003 (Large file handling), FR-004 (Direct transport for small payloads)
**NFRs**: NFR-104 (File upload latency <1s), NFR-105 (File download latency <1s)
**Rationale**:
- <0.5MB payloads use direct transport (<1s latency, FR-004 KPI)
@@ -250,7 +259,8 @@ flowchart TD
### SD-006: Why Transport Abstraction?
**Requirement**: FR-013, FR-014, NFR-201
**Requirement**: FR-013 (Transport publishing), FR-014 (Transport subscription)
**NFR**: NFR-201 (Message delivery at-least-once)
**Rationale**:
- Support multiple broker protocols (NATS, MQTT, WebSocket) transparently
@@ -258,17 +268,40 @@ flowchart TD
- Unified API across all platforms
- At-least-once delivery semantics via transport layer
### SD-007: Why Exponential Backoff?
**Requirement**: FR-010 (Exponential backoff retry)
**NFR**: NFR-202 (File server availability <5% failure rate)
**Rationale**:
- File server may be temporarily unavailable
- Exponential backoff prevents overwhelming server during outages
- Default: 5 retries, 100ms base delay, 5000ms max delay
- 95% successful downloads within retry limit (FR-010 KPI)
### SD-008: Why Correlation ID Propagation?
**Requirement**: FR-011 (Correlation ID propagation)
**NFRs**: NFR-401 (Required logs), NFR-403 (Tracing)
**Rationale**:
- Trace messages across distributed systems
- Correlation ID logged with every message (NFR-401)
- Propagated through all message processing steps (NFR-403)
- Enables debugging and performance analysis in production
---
## 6. Risk Assessment
| Risk | Impact | Probability | Mitigation |
|------|--------|-------------|------------|
| **Performance degradation with >500KB payloads** | High | Medium | Size threshold detection; Link transport fallback |
| **File server availability issues** | Medium | Low | Exponential backoff retry; Graceful degradation |
| **Platform-specific bugs** | Medium | Low | Comprehensive test suite per platform; CI validation |
| **Encoding mismatches between platforms** | High | Low | Strict specification; Test contracts; Validation rules |
| **Transport layer incompatibility** | Medium | Low | Transport-agnostic design; Handler abstraction |
| Risk | Impact | Probability | Mitigation | Requirement IDs | NFR IDs |
|------|--------|-------------|------------|-----------------|---------|
| **Performance degradation with >500KB payloads** | High | Medium | Size threshold detection; Link transport fallback | FR-003, FR-004 | NFR-104, NFR-105 |
| **File server availability issues** | Medium | Low | Exponential backoff retry; Graceful degradation | FR-010 | NFR-202 |
| **Platform-specific bugs** | Medium | Low | Comprehensive test suite per platform; CI validation | FR-001, FR-002, FR-006, FR-007 | - |
| **Encoding mismatches between platforms** | High | Low | Strict specification; Test contracts; Validation rules | FR-012 | NFR-301 |
| **Transport layer incompatibility** | Medium | Low | Transport-agnostic design; Handler abstraction | FR-013, FR-014 | NFR-201 |
| **Correlation ID loss in processing** | Medium | Low | Centralized trace context management | FR-011 | NFR-401, NFR-403 |
---
@@ -276,13 +309,13 @@ flowchart TD
| Solution Component | Decision ID | Requirement ID | Description |
|-------------------|-------------|----------------|-------------|
| **smartpack() function** | SD-001, SD-002, SD-004, SD-005, SD-006 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Unified API for sending messages across all platforms |
| **smartunpack() function** | SD-001, SD-002, SD-004, SD-005, SD-006 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Unified API for receiving messages across all platforms |
| **Direct transport** | SD-002 | FR-004, NFR-101, NFR-102, NFR-104, NFR-105 | Send payloads < threshold directly via transport |
| **Link transport** | SD-001, SD-002 | FR-003, NFR-104, NFR-105 | Upload payloads ≥ threshold to file server |
| **File server handler** | SD-003 | FR-008, FR-009, FR-010 | Pluggable upload/download handlers with retry logic |
| **smartpack() function** | SD-001, SD-002, SD-004, SD-005, SD-006, SD-008 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Unified API for sending messages across all platforms |
| **smartunpack() function** | SD-001, SD-002, SD-004, SD-005, SD-006, SD-007, SD-008 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Unified API for receiving messages across all platforms |
| **Direct transport** | SD-002 | FR-004 | Send payloads < threshold directly via transport |
| **Link transport** | SD-001, SD-002 | FR-003 | Upload payloads ≥ threshold to file server |
| **File server handler** | SD-003, SD-007 | FR-008, FR-009, FR-010 | Pluggable upload/download handlers with retry logic |
| **Payload type preservation** | SD-004 | FR-006, FR-007 | Support text, dictionary, arrowtable, jsontable, image, audio, video, binary |
| **Correlation ID** | SD-001 | FR-011, NFR-401, NFR-403 | Message tracing across distributed systems |
| **Correlation ID propagation** | SD-008 | FR-011 | Message tracing across distributed systems |
| **Multi-payload support** | SD-004 | FR-006, FR-007 | List of (dataname, data, type) tuples |
### Non-Functional Requirements Traceability
@@ -295,11 +328,15 @@ flowchart TD
| **Concurrent connections** | SD-006 | NFR-106 | Support 100+ simultaneous connections |
| **Message throughput** | SD-005, SD-006 | NFR-107 | Handle 1000+ messages/second per instance |
| **At-least-once delivery** | SD-006 | NFR-201 | Transport layer semantics |
| **Graceful degradation** | SD-003 | NFR-202 | File server unavailability handling |
| **Graceful degradation** | SD-003, SD-007 | NFR-202 | File server unavailability handling |
| **Auto-reconnect** | SD-006 | NFR-203 | Transport connection failure recovery |
| **Required logs** | SD-001 | NFR-401 | Correlation ID, msg_id, timestamp, etc. |
| **Payload integrity** | SD-005 | NFR-301 | 100% SHA-256 checksum validation |
| **Transport security** | SD-006 | NFR-302 | 100% TLS connections in production |
| **File server security** | SD-003 | NFR-303 | 100% authenticated file uploads |
| **Required logs** | SD-001, SD-008 | NFR-401 | Correlation ID, msg_id, timestamp, etc. |
| **Critical metrics** | SD-001, SD-005 | NFR-402 | messages_sent_total, file upload/download duration |
| **Tracing** | SD-001 | NFR-403 | Correlation ID propagation |
| **Tracing** | SD-001, SD-008 | NFR-403 | Correlation ID propagation |
| **Alerting** | SD-007 | NFR-404 | <5min alert latency for `download_retry_exceeded` |
---
@@ -307,24 +344,33 @@ flowchart TD
| Stage Transition | Gap-Check Question | Status |
|------------------|-------------------|--------|
| **Requirements → Solution Design** | Does the Solution Design clearly explain how the system solves the user problem, not just what it does? | ✅ Verified - All user stories mapped to solution components with requirement ID and decision ID references |
| **Requirements → Solution Design** | Does the Solution Design clearly explain how the system solves the user problem, not just what it does? | ✅ Verified - All user problems mapped to solution components with requirement ID and decision ID references |
| **Solution Design → Specification** | Does the Specification define all technical details that the solution approach requires? | ⏳ Pending - Specification needs review for completeness |
| **Solution Design → Walkthrough** | Does the Walkthrough reflect the complete flow including error states and timing? | ⏳ Pending - Walkthrough needs validation against design |
### Solution Design Validation
**Problem**: Users need to send mixed payload types (text + image + large file) between Julia, JavaScript, Python, and MicroPython applications.
**User Problems** (from requirements.md):
- **P-001**: Cross-platform data serialization (FR-001, FR-002)
- **P-002**: Large payload handling (FR-003)
- **P-003**: Transport abstraction (FR-013, FR-014)
- **P-004**: Request-response patterns (FR-011)
- **P-005**: File server reliability (FR-010)
- **P-006**: Payload type preservation (FR-006, FR-007)
**Solution Components**:
1. **SD-001** - `smartpack()` - Unified API for all platforms
2. **SD-002** - Tuple format - `(dataname, data, type)` - platform-agnostic
3. **SD-003** - Automatic transport selection - <0.5MB = direct, ≥0.5MB = link
4. **SD-004** - File server handler abstraction - Plik/AWS S3/custom support
5. **SD-005** - Exponential backoff - Reliable file downloads
6. **SD-006** - Correlation ID - Message tracing
1. **SD-001** - `smartpack()` / `smartunpack()` - Unified API for all platforms
2. **SD-002** - Claim-Check pattern - Automatic transport selection based on size threshold
3. **SD-003** - Handler function abstraction - Plik/AWS S3/custom file server support
4. **SD-004** - Tuple format - `(dataname, data, type)` - platform-agnostic
5. **SD-005** - Base64 encoding - JSON-compatible binary data transport
6. **SD-006** - Transport abstraction - Support multiple broker protocols transparently
7. **SD-007** - Exponential backoff - Reliable file downloads with retry logic
8. **SD-008** - Correlation ID propagation - Message tracing across distributed systems
**Requirement Mapping**:
- FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 ✅
- **Functional Requirements**: FR-001 through FR-014 ✅
- **Non-Functional Requirements**: NFR-101 through NFR-405 ✅
**Gap Check**: Does this solution explain *how* users will actually use the system?
@@ -333,7 +379,13 @@ flowchart TD
2. `smartpack()` automatically selects transport based on size (SD-002)
3. Large file (≥0.5MB) → link transport → file server upload (SD-001)
4. Small payload (<0.5MB) → direct transport → base64 encoding (SD-005)
5. Receiver calls `smartunpack()` → receives same tuple format
5. Receiver calls `smartunpack()` → receives same tuple format with preserved types
**NFR Traceability**:
- **Performance**: NFR-101 (serialization <50ms), NFR-102 (deserialization <50ms), NFR-103 (connection <100ms) ✅
- **Reliability**: NFR-201 (at-least-once delivery), NFR-202 (file server <5% failure), NFR-203 (auto-reconnect <30s) ✅
- **Security**: NFR-301 (SHA-256 checksum), NFR-302 (TLS 100%), NFR-303 (authenticated uploads) ✅
- **Observability**: NFR-401 (required logs), NFR-402 (metrics), NFR-403 (tracing), NFR-404 (alerting <5min) ✅
---