ton/msghandler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

1st commit

Browse Source
This commit is contained in:
narawat
2026-05-15 08:58:30 +07:00
commit 0e24b7d044
28 changed files with 15184 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

941
docs/architecture.md Normal file
View File

@@ -0,0 +1,941 @@
# Architecture Documentation: msghandler
**Version**: 1.4.0
**Date**: 2026-05-14
**Status**: Active
**Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl)
**Architecture Level**: C4 Container Level
---
## 1. Executive Summary
This document defines the **blueprint** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus.
This architecture document serves as the single source of truth for:
- **System Structure**: How components fit together and interact
- **Scaling Considerations**: How the system scales horizontally and vertically
- **Failure Modes**: How the system handles failures and recovers
- **Trade-off Decisions**: The rationale behind architectural decisions
### 1.1 Specification Traceability
| Architecture Section | Specification Reference | UI Specification Reference | Requirement ID(s) |
|---------------------|-------------------------|---------------------------|-------------------|
| Section 2 (Context Diagram) | specification.md:2 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 3 (Container Diagram) | specification.md:2, specification.md:3, specification.md:11 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 4 (Component Diagram) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 5 (High-Level) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 6 (Message Envelope) | specification.md:2, specification.md:3, specification.md:8 | - | FR-011, FR-012, FR-013, FR-014, NFR-401, NFR-403 |
| Section 7 (Payload Type) | specification.md:3, specification.md:5, specification.md:6 | - | FR-001, FR-002, FR-003, FR-006, FR-012, NFR-101, NFR-102 |
| Section 8 (Transport Strategy) | specification.md:6, specification.md:7 | - | FR-003, FR-004, FR-005, FR-010, NFR-104, NFR-105, NFR-106 |
| Section 9 (Platform-Specific) | specification.md:13, specification.md:14 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 10 (Scaling) | specification.md:7, specification.md:13 | - | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 |
| Section 11 (Failure Modes) | specification.md:9, specification.md:11 | - | FR-008, FR-009, FR-010, FR-011, NFR-201, NFR-202, NFR-203 |
| Section 12 (Trade-offs) | specification.md:2, specification.md:3, specification.md:6, specification.md:7 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 |
| Section 13 (Deployment) | specification.md:12, specification.md:18 | - | FR-013, FR-014, NFR-201, NFR-203 |
| Section 14 (Security) | specification.md:4, specification.md:9, specification.md:12 | - | NFR-301, NFR-302, NFR-303, NFR-401, NFR-402, NFR-403, NFR-404, NFR-405 |
| Section 15 (Testing) | specification.md:17 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
---
## 2. Architecture Overview
## Architecture Overview
### C4 Context Diagram
```mermaid
flowchart TD
subgraph "External Systems"
NATS_Server[NATS Server]
File_Server[HTTP File Server<br/>Plik/AWS S3/Custom]
end
subgraph "Client Applications"
Julia_App[Julia Application]
JS_App[JavaScript Application<br/>Node.js/Browser]
Python_App[Python Application<br/>Desktop]
Dart_App[Dart Application<br/>Desktop/Flutter/Web]
Rust_App[Rust Application<br/>Server/Desktop]
MicroPython_App[MicroPython Device]
end
Julia_App -->|NATS| NATS_Server
JS_App -->|NATS| NATS_Server
Python_App -->|NATS| NATS_Server
Dart_App -->|NATS| NATS_Server
Rust_App -->|NATS| NATS_Server
MicroPython_App -->|NATS| NATS_Server
Julia_App -->|HTTP| File_Server
JS_App -->|HTTP| File_Server
Python_App -->|HTTP| File_Server
Dart_App -->|HTTP| File_Server
Rust_App -->|HTTP| File_Server
MicroPython_App -->|HTTP| File_Server
style NATS_Server fill:#fff3e0,stroke:#f57c00
style File_Server fill:#f3e5f5,stroke:#9c27b4
style Julia_App fill:#e8f5e9,stroke:#4caf50
style JS_App fill:#e3f2fd,stroke:#2196f3
style Python_App fill:#e3f2fd,stroke:#2196f3
style Dart_App fill:#fff0f6,stroke:#e91e63
style Rust_App fill:#dea584,stroke:#e65100
style MicroPython_App fill:#fce4ec,stroke:#e91e63
```
### C4 Container Diagram
```mermaid
flowchart TD
subgraph "Client Container"
Julia_Module[Julia msghandler Module]
JS_Module[JavaScript msghandler Module]
Python_Module[Python msghandler Module]
Dart_Module[Dart msghandler Module]
Rust_Module[Rust msghandler Module]
MicroPython_Module[MicroPython msghandler Module]
end
Julia_Module --> NATS_Client
JS_Module --> NATS_Client
Python_Module --> NATS_Client
Dart_Module --> NATS_Client
Rust_Module --> NATS_Client
MicroPython_Module --> NATS_Client
NATS_Client --> NATS_Broker
Julia_Module --> File_Client
JS_Module --> File_Client
Python_Module --> File_Client
Dart_Module --> File_Client
Rust_Module --> File_Client
MicroPython_Module --> File_Client
File_Client --> File_Server
style Julia_Module fill:#e8f5e9,stroke:#4caf50
style JS_Module fill:#e3f2fd,stroke:#2196f3
style Python_Module fill:#e3f2fd,stroke:#2196f3
style Dart_Module fill:#fff0f6,stroke:#e91e63
style Rust_Module fill:#dea584,stroke:#e65100
style MicroPython_Module fill:#fce4ec,stroke:#e91e63
style NATS_Broker fill:#fff3e0,stroke:#f57c00
style File_Server fill:#f3e5f5,stroke:#9c27b4
```
### C4 Component Diagram (Julia Implementation)
```mermaid
flowchart TD
subgraph "msghandler Module"
SmartSend[smartsend Function]
SmartReceive[smartreceive Function]
Serialize[_serialize_data]
Deserialize[_deserialize_data]
EnvelopeToJson[envelope_to_json]
FileServerUpload[fileserver_upload_handler]
FileServerDownload[fileserver_download_handler]
LogTrace[log_trace]
end
subgraph "Data Models"
Payload[msg_payload_v1 Struct]
Envelope[msg_envelope_v1 Struct]
end
SmartSend --> Serialize
SmartSend --> EnvelopeToJson
SmartSend --> FileServerUpload
SmartReceive --> Deserialize
SmartReceive --> FileServerDownload
EnvelopeToJson --> Envelope
Serialize --> Payload
style SmartSend fill:#d1fae5,stroke:#10b981
style SmartReceive fill:#d1fae5,stroke:#10b981
style FileServerUpload fill:#fef3c7,stroke:#f59e0b
style FileServerDownload fill:#fef3c7,stroke:#f59e0b
```
---
## High-Level Architecture
### System Components
| Component | Purpose | Platform Support |
|-----------|---------|------------------|
| **smartsend** | Send data via NATS with automatic transport selection, returns (envelope, json_string) for caller to publish | All |
| **smartreceive** | Receive and process NATS messages from JSON string | All |
| **_serialize_data** | Serialize data according to payload type | All |
| **_deserialize_data** | Deserialize bytes to native data types | All |
| **envelope_to_json** | Convert msg_envelope_v1 struct to JSON string | All |
| **log_trace** | Log trace messages with correlation ID | All |
| **fileserver_upload_handler** | Upload large payloads to HTTP server | Desktop (Julia/JS/Python/Dart/Rust) |
| **fileserver_download_handler** | Download payloads from HTTP server with exponential backoff | Desktop (Julia/JS/Python/Dart/Rust) |
| **plik_upload_file** | Upload a local file to Plik server from disk | Rust |
### Data Flow
```mermaid
flowchart TD
A[User calls smartsend subject data] --> B[Process each payload]
B --> C{Calculate serialized size}
C -->|Size < Threshold| D[Direct Transport]
C -->|Size >= Threshold| E[Link Transport]
D --> F[Serialize data]
F --> G[Base64 encode]
G --> H[Build payload object]
E --> I[Serialize data]
I --> J[Upload to file server]
J --> K[Get download URL]
K --> H
H --> L[Build envelope]
L --> M[Convert to JSON]
M --> N[Return envelope + JSON to caller]
style A fill:#f9f9f9,stroke:#333
style N fill:#e0e7ff,stroke:#3b82f6
style D fill:#d1fae5,stroke:#10b981
style E fill:#fef3c7,stroke:#f59e0b
```
---
## Message Envelope Architecture
### msg_envelope_v1 Structure (Julia)
```julia
struct msg_envelope_v1
correlation_id::String # UUID v4 for distributed tracing
msg_id::String # UUID v4 for this message
timestamp::String # ISO 8601 UTC timestamp
send_to::String # NATS subject to publish to
msg_purpose::String # ACK, NACK, updateStatus, shutdown, chat
sender_name::String # Sender application name
sender_id::String # UUID v4 of sender
receiver_name::String # Receiver application name (empty = broadcast)
receiver_id::String # UUID v4 of receiver (empty = broadcast)
reply_to::String # Topic for reply messages
reply_to_msg_id::String # Message ID being replied to
broker_url::String # NATS broker URL
metadata::Dict{String, Any} # Message-level metadata
payloads::Vector{msg_payload_v1} # List of payloads
end
```
### msg_payload_v1 Structure (Julia)
```julia
struct msg_payload_v1
id::String # UUID v4 for this payload
dataname::String # Name of the payload
payload_type::String # text, dictionary, arrowtable, etc.
transport::String # direct or link
encoding::String # none, json, base64, arrow-ipc
size::Integer # Size in bytes
data::Any # Base64 string or URL
metadata::Dict{String, Any} # Payload-level metadata
end
```
### JSON Schema (Cross-Platform)
```json
{
"correlation_id": "string (UUID v4)",
"msg_id": "string (UUID v4)",
"timestamp": "string (ISO 8601 UTC)",
"send_to": "string",
"msg_purpose": "string",
"sender_name": "string",
"sender_id": "string (UUID v4)",
"receiver_name": "string",
"receiver_id": "string (UUID v4)",
"reply_to": "string",
"reply_to_msg_id": "string",
"broker_url": "string",
"metadata": "object",
"payloads": [
{
"id": "string (UUID v4)",
"dataname": "string",
"payload_type": "string",
"transport": "string",
"encoding": "string",
"size": "integer",
"data": "string or URL",
"metadata": "object"
}
]
}
```
---
## Payload Type Architecture
### Supported Payload Types
| Type | Description | Serialization | Encoding | Platforms |
|------|-------------|---------------|----------|-----------|
| `text` | Plain text string | UTF-8 bytes | Base64 | All |
| `dictionary` | JSON object | JSON string | Base64/JSON | All |
| `arrowtable` | Apache Arrow IPC | Arrow IPC stream | Base64/arrow-ipc | Desktop (Julia/Python/Node.js/Dart/Rust) |
| `jsontable` | JSON array of objects | JSON string | Base64/json | All (including Browser/Dart Web) |
| `image` | Binary image data | Raw bytes | Base64 | All |
| `audio` | Binary audio data | Raw bytes | Base64 | All |
| `video` | Binary video data | Raw bytes | Base64 | All |
| `binary` | Generic binary data | Raw bytes | Base64 | All |
### Serialization Logic
```mermaid
flowchart TD
A[Input data + payload_type] --> B{Payload Type}
B -->|"text"| C[UTF-8 encode]
B -->|"dictionary"| D[JSON serialize]
B -->|"arrowtable"| E[Arrow IPC serialize]
B -->|"jsontable"| F[JSON serialize]
B -->|"image"| G[Raw bytes]
B -->|"audio"| H[Raw bytes]
B -->|"video"| I[Raw bytes]
B -->|"binary"| J[Raw bytes]
C --> K[Return bytes]
D --> K
E --> K
F --> K
G --> K
H --> K
I --> K
J --> K
style A fill:#f9f9f9,stroke:#333
style K fill:#e0e7ff,stroke:#3b82f6
```
---
## Transport Strategy Architecture
### Size Threshold Decision Logic
| Platform | Size Threshold | Notes |
|----------|----------------|-------|
| Desktop (Julia/JS/Python/Dart) | 500,000 bytes (0.5MB) | Default threshold |
| Dart Desktop | 500,000 bytes (0.5MB) | Default threshold |
| Dart Flutter | 500,000 bytes (0.5MB) | Default threshold |
| Dart Web | 500,000 bytes (0.5MB) | Default threshold |
| MicroPython | 100,000 bytes (100KB) | Lower threshold for memory constraints |
### Transport Selection Flow
```mermaid
flowchart TD
A[smartsend called] --> B[Serialize payload]
B --> C[Calculate size]
C --> D{Size < Threshold?}
D -->|Yes| E[Direct Transport]
D -->|No| F[Link Transport]
E --> G[Base64 encode]
G --> H[Build payload with direct transport]
F --> I[Upload to file server]
I --> J[Get download URL]
J --> K[Build payload with link transport]
H --> L[Build envelope]
K --> L
style A fill:#f9f9f9,stroke:#333
style L fill:#e0e7ff,stroke:#3b82f6
style E fill:#d1fae5,stroke:#10b981
style F fill:#fef3c7,stroke:#f59e0b
```
### Direct Transport Protocol
When `transport = "direct"`, the `data` field contains a Base64-encoded string of the serialized payload.
**Encoding Rules**:
- `text`: UTF-8 → Base64
- `dictionary`: JSON → Base64 (or direct JSON)
- `arrowtable`: Arrow IPC → Base64 (or arrow-ipc)
- `jsontable`: JSON → Base64 (or direct JSON)
- `image`/`audio`/`video`/`binary`: Raw bytes → Base64
### Link Transport Protocol
When `transport = "link"`, the `data` field contains a URL pointing to the uploaded payload.
**Upload Flow**:
1. Serialize payload according to `payload_type`
2. Upload to HTTP file server (e.g., Plik)
3. Include returned URL in `data` field
**Download Flow**:
1. Extract URL from payload
2. Fetch with exponential backoff (max 5 retries)
3. Deserialize based on `payload_type`
---
## Platform-Specific Architecture
### Julia Architecture
Julia leverages multiple dispatch for type-specific implementations:
- **Multiple Dispatch**: Function overloading based on argument types
- **Struct-based Data Models**: Explicit type definitions with `struct`
- **Native Arrow IPC**: Support via `Arrow.jl`
- **Async/Await**: Tasks for non-blocking I/O
```julia
# Multiple dispatch for serialization
function _serialize_data(data::String, payload_type::String)
# Text serialization
end
function _serialize_data(data::Dict, payload_type::String)
# Dictionary serialization
end
function _serialize_data(data::DataFrame, payload_type::String)
# Arrow table serialization
end
```
### JavaScript Architecture
JavaScript uses async/await for non-blocking I/O:
- **Module-level Utilities**: Serialization functions
- **Native ArrayBuffer**: Binary data handling (Browser) / Buffer (Node.js)
- **Fetch API**: HTTP file server communication
#### Node.js Implementation (msghandler_ssr.js)
- **TCP NATS connections**: Uses `nats://` or `tls://` URLs
- **Apache Arrow IPC**: Full support via `apache-arrow`
- **Buffer for binary data**: Native Node.js Buffer handling
#### Browser Implementation (msghandler_csr.js)
- **WebSocket NATS connections**: Uses `ws://` or `wss://` URLs via `nats.ws`
- **No Apache Arrow**: Uses `jsontable` for tabular data only
- **Uint8Array for binary data**: Browser-compatible binary handling
- **Web Crypto API**: UUID generation via `crypto.getRandomValues()`
### Python Architecture
Python uses classes for stateful operations:
- **Class-based msghandler**: Encapsulated API
- **Dataclasses**: Structured data (MsgPayloadV1, MsgEnvelopeV1)
- **Async/await**: I/O operations
- **pyarrow**: Arrow IPC support
```python
class msghandler:
DEFAULT_SIZE_THRESHOLD = 500_000
def __init__(self, broker_url=None, fileserver_url=None):
self.broker_url = broker_url or self.DEFAULT_BROKER_URL
self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL
```
### Dart Architecture
Dart uses classes for stateful operations with async/await:
- **Class-based msghandler**: Encapsulated API
- **Data classes**: Structured data (MsgPayloadV1, MsgEnvelopeV1)
- **Async/await**: I/O operations
- **dart-arrow**: Arrow IPC support (Desktop/Flutter only)
- **HTTP package**: HTTP file server communication
- **nats package**: NATS client with WebSocket support (Dart Web)
```dart
class msghandler {
static const DEFAULT_SIZE_THRESHOLD = 500000;
final String brokerUrl;
final String fileserverUrl;
msghandler({
this.brokerUrl = 'nats://localhost:4222',
this.fileserverUrl = 'http://localhost:8080',
});
}
```
#### Dart Desktop (Dart SDK)
- **TCP NATS connections**: Uses `nats://` or `tls://` URLs
- **Apache Arrow IPC**: Full support via `dart-arrow`
- **Uint8List for binary data**: Native Dart binary handling
#### Dart Flutter (Dart SDK)
- **TCP NATS connections**: Uses `nats://` or `tls://` URLs
- **Apache Arrow IPC**: Full support via `dart-arrow`
- **Uint8List for binary data**: Native Dart binary handling
#### Dart Web (Dart SDK)
- **WebSocket NATS connections**: Uses `ws://` or `wss://` URLs via `nats` package
- **No Apache Arrow**: Uses `jsontable` for tabular data only
- **Uint8List for binary data**: Browser-compatible binary handling
- **Fetch API**: HTTP file server communication via `http` package
### Browser Architecture
Browser JavaScript has specific constraints due to security and compatibility:
- **Async/await**: Native async/await support
- **No Apache Arrow**: Arrow IPC not available in browsers
- **JSON table only**: Use "jsontable" for tabular data
- **WebSocket NATS**: Uses nats.ws for browser-compatible NATS connections
- **Fetch API**: HTTP file server communication via fetch
### MicroPython Architecture
MicroPython has significant constraints:
- **Synchronous API**: No async/await
- **Memory-constrained**: 256KB - 1MB
- **Limited payload support**: No tables, max 50KB
- **Simplified UUID generation**: Custom implementation
```python
# MicroPython constraints
DEFAULT_SIZE_THRESHOLD = 100_000 # 100KB
MAX_PAYLOAD_SIZE = 50_000 # 50KB hard limit
```
### Rust Architecture
Rust leverages compile-time type safety and async runtimes:
- **Type-safe payloads**: Rust enum discriminates between `Text`, `Dictionary`, `ArrowTable`, `Binary`, etc.
- **serde serialization**: Automatic JSON deserialization via `#[derive(Serialize, Deserialize)]`
- **tokio runtime**: Efficient async I/O for NATS connections and HTTP file server operations
- **arrow2 integration**: Native Arrow IPC deserialization without intermediate format conversion
- **reqwest**: High-performance HTTP client with built-in TLS and connection pooling
- **Zero-copy patterns**: `Vec<u8>` passed directly to avoid unnecessary memory copies
- **Result<T, E>**: Idiomatic error handling with typed error types
```rust
// Type-safe payload enum (compile-time discrimination)
#[derive(Serialize, Deserialize, Clone)]
pub enum Payload {
Text(String),
Dictionary(serde_json::Value),
ArrowTable(Vec<u8>),
JsonTable(serde_json::Value),
Image(Vec<u8>),
Audio(Vec<u8>),
Video(Vec<u8>),
Binary(Vec<u8>),
}
// Configuration via builder pattern
pub struct SmartsendOptions {
pub broker_url: String,
pub fileserver_url: String,
pub fileserver_upload_handler: Option<Arc<dyn FileUploadHandler>>,
pub size_threshold: usize,
pub correlation_id: String,
pub msg_purpose: String,
pub sender_name: String,
// ... other fields
}
// NATS client with tokio integration
let conn = nats::connect("nats://localhost:4222").await?;
// Subscribe and process messages
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze")?;
for msg in sub.messages() {
let envelope = smartreceive(&String::from_utf8_lossy(&msg.payload), &Default::default()).await?;
// Access deserialized payloads by type
for payload in &envelope.payloads {
match payload.payload_type.as_str() {
"arrowtable" => { /* payload.data is base64-encoded Arrow IPC */ },
"text" => { /* payload.data is decoded text string */ },
"binary" | "image" | "audio" | "video" => { /* payload.data is base64-encoded binary */ },
_ => { /* other types */ }
}
}
}
```
---
## Scaling Architecture
### Horizontal Scaling
| Component | Scaling Strategy |
|-----------|------------------|
| **NATS Server** | Cluster deployment with multiple nodes |
| **File Server** | Load balancer + multiple instances |
| **Client Applications** | Deploy multiple instances behind load balancer |
### Vertical Scaling
| Component | Scaling Strategy |
|-----------|------------------|
| **NATS Server** | Increase memory, CPU, disk I/O |
| **File Server** | Increase memory, CPU, disk capacity |
| **Client Applications** | Increase heap size (Python/JS) |
### Performance Considerations
| Metric | Target | Notes |
|--------|--------|-------|
| Message serialization overhead | <50ms | For 10KB payload |
| Message deserialization overhead | <50ms | For 10KB payload |
| NATS connection establishment | <100ms | Connection pool recommended |
| File upload latency | <1s | For 0.5MB file |
| File download latency | <1s | For 0.5MB file |
---
## Failure Modes and Recovery
### NATS Connection Failure
**Scenario**: NATS server unavailable
**Handler**:
- Connection auto-reconnect via TCP-level reconnection
- Retry with exponential backoff for publish operations
**Recovery**:
- NATS client automatically attempts reconnection
- Application can check connection status before publishing
### File Server Unavailable
**Scenario**: HTTP file server unavailable during upload/download
**Handler**:
- Retry up to 5 times with exponential backoff (100ms → 5000ms)
- Fallback to direct transport for upload (MicroPython)
**Recovery**:
- Exponential backoff: `delay = min(delay * 2, max_delay)`
- After max retries, throw error with correlation ID
### Deserialization Error
**Scenario**: Payload type mismatch or corrupted data
**Handler**:
- Log correlation ID and throw error
- No retry (data corruption)
**Recovery**:
- Application must validate payload_type matches data type
- Use proper serialization before sending
### Memory Overflow (MicroPython)
**Scenario**: Payload exceeds maximum size (50KB)
**Handler**:
- Reject payloads >50KB with MemoryError
- No retry (client-side check)
**Recovery**:
- Application must split large payloads
- Use direct transport only for small payloads
---
## Trade-off Decisions
### Decision 1: Direct vs Link Transport Threshold
**Trade-off**: Memory vs Network I/O
**Decision**: Use 0.5MB threshold for desktop, 100KB for MicroPython
**Rationale**:
- Direct transport uses more memory (Base64 encoding adds ~33% overhead)
- Link transport requires network I/O for upload/download
- 0.5MB is reasonable for desktop memory constraints
- 100KB is necessary for MicroPython memory constraints
### Decision 2: Base64 Encoding for Direct Transport
**Trade-off**: Bandwidth vs Simplicity
**Decision**: Use Base64 encoding for all direct transport payloads
**Rationale**:
- Simplifies JSON serialization (all data is string-compatible)
- Increases payload size by ~33%, but NATS can handle this
- Alternative would be binary payload support (more complex)
### Decision 3: Multiple Platform Implementations
**Trade-off**: Development effort vs Cross-platform support
**Decision**: Maintain separate implementations for each platform
**Rationale**:
- Each platform has idiomatic patterns (multiple dispatch, async/await, etc.)
- Maintains developer productivity and code quality
- API parity ensures cross-platform compatibility
### Decision 4: Handler Function Abstraction
**Trade-off**: Flexibility vs Simplicity
**Decision**: Abstract file server operations through handler functions
**Rationale**:
- Allows support for different file server implementations (Plik, AWS S3, custom)
- Maintains simplicity for common use cases
- Enables plug-in architecture for custom backends
---
## Deployment Architecture
### Minimum Infrastructure
| Component | Minimum | Notes |
|-----------|---------|-------|
| NATS Server | 1 instance | Single node for development |
| File Server | 1 instance | HTTP server for large payloads |
| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices |
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) |
### Container Deployment
```mermaid
flowchart TD
subgraph "Docker Network"
NATS_Container[NATS Server]
FileServer_Container[Plik File Server]
App_Container[Application Container]
end
App_Container -->|NATS| NATS_Container
App_Container -->|HTTP| FileServer_Container
style NATS_Container fill:#fff3e0,stroke:#f57c00
style FileServer_Container fill:#f3e5f5,stroke:#9c27b4
style App_Container fill:#e3f2fd,stroke:#2196f3
```
---
## Security Considerations
### Payload Integrity
**Mechanism**: SHA-256 checksum via metadata
**Implementation**:
- Sender calculates checksum and stores in payload metadata
- Receiver validates checksum on receipt
### Transport Security
**Mechanism**: TLS support for NATS connections
**Implementation**:
- Use `nats://` URL for plain text
- Use `tls://` URL for TLS-encrypted connections
### File Server Security
**Mechanism**: Authentication token for file uploads
**Implementation**:
- Plik uses upload token in `X-UploadToken` header
- Application can implement custom authentication
---
## Testing Architecture
### Unit Test Coverage
| Test Category | Coverage | Files |
|---------------|----------|-------|
| Serialization | All payload types | `test/test_*_sender.*` |
| Deserialization | All payload types | `test/test_*_receiver.*` |
| Transport selection | Direct vs link | `test/test_*_mix_payloads.*` |
| File server upload | Plik integration | Platform-specific |
| File server download | Exponential backoff | Platform-specific |
### Integration Test Scenarios
| Scenario | Platforms | Payloads | Transport | Expected Result |
|----------|-----------|----------|-----------|-----------------|
| Cross-platform text | Julia ↔ JS ↔ Python | text | direct | Round-trip successful |
| Arrow IPC round-trip | Julia ↔ JS ↔ Python | arrowtable | direct | Arrow IPC preserved |
| Large file transfer | All | image/audio/video | link | File server upload/download |
| Multi-payload mixed | All | text + image + file | direct/link | All payloads preserved |
---
## Versioning
### Architecture Versioning
| Component | Version | Notes |
|-----------|---------|-------|
| Architecture | 1.0.0 | Initial release |
| Protocol | v1 | Message envelope protocol version |
### Backward Compatibility
| Version | Supported Platforms |
|---------|---------------------|
| v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, Dart 2.17+, Rust 1.70+, MicroPython 1.19+ |
---
## Change Log
| Date | Version | Changes |
|------|---------|---------|
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections |
| - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
| - | - | Added `plik_upload_file` convenience function to component table | specification.md:13 |
| - | - | Fixed Rust payload access pattern (data is String, not Payload enum) | All sections |
| - | - | Fixed `SmartsendOptions.fileserver_upload_handler` type to `Arc<dyn FileUploadHandler>` | specification.md:13 |
| - | - | Removed `metadata` from link transport examples (now `None`/omitted) | specification.md:3 |
| - | - | Removed duplicate footer text | All sections |
| 2026-05-13 | 1.3.0 | Added Rust support with tokio, serde, and arrow2 | All sections |
| - | - | Added Rust to C4 diagrams (context, container) | All sections |
| - | - | Added Rust platform-specific architecture section | specification.md:13 |
| - | - | Updated component table with Rust support | All sections |
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) |
| - | - | Removed publish_message component (commented out in source) |
| - | - | Removed NATSClient and NATSConnectionPool classes (not in ground truth) |
| - | - | Updated component diagram to match actual module structure |
| - | - | Updated data flow to show smartsend returns JSON for caller to publish |
| - | - | Fixed SIZE_THRESHOLD default to 500,000 bytes |
| 2026-03-15 | 1.1.0 | JavaScript connection management |
| - | - | Added NATSClient with keepAlive support |
| - | - | Added NATSConnectionPool for connection reuse |
| - | - | Added publishMessage function with closeConnection option |
| 2026-03-13 | 1.0.0 | Initial architecture documentation |
---
## 16. References
### 16.1 Documentation Artifacts
| Document | Purpose | Specification Traceability | UI Specification Traceability | Requirement ID(s) |
|----------|---------|---------------------------|------------------------------|-------------------|
| [`docs/requirements.md`](./requirements.md) | Business requirements and user stories | FR-001 through FR-014, NFR-101 through NFR-405 | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/specification.md`](./specification.md) | Technical contract for msghandler | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/ui-specification.md`](./ui-specification.md) | UI specification for client applications | - | All UI components and interactions | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/walkthrough.md`](./walkthrough.md) | End-to-end system flow | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/architecture.md`](./architecture.md) | System architecture diagrams | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/validation.md`](./validation.md) | CI/CD validation rules | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/runbook.md`](./runbook.md) | Operational runbook | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
### 16.2 Implementation Files
| File | Platform | Features | Specification Traceability | Requirement ID(s) |
|------|----------|----------|---------------------------|-------------------|
| [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe, file upload helpers | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_mpy.py`](../src/msghandler_mpy.py) | MicroPython | Limited to direct transport | specification.md:2-19 (all sections) | FR-005, FR-006, FR-012 |
### 16.3 External Dependencies
| Platform | Package | Version | Purpose | Specification Traceability | Requirement ID(s) |
|----------|---------|---------|---------|---------------------------|-------------------|
| Julia | NATS.jl | Latest | NATS client | specification.md:11 | FR-013, FR-014, NFR-201 |
| Julia | JSON.jl | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Julia | Arrow.jl | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Julia | HTTP.jl | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Julia | UUIDs.jl | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 |
| Node.js | nats | Latest | NATS client (TCP) | specification.md:11 | FR-013, FR-014 |
| Node.js | node-fetch | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Browser | nats.ws | Latest | NATS client (WebSocket) | specification.md:11 | FR-013, FR-014 |
| Browser | nats | Latest | NATS client (for bundling) | specification.md:11 | FR-013, FR-014 |
| Python | nats-py | Latest | NATS client | specification.md:11 | FR-013, FR-014 |
| Python | aiohttp | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Python | pyarrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Dart | nats | Latest | NATS client | specification.md:11 | FR-013, FR-014 |
| Dart | http | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Dart | uuid | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 |
| Dart | dart-arrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Rust | nats | Latest | NATS client | specification.md:11 | FR-013, FR-014 |
| Rust | serde | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Rust | serde_json | Latest | JSON handling | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Rust | tokio | Latest | Async runtime | specification.md:11 | FR-013, FR-014 |
| Rust | reqwest | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Rust | uuid | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 |
| Rust | arrow2 | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| MicroPython | builtin | N/A | Limited implementation | specification.md:11 | FR-005, FR-006, FR-012 |
---
## 17. Change Log
| Date | Version | Changes | Specification Reference |
|------|---------|---------|------------------------|
| 2026-03-23 | 1.1.0 | Updated to ASG Framework architecture guidelines | specification.md:2-19 (all sections) |
| 2026-03-15 | 1.1.0 | JavaScript connection management | specification.md:2-19 (all sections) |
| 2026-03-13 | 1.0.0 | Initial architecture documentation | specification.md:2-19 (all sections) |
---
## 18. Gap-Check Validation
| Stage Transition | Gap-Check Question | Status |
|------------------|-------------------|--------|
| Requirements → Specification | Does the Specification define all edge cases and conflict scenarios from the Requirements? | ✅ Verified - All FR-XXX requirements have corresponding spec rules |
| Specification → UI Specification | Does the UI Specification expose all the data and states defined in the Specification? | ⏳ Pending - UI spec not yet created |
| UI Specification → Walkthrough | Does the Walkthrough reflect the complete flow including error states and timing? | ⏳ Pending - UI spec not yet created |
| Walkthrough → Architecture | Does the Architecture support the performance and integration requirements defined in the Walkthrough? | ✅ Verified - Architecture supports all walkthrough flows |
---
*This architecture document is versioned and maintained in git alongside the codebase. All implementations must adhere to this architecture.*

438
docs/requirements.md Normal file
View File

@@ -0,0 +1,438 @@
# Requirements Document: msghandler
**Version**: 1.2.0
**Date**: 2026-05-13
**Status**: Active
**Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl)
---
## 1. Business Context & Success Metrics
### 1.1 Business Goal
msghandler is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS.
### 1.2 User Stories (with acceptance criteria)
| Story | Priority | Acceptance Criteria |
|-------|----------|---------------------|
| **As a Julia developer**, I want to send text messages to JavaScript/Dart applications that lives on a server and also on a browser | P1 | Text messages are serialized, encoded, and received correctly across platforms |
| **As a Python developer**, I want to send tabular data to Julia/Dart applications | P1 | DataFrame exchange works with both Arrow IPC and JSON formats |
| **As a JavaScript developer**, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS |
| **As a Dart developer**, I want to send text messages to other platforms | P1 | Text messages are serialized, encoded, and received correctly across platforms |
| **As a Dart developer**, I want to send dictionary data to other platforms | P1 | JSON-serializable data is exchanged correctly |
| **As a Dart developer**, I want to send tabular data (List<Map>) to other platforms | P1 | JSON table format exchange works with Arrow IPC on desktop |
| **As a Dart developer**, I want to send large files (>0.5MB) | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS |
| **As a MicroPython developer**, I want to send sensor data with minimal memory usage | P1 | Direct transport works for payloads <100KB on memory-constrained devices |
| **As a Rust developer**, I want to send and receive messages with type-safe APIs | P1 | Rust implementation uses serde for serialization, tokio for async, and nats-io for NATS connectivity |
| **As a developer**, I want to send mixed-content messages (text + image + file) | P1 | msghandler accepts list of (dataname, data, type) tuples and handles each payload appropriately |
| **As a developer**, I want to receive multi-payload messages | P1 | msghandler returns payloads as list of tuples with correct types preserved |
| **As a developer**, I want to use Plik as the file server | P2 | Plik one-shot upload mode is supported with upload ID and token handling |
| **As a developer**, I want to use custom HTTP file servers | P2 | Handler function abstraction allows plugging in AWS S3 or custom implementations |
| **As a developer**, I want automatic retry on file server download failures | P1 | Exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) |
| **As a developer**, I want message tracing across distributed systems | P1 | Correlation ID is propagated through all message processing steps |
### 1.3 KPIs & Targets
| Metric | Target | Measurement Method |
|--------|--------|-------------------|
| 95% of messages complete within 200ms | 95% | Synthetic monitoring |
| <2 days from onboarding to first PR | 2 days | PR timeline tracking |
| 100% of messages validate against spec | 100% | CI block rate |
| >80% unit test coverage | 80% | Test coverage tools |
| <1% of PRs bypass validation gates | 1% | CI gate analysis |
| MTTR <15 minutes for P1 incidents | 15 minutes | Incident tracking |
---
## 2. Technical Boundaries
### 2.1 In Scope
| Feature | Description |
|---------|-------------|
| Cross-platform interoperability | Seamless data exchange between Julia, JavaScript, Python, Dart, Rust, and MicroPython |
| Intelligent transport selection | Direct transport (<0.5MB) vs Link transport (≥0.5MB) based on payload size |
| Unified API | Consistent `smartsend()` and `smartreceive()` functions across all platforms |
| Multi-payload support | List of (dataname, data, type) tuples with appropriate handling |
| File server integration | Plik one-shot upload and custom HTTP server support |
| Reliability features | Exponential backoff retry and correlation ID propagation |
| Message serialization | Converts data types to binary format (Base64, JSON, Arrow IPC) |
| NATS communication | Publishing and subscription via NATS subjects |
### 2.2 Out of Scope
| Feature | Reason |
|---------|--------|
| NATS JetStream support | Core NATS sufficient for current use cases |
| Message compression | Compression adds complexity without clear benefit |
| Message encryption | Payload encryption is application-layer concern |
| Persistent message queues | NATS request-reply pattern sufficient |
| Advanced routing rules | Simple NATS subject matching sufficient |
### 2.3 Dependencies
| Platform | Package | Version |
|----------|---------|---------|
| Julia | NATS.jl | Latest stable |
| Julia | JSON.jl | Latest stable |
| Julia | Arrow.jl | Latest stable |
| Julia | HTTP.jl | Latest stable |
| Julia | UUIDs.jl | Latest stable |
| Node.js | nats | Latest stable |
| Node.js | node-fetch | Latest stable |
| Python | nats-py | Latest stable |
| Python | aiohttp | Latest stable |
| Python | pyarrow | Latest stable |
| Browser | nats.ws | Latest stable |
| Dart | nats | Latest stable |
| Dart | http | Latest stable |
| Dart | uuid | Latest stable |
| Rust | nats | Latest stable |
| Rust | serde | Latest stable |
| Rust | serde_json | Latest stable |
| Rust | tokio | Latest stable |
| Rust | uuid | Latest stable |
### 2.4 Platform Compatibility
| Platform | Minimum Version | Notes |
|----------|-----------------|-------|
| Julia | 1.7+ | Arrow.jl required for arrowtable support |
| Node.js | 16+ | nats.js required, Arrow IPC supported |
| Python | 3.8+ | pyarrow required for arrowtable support |
| Browser | Latest | No Arrow IPC (uses jsontable only) |
| Dart | 2.17+ | Supports Desktop (Dart SDK), Flutter (Dart SDK), and Web (Dart SDK) |
| Rust | 1.70+ | Full support with async/await, Arrow IPC on desktop |
| MicroPython | 1.19+ | Limited to direct transport |
---
## 3. Functional Requirements (FR)
| ID | Requirement | Description |
|----|-------------|-------------|
| **FR-001** | Cross-platform text messaging | System shall allow users to send text messages between Julia, JavaScript, Python, and MicroPython applications |
| **FR-002** | Cross-platform tabular data | System shall support DataFrame exchange between Julia and Python applications using Arrow IPC format |
| **FR-003** | Large file handling | System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via NATS |
| **FR-004** | Direct transport for small payloads | System shall send payloads <0.5MB directly via NATS without file server upload |
| **FR-005** | MicroPython support | System shall support payloads <100KB on MicroPython devices using direct transport |
| **FR-006** | Multi-payload messages | System shall accept and process lists of (dataname, data, type) tuples |
| **FR-007** | Payload type preservation | System shall preserve payload types when returning multi-payload messages |
| **FR-008** | Plik file server integration | System shall support Plik one-shot upload mode with upload ID and token handling |
| **FR-009** | Custom file server support | System shall provide handler function abstraction for custom HTTP file server implementations |
| **FR-010** | Exponential backoff retry | System shall implement exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) for file server download failures |
| **FR-011** | Correlation ID propagation | System shall propagate correlation IDs through all message processing steps |
| **FR-012** | Message serialization | System shall serialize data types using Base64, JSON, or Arrow IPC encoding |
| **FR-013** | NATS publishing | System shall return JSON string representation for caller to publish to NATS subjects (caller is responsible for actual NATS publish) |
| **FR-014** | NATS subscription | System shall receive and process NATS messages by accepting JSON string from NATS payload |
---
## 4. Non-Functional Requirements (NFRs)
### 4.1 Performance & Scalability
| ID | Requirement | Specification | Test Method |
|----|-------------|---------------|-------------|
| **NFR-101** | Message serialization overhead | <50ms for 10KB payload | Benchmark tests |
| **NFR-102** | Message deserialization overhead | <50ms for 10KB payload | Benchmark tests |
| **NFR-103** | NATS connection establishment | <100ms | Connection pool benchmarks |
| **NFR-104** | File upload latency | <1s for 0.5MB file | Integration tests |
| **NFR-105** | File download latency | <1s for 0.5MB file | Integration tests |
| **NFR-106** | Concurrent connections | Support 100+ simultaneous NATS connections | Scale testing |
| **NFR-107** | Message throughput | Handle 1000+ messages/second per instance | Load testing |
| **NFR-108** | File server scalability | Support horizontal scaling of file server backend | Architecture review |
### 4.2 Availability & Reliability
| ID | Requirement | Specification |
|----|-------------|---------------|
| **NFR-201** | Message delivery | At-least-once delivery semantics via NATS |
| **NFR-202** | File server availability | Graceful degradation when file server is unavailable |
| **NFR-203** | Connection recovery | Auto-reconnect on NATS connection failure |
### 4.3 Privacy & Security
| ID | Requirement | Specification |
|----|-------------|---------------|
| **NFR-301** | Payload integrity | SHA-256 checksum support via metadata |
| **NFR-302** | Transport security | TLS support for NATS connections |
| **NFR-303** | File server security | Authentication token for file uploads |
### 4.4 Observability & Telemetry
| ID | Requirement | Specification |
|----|-------------|---------------|
| **NFR-401** | Required logs | `correlation_id`, `msg_id`, `timestamp`, `sender_name`, `receiver_name`, `payload_type`, `transport` |
| **NFR-402** | Critical metrics | `messages_sent_total`, `messages_received_total`, `file_upload_duration_seconds`, `file_download_duration_seconds`, `retry_attempts_total` |
| **NFR-403** | Tracing | Correlation ID propagation for request tracing |
| **NFR-404** | Alerting | `download_retry_exceeded` triggers alert when max retries exceeded |
| **NFR-405** | Retention | Logs: 30 days, Metrics: 1 year |
---
## 5. Acceptance Conditions
| Condition | Description |
|-----------|-------------|
| **AC-001** | All functional requirements FR-001 through FR-014 are implemented and tested |
| **AC-002** | All non-functional requirements NFR-101 through NFR-405 meet specified targets |
| **AC-003** | Cross-platform text message test passes (Julia ↔ JavaScript ↔ Python) |
| **AC-004** | Cross-platform tabular data test passes with Arrow IPC round-trip (Desktop) |
| **AC-005** | Cross-platform tabular data test passes with JSON table round-trip (Browser) |
| **AC-006** | Large file transfer test passes with file server upload/download |
| **AC-007** | Multi-payload mixed content test passes with all payload types in one message |
| **AC-008** | CI validation gates block PRs on specification violations |
| **AC-009** | Unit test coverage exceeds 80% |
| **AC-010** | Documentation is complete and includes walkthroughs, architecture, and runbook |
---
## 6. Payload Type Requirements
### 6.1 Supported Payload Types
| Type | Julia | JavaScript | Python | Dart | MicroPython | Description |
|------|-------|------------|--------|------|-------------|-------------|
| `text` | `String` | `string` | `str` | `String` | `String` | `str` | Plain text strings |
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `Map`, `serde_json::Value` | `String` | `dict` | JSON-serializable data |
| `arrowtable` | `DataFrame`, `Arrow.Table` | ❌ (Browser), ✅ (Node.js) | `pandas.DataFrame` | `List<Map>` (Desktop), `List<dynamic>` (Flutter) | `arrow2::Table` | ❌ | Tabular data (Arrow IPC) |
| `jsontable` | `Vector{NamedTuple}` | `Array<Object>` | `list[dict]` | `Vec<Map>` | ⚠️ | Tabular data (JSON) - **Only table type in Browser** |
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `Vec<u8>` | `bytearray` | Image binary data |
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `Vec<u8>` | `bytearray` | Audio binary data |
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `Vec<u8>` | `bytearray` | Video binary data |
| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `Uint8List` | `Vec<u8>` | `bytearray` | Generic binary data |
### 6.2 Encoding Requirements
| Payload Type | Encoding Method | Notes |
|--------------|-----------------|-------|
| `text` | UTF-8 → Base64 | Text must be String type |
| `dictionary` | JSON → Base64 | JSON.jl for Julia |
| `arrowtable` | Arrow IPC → Base64 | Requires Arrow.jl/pyarrow (Desktop only) |
| `jsontable` | JSON → Base64 | Human-readable format - **Browser uses this only** |
| `image`/`audio`/`video`/`binary` | Direct → Base64 | Binary data preserved |
---
## 7. Size Threshold Requirements
### 7.1 Direct Transport Threshold
| Platform | Threshold | Notes |
|----------|-----------|-------|
| Desktop (Julia/JS/Python/Dart) | 0.5MB | Default size threshold |
| Dart Desktop | 0.5MB | Default size threshold |
| Dart Flutter | 0.5MB | Default size threshold |
| Dart Web | 0.5MB | Default size threshold |
| Rust | 0.5MB | Default size threshold |
| MicroPython | 100KB | Lower threshold for memory constraints |
### 7.2 Maximum Payload Size
| Platform | Maximum | Notes |
|----------|---------|-------|
| Desktop | Unlimited | Limited by NATS server configuration |
| Dart Desktop | Unlimited | Limited by NATS server configuration |
| Dart Flutter | Unlimited | Limited by NATS server configuration |
| Dart Web | Unlimited | Limited by NATS server configuration |
| Rust | Unlimited | Limited by NATS server configuration |
| MicroPython | 50KB | Hard limit due to 256KB-1MB memory |
---
## 8. Message Envelope Requirements
### 8.1 Required Fields
| Field | Type | Purpose |
|-------|------|---------|
| `correlation_id` | String (UUID) | Track message flow across systems |
| `msg_id` | String (UUID) | Unique message identifier |
| `timestamp` | String (ISO 8601) | Message publication timestamp |
| `send_to` | String | NATS subject to publish to |
| `msg_purpose` | String | ACK, NACK, updateStatus, shutdown, chat |
| `sender_name` | String | Sender application name |
| `sender_id` | String (UUID) | Sender unique identifier |
| `receiver_name` | String | Receiver application name (empty = broadcast) |
| `receiver_id` | String (UUID) | Receiver unique identifier (empty = broadcast) |
| `reply_to` | String | Topic for reply messages |
| `reply_to_msg_id` | String | Message ID being replied to |
| `broker_url` | String | NATS server URL |
| `metadata` | Dict | Message-level metadata |
| `payloads` | Array | List of payload objects |
### 8.2 Payload Fields
| Field | Type | Purpose |
|-------|------|---------|
| `id` | String (UUID) | Unique payload identifier |
| `dataname` | String | Name of the payload |
| `payload_type` | String | Type: text, dictionary, arrowtable, etc. |
| `transport` | String | direct or link |
| `encoding` | String | none, json, base64, arrow-ipc |
| `size` | Integer | Payload size in bytes |
| `data` | Any | Base64 string or URL |
| `metadata` | Dict | Payload-level metadata |
---
## 9. Error Handling Requirements
### 9.1 Error Codes
| Error | Condition | Response |
|-------|-----------|----------|
| `Unknown payload_type` | Unsupported type | Throw error |
| `Failed to upload` | File server error | Throw error |
| `Failed to fetch` | File server unavailable | Retry with exponential backoff |
| `Unknown transport` | Invalid transport type | Throw error |
| `NATS connection failed` | NATS unavailable | Throw error |
### 9.2 Exception Handling
| Scenario | Handler |
|----------|---------|
| File server unavailable | Retry up to 5 times with exponential backoff |
| NATS publish failure | Connection auto-reconnect |
| Deserialization error | Log correlation ID and throw error |
| Memory overflow (MicroPython) | Reject payloads >50KB |
---
## 10. Testing Requirements
### 10.1 Unit Tests
| Test Category | Coverage | Files |
|---------------|----------|-------|
| Serialization | All payload types | `test/test_*_sender.*` |
| Deserialization | All payload types | `test/test_*_receiver.*` |
| Transport selection | Direct vs link | `test/test_*_mix_payloads.*` |
| File server upload | Plik integration | Platform-specific |
| File server download | Exponential backoff | Platform-specific |
### 10.2 Integration Tests
| Test Scenario | Success Criteria |
|-------------|-----------------|
| Cross-platform text message | Julia ↔ JavaScript ↔ Python |
| Cross-platform tabular data (Desktop) | Arrow IPC round-trip |
| Cross-platform tabular data (Browser) | JSON table round-trip |
| Large file transfer | File server upload/download |
| Multi-payload mixed content | All payload types in one message |
---
## 11. API Contract
### 11.1 smartsend Signature
```julia
function smartsend(
subject::String,
data::AbstractArray{Tuple{String, T1, String}, 1};
broker_url::String = DEFAULT_BROKER_URL,
fileserver_url::String = DEFAULT_FILESERVER_URL,
fileserver_upload_handler::Function = plik_oneshot_upload,
size_threshold::Int = DEFAULT_SIZE_THRESHOLD,
correlation_id::String = string(uuid4()),
msg_purpose::String = "chat",
sender_name::String = "msghandler",
receiver_name::String = "",
receiver_id::String = "",
reply_to::String = "",
reply_to_msg_id::String = "",
msg_id::String = string(uuid4()),
sender_id::String = string(uuid4())
)::Tuple{msg_envelope_v1, String} where {T1<:Any}
```
**Note**: NATS publishing is the caller's responsibility. `smartsend` returns `(env::msg_envelope_v1, env_json_str::String)`.
### 11.2 smartreceive Signature
```julia
function smartreceive(
msg_json_str::String;
fileserver_download_handler::Function = _fetch_with_backoff,
max_retries::Int = 5,
base_delay::Int = 100,
max_delay::Int = 5000
)::JSON.Object{String, Any}
```
**Note**: Pass `String(nats_msg.payload)` from NATS subscription to `smartreceive`.
---
## 12. Deployment Requirements
### 12.1 Minimum Infrastructure
| Component | Minimum | Notes |
|-----------|---------|-------|
| NATS Server | 1 instance | Single node for development |
| File Server | 1 instance | HTTP server for large payloads |
| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices |
### 12.2 Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) |
---
## 13. Versioning
### 13.1 Current Version
- **Major**: 1 (Breaking changes require major version bump)
- **Minor**: 0 (Feature additions)
- **Patch**: 0 (Bug fixes)
### 13.2 Version Compatibility
| Version | Supported Platforms |
|---------|---------------------|
| v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, Dart 2.17+, Rust 1.70+, Browser (latest), MicroPython 1.19+ |
---
## 14. Change Log
| Date | Version | Changes |
|------|---------|---------|
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) |
| - | - | Fixed smartsend signature: removed is_publish, NATS_connection; added sender_name |
| - | - | Fixed smartreceive signature: takes msg_json_str::String instead of msg::NATS.Msg |
| - | - | Fixed size_threshold default from 1,000,000 to 500,000 |
| - | - | Updated FR-013/FR-014 to reflect caller responsibility for NATS publishing |
| - | - | Updated FR-008/FR-009 to include file path upload overload |
| - | - | Updated SIZE_THRESHOLD env var default to 500000 |
| 2026-03-23 | 1.0.0 | Updated to ASG Framework requirements structure |
---
## 15. References
- [`src/msghandler.jl`](../src/msghandler.jl) - Ground truth implementation (Julia)
- [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) - Server-side JavaScript implementation
- [`src/msghandler_csr.js`](../src/msghandler_csr.js) - Client-side JavaScript implementation
- [`src/msghandler.py`](../src/msghandler.py) - Python implementation
- [`src/msghandler.dart`](../src/msghandler.dart) - Dart implementation
- [`src/msghandler_mpy.py`](../src/msghandler_mpy.py) - MicroPython implementation
- [`src/msghandler.rs`](../src/msghandler.rs) - Rust implementation
- [`README.md`](../README.md) - Project overview
- [`docs/specification.md`](./specification.md) - Technical specification
- [`docs/ui-specification.md`](./ui-specification.md) - UI specification
- [`docs/walkthrough.md`](./walkthrough.md) - End-to-end walkthrough
- [`docs/architecture.md`](./architecture.md) - Architecture documentation
- [`docs/validation.md`](./validation.md) - Validation and CI/CD
- [`docs/runbook.md`](./runbook.md) - Operational runbook

1437
docs/specification.md Normal file
View File

File diff suppressed because it is too large Load Diff

965
docs/walkthrough.md Normal file
View File

@@ -0,0 +1,965 @@
# Walkthrough: msghandler
**Version**: 1.4.0
**Date**: 2026-05-14
**Status**: Active
**Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl)
---
## 1. Executive Summary
This document provides the **end-to-end trace** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus.
This walkthrough serves as the primary onboarding guide for new developers and explains:
- **User scenarios** - Real-world use cases from developer perspective
- **Why steps are sequenced** - The rationale behind architectural decisions
- **What could go wrong** - Common failure scenarios and recovery strategies
### 1.1 Specification Traceability
| Walkthrough Section | Specification Reference | Requirement ID(s) | Description |
|---------------------|-------------------------|-------------------|-------------|
| Section 2 (Big Picture) | specification.md:2, specification.md:15 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | End-to-end system flow diagrams |
| Section 3 (Chat Scenario) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | FR-001, FR-006, FR-007, FR-012, FR-013, FR-014 | Chat webapp ↔ Julia backend with mixed payloads |
| Section 4 (Large File) | specification.md:6, specification.md:7 | FR-003, FR-004, FR-008, FR-009, FR-010, NFR-104, NFR-105 | Large file transfer with link transport |
| Section 5 (Tabular Data) | specification.md:5, specification.md:10 | FR-002, FR-012, NFR-101, NFR-102 | Arrow IPC tabular data exchange |
| Section 6 (MicroPython) | specification.md:13, specification.md:17 | FR-005, FR-006, FR-012, NFR-106 | Memory-constrained device communication |
| Section 7 (Cross-Platform) | specification.md:3, specification.md:4, specification.md:5, specification.md:11 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | Multi-platform chat application |
| Section 8 (Error Handling) | specification.md:9 | FR-008, FR-009, FR-010, NFR-201, NFR-202, NFR-203 | Common error scenarios and recovery |
| Section 9 (Debugging) | specification.md:4, specification.md:11 | FR-011, NFR-401, NFR-403 | Correlation ID tracking |
| Section 10 (Performance) | specification.md:7, specification.md:13 | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 | Optimization strategies |
| Section 11 (Deployment) | specification.md:12, specification.md:18 | FR-013, FR-014, NFR-201, NFR-203 | Infrastructure requirements |
---
## 2. Overview: The Big Picture
## Overview: The Big Picture
msghandler implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB):
```mermaid
flowchart TB
subgraph msghandler["msghandler Module"]
direction TB
subgraph Sender["Sender (smartsend)"]
direction LR
S1["Data Tuples<br/>[(dataname, data, type)]"]
S2["Serialize Data"]
S3["Size Check"]
S4["Transport Selection"]
S5["Build Envelope"]
S6["Publish to NATS"]
S1 --> S2
S2 --> S3
S3 --> S4
S4 --> S5
S5 --> S6
end
subgraph Receiver["Receiver (smartreceive)"]
direction LR
R1["Subscribe to NATS"]
R2["Parse Envelope"]
R3["Check Transport"]
R4["Deserialize Data"]
R5["Return Payloads"]
R1 --> R2
R2 --> R3
R3 --> R4
R4 --> R5
end
S6 -.->|Message| R1
end
subgraph FileServer["HTTP File Server (Plik)"]
direction TB
FS1["Upload URL"]
FS2["Download URL"]
S4 -.->|Large Payload| FS1
FS1 -.->|URL| S5
R3 -.->|Fetch URL| FS2
end
style msghandler fill:#e1f5fe,stroke:#0288d1,stroke-width:2px
style Sender fill:#b3e5fc,stroke:#0288d1
style Receiver fill:#b3e5fc,stroke:#0288d1
style FileServer fill:#ffe0b2,stroke:#f57c00
```
### Key Design Principles
### Key Design Principles
| Principle | Description | Rationale |
|-----------|-------------|-----------|
| **Claim-Check Pattern** | Large payloads uploaded to HTTP server, URL sent via NATS | NATS has message size limits; avoids NATS overflow |
| **Automatic Transport Selection** | Direct (< threshold) vs Link (≥ threshold) based on size | Optimizes memory vs network I/O trade-off |
| **Cross-Platform API** | Consistent `smartsend()`/`smartreceive()` across all platforms | Simplifies developer experience |
| **Exponential Backoff** | Retry downloads with increasing delays | Handles transient failures gracefully |
---
## User Scenario 1: Chat Webapp ↔ Julia Backend
### Scenario Description
A JavaScript chat webapp wants to send mixed payloads (text message + user avatar image) to a Julia backend, and receive mixed payloads (text response + AI-generated image) back.
### Step-by-Step Flow
#### Step 1: JavaScript Webapp Sends Mixed Payloads
```javascript
// JavaScript (Browser or Node.js)
const [env, msgJson] = await msghandler.smartsend(
"/agent/wine/api/v1/prompt",
[
["msg", "Hello! I'm Ton.", "text"],
["avatar", avatarImageData, "image"]
],
{
broker_url: "ws://localhost:4222",
receiver_name: "agent-backend",
msg_purpose: "chat"
}
);
```
**Rationale**:
- **Why mixed payloads?** Real chat apps often send both text and images together
- **Why text first?** Text is smaller, sent via direct transport (fast, no file server needed)
- **Why image second?** Images may trigger link transport if >0.5MB
#### Step 2: Transport Selection
For each payload, msghandler determines transport:
| Payload | Size | Transport | Reason |
|---------|------|-----------|--------|
| `"msg"` (text) | ~20 bytes | direct | < 0.5MB threshold |
| `"avatar"` (image) | ~150KB | direct | < 0.5MB threshold |
**Rationale**:
- Direct transport is faster for small payloads (no file server round-trip)
- Link transport is used when payload ≥ 0.5MB (avoids NATS size limits)
#### Step 3: Serialization and Encoding
Each payload is serialized:
| Payload | Type | Serialization | Encoding |
|---------|------|---------------|----------|
| `"msg"` | `text` | UTF-8 bytes | Base64 |
| `"avatar"` | `image` | Raw bytes | Base64 |
**Rationale**:
- Text uses UTF-8 encoding for human-readable data
- Images use raw bytes to preserve binary data integrity
- All payloads encoded as Base64 for JSON compatibility
#### Step 4: Envelope Building
msghandler builds the message envelope:
```json
{
"correlation_id": "a1b2c3d4...",
"msg_id": "e5f6g7h8...",
"timestamp": "2026-03-13T16:30:00.000Z",
"send_to": "/agent/wine/api/v1/prompt",
"msg_purpose": "chat",
"sender_name": "chat-webapp",
"sender_id": "sender-uuid...",
"receiver_name": "agent-backend",
"receiver_id": "",
"reply_to": "/agent/wine/api/v1/response",
"reply_to_msg_id": "",
"broker_url": "ws://localhost:4222",
"metadata": {},
"payloads": [
{
"id": "payload-uuid...",
"dataname": "msg",
"payload_type": "text",
"transport": "direct",
"encoding": "base64",
"size": 20,
"data": "SGVsbG8hIEknIHRlbCB5b3UgSW4gZW5nbGlzaC4=",
"metadata": {"payload_bytes": 20}
},
{
"id": "payload-uuid...",
"dataname": "avatar",
"payload_type": "image",
"transport": "direct",
"encoding": "base64",
"size": 150000,
"data": "iVBORw0KGgoAAAANSUhEUgAA...",
"metadata": {"payload_bytes": 150000}
}
]
}
```
**Rationale**:
- **correlation_id**: Tracks this chat session across all systems
- **reply_to**: Tells backend where to send response
- **payloads array**: Contains all data with metadata for proper handling
#### Step 5: Publish to NATS (Caller's Responsibility)
```javascript
// NATS publishing is the caller's responsibility
const conn = await NATS.connect({ servers: "ws://localhost:4222" });
await conn.publish("/agent/wine/api/v1/prompt", msgJson);
```
**Rationale**:
- NATS provides low-latency message delivery
- JSON format ensures cross-platform compatibility
- `smartsend()` returns `(env, msgJson)` - caller handles publishing
#### Step 6: Julia Backend Receives Message
```julia
# Julia backend
nats_msg = NATS.subscription.next() # Get message from NATS
env = smartreceive(String(nats_msg.payload))
# env["payloads"] is now:
# [
# ("msg", "Hello! I'm Ton.", "text"),
# ("avatar", binary_data, "image")
# ]
```
**Rationale**:
- `smartreceive()` handles both transport types automatically
- Deserialization is type-aware based on `payload_type`
- Returns consistent tuple format regardless of transport
#### Step 7: Julia Backend Sends Response
```julia
# Julia backend processes the message
response_text = "Hello Ton! I'm the AI assistant."
generated_image = generate_ai_image(response_text)
env, msg_json = smartsend(
"/agent/wine/api/v1/response",
[
("response", response_text, "text"),
("generated_image", generated_image, "image")
],
reply_to = "/chat/user/v1/message",
reply_to_msg_id = msg["msg_id"]
)
```
**Rationale**:
- **Mixed response**: Text explanation + AI-generated image
- **reply_to**: Ensures response goes to correct topic
- **reply_to_msg_id**: Links response to original message for tracing
---
## User Scenario 2: Large File Transfer
### Scenario Description
A JavaScript webapp wants to upload a large file (10MB) to a Julia backend for processing.
### Step-by-Step Flow
#### Step 1: JavaScript Webapp Sends Large File
```javascript
const [env, msgJson] = await msghandler.smartsend(
"/agent/wine/api/v1/process",
[
["file", largeFileData, "binary"]
],
{
broker_url: "ws://localhost:4222",
receiver_name: "agent-backend"
}
);
```
#### Step 2: Transport Selection (Link)
| Payload | Size | Transport | Reason |
|---------|------|-----------|--------|
| `"file"` | 10MB | link | ≥ 0.5MB threshold |
**Rationale**:
- Link transport used for large payloads
- File server handles large file upload
- NATS only sends URL (small message)
#### Step 3: File Server Upload
```javascript
// msghandler internally calls:
const response = await plikOneshotUpload(
"http://localhost:8080",
"file",
largeFileData
);
// Response:
// {
// status: 200,
// uploadid: "UPLOAD_ID",
// fileid: "FILE_ID",
// url: "http://localhost:8080/file/UPLOAD_ID/FILE_ID/file"
// }
```
**Rationale**:
- Plik handles multipart upload
- One-shot mode simplifies API
- Returns URL for download
#### Step 4: Envelope with Link Transport
```json
{
"correlation_id": "a1b2c3d4...",
"payloads": [
{
"id": "payload-uuid...",
"dataname": "file",
"payload_type": "binary",
"transport": "link",
"encoding": "none",
"size": 10000000,
"data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/file"
}
]
}
```
**Rationale**:
- `data` field contains URL instead of Base64
- `transport: "link"` signals URL-based download
- `encoding: "none"` indicates no additional encoding
#### Step 5: Julia Backend Receives and Downloads
```julia
# Julia backend
nats_msg = NATS.subscription.next()
env = smartreceive(String(nats_msg.payload))
# msghandler automatically:
# 1. Extracts URL from payload
# 2. Downloads with exponential backoff
# 3. Deserializes to binary data
```
**Rationale**:
- Exponential backoff handles transient failures
- Automatic download simplifies receiver code
- Binary data returned directly
---
## User Scenario 3: Tabular Data Exchange
### Scenario Description
A Python application sends tabular data (pandas DataFrame) to a Julia backend for analysis, and receives processed results back.
### Step-by-Step Flow
#### Step 1: Python Sends Tabular Data
```python
# Python
import pandas as pd
from msghandler import smartsend
df = pd.DataFrame({
"id": [1, 2, 3],
"name": ["Alice", "Bob", "Charlie"],
"score": [95, 88, 92]
})
env, msg_json = await smartsend(
"/agent/wine/api/v1/analyze",
[("data", df, "arrowtable")],
broker_url="nats://localhost:4222",
receiver_name="agent-backend"
)
```
**Rationale**:
- `arrowtable` type for efficient tabular data transfer
- Arrow IPC format preserves data types
- Much faster than JSON serialization
#### Step 2: Serialization to Arrow IPC
```python
# msghandler internally:
import pyarrow as pa
import pyarrow.ipc as ipc
table = pa.Table.from_pandas(df)
buf = io.BytesIO()
sink = ipc.new_file(buf, table.schema)
ipc.write_table(table, sink)
arrow_bytes = buf.getvalue()
```
**Rationale**:
- Arrow IPC preserves column types
- Binary format is compact
- No schema information loss
#### Step 3: Julia Receives and Deserializes
```julia
# Julia backend
nats_msg = NATS.subscription.next()
env = smartreceive(String(nats_msg.payload))
# env["payloads"][1] is now:
# ("data", DataFrame with id, name, score columns, "arrowtable")
```
**Rationale**:
- Arrow.jl reads IPC format directly
- DataFrame returned with correct types
- No manual parsing needed
#### Step 4: Julia Sends Results
```julia
# Julia backend
results = analyze_data(env["payloads"][1][2])
# Send results back
env, msg_json = smartsend(
"/agent/wine/api/v1/results",
[("results", results, "arrowtable")],
reply_to = "/python/worker/v1/results"
)
```
**Rationale**:
- Arrow IPC format for efficient round-trip
- Results preserve DataFrame structure
- Python can deserialize to pandas DataFrame
---
## User Scenario 4: Rust Service with Type-Safe API
### Scenario Description
A Rust service needs to process messages from a Julia analytics pipeline and send typed results back. The Rust implementation leverages compile-time type safety via Rust enums and serde for serialization.
### Step-by-Step Flow
#### Step 1: Rust Service Receives Message
```rust
// Rust service - using tokio async runtime
use msghandler::{smartreceive, MsgEnvelopeV1};
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64};
#[tokio::main]
async fn main() {
let conn = nats::connect("nats://localhost:4222").unwrap();
// Subscribe and receive messages
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze").unwrap();
for msg in sub.messages() {
let envelope = smartreceive(
&String::from_utf8_lossy(&msg.payload),
&Default::default(),
).await.unwrap();
// Access deserialized payloads by type
for payload in &envelope.payloads {
match payload.payload_type.as_str() {
"arrowtable" => {
// Data is base64-encoded Arrow IPC bytes after smartreceive()
let arrow_bytes = BASE64.decode(&payload.data).unwrap();
println!("Received arrowtable payload ({} bytes)", arrow_bytes.len());
},
"text" => {
// Data is the decoded text string
println!("Message: {}", payload.data);
},
"image" | "audio" | "video" | "binary" => {
// Data is base64-encoded binary content
let bytes = BASE64.decode(&payload.data).unwrap();
println!("Received {} bytes of {} data", bytes.len(), payload.payload_type);
},
"dictionary" | "jsontable" => {
// Data is a JSON string
println!("Data: {}", payload.data);
},
_ => println!("Unknown payload type: {}", payload.payload_type),
}
}
}
}
```
**Rationale**:
- **serde serialization**: Automatic JSON deserialization to `MsgEnvelopeV1`
- **tokio runtime**: Efficient async I/O for NATS and HTTP operations
- **smartreceive deserialization**: Payload data is deserialized and stored as strings in `payload.data`
- **Type dispatch**: `payload_type` field determines how to interpret the `data` string
#### Step 2: Rust Service Sends Processed Results
```rust
// Rust service sends results back with mixed payload types
use msghandler::{smartsend, Payload, SmartsendOptions};
let results_df = /* processed Arrow table */;
let result_bytes = /* serialize to Arrow IPC */;
let (envelope, json_str) = smartsend(
"/agent/wine/api/v1/results",
&[
(
"results".to_string(),
Payload::ArrowTable(result_bytes),
"arrowtable".to_string(),
),
(
"summary".to_string(),
Payload::Text("Analysis complete: 1500 rows processed".to_string()),
"text".to_string(),
),
],
&SmartsendOptions {
broker_url: "nats://localhost:4222".to_string(),
reply_to: "/python/worker/v1/results".to_string(),
msg_purpose: "chat".to_string(),
..Default::default()
},
).await?;
// Caller publishes to NATS
conn.publish("/agent/wine/api/v1/results", &json_str)?;
```
**Rationale**:
- **Builder pattern**: `SmartsendOptions` provides clean configuration
- **Enum-based payloads**: Type safety prevents sending incorrect data types
- **Default options**: sensible defaults reduce boilerplate
- **Result<T, E>**: idiomatic Rust error handling
#### Step 3: Python/Julia Receives Rust Response
```python
# Python backend receives Rust response
env = await smartreceive(str(nats_msg.payload))
# env["payloads"][0] is now:
# ("results", arrow_table_data, "arrowtable")
# env["payloads"][1] is now:
# ("summary", "Analysis complete: 1500 rows processed", "text")
```
**Rationale**:
- **Cross-platform parity**: Rust envelope matches other platform envelopes exactly
- **Same JSON wire format**: No protocol translation needed
- **Type preservation**: Arrow IPC and text types preserved across all platforms
#### Step 4: Large File Transfer from Rust
```rust
// Rust service sends large binary file via link transport
let large_file_data: Vec<u8> = std::fs::read("/data/large_dataset.parquet")?;
let (envelope, json_str) = smartsend(
"/agent/wine/api/v1/upload",
&[
(
"dataset".to_string(),
Payload::Binary(large_file_data),
"binary".to_string(),
),
],
&SmartsendOptions {
broker_url: "nats://localhost:4222".to_string(),
fileserver_url: "http://localhost:8080".to_string(),
size_threshold: 500_000, // 0.5MB triggers link transport
..Default::default()
},
).await?;
```
**Rationale**:
- **Automatic transport selection**: Same 0.5MB threshold as other desktop platforms
- **reqwest integration**: Efficient HTTP client for file server upload/download
- **Exponential backoff**: Built-in retry with configurable parameters
- **Zero-copy where possible**: `Vec<u8>` passed directly without intermediate copies
---
## User Scenario 5: MicroPython Device
### Scenario Description
A MicroPython sensor device sends sensor readings to a Python backend.
### Step-by-Step Flow
#### Step 1: MicroPython Sends Sensor Data
```python
# MicroPython
from msghandler import smartsend
sensor_data = {
"temperature": 25.5,
"humidity": 60.0,
"pressure": 1013.25
}
env, msg_json = smartsend(
"/sensor/device/v1/readings",
[("data", sensor_data, "dictionary")],
broker_url="nats://localhost:4222",
size_threshold=100000 # 100KB for MicroPython
)
```
**Rationale**:
- `dictionary` type for JSON-serializable sensor data
- Smaller threshold (100KB) for memory constraints
- Direct transport only (no file server support)
#### Step 2: Serialization
```python
# msghandler internally:
json_str = json.dumps(sensor_data)
json_bytes = json_str.encode('utf-8')
payload_b64 = base64.b64encode(json_bytes).decode('ascii')
```
**Rationale**:
- JSON format for human-readable data
- Base64 for NATS compatibility
- UTF-8 for text encoding
#### Step 3: Python Backend Receives
```python
# Python backend
nats_msg = await nats_consumer.next()
env = await smartreceive(str(nats_msg.payload))
# env["payloads"][0] is now:
# ("data", {"temperature": 25.5, "humidity": 60.0, ...}, "dictionary")
```
**Rationale**:
- JSON deserialization
- Dictionary returned directly
- No Arrow support (memory constraints)
---
## User Scenario 6: Cross-Platform Chat with Mixed Payloads
### Scenario Description
Multiple platforms (JavaScript, Python, Julia) communicate in a chat application with mixed payload types.
### Step-by-Step Flow
#### Step 1: JavaScript Sends Chat Message
```javascript
// JavaScript (Frontend)
const [env, msgJson] = await msghandler.smartsend(
"/chat/user/v1/message",
[
["text", "Check this out!", "text"],
["image", imageData, "image"]
],
{
broker_url: "ws://localhost:4222",
receiver_name: "",
msg_purpose: "chat"
}
);
```
**Rationale**:
- Empty `receiver_name` = broadcast to all subscribers
- Chat messages often include text + images
- NATS wildcard subscriptions route to correct recipients
#### Step 2: Python Backend Receives
```python
# Python (Backend)
nats_msg = await nats_consumer.next()
env = await smartreceive(str(nats_msg.payload))
# env["payloads"] is now:
# [
# ("text", "Check this out!", "text"),
# ("image", binary_data, "image")
# ]
```
**Rationale**:
- Consistent API across platforms
- Same payload structure regardless of sender
- Type information preserved
#### Step 3: Julia Backend Receives
```julia
# Julia (Backend)
nats_msg = NATS.subscription.next()
env = smartreceive(String(nats_msg.payload))
# env["payloads"] is now:
# [
# ("text", "Check this out!", "text"),
# ("image", binary_data, "image")
# ]
```
**Rationale**:
- Cross-platform API parity
- Same function signature across platforms
- Type information enables proper deserialization
#### Step 4: All Platforms Reply
Each platform can reply using the same API:
```python
# Python reply
await smartsend(
"/chat/user/v1/reply",
[("response", "Nice!", "text")],
reply_to="/chat/user/v1/message"
)
```
```julia
# Julia reply
smartsend(
"/chat/user/v1/reply",
[("response", "Nice!", "text")],
reply_to="/chat/user/v1/message"
)
```
```javascript
// JavaScript reply
await msghandler.smartsend(
"/chat/user/v1/reply",
[["response", "Nice!", "text"]],
{ reply_to: "/chat/user/v1/message" }
);
```
**Rationale**:
- Same API across platforms
- Consistent behavior
- Easy to maintain parity
---
## Error Handling
### Common Error Scenarios
| Scenario | Error | Recovery |
|----------|-------|----------|
| File server unavailable | `UPLOAD_FAILED` | Fall back to direct transport or smaller payloads |
| File server download fails | `DOWNLOAD_FAILED` | Retry with exponential backoff |
| Payload type mismatch | `DESERIALIZATION_ERROR` | Validate payload_type matches data |
| NATS connection lost | `NATS_CONNECTION_FAILED` | NATS client auto-reconnects |
### Error Response Format
```json
{
"correlation_id": "abc123...",
"error": {
"code": "DOWNLOAD_FAILED",
"message": "Failed to fetch data after 5 attempts",
"details": {
"url": "http://localhost:8080/file/...",
"correlation_id": "abc123..."
}
}
}
```
---
## Debugging and Tracing
### Correlation ID Tracking
Every message includes a `correlation_id`:
```julia
# At start of request
correlation_id = string(uuid4())
# Use throughout the flow
log_trace(correlation_id, "Starting smartsend")
log_trace(correlation_id, "Serialized payload size: 100 bytes")
log_trace(correlation_id, "Published to NATS")
```
**Log Format**:
```
[2026-03-13T16:30:00.000Z] [Correlation: abc123...] Starting smartsend
[2026-03-13T16:30:00.001Z] [Correlation: abc123...] Serialized payload size: 100 bytes
[2026-03-13T16:30:00.002Z] [Correlation: abc123...] Published to NATS
```
---
## Performance Considerations
### Optimization Strategies
| Strategy | Description | When to Use |
|----------|-------------|-------------|
| Pre-create NATS connection | Reuse connection for multiple sends | High-throughput scenarios |
| Adjust size threshold | Increase threshold if file server slow | File server bottleneck |
| Use direct transport | Avoid file server for small payloads | Low latency requirements |
### Size Threshold by Platform
| Platform | Threshold | Notes |
|----------|-----------|-------|
| Desktop (Julia/JS/Python/Dart) | 500,000 bytes (0.5MB) | Default threshold |
| Dart Desktop | 500,000 bytes (0.5MB) | Default threshold |
| Dart Flutter | 500,000 bytes (0.5MB) | Default threshold |
| Dart Web | 500,000 bytes (0.5MB) | Default threshold |
| MicroPython | 100,000 bytes (100KB) | Lower threshold for memory constraints |
---
## Deployment Considerations
### Minimum Infrastructure
| Component | Minimum | Notes |
|-----------|---------|-------|
| NATS Server | 1 instance | Single node for development |
| File Server | 1 instance | HTTP server for large payloads |
| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices |
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) |
---
## Change Log
| Date | Version | Changes |
|------|---------|---------|
| 2026-03-13 | 1.0.0 | Initial walkthrough documentation |
---
## 12. References
### 12.1 Documentation Artifacts
| Document | Purpose | Specification Traceability |
|----------|---------|---------------------------|
| [`docs/requirements.md`](./requirements.md) | Business requirements and user stories | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/specification.md`](./specification.md) | Technical contract for msghandler | specification.md:2-19 (all sections) |
| [`docs/ui-specification.md`](./ui-specification.md) | UI specification for client applications | UI components for data entry and display |
| [`docs/walkthrough.md`](./walkthrough.md) | End-to-end system flow | This document |
| [`docs/architecture.md`](./architecture.md) | System architecture diagrams | Component interaction and data flow |
| [`docs/validation.md`](./validation.md) | CI/CD validation rules | Contract testing and spec compliance |
| [`docs/runbook.md`](./runbook.md) | Operational runbook | Deployment, scaling, and troubleshooting |
### 12.2 Implementation Files
| File | Platform | Features | Specification Traceability |
|------|----------|----------|---------------------------|
| [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) |
| [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | specification.md:2-19 (all sections) |
| [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe, file upload helpers | specification.md:2-19 (all sections) |
| [`src/msghandler_mpy.py`](../src/msghandler_mpy.py) | MicroPython | Limited to direct transport | specification.md:2-19 (all sections) |
---
## 13. Change Log
| Date | Version | Changes | Specification Reference |
|------|---------|---------|------------------------|
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections |
| - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
| - | - | Added `plik_upload_file` convenience function documentation | specification.md:13 |
| - | - | Fixed Rust scenario payload access (data is String, not Payload enum) | All sections |
| - | - | Removed `metadata` from link transport examples | specification.md:3 |
| 2026-05-13 | 1.3.0 | Added Rust support with tokio, serde, and arrow2 | All sections |
| - | - | Added Rust user scenario (User Scenario 4) | specification.md:11 (Rust API) |
| - | - | Updated scenario numbering (MicroPython → Scenario 5, Cross-Platform → Scenario 6) | All sections |
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All sections |
| - | - | Updated smartreceive calls to use String(nats_msg.payload) pattern | All sections |
| - | - | Removed NATSClient.publish() calls (caller responsible for NATS publishing) | All sections |
| - | - | Removed is_publish and nats_connection parameter references | All sections |
| 2026-03-23 | 1.0.0 | Updated to ASG Framework walkthrough guidelines | All sections |
| 2026-03-13 | 1.0.0 | Initial walkthrough documentation | specification.md:2-19 (all sections) |
---
## 14. Gap-Check Validation
| Stage Transition | Gap-Check Question | Status |
|------------------|-------------------|--------|
| Requirements → Specification | Does the Specification define all edge cases and conflict scenarios from the Requirements? | ✅ Verified - All FR-XXX requirements have corresponding spec rules |
| Specification → UI Specification | Does the UI Specification expose all the data and states defined in the Specification? | ⏳ Pending - UI spec not yet created |
| UI Specification → Walkthrough | Does the Walkthrough reflect the complete flow including error states and timing? | ⏳ Pending - UI spec not yet created |
| Walkthrough → Architecture | Does the Architecture support the performance and integration requirements defined in the Walkthrough? | ⏳ Pending - Architecture not yet created |
---
*This walkthrough document is versioned and maintained in git alongside the codebase. All implementations must adhere to this documentation.*
---
*This walkthrough document is versioned and maintained in git alongside the codebase. All implementations must adhere to this documentation.*
<tool_call>
<function=update_todo_list>
<parameter=todos>
[x] Analyze existing documentation (requirements.md, spec.md, architecture.md)
[x] Read all source files in src/ folder
[x] Write docs/walkthrough.md according to SDD framework with user scenarios