ton/msghandler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

update docs

Browse Source
This commit is contained in:
narawat
2026-05-15 13:25:48 +07:00
parent df9012e0eb
commit d0910ccc3f
4 changed files with 206 additions and 205 deletions
Download Patch File Download Diff File Expand all files Collapse all files

113
docs/architecture.md
View File

@@ -10,7 +10,7 @@
## 1. Executive Summary ## 1. Executive Summary
This document defines the **blueprint** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus. This document defines the **blueprint** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer.
This architecture document serves as the single source of truth for: This architecture document serves as the single source of truth for:
- **System Structure**: How components fit together and interact - **System Structure**: How components fit together and interact
@@ -48,11 +48,10 @@ This architecture document serves as the single source of truth for:
```mermaid ```mermaid
flowchart TD flowchart TD
subgraph "External Systems" subgraph "External Systems"
NATS_Server[NATS Server] Message_Broker[Message Broker<br/>NATS/MQTT/WebSocket/Custom]
File_Server[HTTP File Server<br/>Plik/AWS S3/Custom] File_Server[HTTP File Server<br/>Plik/AWS S3/Custom]
end end
subgraph "Client Applications"
Julia_App[Julia Application] Julia_App[Julia Application]
JS_App[JavaScript Application<br/>Node.js/Browser] JS_App[JavaScript Application<br/>Node.js/Browser]
Python_App[Python Application<br/>Desktop] Python_App[Python Application<br/>Desktop]
@@ -61,12 +60,12 @@ flowchart TD
MicroPython_App[MicroPython Device] MicroPython_App[MicroPython Device]
end end
Julia_App -->|NATS| NATS_Server Julia_App -->|Transport| Message_Broker
JS_App -->|NATS| NATS_Server JS_App -->|Transport| Message_Broker
Python_App -->|NATS| NATS_Server Python_App -->|Transport| Message_Broker
Dart_App -->|NATS| NATS_Server Dart_App -->|Transport| Message_Broker
Rust_App -->|NATS| NATS_Server Rust_App -->|Transport| Message_Broker
MicroPython_App -->|NATS| NATS_Server MicroPython_App -->|Transport| Message_Broker
Julia_App -->|HTTP| File_Server Julia_App -->|HTTP| File_Server
JS_App -->|HTTP| File_Server JS_App -->|HTTP| File_Server
@@ -75,7 +74,7 @@ flowchart TD
Rust_App -->|HTTP| File_Server Rust_App -->|HTTP| File_Server
MicroPython_App -->|HTTP| File_Server MicroPython_App -->|HTTP| File_Server
style NATS_Server fill:#fff3e0,stroke:#f57c00 style Message_Broker fill:#fff3e0,stroke:#f57c00
style File_Server fill:#f3e5f5,stroke:#9c27b4 style File_Server fill:#f3e5f5,stroke:#9c27b4
style Julia_App fill:#e8f5e9,stroke:#4caf50 style Julia_App fill:#e8f5e9,stroke:#4caf50
style JS_App fill:#e3f2fd,stroke:#2196f3 style JS_App fill:#e3f2fd,stroke:#2196f3
@@ -98,14 +97,14 @@ flowchart TD
MicroPython_Module[MicroPython msghandler Module] MicroPython_Module[MicroPython msghandler Module]
end end
Julia_Module --> NATS_Client Julia_Module --> Transport_Client
JS_Module --> NATS_Client JS_Module --> Transport_Client
Python_Module --> NATS_Client Python_Module --> Transport_Client
Dart_Module --> NATS_Client Dart_Module --> Transport_Client
Rust_Module --> NATS_Client Rust_Module --> Transport_Client
MicroPython_Module --> NATS_Client MicroPython_Module --> Transport_Client
NATS_Client --> NATS_Broker Transport_Client --> Message_Broker
Julia_Module --> File_Client Julia_Module --> File_Client
JS_Module --> File_Client JS_Module --> File_Client
@@ -122,7 +121,7 @@ flowchart TD
style Dart_Module fill:#fff0f6,stroke:#e91e63 style Dart_Module fill:#fff0f6,stroke:#e91e63
style Rust_Module fill:#dea584,stroke:#e65100 style Rust_Module fill:#dea584,stroke:#e65100
style MicroPython_Module fill:#fce4ec,stroke:#e91e63 style MicroPython_Module fill:#fce4ec,stroke:#e91e63
style NATS_Broker fill:#fff3e0,stroke:#f57c00 style Message_Broker fill:#fff3e0,stroke:#f57c00
style File_Server fill:#f3e5f5,stroke:#9c27b4 style File_Server fill:#f3e5f5,stroke:#9c27b4
``` ```
@@ -174,8 +173,8 @@ flowchart TD
| Component | Purpose | Platform Support | | Component | Purpose | Platform Support |
|-----------|---------|------------------| |-----------|---------|------------------|
| **smartsend** | Send data via NATS with automatic transport selection, returns (envelope, json_string) for caller to publish | All | | **smartsend** | Send data with automatic transport selection, returns (envelope, json_string) for caller to publish via transport | All |
| **smartreceive** | Receive and process NATS messages from JSON string | All | | **smartreceive** | Receive and process messages from JSON string | All |
| **_serialize_data** | Serialize data according to payload type | All | | **_serialize_data** | Serialize data according to payload type | All |
| **_deserialize_data** | Deserialize bytes to native data types | All | | **_deserialize_data** | Deserialize bytes to native data types | All |
| **envelope_to_json** | Convert msg_envelope_v1 struct to JSON string | All | | **envelope_to_json** | Convert msg_envelope_v1 struct to JSON string | All |
@@ -224,7 +223,7 @@ struct msg_envelope_v1
msg_id::String # UUID v4 for this message msg_id::String # UUID v4 for this message
timestamp::String # ISO 8601 UTC timestamp timestamp::String # ISO 8601 UTC timestamp
send_to::String # NATS subject to publish to send_to::String # Topic/subject to publish to
msg_purpose::String # ACK, NACK, updateStatus, shutdown, chat msg_purpose::String # ACK, NACK, updateStatus, shutdown, chat
sender_name::String # Sender application name sender_name::String # Sender application name
sender_id::String # UUID v4 of sender sender_id::String # UUID v4 of sender
@@ -233,7 +232,7 @@ struct msg_envelope_v1
reply_to::String # Topic for reply messages reply_to::String # Topic for reply messages
reply_to_msg_id::String # Message ID being replied to reply_to_msg_id::String # Message ID being replied to
broker_url::String # NATS broker URL broker_url::String # Broker URL for the transport layer
metadata::Dict{String, Any} # Message-level metadata metadata::Dict{String, Any} # Message-level metadata
payloads::Vector{msg_payload_v1} # List of payloads payloads::Vector{msg_payload_v1} # List of payloads
@@ -436,13 +435,13 @@ JavaScript uses async/await for non-blocking I/O:
#### Node.js Implementation (msghandler_ssr.js) #### Node.js Implementation (msghandler_ssr.js)
- **TCP NATS connections**: Uses `nats://` or `tls://` URLs - **Transport connections**: Uses broker URLs (e.g., `nats://`, `mqtt://`, `ws://`)
- **Apache Arrow IPC**: Full support via `apache-arrow` - **Apache Arrow IPC**: Full support via `apache-arrow`
- **Buffer for binary data**: Native Node.js Buffer handling - **Buffer for binary data**: Native Node.js Buffer handling
#### Browser Implementation (msghandler_csr.js) #### Browser Implementation (msghandler_csr.js)
- **WebSocket NATS connections**: Uses `ws://` or `wss://` URLs via `nats.ws` - **WebSocket connections**: Uses `ws://` or `wss://` URLs (transport-agnostic)
- **No Apache Arrow**: Uses `jsontable` for tabular data only - **No Apache Arrow**: Uses `jsontable` for tabular data only
- **Uint8Array for binary data**: Browser-compatible binary handling - **Uint8Array for binary data**: Browser-compatible binary handling
- **Web Crypto API**: UUID generation via `crypto.getRandomValues()` - **Web Crypto API**: UUID generation via `crypto.getRandomValues()`
@@ -474,7 +473,7 @@ Dart uses classes for stateful operations with async/await:
- **Async/await**: I/O operations - **Async/await**: I/O operations
- **dart-arrow**: Arrow IPC support (Desktop/Flutter only) - **dart-arrow**: Arrow IPC support (Desktop/Flutter only)
- **HTTP package**: HTTP file server communication - **HTTP package**: HTTP file server communication
- **nats package**: NATS client with WebSocket support (Dart Web) - **Transport package**: Transport client with WebSocket support (Dart Web)
```dart ```dart
class msghandler { class msghandler {
@@ -484,7 +483,7 @@ class msghandler {
final String fileserverUrl; final String fileserverUrl;
msghandler({ msghandler({
this.brokerUrl = 'nats://localhost:4222', this.brokerUrl = DEFAULT_BROKER_URL,
this.fileserverUrl = 'http://localhost:8080', this.fileserverUrl = 'http://localhost:8080',
}); });
} }
@@ -492,19 +491,19 @@ class msghandler {
#### Dart Desktop (Dart SDK) #### Dart Desktop (Dart SDK)
- **TCP NATS connections**: Uses `nats://` or `tls://` URLs - **Transport connections**: Uses broker URLs (e.g., `nats://`, `mqtt://`)
- **Apache Arrow IPC**: Full support via `dart-arrow` - **Apache Arrow IPC**: Full support via `dart-arrow`
- **Uint8List for binary data**: Native Dart binary handling - **Uint8List for binary data**: Native Dart binary handling
#### Dart Flutter (Dart SDK) #### Dart Flutter (Dart SDK)
- **TCP NATS connections**: Uses `nats://` or `tls://` URLs - **Transport connections**: Uses broker URLs (e.g., `nats://`, `mqtt://`)
- **Apache Arrow IPC**: Full support via `dart-arrow` - **Apache Arrow IPC**: Full support via `dart-arrow`
- **Uint8List for binary data**: Native Dart binary handling - **Uint8List for binary data**: Native Dart binary handling
#### Dart Web (Dart SDK) #### Dart Web (Dart SDK)
- **WebSocket NATS connections**: Uses `ws://` or `wss://` URLs via `nats` package - **WebSocket connections**: Uses `ws://` or `wss://` URLs (transport-agnostic)
- **No Apache Arrow**: Uses `jsontable` for tabular data only - **No Apache Arrow**: Uses `jsontable` for tabular data only
- **Uint8List for binary data**: Browser-compatible binary handling - **Uint8List for binary data**: Browser-compatible binary handling
- **Fetch API**: HTTP file server communication via `http` package - **Fetch API**: HTTP file server communication via `http` package
@@ -516,7 +515,7 @@ Browser JavaScript has specific constraints due to security and compatibility:
- **Async/await**: Native async/await support - **Async/await**: Native async/await support
- **No Apache Arrow**: Arrow IPC not available in browsers - **No Apache Arrow**: Arrow IPC not available in browsers
- **JSON table only**: Use "jsontable" for tabular data - **JSON table only**: Use "jsontable" for tabular data
- **WebSocket NATS**: Uses nats.ws for browser-compatible NATS connections - **WebSocket transport**: Uses transport client for browser-compatible connections
- **Fetch API**: HTTP file server communication via fetch - **Fetch API**: HTTP file server communication via fetch
### MicroPython Architecture ### MicroPython Architecture
@@ -540,7 +539,7 @@ Rust leverages compile-time type safety and async runtimes:
- **Type-safe payloads**: Rust enum discriminates between `Text`, `Dictionary`, `ArrowTable`, `Binary`, etc. - **Type-safe payloads**: Rust enum discriminates between `Text`, `Dictionary`, `ArrowTable`, `Binary`, etc.
- **serde serialization**: Automatic JSON deserialization via `#[derive(Serialize, Deserialize)]` - **serde serialization**: Automatic JSON deserialization via `#[derive(Serialize, Deserialize)]`
- **tokio runtime**: Efficient async I/O for NATS connections and HTTP file server operations - **tokio runtime**: Efficient async I/O for transport connections and HTTP file server operations
- **arrow2 integration**: Native Arrow IPC deserialization without intermediate format conversion - **arrow2 integration**: Native Arrow IPC deserialization without intermediate format conversion
- **reqwest**: High-performance HTTP client with built-in TLS and connection pooling - **reqwest**: High-performance HTTP client with built-in TLS and connection pooling
- **Zero-copy patterns**: `Vec<u8>` passed directly to avoid unnecessary memory copies - **Zero-copy patterns**: `Vec<u8>` passed directly to avoid unnecessary memory copies
@@ -572,8 +571,8 @@ pub struct SmartsendOptions {
// ... other fields // ... other fields
} }
// NATS client with tokio integration // Transport client with tokio integration
let conn = nats::connect("nats://localhost:4222").await?; let conn = transport_client::connect(DEFAULT_BROKER_URL).await?;
// Subscribe and process messages // Subscribe and process messages
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze")?; let mut sub = conn.subscribe("/agent/wine/api/v1/analyze")?;
@@ -599,7 +598,7 @@ for msg in sub.messages() {
| Component | Scaling Strategy | | Component | Scaling Strategy |
|-----------|------------------| |-----------|------------------|
| **NATS Server** | Cluster deployment with multiple nodes | | **Message Broker** | Cluster deployment with multiple nodes |
| **File Server** | Load balancer + multiple instances | | **File Server** | Load balancer + multiple instances |
| **Client Applications** | Deploy multiple instances behind load balancer | | **Client Applications** | Deploy multiple instances behind load balancer |
@@ -607,7 +606,7 @@ for msg in sub.messages() {
| Component | Scaling Strategy | | Component | Scaling Strategy |
|-----------|------------------| |-----------|------------------|
| **NATS Server** | Increase memory, CPU, disk I/O | | **Message Broker** | Increase memory, CPU, disk I/O |
| **File Server** | Increase memory, CPU, disk capacity | | **File Server** | Increase memory, CPU, disk capacity |
| **Client Applications** | Increase heap size (Python/JS) | | **Client Applications** | Increase heap size (Python/JS) |
@@ -617,7 +616,7 @@ for msg in sub.messages() {
|--------|--------|-------| |--------|--------|-------|
| Message serialization overhead | <50ms | For 10KB payload | | Message serialization overhead | <50ms | For 10KB payload |
| Message deserialization overhead | <50ms | For 10KB payload | | Message deserialization overhead | <50ms | For 10KB payload |
| NATS connection establishment | <100ms | Connection pool recommended | | Transport connection establishment | <100ms | Connection pool recommended |
| File upload latency | <1s | For 0.5MB file | | File upload latency | <1s | For 0.5MB file |
| File download latency | <1s | For 0.5MB file | | File download latency | <1s | For 0.5MB file |
@@ -625,16 +624,16 @@ for msg in sub.messages() {
## Failure Modes and Recovery ## Failure Modes and Recovery
### NATS Connection Failure ### Transport Connection Failure
**Scenario**: NATS server unavailable **Scenario**: Message broker unavailable
**Handler**: **Handler**:
- Connection auto-reconnect via TCP-level reconnection - Connection auto-reconnect via transport-level reconnection
- Retry with exponential backoff for publish operations - Retry with exponential backoff for publish operations
**Recovery**: **Recovery**:
- NATS client automatically attempts reconnection - Transport client automatically attempts reconnection
- Application can check connection status before publishing - Application can check connection status before publishing
### File Server Unavailable ### File Server Unavailable
@@ -697,7 +696,7 @@ for msg in sub.messages() {
**Rationale**: **Rationale**:
- Simplifies JSON serialization (all data is string-compatible) - Simplifies JSON serialization (all data is string-compatible)
- Increases payload size by ~33%, but NATS can handle this - Increases payload size by ~33%, but transport can handle this
- Alternative would be binary payload support (more complex) - Alternative would be binary payload support (more complex)
### Decision 3: Multiple Platform Implementations ### Decision 3: Multiple Platform Implementations
@@ -730,7 +729,7 @@ for msg in sub.messages() {
| Component | Minimum | Notes | | Component | Minimum | Notes |
|-----------|---------|-------| |-----------|---------|-------|
| NATS Server | 1 instance | Single node for development | | Message Broker | 1 instance | Single node for development |
| File Server | 1 instance | HTTP server for large payloads | | File Server | 1 instance | HTTP server for large payloads |
| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) | | Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices | | Client Memory | 256KB | MicroPython devices |
@@ -739,7 +738,7 @@ for msg in sub.messages() {
| Variable | Default | Description | | Variable | Default | Description |
|----------|---------|-------------| |----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL | | `BROKER_URL` | `ws://localhost:4222` | Message broker URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL | | `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) | | `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) |
@@ -748,15 +747,15 @@ for msg in sub.messages() {
```mermaid ```mermaid
flowchart TD flowchart TD
subgraph "Docker Network" subgraph "Docker Network"
NATS_Container[NATS Server] Broker_Container[Message Broker]
FileServer_Container[Plik File Server] FileServer_Container[Plik File Server]
App_Container[Application Container] App_Container[Application Container]
end end
App_Container -->|NATS| NATS_Container App_Container -->|Transport| Broker_Container
App_Container -->|HTTP| FileServer_Container App_Container -->|HTTP| FileServer_Container
style NATS_Container fill:#fff3e0,stroke:#f57c00 style Broker_Container fill:#fff3e0,stroke:#f57c00
style FileServer_Container fill:#f3e5f5,stroke:#9c27b4 style FileServer_Container fill:#f3e5f5,stroke:#9c27b4
style App_Container fill:#e3f2fd,stroke:#2196f3 style App_Container fill:#e3f2fd,stroke:#2196f3
``` ```
@@ -775,11 +774,12 @@ flowchart TD
### Transport Security ### Transport Security
**Mechanism**: TLS support for NATS connections **Mechanism**: TLS support for transport connections
**Implementation**: **Implementation**:
- Use `nats://` URL for plain text - Use `nats://` URL for plain text
- Use `tls://` URL for TLS-encrypted connections - Use `tls://` URL for TLS-encrypted connections
- Use `ws://` or `wss://` for WebSocket connections
### File Server Security ### File Server Security
@@ -835,6 +835,11 @@ flowchart TD
| Date | Version | Changes | | Date | Version | Changes |
|------|---------|---------| |------|---------|---------|
| 2026-05-15 | 1.5.0 | Made transport layer agnostic | All sections |
| - | - | Removed all NATS-specific references from architecture docs | All sections |
| - | - | Updated diagrams to use generic "Message Broker" instead of "NATS Server" | All sections |
| - | - | Updated code examples to use transport-agnostic patterns | All sections |
| - | - | Removed NATS client packages from external dependencies | All sections |
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections | | 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections |
| - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 | | - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
| - | - | Added `plik_upload_file` convenience function to component table | specification.md:13 | | - | - | Added `plik_upload_file` convenience function to component table | specification.md:13 |
@@ -849,6 +854,7 @@ flowchart TD
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | | 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) |
| - | - | Removed publish_message component (commented out in source) | | - | - | Removed publish_message component (commented out in source) |
| - | - | Removed NATSClient and NATSConnectionPool classes (not in ground truth) | | - | - | Removed NATSClient and NATSConnectionPool classes (not in ground truth) |
| - | - | Updated smartsend to return JSON for caller to publish via transport |
| - | - | Updated component diagram to match actual module structure | | - | - | Updated component diagram to match actual module structure |
| - | - | Updated data flow to show smartsend returns JSON for caller to publish | | - | - | Updated data flow to show smartsend returns JSON for caller to publish |
| - | - | Fixed SIZE_THRESHOLD default to 500,000 bytes | | - | - | Fixed SIZE_THRESHOLD default to 500,000 bytes |
@@ -856,6 +862,7 @@ flowchart TD
| - | - | Added NATSClient with keepAlive support | | - | - | Added NATSClient with keepAlive support |
| - | - | Added NATSConnectionPool for connection reuse | | - | - | Added NATSConnectionPool for connection reuse |
| - | - | Added publishMessage function with closeConnection option | | - | - | Added publishMessage function with closeConnection option |
| (Historical - pre-transport-agnostic refactor) | | |
| 2026-03-13 | 1.0.0 | Initial architecture documentation | | 2026-03-13 | 1.0.0 | Initial architecture documentation |
--- ---
@@ -880,7 +887,7 @@ flowchart TD
|------|----------|----------|---------------------------|-------------------| |------|----------|----------|---------------------------|-------------------|
| [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe, file upload helpers | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe, file upload helpers | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
@@ -889,24 +896,18 @@ flowchart TD
### 16.3 External Dependencies ### 16.3 External Dependencies
| Platform | Package | Version | Purpose | Specification Traceability | Requirement ID(s) | | Platform | Package | Version | Purpose | Specification Traceability | Requirement ID(s) |
|----------|---------|---------|---------|---------------------------|-------------------| |----------|---------|---------|---------|--------------------------|-------------------|
| Julia | NATS.jl | Latest | NATS client | specification.md:11 | FR-013, FR-014, NFR-201 |
| Julia | JSON.jl | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 | | Julia | JSON.jl | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Julia | Arrow.jl | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 | | Julia | Arrow.jl | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Julia | HTTP.jl | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 | | Julia | HTTP.jl | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Julia | UUIDs.jl | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 | | Julia | UUIDs.jl | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 |
| Node.js | nats | Latest | NATS client (TCP) | specification.md:11 | FR-013, FR-014 |
| Node.js | node-fetch | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 | | Node.js | node-fetch | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Browser | nats.ws | Latest | NATS client (WebSocket) | specification.md:11 | FR-013, FR-014 | | Browser | - | - | Transport-agnostic (caller provides) | specification.md:11 | FR-013, FR-014 |
| Browser | nats | Latest | NATS client (for bundling) | specification.md:11 | FR-013, FR-014 |
| Python | nats-py | Latest | NATS client | specification.md:11 | FR-013, FR-014 |
| Python | aiohttp | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 | | Python | aiohttp | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Python | pyarrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 | | Python | pyarrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Dart | nats | Latest | NATS client | specification.md:11 | FR-013, FR-014 |
| Dart | http | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 | | Dart | http | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Dart | uuid | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 | | Dart | uuid | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 |
| Dart | dart-arrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 | | Dart | dart-arrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Rust | nats | Latest | NATS client | specification.md:11 | FR-013, FR-014 |
| Rust | serde | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 | | Rust | serde | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Rust | serde_json | Latest | JSON handling | specification.md:11 | FR-012, NFR-101, NFR-102 | | Rust | serde_json | Latest | JSON handling | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Rust | tokio | Latest | Async runtime | specification.md:11 | FR-013, FR-014 | | Rust | tokio | Latest | Async runtime | specification.md:11 | FR-013, FR-014 |

75
docs/requirements.md
View File

@@ -11,7 +11,7 @@
### 1.1 Business Goal ### 1.1 Business Goal
msghandler is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS. msghandler is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over the transport layer.
### 1.2 User Stories (with acceptance criteria) ### 1.2 User Stories (with acceptance criteria)
@@ -19,13 +19,13 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
|-------|----------|---------------------| |-------|----------|---------------------|
| **As a Julia developer**, I want to send text messages to JavaScript/Dart applications that lives on a server and also on a browser | P1 | Text messages are serialized, encoded, and received correctly across platforms | | **As a Julia developer**, I want to send text messages to JavaScript/Dart applications that lives on a server and also on a browser | P1 | Text messages are serialized, encoded, and received correctly across platforms |
| **As a Python developer**, I want to send tabular data to Julia/Dart applications | P1 | DataFrame exchange works with both Arrow IPC and JSON formats | | **As a Python developer**, I want to send tabular data to Julia/Dart applications | P1 | DataFrame exchange works with both Arrow IPC and JSON formats |
| **As a JavaScript developer**, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS | | **As a JavaScript developer**, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via the transport layer |
| **As a Dart developer**, I want to send text messages to other platforms | P1 | Text messages are serialized, encoded, and received correctly across platforms | | **As a Dart developer**, I want to send text messages to other platforms | P1 | Text messages are serialized, encoded, and received correctly across platforms |
| **As a Dart developer**, I want to send dictionary data to other platforms | P1 | JSON-serializable data is exchanged correctly | | **As a Dart developer**, I want to send dictionary data to other platforms | P1 | JSON-serializable data is exchanged correctly |
| **As a Dart developer**, I want to send tabular data (List<Map>) to other platforms | P1 | JSON table format exchange works with Arrow IPC on desktop | | **As a Dart developer**, I want to send tabular data (List<Map>) to other platforms | P1 | JSON table format exchange works with Arrow IPC on desktop |
| **As a Dart developer**, I want to send large files (>0.5MB) | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS | | **As a Dart developer**, I want to send large files (>0.5MB) | P1 | Large files are automatically uploaded to file server and URLs are sent via the transport layer |
| **As a MicroPython developer**, I want to send sensor data with minimal memory usage | P1 | Direct transport works for payloads <100KB on memory-constrained devices | | **As a MicroPython developer**, I want to send sensor data with minimal memory usage | P1 | Direct transport works for payloads <100KB on memory-constrained devices |
| **As a Rust developer**, I want to send and receive messages with type-safe APIs | P1 | Rust implementation uses serde for serialization, tokio for async, and nats-io for NATS connectivity | | **As a Rust developer**, I want to send and receive messages with type-safe APIs | P1 | Rust implementation uses serde for serialization, tokio for async, and transport-agnostic client for connectivity |
| **As a developer**, I want to send mixed-content messages (text + image + file) | P1 | msghandler accepts list of (dataname, data, type) tuples and handles each payload appropriately | | **As a developer**, I want to send mixed-content messages (text + image + file) | P1 | msghandler accepts list of (dataname, data, type) tuples and handles each payload appropriately |
| **As a developer**, I want to receive multi-payload messages | P1 | msghandler returns payloads as list of tuples with correct types preserved | | **As a developer**, I want to receive multi-payload messages | P1 | msghandler returns payloads as list of tuples with correct types preserved |
| **As a developer**, I want to use Plik as the file server | P2 | Plik one-shot upload mode is supported with upload ID and token handling | | **As a developer**, I want to use Plik as the file server | P2 | Plik one-shot upload mode is supported with upload ID and token handling |
@@ -59,37 +59,32 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| File server integration | Plik one-shot upload and custom HTTP server support | | File server integration | Plik one-shot upload and custom HTTP server support |
| Reliability features | Exponential backoff retry and correlation ID propagation | | Reliability features | Exponential backoff retry and correlation ID propagation |
| Message serialization | Converts data types to binary format (Base64, JSON, Arrow IPC) | | Message serialization | Converts data types to binary format (Base64, JSON, Arrow IPC) |
| NATS communication | Publishing and subscription via NATS subjects | | Transport communication | Publishing and subscription via message broker (NATS, MQTT, WebSocket, etc.) |
### 2.2 Out of Scope ### 2.2 Out of Scope
| Feature | Reason | | Feature | Reason |
|---------|--------| |---------|--------|
| NATS JetStream support | Core NATS sufficient for current use cases | | Advanced transport features | Basic transport sufficient for current use cases |
| Message compression | Compression adds complexity without clear benefit | | Message compression | Compression adds complexity without clear benefit |
| Message encryption | Payload encryption is application-layer concern | | Message encryption | Payload encryption is application-layer concern |
| Persistent message queues | NATS request-reply pattern sufficient | | Persistent message queues | Transport request-reply pattern sufficient |
| Advanced routing rules | Simple NATS subject matching sufficient | | Advanced routing rules | Simple topic matching sufficient |
### 2.3 Dependencies ### 2.3 Dependencies
| Platform | Package | Version | | Platform | Package | Version |
|----------|---------|---------| |----------|---------|---------|
| Julia | NATS.jl | Latest stable |
| Julia | JSON.jl | Latest stable | | Julia | JSON.jl | Latest stable |
| Julia | Arrow.jl | Latest stable | | Julia | Arrow.jl | Latest stable |
| Julia | HTTP.jl | Latest stable | | Julia | HTTP.jl | Latest stable |
| Julia | UUIDs.jl | Latest stable | | Julia | UUIDs.jl | Latest stable |
| Node.js | nats | Latest stable |
| Node.js | node-fetch | Latest stable | | Node.js | node-fetch | Latest stable |
| Python | nats-py | Latest stable |
| Python | aiohttp | Latest stable | | Python | aiohttp | Latest stable |
| Python | pyarrow | Latest stable | | Python | pyarrow | Latest stable |
| Browser | nats.ws | Latest stable | | Browser | - | Transport-agnostic (caller provides) |
| Dart | nats | Latest stable |
| Dart | http | Latest stable | | Dart | http | Latest stable |
| Dart | uuid | Latest stable | | Dart | uuid | Latest stable |
| Rust | nats | Latest stable |
| Rust | serde | Latest stable | | Rust | serde | Latest stable |
| Rust | serde_json | Latest stable | | Rust | serde_json | Latest stable |
| Rust | tokio | Latest stable | | Rust | tokio | Latest stable |
@@ -100,7 +95,7 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| Platform | Minimum Version | Notes | | Platform | Minimum Version | Notes |
|----------|-----------------|-------| |----------|-----------------|-------|
| Julia | 1.7+ | Arrow.jl required for arrowtable support | | Julia | 1.7+ | Arrow.jl required for arrowtable support |
| Node.js | 16+ | nats.js required, Arrow IPC supported | | Node.js | 16+ | Transport client required, Arrow IPC supported |
| Python | 3.8+ | pyarrow required for arrowtable support | | Python | 3.8+ | pyarrow required for arrowtable support |
| Browser | Latest | No Arrow IPC (uses jsontable only) | | Browser | Latest | No Arrow IPC (uses jsontable only) |
| Dart | 2.17+ | Supports Desktop (Dart SDK), Flutter (Dart SDK), and Web (Dart SDK) | | Dart | 2.17+ | Supports Desktop (Dart SDK), Flutter (Dart SDK), and Web (Dart SDK) |
@@ -115,8 +110,8 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
|----|-------------|-------------| |----|-------------|-------------|
| **FR-001** | Cross-platform text messaging | System shall allow users to send text messages between Julia, JavaScript, Python, and MicroPython applications | | **FR-001** | Cross-platform text messaging | System shall allow users to send text messages between Julia, JavaScript, Python, and MicroPython applications |
| **FR-002** | Cross-platform tabular data | System shall support DataFrame exchange between Julia and Python applications using Arrow IPC format | | **FR-002** | Cross-platform tabular data | System shall support DataFrame exchange between Julia and Python applications using Arrow IPC format |
| **FR-003** | Large file handling | System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via NATS | | **FR-003** | Large file handling | System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via transport |
| **FR-004** | Direct transport for small payloads | System shall send payloads <0.5MB directly via NATS without file server upload | | **FR-004** | Direct transport for small payloads | System shall send payloads <0.5MB directly via transport without file server upload |
| **FR-005** | MicroPython support | System shall support payloads <100KB on MicroPython devices using direct transport | | **FR-005** | MicroPython support | System shall support payloads <100KB on MicroPython devices using direct transport |
| **FR-006** | Multi-payload messages | System shall accept and process lists of (dataname, data, type) tuples | | **FR-006** | Multi-payload messages | System shall accept and process lists of (dataname, data, type) tuples |
| **FR-007** | Payload type preservation | System shall preserve payload types when returning multi-payload messages | | **FR-007** | Payload type preservation | System shall preserve payload types when returning multi-payload messages |
@@ -125,8 +120,8 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| **FR-010** | Exponential backoff retry | System shall implement exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) for file server download failures | | **FR-010** | Exponential backoff retry | System shall implement exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) for file server download failures |
| **FR-011** | Correlation ID propagation | System shall propagate correlation IDs through all message processing steps | | **FR-011** | Correlation ID propagation | System shall propagate correlation IDs through all message processing steps |
| **FR-012** | Message serialization | System shall serialize data types using Base64, JSON, or Arrow IPC encoding | | **FR-012** | Message serialization | System shall serialize data types using Base64, JSON, or Arrow IPC encoding |
| **FR-013** | NATS publishing | System shall return JSON string representation for caller to publish to NATS subjects (caller is responsible for actual NATS publish) | | **FR-013** | Transport publishing | System shall return JSON string representation for caller to publish via transport layer (caller is responsible for actual transport publish) |
| **FR-014** | NATS subscription | System shall receive and process NATS messages by accepting JSON string from NATS payload | | **FR-014** | Transport subscription | System shall receive and process messages by accepting JSON string from transport payload |
--- ---
@@ -138,10 +133,10 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
|----|-------------|---------------|-------------| |----|-------------|---------------|-------------|
| **NFR-101** | Message serialization overhead | <50ms for 10KB payload | Benchmark tests | | **NFR-101** | Message serialization overhead | <50ms for 10KB payload | Benchmark tests |
| **NFR-102** | Message deserialization overhead | <50ms for 10KB payload | Benchmark tests | | **NFR-102** | Message deserialization overhead | <50ms for 10KB payload | Benchmark tests |
| **NFR-103** | NATS connection establishment | <100ms | Connection pool benchmarks | | **NFR-103** | Transport connection establishment | <100ms | Connection pool benchmarks |
| **NFR-104** | File upload latency | <1s for 0.5MB file | Integration tests | | **NFR-104** | File upload latency | <1s for 0.5MB file | Integration tests |
| **NFR-105** | File download latency | <1s for 0.5MB file | Integration tests | | **NFR-105** | File download latency | <1s for 0.5MB file | Integration tests |
| **NFR-106** | Concurrent connections | Support 100+ simultaneous NATS connections | Scale testing | | **NFR-106** | Concurrent connections | Support 100+ simultaneous transport connections | Scale testing |
| **NFR-107** | Message throughput | Handle 1000+ messages/second per instance | Load testing | | **NFR-107** | Message throughput | Handle 1000+ messages/second per instance | Load testing |
| **NFR-108** | File server scalability | Support horizontal scaling of file server backend | Architecture review | | **NFR-108** | File server scalability | Support horizontal scaling of file server backend | Architecture review |
@@ -149,16 +144,16 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| ID | Requirement | Specification | | ID | Requirement | Specification |
|----|-------------|---------------| |----|-------------|---------------|
| **NFR-201** | Message delivery | At-least-once delivery semantics via NATS | | **NFR-201** | Message delivery | At-least-once delivery semantics via transport |
| **NFR-202** | File server availability | Graceful degradation when file server is unavailable | | **NFR-202** | File server availability | Graceful degradation when file server is unavailable |
| **NFR-203** | Connection recovery | Auto-reconnect on NATS connection failure | | **NFR-203** | Connection recovery | Auto-reconnect on transport connection failure |
### 4.3 Privacy & Security ### 4.3 Privacy & Security
| ID | Requirement | Specification | | ID | Requirement | Specification |
|----|-------------|---------------| |----|-------------|---------------|
| **NFR-301** | Payload integrity | SHA-256 checksum support via metadata | | **NFR-301** | Payload integrity | SHA-256 checksum support via metadata |
| **NFR-302** | Transport security | TLS support for NATS connections | | **NFR-302** | Transport security | TLS support for transport connections |
| **NFR-303** | File server security | Authentication token for file uploads | | **NFR-303** | File server security | Authentication token for file uploads |
### 4.4 Observability & Telemetry ### 4.4 Observability & Telemetry
@@ -234,11 +229,11 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| Platform | Maximum | Notes | | Platform | Maximum | Notes |
|----------|---------|-------| |----------|---------|-------|
| Desktop | Unlimited | Limited by NATS server configuration | | Desktop | Unlimited | Limited by transport server configuration |
| Dart Desktop | Unlimited | Limited by NATS server configuration | | Dart Desktop | Unlimited | Limited by transport server configuration |
| Dart Flutter | Unlimited | Limited by NATS server configuration | | Dart Flutter | Unlimited | Limited by transport server configuration |
| Dart Web | Unlimited | Limited by NATS server configuration | | Dart Web | Unlimited | Limited by transport server configuration |
| Rust | Unlimited | Limited by NATS server configuration | | Rust | Unlimited | Limited by transport server configuration |
| MicroPython | 50KB | Hard limit due to 256KB-1MB memory | | MicroPython | 50KB | Hard limit due to 256KB-1MB memory |
--- ---
@@ -252,7 +247,7 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| `correlation_id` | String (UUID) | Track message flow across systems | | `correlation_id` | String (UUID) | Track message flow across systems |
| `msg_id` | String (UUID) | Unique message identifier | | `msg_id` | String (UUID) | Unique message identifier |
| `timestamp` | String (ISO 8601) | Message publication timestamp | | `timestamp` | String (ISO 8601) | Message publication timestamp |
| `send_to` | String | NATS subject to publish to | | `send_to` | String | Topic/subject to publish to |
| `msg_purpose` | String | ACK, NACK, updateStatus, shutdown, chat | | `msg_purpose` | String | ACK, NACK, updateStatus, shutdown, chat |
| `sender_name` | String | Sender application name | | `sender_name` | String | Sender application name |
| `sender_id` | String (UUID) | Sender unique identifier | | `sender_id` | String (UUID) | Sender unique identifier |
@@ -260,7 +255,7 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| `receiver_id` | String (UUID) | Receiver unique identifier (empty = broadcast) | | `receiver_id` | String (UUID) | Receiver unique identifier (empty = broadcast) |
| `reply_to` | String | Topic for reply messages | | `reply_to` | String | Topic for reply messages |
| `reply_to_msg_id` | String | Message ID being replied to | | `reply_to_msg_id` | String | Message ID being replied to |
| `broker_url` | String | NATS server URL | | `broker_url` | String | Broker URL for the transport layer |
| `metadata` | Dict | Message-level metadata | | `metadata` | Dict | Message-level metadata |
| `payloads` | Array | List of payload objects | | `payloads` | Array | List of payload objects |
@@ -289,14 +284,14 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| `Failed to upload` | File server error | Throw error | | `Failed to upload` | File server error | Throw error |
| `Failed to fetch` | File server unavailable | Retry with exponential backoff | | `Failed to fetch` | File server unavailable | Retry with exponential backoff |
| `Unknown transport` | Invalid transport type | Throw error | | `Unknown transport` | Invalid transport type | Throw error |
| `NATS connection failed` | NATS unavailable | Throw error | | `Transport connection failed` | Transport/broker unavailable | Throw error |
### 9.2 Exception Handling ### 9.2 Exception Handling
| Scenario | Handler | | Scenario | Handler |
|----------|---------| |----------|---------|
| File server unavailable | Retry up to 5 times with exponential backoff | | File server unavailable | Retry up to 5 times with exponential backoff |
| NATS publish failure | Connection auto-reconnect | | Transport publish failure | Handled by caller |
| Deserialization error | Log correlation ID and throw error | | Deserialization error | Log correlation ID and throw error |
| Memory overflow (MicroPython) | Reject payloads >50KB | | Memory overflow (MicroPython) | Reject payloads >50KB |
@@ -350,7 +345,7 @@ function smartsend(
)::Tuple{msg_envelope_v1, String} where {T1<:Any} )::Tuple{msg_envelope_v1, String} where {T1<:Any}
``` ```
**Note**: NATS publishing is the caller's responsibility. `smartsend` returns `(env::msg_envelope_v1, env_json_str::String)`. **Note**: Publishing via the transport layer is the caller's responsibility. `smartsend` returns `(env::msg_envelope_v1, env_json_str::String)`.
### 11.2 smartreceive Signature ### 11.2 smartreceive Signature
@@ -364,7 +359,9 @@ function smartreceive(
)::JSON.Object{String, Any} )::JSON.Object{String, Any}
``` ```
**Note**: Pass `String(nats_msg.payload)` from NATS subscription to `smartreceive`. **Note**: Pass the payload string from the transport subscription to `smartreceive`. The input is the JSON string payload from the transport message, not the transport message object directly.
**Note**: Pass the payload from the transport subscription to `smartreceive`.
--- ---
@@ -374,7 +371,7 @@ function smartreceive(
| Component | Minimum | Notes | | Component | Minimum | Notes |
|-----------|---------|-------| |-----------|---------|-------|
| NATS Server | 1 instance | Single node for development | | Message Broker | 1 instance | Single node for development |
| File Server | 1 instance | HTTP server for large payloads | | File Server | 1 instance | HTTP server for large payloads |
| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) | | Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices | | Client Memory | 256KB | MicroPython devices |
@@ -383,7 +380,7 @@ function smartreceive(
| Variable | Default | Description | | Variable | Default | Description |
|----------|---------|-------------| |----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL | | `BROKER_URL` | `ws://localhost:4222` | Message broker URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL | | `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) | | `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) |
@@ -409,6 +406,10 @@ function smartreceive(
| Date | Version | Changes | | Date | Version | Changes |
|------|---------|---------| |------|---------|---------|
| 2026-05-15 | 1.3.0 | Made transport layer agnostic |
| - | - | Removed all NATS-specific dependencies and references |
| - | - | Updated all NATS references to generic "transport layer"/"message broker" |
| - | - | Removed NATS client packages from dependencies tables |
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | | 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) |
| - | - | Fixed smartsend signature: removed is_publish, NATS_connection; added sender_name | | - | - | Fixed smartsend signature: removed is_publish, NATS_connection; added sender_name |
| - | - | Fixed smartreceive signature: takes msg_json_str::String instead of msg::NATS.Msg | | - | - | Fixed smartreceive signature: takes msg_json_str::String instead of msg::NATS.Msg |

114
docs/specification.md
View File

@@ -10,7 +10,7 @@
## 1. Technical Contract Overview ## 1. Technical Contract Overview
This document defines the **technical contract** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus. This document defines the **technical contract** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer.
This specification serves as the single source of truth for: This specification serves as the single source of truth for:
- **Inputs**: What data structures are accepted by `smartsend()` - **Inputs**: What data structures are accepted by `smartsend()`
@@ -28,7 +28,7 @@ This specification serves as the single source of truth for:
| Section 5 (Enumerations) | FR-003, FR-004, FR-006, NFR-101 | Enumerations for transport and encoding | | Section 5 (Enumerations) | FR-003, FR-004, FR-006, NFR-101 | Enumerations for transport and encoding |
| Section 6 (Transport Protocols) | FR-003, FR-004, NFR-104, NFR-105 | Direct and link transport protocols | | Section 6 (Transport Protocols) | FR-003, FR-004, NFR-104, NFR-105 | Direct and link transport protocols |
| Section 7 (Size Thresholds) | FR-004, FR-005, NFR-104, NFR-105 | Size thresholds for transport selection | | Section 7 (Size Thresholds) | FR-004, FR-005, NFR-104, NFR-105 | Size thresholds for transport selection |
| Section 8 (NATS Subject Convention) | FR-013, FR-014 | NATS subject naming patterns | | Section 8 (Topic Convention) | FR-013, FR-014 | Topic/subject naming patterns |
| Section 9 (Error Handling) | FR-010, FR-011, NFR-201, NFR-202, NFR-203 | Error codes and exception handling | | Section 9 (Error Handling) | FR-010, FR-011, NFR-201, NFR-202, NFR-203 | Error codes and exception handling |
| Section 10 (Serialization Rules) | FR-001, FR-002, FR-003, FR-012, NFR-101, NFR-102 | Serialization and encoding rules | | Section 10 (Serialization Rules) | FR-001, FR-002, FR-003, FR-012, NFR-101, NFR-102 | Serialization and encoding rules |
| Section 11 (API Contract) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Function signatures for all platforms | | Section 11 (API Contract) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Function signatures for all platforms |
@@ -95,7 +95,7 @@ This specification serves as the single source of truth for:
| `correlation_id` | `string` | Yes | UUID v4 format | Track message flow across distributed systems | FR-011, NFR-401 | | `correlation_id` | `string` | Yes | UUID v4 format | Track message flow across distributed systems | FR-011, NFR-401 |
| `msg_id` | `string` | Yes | UUID v4 format | Unique identifier for this specific message | FR-012, NFR-401 | | `msg_id` | `string` | Yes | UUID v4 format | Unique identifier for this specific message | FR-012, NFR-401 |
| `timestamp` | `string` | Yes | ISO 8601 UTC | Message publication timestamp | FR-012, NFR-401 | | `timestamp` | `string` | Yes | ISO 8601 UTC | Message publication timestamp | FR-012, NFR-401 |
| `send_to` | `string` | Yes | Non-empty string | NATS subject/topic to publish the message to | FR-013 | | `send_to` | `string` | Yes | Non-empty string | Topic/subject to publish the message to | FR-013 |
| `msg_purpose` | `string` | Yes | Enum | Purpose of the message | FR-013 | | `msg_purpose` | `string` | Yes | Enum | Purpose of the message | FR-013 |
| `sender_name` | `string` | Yes | Non-empty string | Name of the sender application | FR-013 | | `sender_name` | `string` | Yes | Non-empty string | Name of the sender application | FR-013 |
| `sender_id` | `string` | Yes | UUID v4 format | Unique identifier for the sender | FR-013 | | `sender_id` | `string` | Yes | UUID v4 format | Unique identifier for the sender | FR-013 |
@@ -103,7 +103,7 @@ This specification serves as the single source of truth for:
| `receiver_id` | `string` | Yes | Any string | UUID of the receiver (empty = broadcast) | FR-013 | | `receiver_id` | `string` | Yes | Any string | UUID of the receiver (empty = broadcast) | FR-013 |
| `reply_to` | `string` | Yes | Any string | Topic where receiver should reply | FR-013 | | `reply_to` | `string` | Yes | Any string | Topic where receiver should reply | FR-013 |
| `reply_to_msg_id` | `string` | Yes | Any string | Message ID this message is replying to | FR-013 | | `reply_to_msg_id` | `string` | Yes | Any string | Message ID this message is replying to | FR-013 |
| `broker_url` | `string` | Yes | Valid URL | NATS broker URL | FR-013 | | `broker_url` | `string` | Yes | Valid URL | Broker URL for the transport layer | FR-013 |
| `metadata` | `object` | No | Any JSON object | Message-level metadata | NFR-401 | | `metadata` | `object` | No | Any JSON object | Message-level metadata | NFR-401 |
| `payloads` | `array` | Yes | Non-empty array | List of payload objects | FR-012, FR-013 | | `payloads` | `array` | Yes | Non-empty array | List of payload objects | FR-012, FR-013 |
@@ -240,7 +240,7 @@ await smartsend("/agent/v1/process", data)
| Value | Description | Data Format | Use Case | | Value | Description | Data Format | Use Case |
|-------|-------------|-------------|----------| |-------|-------------|-------------|----------|
| `direct` | Payload sent directly via NATS | Base64-encoded string | Payloads < size_threshold | | `direct` | Payload sent directly via the transport layer | Base64-encoded string | Payloads < size_threshold |
| `link` | Payload uploaded to file server | HTTP URL | Payloads ≥ size_threshold | | `link` | Payload uploaded to file server | HTTP URL | Payloads ≥ size_threshold |
### `encoding` Enum ### `encoding` Enum
@@ -312,7 +312,7 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa
--- ---
## NATS Subject Convention ## Topic Convention
### Subject Naming Pattern ### Subject Naming Pattern
@@ -357,7 +357,7 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa
| `INVALID_TRANSPORT` | 400 | Unsupported transport type | Use `direct` or `link` | FR-003, FR-004, FR-006 | | `INVALID_TRANSPORT` | 400 | Unsupported transport type | Use `direct` or `link` | FR-003, FR-004, FR-006 |
| `UPLOAD_FAILED` | 500 | File server upload failed | Retry or use direct transport | FR-008, FR-009 | | `UPLOAD_FAILED` | 500 | File server upload failed | Retry or use direct transport | FR-008, FR-009 |
| `DOWNLOAD_FAILED` | 503 | File server download failed | Retry with exponential backoff | FR-010, FR-011, NFR-201, NFR-202 | | `DOWNLOAD_FAILED` | 503 | File server download failed | Retry with exponential backoff | FR-010, FR-011, NFR-201, NFR-202 |
| `NATS_CONNECTION_FAILED` | 503 | NATS connection failed | Check NATS server availability | FR-013, FR-014, NFR-201, NFR-203 | | `TRANSPORT_CONNECTION_FAILED` | 503 | Transport connection failed | Check broker/server availability | FR-013, FR-014, NFR-201, NFR-203 |
| `DESERIALIZATION_ERROR` | 500 | Payload deserialization failed | Check payload_type matches data | FR-001, FR-002, FR-003, FR-012 | | `DESERIALIZATION_ERROR` | 500 | Payload deserialization failed | Check payload_type matches data | FR-001, FR-002, FR-003, FR-012 |
| `SIZE_EXCEEDED` | 413 | Payload exceeds maximum size | Split payload or use link transport | FR-003, FR-004, FR-005, NFR-104, NFR-105 | | `SIZE_EXCEEDED` | 413 | Payload exceeds maximum size | Split payload or use link transport | FR-003, FR-004, FR-005, NFR-104, NFR-105 |
@@ -366,7 +366,7 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa
| Scenario | Handler | Retry Policy | Requirement ID | | Scenario | Handler | Retry Policy | Requirement ID |
|----------|---------|--------------|----------------| |----------|---------|--------------|----------------|
| File server unavailable | Retry up to 5 times | Exponential backoff (100ms → 5000ms) | FR-010, NFR-202 | | File server unavailable | Retry up to 5 times | Exponential backoff (100ms → 5000ms) | FR-010, NFR-202 |
| NATS publish failure | Connection auto-reconnect | TCP-level reconnection | FR-013, FR-014, NFR-201, NFR-203 | | Transport publish failure | Handled by caller | Caller's retry logic | FR-013, FR-014, NFR-201, NFR-203 |
| Deserialization error | Log correlation ID and throw | No retry (data corruption) | FR-001, FR-002, FR-003, FR-012, NFR-401 | | Deserialization error | Log correlation ID and throw | No retry (data corruption) | FR-001, FR-002, FR-003, FR-012, NFR-401 |
| Memory overflow (MicroPython) | Reject payloads >50KB | No retry (client-side check) | FR-005, NFR-106 | | Memory overflow (MicroPython) | Reject payloads >50KB | No retry (client-side check) | FR-005, NFR-106 |
@@ -435,7 +435,7 @@ function smartsend(
)::Tuple{msg_envelope_v1, String} where {T1<:Any} )::Tuple{msg_envelope_v1, String} where {T1<:Any}
``` ```
**Note**: NATS publishing is the caller's responsibility. Returns `(env::msg_envelope_v1, env_json_str::String)`. **Note**: Publishing via the transport layer is the caller's responsibility. Returns `(env::msg_envelope_v1, env_json_str::String)`.
#### Python #### Python
@@ -443,7 +443,7 @@ function smartsend(
async def smartsend( async def smartsend(
subject: str, subject: str,
data: List[Tuple[str, Any, str]], data: List[Tuple[str, Any, str]],
broker_url: str = "nats://localhost:4222", broker_url: str = DEFAULT_BROKER_URL,
fileserver_url: str = "http://localhost:8080", fileserver_url: str = "http://localhost:8080",
fileserver_upload_handler: Callable = plik_oneshot_upload, fileserver_upload_handler: Callable = plik_oneshot_upload,
size_threshold: int = 500_000, size_threshold: int = 500_000,
@@ -459,7 +459,7 @@ async def smartsend(
) -> Tuple[Dict, str]: ) -> Tuple[Dict, str]:
``` ```
**Note**: NATS publishing is the caller's responsibility. **Note**: Publishing via the transport layer is the caller's responsibility.
#### JavaScript (Node.js) #### JavaScript (Node.js)
@@ -485,7 +485,7 @@ async function smartsend(
): Promise<[Object, string]>; ): Promise<[Object, string]>;
``` ```
**Note**: NATS publishing is the caller's responsibility. **Note**: Publishing via the transport layer is the caller's responsibility.
#### JavaScript (Browser) #### JavaScript (Browser)
@@ -511,7 +511,7 @@ async function smartsend(
): Promise<[Object, string]>; ): Promise<[Object, string]>;
``` ```
**Note**: NATS publishing is the caller's responsibility. **Note**: Publishing via the transport layer is the caller's responsibility.
#### MicroPython #### MicroPython
@@ -524,7 +524,7 @@ def smartsend(
) -> Tuple[Dict, str]: ) -> Tuple[Dict, str]:
``` ```
**Note**: NATS publishing is the caller's responsibility. **Note**: Publishing via the transport layer is the caller's responsibility.
#### Dart (Desktop/Flutter) #### Dart (Desktop/Flutter)
@@ -532,10 +532,10 @@ def smartsend(
Future<[Map<String, dynamic>, String]> smartsend( Future<[Map<String, dynamic>, String]> smartsend(
String subject, String subject,
List<List<dynamic>> data, { List<List<dynamic>> data, {
String brokerUrl = 'nats://localhost:4222', String brokerUrl = DEFAULT_BROKER_URL,
String fileserverUrl = 'http://localhost:8080', String fileserverUrl = DEFAULT_FILESERVER_URL,
Function? fileserverUploadHandler, Function? fileserverUploadHandler,
int sizeThreshold = 500000, int sizeThreshold = DEFAULT_SIZE_THRESHOLD,
String? correlationId, String? correlationId,
String msgPurpose = 'chat', String msgPurpose = 'chat',
String senderName = 'msghandler', String senderName = 'msghandler',
@@ -547,9 +547,8 @@ Future<[Map<String, dynamic>, String]> smartsend(
String? senderId, String? senderId,
}) async { }) async {
// Returns [envelope, jsonString] // Returns [envelope, jsonString]
// NATS publishing is caller's responsibility // Publishing via transport layer is caller's responsibility
} }
```
#### Dart Web #### Dart Web
@@ -557,7 +556,7 @@ Future<[Map<String, dynamic>, String]> smartsend(
Future<[Map<String, dynamic>, String]> smartsend( Future<[Map<String, dynamic>, String]> smartsend(
String subject, String subject,
List<List<dynamic>> data, { List<List<dynamic>> data, {
String brokerUrl = 'nats://localhost:4222', String brokerUrl = DEFAULT_BROKER_URL,
String fileserverUrl = 'http://localhost:8080', String fileserverUrl = 'http://localhost:8080',
Function? fileserverUploadHandler, Function? fileserverUploadHandler,
int sizeThreshold = 500000, int sizeThreshold = 500000,
@@ -572,7 +571,7 @@ Future<[Map<String, dynamic>, String]> smartsend(
String? senderId, String? senderId,
}) async { }) async {
// Returns [envelope, jsonString] // Returns [envelope, jsonString]
// NATS publishing is caller's responsibility // Publishing via transport layer is caller's responsibility
} }
``` ```
@@ -635,7 +634,7 @@ pub struct MsgEnvelopeV1 {
} }
``` ```
**Note**: NATS publishing is the caller's responsibility. Returns `Result<(MsgEnvelopeV1, String), msghandlerError>`. Uses `serde` for JSON serialization. **Note**: Publishing via the transport layer is the caller's responsibility. Returns `Result<(MsgEnvelopeV1, String), msghandlerError>`. Uses `serde` for JSON serialization.
### `smartreceive` Function Signature ### `smartreceive` Function Signature
@@ -643,7 +642,7 @@ pub struct MsgEnvelopeV1 {
```julia ```julia
function smartreceive( function smartreceive(
msg_json_str::String; # Pass String(nats_msg.payload) from NATS subscription msg_json_str::String; # Pass payload from transport subscription
fileserver_download_handler::Function = _fetch_with_backoff, fileserver_download_handler::Function = _fetch_with_backoff,
max_retries::Int = 5, max_retries::Int = 5,
base_delay::Int = 100, base_delay::Int = 100,
@@ -651,13 +650,13 @@ function smartreceive(
)::JSON.Object{String, Any} )::JSON.Object{String, Any}
``` ```
**Note**: Input is JSON string from NATS message payload, not NATS.Msg directly. **Note**: Input is the JSON string payload from the transport subscription, not the transport message object directly.
#### Python #### Python
```python ```python
async def smartreceive( async def smartreceive(
msg_json_str: str, # JSON string from NATS message payload msg_json_str: str, # JSON string from transport message payload
fileserver_download_handler: Callable = fetch_with_backoff, fileserver_download_handler: Callable = fetch_with_backoff,
max_retries: int = 5, max_retries: int = 5,
base_delay: int = 100, base_delay: int = 100,
@@ -665,13 +664,13 @@ async def smartreceive(
) -> Dict[str, Any]: ) -> Dict[str, Any]:
``` ```
**Note**: Input is JSON string from NATS message payload. **Note**: Input is the JSON string payload from the transport message.
#### JavaScript (Node.js) #### JavaScript (Node.js)
```typescript ```typescript
async function smartreceive( async function smartreceive(
msg_json_str: string, // JSON string from NATS message payload msg_json_str: string, // JSON string from transport message payload
options?: { options?: {
fileserver_download_handler?: Function; fileserver_download_handler?: Function;
max_retries?: number; max_retries?: number;
@@ -685,7 +684,7 @@ async function smartreceive(
```typescript ```typescript
async function smartreceive( async function smartreceive(
msg_json_str: string, // JSON string from NATS message payload msg_json_str: string, // JSON string from transport message payload
options?: { options?: {
fileserver_download_handler?: Function; fileserver_download_handler?: Function;
max_retries?: number; max_retries?: number;
@@ -695,7 +694,7 @@ async function smartreceive(
): Promise<Object>; ): Promise<Object>;
``` ```
**Note**: Input is JSON string from NATS message payload. **Note**: Input is the JSON string payload from the transport message.
#### MicroPython #### MicroPython
@@ -703,13 +702,13 @@ async function smartreceive(
def smartreceive(msg_json_str: str, **kwargs) -> Dict[str, Any]: def smartreceive(msg_json_str: str, **kwargs) -> Dict[str, Any]:
``` ```
**Note**: Input is JSON string from NATS message payload. **Note**: Input is the JSON string payload from the transport message.
#### Dart (Desktop/Flutter) #### Dart (Desktop/Flutter)
```dart ```dart
Future<Map<String, dynamic>> smartreceive( Future<Map<String, dynamic>> smartreceive(
Map<String, dynamic> msg_json_str, // JSON object from NATS message payload Map<String, dynamic> msg_json_str, // JSON object from transport message payload
{ {
Function? fileserverDownloadHandler, Function? fileserverDownloadHandler,
int maxRetries = 5, int maxRetries = 5,
@@ -724,7 +723,7 @@ Future<Map<String, dynamic>> smartreceive(
```dart ```dart
Future<Map<String, dynamic>> smartreceive( Future<Map<String, dynamic>> smartreceive(
Map<String, dynamic> msg_json_str, // JSON object from NATS message payload Map<String, dynamic> msg_json_str, // JSON object from transport message payload
{ {
Function? fileserverDownloadHandler, Function? fileserverDownloadHandler,
int maxRetries = 5, int maxRetries = 5,
@@ -739,7 +738,7 @@ Future<Map<String, dynamic>> smartreceive(
```rust ```rust
pub async fn smartreceive( pub async fn smartreceive(
msg_json_str: &str, // JSON string from NATS message payload msg_json_str: &str, // JSON string from transport message payload
options: &SmartreceiveOptions, options: &SmartreceiveOptions,
) -> Result<MsgEnvelopeV1, msghandlerError> ) -> Result<MsgEnvelopeV1, msghandlerError>
@@ -752,7 +751,7 @@ pub struct SmartreceiveOptions {
} }
``` ```
**Note**: Input is JSON string from NATS message payload. Returns `Result<MsgEnvelopeV1, msghandlerError>`. **Note**: Input is the JSON string payload from the transport message. Returns `Result<MsgEnvelopeV1, msghandlerError>`.
--- ---
@@ -897,7 +896,7 @@ function fileserver_download_handler(
|------|----------|----------|-------| |------|----------|----------|-------|
| [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | Ground truth implementation | | [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | Ground truth implementation |
| [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | Server-side JavaScript | | [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | Server-side JavaScript |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | Client-side rendering | | [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | Client-side rendering |
| [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | Desktop Python | | [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | Desktop Python |
| [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | Desktop/Flutter/Web | | [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | Desktop/Flutter/Web |
| [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe | Uses tokio + serde + arrow2 | | [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe | Uses tokio + serde + arrow2 |
@@ -910,7 +909,7 @@ The browser implementation ([`src/msghandler_csr.js`](../src/msghandler_csr.js))
| Constraint | Reason | Workaround | | Constraint | Reason | Workaround |
|------------|--------|------------| |------------|--------|------------|
| No Apache Arrow IPC | Browser-incompatible dependency | Use `jsontable` for tabular data | | No Apache Arrow IPC | Browser-incompatible dependency | Use `jsontable` for tabular data |
| WebSocket NATS only | Browser cannot use TCP directly | Use `ws://` or `wss://` broker URLs | | WebSocket only | Browser cannot use TCP directly | Use `ws://` or `wss://` broker URLs |
| Fetch API for HTTP | Browser fetch() API only | Compatible with Plik and other HTTP servers | | Fetch API for HTTP | Browser fetch() API only | Compatible with Plik and other HTTP servers |
### Payload Type Availability by Platform ### Payload Type Availability by Platform
@@ -941,7 +940,7 @@ flowchart TD
D --> F[Build envelope with metadata] D --> F[Build envelope with metadata]
E --> F E --> F
F --> G[Convert envelope to JSON string] F --> G[Convert envelope to JSON string]
G --> H[Publish to NATS subject] G --> H[Publish to topic via transport]
H --> I[Return envelope and JSON string to caller] H --> I[Return envelope and JSON string to caller]
style A fill:#f9f9f9,stroke:#333 style A fill:#f9f9f9,stroke:#333
@@ -954,7 +953,7 @@ flowchart TD
```mermaid ```mermaid
flowchart TD flowchart TD
A[NATS message arrives] --> B[Parse JSON envelope] A[Transport message arrives] --> B[Parse JSON envelope]
B --> C[For each payload: Check transport type] B --> C[For each payload: Check transport type]
C -->|transport == direct| D[Direct Transport: Extract Base64] C -->|transport == direct| D[Direct Transport: Extract Base64]
C -->|transport == link| E[Link Transport: Fetch from URL] C -->|transport == link| E[Link Transport: Fetch from URL]
@@ -1046,23 +1045,17 @@ flowchart TD
| Platform | Package | Version | Purpose | | Platform | Package | Version | Purpose |
|----------|---------|---------|---------| |----------|---------|---------|---------|
| Julia | NATS.jl | Latest | NATS client |
| Julia | JSON.jl | Latest | JSON serialization | | Julia | JSON.jl | Latest | JSON serialization |
| Julia | Arrow.jl | Latest | Arrow IPC support | | Julia | Arrow.jl | Latest | Arrow IPC support |
| Julia | HTTP.jl | Latest | HTTP file server | | Julia | HTTP.jl | Latest | HTTP file server |
| Julia | UUIDs.jl | Latest | UUID generation | | Julia | UUIDs.jl | Latest | UUID generation |
| Node.js | nats | Latest | NATS client (TCP) |
| Node.js | node-fetch | Latest | HTTP file server | | Node.js | node-fetch | Latest | HTTP file server |
| Browser | nats.ws | Latest | NATS client (WebSocket) | | Browser | - | - | Transport-agnostic (caller provides) |
| Browser | nats | Latest | NATS client (for bundling) |
| Python | nats-py | Latest | NATS client |
| Python | aiohttp | Latest | HTTP file server | | Python | aiohttp | Latest | HTTP file server |
| Python | pyarrow | Latest | Arrow IPC support | | Python | pyarrow | Latest | Arrow IPC support |
| Dart | nats | Latest | NATS client |
| Dart | http | Latest | HTTP file server | | Dart | http | Latest | HTTP file server |
| Dart | uuid | Latest | UUID generation | | Dart | uuid | Latest | UUID generation |
| Dart | dart-arrow | Latest | Arrow IPC support (Desktop/Flutter) | | Dart | dart-arrow | Latest | Arrow IPC support (Desktop/Flutter) |
| Rust | nats | Latest | NATS client |
| Rust | serde | Latest | JSON serialization | | Rust | serde | Latest | JSON serialization |
| Rust | serde_json | Latest | JSON handling | | Rust | serde_json | Latest | JSON handling |
| Rust | tokio | Latest | Async runtime | | Rust | tokio | Latest | Async runtime |
@@ -1084,6 +1077,10 @@ flowchart TD
| Date | Version | Changes | | Date | Version | Changes |
|------|---------|---------| |------|---------|---------|
| 2026-05-15 | 1.3.0 | Made transport layer agnostic |
| - | - | Removed all NATS-specific dependencies (NATS.jl, nats, nats-py, nats.ws) |
| - | - | Updated docs to reference generic message broker/transport |
| - | - | broker_url is now metadata only, not used for active connections |
| 2026-03-15 | 1.1.0 | Browser connection management | | 2026-03-15 | 1.1.0 | Browser connection management |
| - | - | Added NATSClient class with keepAlive support | | - | - | Added NATSClient class with keepAlive support |
| - | - | Added NATSConnectionPool for connection reuse | | - | - | Added NATSConnectionPool for connection reuse |
@@ -1119,7 +1116,7 @@ flowchart TD
|------|----------|----------|--------------------------| |------|----------|----------|--------------------------|
| [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe | FR-001 through FR-014, NFR-101 through NFR-405 |
@@ -1129,23 +1126,17 @@ flowchart TD
| Platform | Package | Version | Purpose | Requirements Traceability | | Platform | Package | Version | Purpose | Requirements Traceability |
|----------|---------|---------|---------|--------------------------| |----------|---------|---------|---------|--------------------------|
| Julia | NATS.jl | Latest | NATS client | FR-013, FR-014, NFR-201 |
| Julia | JSON.jl | Latest | JSON serialization | FR-012, NFR-101, NFR-102 | | Julia | JSON.jl | Latest | JSON serialization | FR-012, NFR-101, NFR-102 |
| Julia | Arrow.jl | Latest | Arrow IPC support | FR-002, FR-012 | | Julia | Arrow.jl | Latest | Arrow IPC support | FR-002, FR-012 |
| Julia | HTTP.jl | Latest | HTTP file server | FR-008, FR-009 | | Julia | HTTP.jl | Latest | HTTP file server | FR-008, FR-009 |
| Julia | UUIDs.jl | Latest | UUID generation | FR-011, NFR-401 | | Julia | UUIDs.jl | Latest | UUID generation | FR-011, NFR-401 |
| Node.js | nats | Latest | NATS client (TCP) | FR-013, FR-014 |
| Node.js | node-fetch | Latest | HTTP file server | FR-008, FR-009 | | Node.js | node-fetch | Latest | HTTP file server | FR-008, FR-009 |
| Browser | nats.ws | Latest | NATS client (WebSocket) | FR-013, FR-014 | | Browser | - | - | Transport-agnostic (caller provides) | FR-013, FR-014 |
| Browser | nats | Latest | NATS client (for bundling) | FR-013, FR-014 |
| Python | nats-py | Latest | NATS client | FR-013, FR-014 |
| Python | aiohttp | Latest | HTTP file server | FR-008, FR-009 | | Python | aiohttp | Latest | HTTP file server | FR-008, FR-009 |
| Python | pyarrow | Latest | Arrow IPC support | FR-002, FR-012 | | Python | pyarrow | Latest | Arrow IPC support | FR-002, FR-012 |
| Dart | nats | Latest | NATS client | FR-013, FR-014 |
| Dart | http | Latest | HTTP file server | FR-008, FR-009 | | Dart | http | Latest | HTTP file server | FR-008, FR-009 |
| Dart | uuid | Latest | UUID generation | FR-011, NFR-401 | | Dart | uuid | Latest | UUID generation | FR-011, NFR-401 |
| Dart | dart-arrow | Latest | Arrow IPC support | FR-002, FR-012 | | Dart | dart-arrow | Latest | Arrow IPC support | FR-002, FR-012 |
| Rust | nats | Latest | NATS client | FR-013, FR-014 |
| Rust | serde | Latest | JSON serialization | FR-012, NFR-101, NFR-102 | | Rust | serde | Latest | JSON serialization | FR-012, NFR-101, NFR-102 |
| Rust | serde_json | Latest | JSON handling | FR-012, NFR-101, NFR-102 | | Rust | serde_json | Latest | JSON handling | FR-012, NFR-101, NFR-102 |
| Rust | tokio | Latest | Async runtime | FR-013, FR-014 | | Rust | tokio | Latest | Async runtime | FR-013, FR-014 |
@@ -1160,6 +1151,10 @@ flowchart TD
| Date | Version | Changes | Requirement ID(s) | | Date | Version | Changes | Requirement ID(s) |
|------|---------|---------|-------------------| |------|---------|---------|-------------------|
| 2026-05-15 | 1.3.0 | Made transport layer agnostic | All |
| - | - | Removed NATS-specific dependencies and references from all docs | All |
| - | - | Updated all NATS references to generic "transport layer"/"message broker" | All |
| - | - | Removed NATS client packages from dependencies tables | All |
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All | | 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All |
| - | - | Updated smartsend signatures: removed is_publish, nats_connection; added sender_name | FR-001 through FR-014 | | - | - | Updated smartsend signatures: removed is_publish, nats_connection; added sender_name | FR-001 through FR-014 |
| - | - | Updated smartreceive signatures: takes msg_json_str::String instead of msg | FR-001 through FR-014 | | - | - | Updated smartreceive signatures: takes msg_json_str::String instead of msg | FR-001 through FR-014 |
@@ -1200,7 +1195,7 @@ flowchart TD
"send_to": { "send_to": {
"type": "string", "type": "string",
"minLength": 1, "minLength": 1,
"description": "NATS subject to publish to" "description": "Topic/subject to publish to"
}, },
"msg_purpose": { "msg_purpose": {
"type": "string", "type": "string",
@@ -1236,8 +1231,7 @@ flowchart TD
}, },
"broker_url": { "broker_url": {
"type": "string", "type": "string",
"pattern": "^nats://[^\\s]+$", "description": "Broker URL for the transport layer"
"description": "NATS broker URL"
}, },
"metadata": { "metadata": {
"type": "object", "type": "object",
@@ -1302,14 +1296,14 @@ flowchart TD
} }
``` ```
### B. AsyncAPI Specification (NATS) ### B. AsyncAPI Specification
```yaml ```yaml
asyncapi: '2.6.0' asyncapi: '2.6.0'
info: info:
title: msghandler API title: msghandler API
version: '1.0.0' version: '1.0.0'
description: Cross-platform bi-directional data bridge using NATS description: Cross-platform bi-directional data bridge using a message broker
contact: contact:
name: msghandler Team name: msghandler Team
url: https://github.com/your-org/msghandler url: https://github.com/your-org/msghandler
@@ -1331,12 +1325,12 @@ channels:
schema: schema:
type: string type: string
publish: publish:
summary: Publish message to NATS summary: Publish message to transport
operationId: publishMessage operationId: publishMessage
message: message:
$ref: '#/components/message' $ref: '#/components/message'
subscribe: subscribe:
summary: Subscribe to NATS messages summary: Subscribe to messages from transport
operationId: subscribeMessage operationId: subscribeMessage
message: message:
$ref: '#/components/message' $ref: '#/components/message'

95
docs/walkthrough.md
View File

@@ -9,7 +9,7 @@
## 1. Executive Summary ## 1. Executive Summary
This document provides the **end-to-end trace** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus. This document provides the **end-to-end trace** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer.
This walkthrough serves as the primary onboarding guide for new developers and explains: This walkthrough serves as the primary onboarding guide for new developers and explains:
- **User scenarios** - Real-world use cases from developer perspective - **User scenarios** - Real-world use cases from developer perspective
@@ -51,7 +51,7 @@ flowchart TB
S3["Size Check"] S3["Size Check"]
S4["Transport Selection"] S4["Transport Selection"]
S5["Build Envelope"] S5["Build Envelope"]
S6["Publish to NATS"] S6["Publish to transport"]
S1 --> S2 S1 --> S2
S2 --> S3 S2 --> S3
@@ -62,7 +62,7 @@ flowchart TB
subgraph Receiver["Receiver (smartreceive)"] subgraph Receiver["Receiver (smartreceive)"]
direction LR direction LR
R1["Subscribe to NATS"] R1["Subscribe via transport"]
R2["Parse Envelope"] R2["Parse Envelope"]
R3["Check Transport"] R3["Check Transport"]
R4["Deserialize Data"] R4["Deserialize Data"]
@@ -99,7 +99,7 @@ flowchart TB
| Principle | Description | Rationale | | Principle | Description | Rationale |
|-----------|-------------|-----------| |-----------|-------------|-----------|
| **Claim-Check Pattern** | Large payloads uploaded to HTTP server, URL sent via NATS | NATS has message size limits; avoids NATS overflow | | **Claim-Check Pattern** | Large payloads uploaded to HTTP server, URL sent via transport | Transport has message size limits; avoids overflow |
| **Automatic Transport Selection** | Direct (< threshold) vs Link (≥ threshold) based on size | Optimizes memory vs network I/O trade-off | | **Automatic Transport Selection** | Direct (< threshold) vs Link (≥ threshold) based on size | Optimizes memory vs network I/O trade-off |
| **Cross-Platform API** | Consistent `smartsend()`/`smartreceive()` across all platforms | Simplifies developer experience | | **Cross-Platform API** | Consistent `smartsend()`/`smartreceive()` across all platforms | Simplifies developer experience |
| **Exponential Backoff** | Retry downloads with increasing delays | Handles transient failures gracefully | | **Exponential Backoff** | Retry downloads with increasing delays | Handles transient failures gracefully |
@@ -148,7 +148,7 @@ For each payload, msghandler determines transport:
**Rationale**: **Rationale**:
- Direct transport is faster for small payloads (no file server round-trip) - Direct transport is faster for small payloads (no file server round-trip)
- Link transport is used when payload ≥ 0.5MB (avoids NATS size limits) - Link transport is used when payload ≥ 0.5MB (avoids transport size limits)
#### Step 3: Serialization and Encoding #### Step 3: Serialization and Encoding
@@ -213,25 +213,26 @@ msghandler builds the message envelope:
- **reply_to**: Tells backend where to send response - **reply_to**: Tells backend where to send response
- **payloads array**: Contains all data with metadata for proper handling - **payloads array**: Contains all data with metadata for proper handling
#### Step 5: Publish to NATS (Caller's Responsibility) #### Step 5: Publish to Transport (Caller's Responsibility)
```javascript ```javascript
// NATS publishing is the caller's responsibility // Publishing via the transport layer is the caller's responsibility
const conn = await NATS.connect({ servers: "ws://localhost:4222" }); // Example with any transport (NATS, MQTT, WebSocket, etc.)
await conn.publish("/agent/wine/api/v1/prompt", msgJson); // const conn = await transportClient.connect({ servers: "ws://localhost:4222" });
// await conn.publish("/agent/wine/api/v1/prompt", msgJson);
``` ```
**Rationale**: **Rationale**:
- NATS provides low-latency message delivery - The transport layer provides message delivery (NATS, MQTT, WebSocket, etc.)
- JSON format ensures cross-platform compatibility - JSON format ensures cross-platform compatibility
- `smartsend()` returns `(env, msgJson)` - caller handles publishing - `smartsend()` returns `(env, msgJson)` - caller handles publishing via their chosen transport
#### Step 6: Julia Backend Receives Message #### Step 6: Julia Backend Receives Message
```julia ```julia
# Julia backend # Julia backend
nats_msg = NATS.subscription.next() # Get message from NATS transport_msg = transport_subscription.next() # Get message from transport
env = smartreceive(String(nats_msg.payload)) env = smartreceive(String(transport_msg.payload))
# env["payloads"] is now: # env["payloads"] is now:
# [ # [
@@ -302,7 +303,7 @@ const [env, msgJson] = await msghandler.smartsend(
**Rationale**: **Rationale**:
- Link transport used for large payloads - Link transport used for large payloads
- File server handles large file upload - File server handles large file upload
- NATS only sends URL (small message) - Transport only sends URL (small message)
#### Step 3: File Server Upload #### Step 3: File Server Upload
@@ -356,8 +357,8 @@ const response = await plikOneshotUpload(
```julia ```julia
# Julia backend # Julia backend
nats_msg = NATS.subscription.next() transport_msg = transport_subscription.next()
env = smartreceive(String(nats_msg.payload)) env = smartreceive(String(transport_msg.payload))
# msghandler automatically: # msghandler automatically:
# 1. Extracts URL from payload # 1. Extracts URL from payload
@@ -396,7 +397,7 @@ df = pd.DataFrame({
env, msg_json = await smartsend( env, msg_json = await smartsend(
"/agent/wine/api/v1/analyze", "/agent/wine/api/v1/analyze",
[("data", df, "arrowtable")], [("data", df, "arrowtable")],
broker_url="nats://localhost:4222", broker_url=DEFAULT_BROKER_URL,
receiver_name="agent-backend" receiver_name="agent-backend"
) )
``` ```
@@ -429,8 +430,8 @@ arrow_bytes = buf.getvalue()
```julia ```julia
# Julia backend # Julia backend
nats_msg = NATS.subscription.next() transport_msg = transport_subscription.next()
env = smartreceive(String(nats_msg.payload)) env = smartreceive(String(transport_msg.payload))
# env["payloads"][1] is now: # env["payloads"][1] is now:
# ("data", DataFrame with id, name, score columns, "arrowtable") # ("data", DataFrame with id, name, score columns, "arrowtable")
@@ -479,7 +480,7 @@ use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64};
#[tokio::main] #[tokio::main]
async fn main() { async fn main() {
let conn = nats::connect("nats://localhost:4222").unwrap(); let conn = transport_client::connect("ws://localhost:4222").unwrap();
// Subscribe and receive messages // Subscribe and receive messages
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze").unwrap(); let mut sub = conn.subscribe("/agent/wine/api/v1/analyze").unwrap();
@@ -520,7 +521,7 @@ async fn main() {
**Rationale**: **Rationale**:
- **serde serialization**: Automatic JSON deserialization to `MsgEnvelopeV1` - **serde serialization**: Automatic JSON deserialization to `MsgEnvelopeV1`
- **tokio runtime**: Efficient async I/O for NATS and HTTP operations - **tokio runtime**: Efficient async I/O for transport and HTTP operations
- **smartreceive deserialization**: Payload data is deserialized and stored as strings in `payload.data` - **smartreceive deserialization**: Payload data is deserialized and stored as strings in `payload.data`
- **Type dispatch**: `payload_type` field determines how to interpret the `data` string - **Type dispatch**: `payload_type` field determines how to interpret the `data` string
@@ -548,14 +549,14 @@ let (envelope, json_str) = smartsend(
), ),
], ],
&SmartsendOptions { &SmartsendOptions {
broker_url: "nats://localhost:4222".to_string(), broker_url: DEFAULT_BROKER_URL.to_string(),
reply_to: "/python/worker/v1/results".to_string(), reply_to: "/python/worker/v1/results".to_string(),
msg_purpose: "chat".to_string(), msg_purpose: "chat".to_string(),
..Default::default() ..Default::default()
}, },
).await?; ).await?;
// Caller publishes to NATS // Caller publishes via transport
conn.publish("/agent/wine/api/v1/results", &json_str)?; conn.publish("/agent/wine/api/v1/results", &json_str)?;
``` ```
@@ -569,7 +570,7 @@ conn.publish("/agent/wine/api/v1/results", &json_str)?;
```python ```python
# Python backend receives Rust response # Python backend receives Rust response
env = await smartreceive(str(nats_msg.payload)) env = await smartreceive(str(transport_msg.payload))
# env["payloads"][0] is now: # env["payloads"][0] is now:
# ("results", arrow_table_data, "arrowtable") # ("results", arrow_table_data, "arrowtable")
@@ -598,9 +599,9 @@ let (envelope, json_str) = smartsend(
), ),
], ],
&SmartsendOptions { &SmartsendOptions {
broker_url: "nats://localhost:4222".to_string(), broker_url: DEFAULT_BROKER_URL.to_string(),
fileserver_url: "http://localhost:8080".to_string(), fileserver_url: DEFAULT_FILESERVER_URL.to_string(),
size_threshold: 500_000, // 0.5MB triggers link transport size_threshold: DEFAULT_SIZE_THRESHOLD, // threshold triggers link transport
..Default::default() ..Default::default()
}, },
).await?; ).await?;
@@ -637,7 +638,7 @@ sensor_data = {
env, msg_json = smartsend( env, msg_json = smartsend(
"/sensor/device/v1/readings", "/sensor/device/v1/readings",
[("data", sensor_data, "dictionary")], [("data", sensor_data, "dictionary")],
broker_url="nats://localhost:4222", broker_url=DEFAULT_BROKER_URL,
size_threshold=100000 # 100KB for MicroPython size_threshold=100000 # 100KB for MicroPython
) )
``` ```
@@ -658,15 +659,15 @@ payload_b64 = base64.b64encode(json_bytes).decode('ascii')
**Rationale**: **Rationale**:
- JSON format for human-readable data - JSON format for human-readable data
- Base64 for NATS compatibility - Base64 for transport compatibility
- UTF-8 for text encoding - UTF-8 for text encoding
#### Step 3: Python Backend Receives #### Step 3: Python Backend Receives
```python ```python
# Python backend # Python backend
nats_msg = await nats_consumer.next() transport_msg = await transport_consumer.next()
env = await smartreceive(str(nats_msg.payload)) env = await smartreceive(str(transport_msg.payload))
# env["payloads"][0] is now: # env["payloads"][0] is now:
# ("data", {"temperature": 25.5, "humidity": 60.0, ...}, "dictionary") # ("data", {"temperature": 25.5, "humidity": 60.0, ...}, "dictionary")
@@ -708,14 +709,14 @@ const [env, msgJson] = await msghandler.smartsend(
**Rationale**: **Rationale**:
- Empty `receiver_name` = broadcast to all subscribers - Empty `receiver_name` = broadcast to all subscribers
- Chat messages often include text + images - Chat messages often include text + images
- NATS wildcard subscriptions route to correct recipients - Transport wildcard subscriptions route to correct recipients
#### Step 2: Python Backend Receives #### Step 2: Python Backend Receives
```python ```python
# Python (Backend) # Python (Backend)
nats_msg = await nats_consumer.next() transport_msg = await transport_consumer.next()
env = await smartreceive(str(nats_msg.payload)) env = await smartreceive(str(transport_msg.payload))
# env["payloads"] is now: # env["payloads"] is now:
# [ # [
@@ -733,8 +734,8 @@ env = await smartreceive(str(nats_msg.payload))
```julia ```julia
# Julia (Backend) # Julia (Backend)
nats_msg = NATS.subscription.next() transport_msg = transport_subscription.next()
env = smartreceive(String(nats_msg.payload)) env = smartreceive(String(transport_msg.payload))
# env["payloads"] is now: # env["payloads"] is now:
# [ # [
@@ -795,7 +796,7 @@ await msghandler.smartsend(
| File server unavailable | `UPLOAD_FAILED` | Fall back to direct transport or smaller payloads | | File server unavailable | `UPLOAD_FAILED` | Fall back to direct transport or smaller payloads |
| File server download fails | `DOWNLOAD_FAILED` | Retry with exponential backoff | | File server download fails | `DOWNLOAD_FAILED` | Retry with exponential backoff |
| Payload type mismatch | `DESERIALIZATION_ERROR` | Validate payload_type matches data | | Payload type mismatch | `DESERIALIZATION_ERROR` | Validate payload_type matches data |
| NATS connection lost | `NATS_CONNECTION_FAILED` | NATS client auto-reconnects | | Transport connection lost | `TRANSPORT_CONNECTION_FAILED` | Transport client auto-reconnects |
### Error Response Format ### Error Response Format
@@ -828,14 +829,14 @@ correlation_id = string(uuid4())
# Use throughout the flow # Use throughout the flow
log_trace(correlation_id, "Starting smartsend") log_trace(correlation_id, "Starting smartsend")
log_trace(correlation_id, "Serialized payload size: 100 bytes") log_trace(correlation_id, "Serialized payload size: 100 bytes")
log_trace(correlation_id, "Published to NATS") log_trace(correlation_id, "Published to transport")
``` ```
**Log Format**: **Log Format**:
``` ```
[2026-03-13T16:30:00.000Z] [Correlation: abc123...] Starting smartsend [2026-03-13T16:30:00.000Z] [Correlation: abc123...] Starting smartsend
[2026-03-13T16:30:00.001Z] [Correlation: abc123...] Serialized payload size: 100 bytes [2026-03-13T16:30:00.001Z] [Correlation: abc123...] Serialized payload size: 100 bytes
[2026-03-13T16:30:00.002Z] [Correlation: abc123...] Published to NATS [2026-03-13T16:30:00.002Z] [Correlation: abc123...] Published to transport
``` ```
--- ---
@@ -846,7 +847,7 @@ log_trace(correlation_id, "Published to NATS")
| Strategy | Description | When to Use | | Strategy | Description | When to Use |
|----------|-------------|-------------| |----------|-------------|-------------|
| Pre-create NATS connection | Reuse connection for multiple sends | High-throughput scenarios | | Pre-create transport connection | Reuse connection for multiple sends | High-throughput scenarios |
| Adjust size threshold | Increase threshold if file server slow | File server bottleneck | | Adjust size threshold | Increase threshold if file server slow | File server bottleneck |
| Use direct transport | Avoid file server for small payloads | Low latency requirements | | Use direct transport | Avoid file server for small payloads | Low latency requirements |
@@ -868,7 +869,7 @@ log_trace(correlation_id, "Published to NATS")
| Component | Minimum | Notes | | Component | Minimum | Notes |
|-----------|---------|-------| |-----------|---------|-------|
| NATS Server | 1 instance | Single node for development | | Message Broker | 1 instance | Single node for development |
| File Server | 1 instance | HTTP server for large payloads | | File Server | 1 instance | HTTP server for large payloads |
| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) | | Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices | | Client Memory | 256KB | MicroPython devices |
@@ -877,7 +878,7 @@ log_trace(correlation_id, "Published to NATS")
| Variable | Default | Description | | Variable | Default | Description |
|----------|---------|-------------| |----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL | | `BROKER_URL` | `ws://localhost:4222` | Message broker URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL | | `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) | | `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) |
@@ -911,7 +912,7 @@ log_trace(correlation_id, "Published to NATS")
|------|----------|----------|---------------------------| |------|----------|----------|---------------------------|
| [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) | | [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) |
| [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) | | [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | specification.md:2-19 (all sections) | | [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | specification.md:2-19 (all sections) |
| [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) | | [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) | | [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe, file upload helpers | specification.md:2-19 (all sections) | | [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe, file upload helpers | specification.md:2-19 (all sections) |
@@ -923,6 +924,10 @@ log_trace(correlation_id, "Published to NATS")
| Date | Version | Changes | Specification Reference | | Date | Version | Changes | Specification Reference |
|------|---------|---------|------------------------| |------|---------|---------|------------------------|
| 2026-05-15 | 1.5.0 | Made transport layer agnostic | All sections |
| - | - | Removed all NATS-specific references from walkthrough | All sections |
| - | - | Updated code examples to use transport-agnostic patterns | All sections |
| - | - | Updated diagrams to remove NATS-specific labels | All sections |
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections | | 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections |
| - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 | | - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
| - | - | Added `plik_upload_file` convenience function documentation | specification.md:13 | | - | - | Added `plik_upload_file` convenience function documentation | specification.md:13 |
@@ -932,8 +937,8 @@ log_trace(correlation_id, "Published to NATS")
| - | - | Added Rust user scenario (User Scenario 4) | specification.md:11 (Rust API) | | - | - | Added Rust user scenario (User Scenario 4) | specification.md:11 (Rust API) |
| - | - | Updated scenario numbering (MicroPython → Scenario 5, Cross-Platform → Scenario 6) | All sections | | - | - | Updated scenario numbering (MicroPython → Scenario 5, Cross-Platform → Scenario 6) | All sections |
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All sections | | 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All sections |
| - | - | Updated smartreceive calls to use String(nats_msg.payload) pattern | All sections | | - | - | Updated smartreceive calls to use transport payload pattern | All sections |
| - | - | Removed NATSClient.publish() calls (caller responsible for NATS publishing) | All sections | | - | - | Removed NATSClient.publish() calls (caller responsible for transport publishing) | All sections |
| - | - | Removed is_publish and nats_connection parameter references | All sections | | - | - | Removed is_publish and nats_connection parameter references | All sections |
| 2026-03-23 | 1.0.0 | Updated to ASG Framework walkthrough guidelines | All sections | | 2026-03-23 | 1.0.0 | Updated to ASG Framework walkthrough guidelines | All sections |
| 2026-03-13 | 1.0.0 | Initial walkthrough documentation | specification.md:2-19 (all sections) | | 2026-03-13 | 1.0.0 | Initial walkthrough documentation | specification.md:2-19 (all sections) |