diff --git a/docs/architecture.md b/docs/architecture.md index 470c2a6..458677f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -10,7 +10,7 @@ ## 1. Executive Summary -This document defines the **blueprint** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus. +This document defines the **blueprint** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer. This architecture document serves as the single source of truth for: - **System Structure**: How components fit together and interact @@ -47,26 +47,25 @@ This architecture document serves as the single source of truth for: ```mermaid flowchart TD - subgraph "External Systems" - NATS_Server[NATS Server] + subgraph "External Systems" + Message_Broker[Message Broker
NATS/MQTT/WebSocket/Custom] File_Server[HTTP File Server
Plik/AWS S3/Custom] end - subgraph "Client Applications" - Julia_App[Julia Application] - JS_App[JavaScript Application
Node.js/Browser] - Python_App[Python Application
Desktop] - Dart_App[Dart Application
Desktop/Flutter/Web] - Rust_App[Rust Application
Server/Desktop] - MicroPython_App[MicroPython Device] + Julia_App[Julia Application] + JS_App[JavaScript Application
Node.js/Browser] + Python_App[Python Application
Desktop] + Dart_App[Dart Application
Desktop/Flutter/Web] + Rust_App[Rust Application
Server/Desktop] + MicroPython_App[MicroPython Device] end - Julia_App -->|NATS| NATS_Server - JS_App -->|NATS| NATS_Server - Python_App -->|NATS| NATS_Server - Dart_App -->|NATS| NATS_Server - Rust_App -->|NATS| NATS_Server - MicroPython_App -->|NATS| NATS_Server + Julia_App -->|Transport| Message_Broker + JS_App -->|Transport| Message_Broker + Python_App -->|Transport| Message_Broker + Dart_App -->|Transport| Message_Broker + Rust_App -->|Transport| Message_Broker + MicroPython_App -->|Transport| Message_Broker Julia_App -->|HTTP| File_Server JS_App -->|HTTP| File_Server @@ -75,7 +74,7 @@ flowchart TD Rust_App -->|HTTP| File_Server MicroPython_App -->|HTTP| File_Server - style NATS_Server fill:#fff3e0,stroke:#f57c00 + style Message_Broker fill:#fff3e0,stroke:#f57c00 style File_Server fill:#f3e5f5,stroke:#9c27b4 style Julia_App fill:#e8f5e9,stroke:#4caf50 style JS_App fill:#e3f2fd,stroke:#2196f3 @@ -98,14 +97,14 @@ flowchart TD MicroPython_Module[MicroPython msghandler Module] end - Julia_Module --> NATS_Client - JS_Module --> NATS_Client - Python_Module --> NATS_Client - Dart_Module --> NATS_Client - Rust_Module --> NATS_Client - MicroPython_Module --> NATS_Client + Julia_Module --> Transport_Client + JS_Module --> Transport_Client + Python_Module --> Transport_Client + Dart_Module --> Transport_Client + Rust_Module --> Transport_Client + MicroPython_Module --> Transport_Client - NATS_Client --> NATS_Broker + Transport_Client --> Message_Broker Julia_Module --> File_Client JS_Module --> File_Client @@ -122,7 +121,7 @@ flowchart TD style Dart_Module fill:#fff0f6,stroke:#e91e63 style Rust_Module fill:#dea584,stroke:#e65100 style MicroPython_Module fill:#fce4ec,stroke:#e91e63 - style NATS_Broker fill:#fff3e0,stroke:#f57c00 + style Message_Broker fill:#fff3e0,stroke:#f57c00 style File_Server fill:#f3e5f5,stroke:#9c27b4 ``` @@ -174,8 +173,8 @@ flowchart TD | Component | Purpose | Platform Support | |-----------|---------|------------------| -| **smartsend** | Send data via NATS with automatic transport selection, returns (envelope, json_string) for caller to publish | All | -| **smartreceive** | Receive and process NATS messages from JSON string | All | +| **smartsend** | Send data with automatic transport selection, returns (envelope, json_string) for caller to publish via transport | All | +| **smartreceive** | Receive and process messages from JSON string | All | | **_serialize_data** | Serialize data according to payload type | All | | **_deserialize_data** | Deserialize bytes to native data types | All | | **envelope_to_json** | Convert msg_envelope_v1 struct to JSON string | All | @@ -224,7 +223,7 @@ struct msg_envelope_v1 msg_id::String # UUID v4 for this message timestamp::String # ISO 8601 UTC timestamp - send_to::String # NATS subject to publish to + send_to::String # Topic/subject to publish to msg_purpose::String # ACK, NACK, updateStatus, shutdown, chat sender_name::String # Sender application name sender_id::String # UUID v4 of sender @@ -233,7 +232,7 @@ struct msg_envelope_v1 reply_to::String # Topic for reply messages reply_to_msg_id::String # Message ID being replied to - broker_url::String # NATS broker URL + broker_url::String # Broker URL for the transport layer metadata::Dict{String, Any} # Message-level metadata payloads::Vector{msg_payload_v1} # List of payloads @@ -436,13 +435,13 @@ JavaScript uses async/await for non-blocking I/O: #### Node.js Implementation (msghandler_ssr.js) -- **TCP NATS connections**: Uses `nats://` or `tls://` URLs +- **Transport connections**: Uses broker URLs (e.g., `nats://`, `mqtt://`, `ws://`) - **Apache Arrow IPC**: Full support via `apache-arrow` - **Buffer for binary data**: Native Node.js Buffer handling #### Browser Implementation (msghandler_csr.js) -- **WebSocket NATS connections**: Uses `ws://` or `wss://` URLs via `nats.ws` +- **WebSocket connections**: Uses `ws://` or `wss://` URLs (transport-agnostic) - **No Apache Arrow**: Uses `jsontable` for tabular data only - **Uint8Array for binary data**: Browser-compatible binary handling - **Web Crypto API**: UUID generation via `crypto.getRandomValues()` @@ -474,7 +473,7 @@ Dart uses classes for stateful operations with async/await: - **Async/await**: I/O operations - **dart-arrow**: Arrow IPC support (Desktop/Flutter only) - **HTTP package**: HTTP file server communication -- **nats package**: NATS client with WebSocket support (Dart Web) +- **Transport package**: Transport client with WebSocket support (Dart Web) ```dart class msghandler { @@ -484,7 +483,7 @@ class msghandler { final String fileserverUrl; msghandler({ - this.brokerUrl = 'nats://localhost:4222', + this.brokerUrl = DEFAULT_BROKER_URL, this.fileserverUrl = 'http://localhost:8080', }); } @@ -492,19 +491,19 @@ class msghandler { #### Dart Desktop (Dart SDK) -- **TCP NATS connections**: Uses `nats://` or `tls://` URLs +- **Transport connections**: Uses broker URLs (e.g., `nats://`, `mqtt://`) - **Apache Arrow IPC**: Full support via `dart-arrow` - **Uint8List for binary data**: Native Dart binary handling #### Dart Flutter (Dart SDK) -- **TCP NATS connections**: Uses `nats://` or `tls://` URLs +- **Transport connections**: Uses broker URLs (e.g., `nats://`, `mqtt://`) - **Apache Arrow IPC**: Full support via `dart-arrow` - **Uint8List for binary data**: Native Dart binary handling #### Dart Web (Dart SDK) -- **WebSocket NATS connections**: Uses `ws://` or `wss://` URLs via `nats` package +- **WebSocket connections**: Uses `ws://` or `wss://` URLs (transport-agnostic) - **No Apache Arrow**: Uses `jsontable` for tabular data only - **Uint8List for binary data**: Browser-compatible binary handling - **Fetch API**: HTTP file server communication via `http` package @@ -516,7 +515,7 @@ Browser JavaScript has specific constraints due to security and compatibility: - **Async/await**: Native async/await support - **No Apache Arrow**: Arrow IPC not available in browsers - **JSON table only**: Use "jsontable" for tabular data -- **WebSocket NATS**: Uses nats.ws for browser-compatible NATS connections +- **WebSocket transport**: Uses transport client for browser-compatible connections - **Fetch API**: HTTP file server communication via fetch ### MicroPython Architecture @@ -540,7 +539,7 @@ Rust leverages compile-time type safety and async runtimes: - **Type-safe payloads**: Rust enum discriminates between `Text`, `Dictionary`, `ArrowTable`, `Binary`, etc. - **serde serialization**: Automatic JSON deserialization via `#[derive(Serialize, Deserialize)]` -- **tokio runtime**: Efficient async I/O for NATS connections and HTTP file server operations +- **tokio runtime**: Efficient async I/O for transport connections and HTTP file server operations - **arrow2 integration**: Native Arrow IPC deserialization without intermediate format conversion - **reqwest**: High-performance HTTP client with built-in TLS and connection pooling - **Zero-copy patterns**: `Vec` passed directly to avoid unnecessary memory copies @@ -572,8 +571,8 @@ pub struct SmartsendOptions { // ... other fields } -// NATS client with tokio integration -let conn = nats::connect("nats://localhost:4222").await?; +// Transport client with tokio integration +let conn = transport_client::connect(DEFAULT_BROKER_URL).await?; // Subscribe and process messages let mut sub = conn.subscribe("/agent/wine/api/v1/analyze")?; @@ -599,7 +598,7 @@ for msg in sub.messages() { | Component | Scaling Strategy | |-----------|------------------| -| **NATS Server** | Cluster deployment with multiple nodes | +| **Message Broker** | Cluster deployment with multiple nodes | | **File Server** | Load balancer + multiple instances | | **Client Applications** | Deploy multiple instances behind load balancer | @@ -607,7 +606,7 @@ for msg in sub.messages() { | Component | Scaling Strategy | |-----------|------------------| -| **NATS Server** | Increase memory, CPU, disk I/O | +| **Message Broker** | Increase memory, CPU, disk I/O | | **File Server** | Increase memory, CPU, disk capacity | | **Client Applications** | Increase heap size (Python/JS) | @@ -617,7 +616,7 @@ for msg in sub.messages() { |--------|--------|-------| | Message serialization overhead | <50ms | For 10KB payload | | Message deserialization overhead | <50ms | For 10KB payload | -| NATS connection establishment | <100ms | Connection pool recommended | +| Transport connection establishment | <100ms | Connection pool recommended | | File upload latency | <1s | For 0.5MB file | | File download latency | <1s | For 0.5MB file | @@ -625,16 +624,16 @@ for msg in sub.messages() { ## Failure Modes and Recovery -### NATS Connection Failure +### Transport Connection Failure -**Scenario**: NATS server unavailable +**Scenario**: Message broker unavailable **Handler**: -- Connection auto-reconnect via TCP-level reconnection +- Connection auto-reconnect via transport-level reconnection - Retry with exponential backoff for publish operations **Recovery**: -- NATS client automatically attempts reconnection +- Transport client automatically attempts reconnection - Application can check connection status before publishing ### File Server Unavailable @@ -697,7 +696,7 @@ for msg in sub.messages() { **Rationale**: - Simplifies JSON serialization (all data is string-compatible) -- Increases payload size by ~33%, but NATS can handle this +- Increases payload size by ~33%, but transport can handle this - Alternative would be binary payload support (more complex) ### Decision 3: Multiple Platform Implementations @@ -730,7 +729,7 @@ for msg in sub.messages() { | Component | Minimum | Notes | |-----------|---------|-------| -| NATS Server | 1 instance | Single node for development | +| Message Broker | 1 instance | Single node for development | | File Server | 1 instance | HTTP server for large payloads | | Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) | | Client Memory | 256KB | MicroPython devices | @@ -739,7 +738,7 @@ for msg in sub.messages() { | Variable | Default | Description | |----------|---------|-------------| -| `NATS_URL` | `nats://localhost:4222` | NATS server URL | +| `BROKER_URL` | `ws://localhost:4222` | Message broker URL | | `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL | | `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) | @@ -748,15 +747,15 @@ for msg in sub.messages() { ```mermaid flowchart TD subgraph "Docker Network" - NATS_Container[NATS Server] + Broker_Container[Message Broker] FileServer_Container[Plik File Server] App_Container[Application Container] end - App_Container -->|NATS| NATS_Container + App_Container -->|Transport| Broker_Container App_Container -->|HTTP| FileServer_Container - style NATS_Container fill:#fff3e0,stroke:#f57c00 + style Broker_Container fill:#fff3e0,stroke:#f57c00 style FileServer_Container fill:#f3e5f5,stroke:#9c27b4 style App_Container fill:#e3f2fd,stroke:#2196f3 ``` @@ -775,11 +774,12 @@ flowchart TD ### Transport Security -**Mechanism**: TLS support for NATS connections +**Mechanism**: TLS support for transport connections **Implementation**: - Use `nats://` URL for plain text - Use `tls://` URL for TLS-encrypted connections +- Use `ws://` or `wss://` for WebSocket connections ### File Server Security @@ -835,6 +835,11 @@ flowchart TD | Date | Version | Changes | |------|---------|---------| +| 2026-05-15 | 1.5.0 | Made transport layer agnostic | All sections | +| - | - | Removed all NATS-specific references from architecture docs | All sections | +| - | - | Updated diagrams to use generic "Message Broker" instead of "NATS Server" | All sections | +| - | - | Updated code examples to use transport-agnostic patterns | All sections | +| - | - | Removed NATS client packages from external dependencies | All sections | | 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections | | - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 | | - | - | Added `plik_upload_file` convenience function to component table | specification.md:13 | @@ -849,6 +854,7 @@ flowchart TD | 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | | - | - | Removed publish_message component (commented out in source) | | - | - | Removed NATSClient and NATSConnectionPool classes (not in ground truth) | +| - | - | Updated smartsend to return JSON for caller to publish via transport | | - | - | Updated component diagram to match actual module structure | | - | - | Updated data flow to show smartsend returns JSON for caller to publish | | - | - | Fixed SIZE_THRESHOLD default to 500,000 bytes | @@ -856,6 +862,7 @@ flowchart TD | - | - | Added NATSClient with keepAlive support | | - | - | Added NATSConnectionPool for connection reuse | | - | - | Added publishMessage function with closeConnection option | +| (Historical - pre-transport-agnostic refactor) | | | | 2026-03-13 | 1.0.0 | Initial architecture documentation | --- @@ -880,7 +887,7 @@ flowchart TD |------|----------|----------|---------------------------|-------------------| | [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | -| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe, file upload helpers | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | @@ -889,24 +896,18 @@ flowchart TD ### 16.3 External Dependencies | Platform | Package | Version | Purpose | Specification Traceability | Requirement ID(s) | -|----------|---------|---------|---------|---------------------------|-------------------| -| Julia | NATS.jl | Latest | NATS client | specification.md:11 | FR-013, FR-014, NFR-201 | +|----------|---------|---------|---------|--------------------------|-------------------| | Julia | JSON.jl | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 | | Julia | Arrow.jl | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 | | Julia | HTTP.jl | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 | | Julia | UUIDs.jl | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 | -| Node.js | nats | Latest | NATS client (TCP) | specification.md:11 | FR-013, FR-014 | | Node.js | node-fetch | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 | -| Browser | nats.ws | Latest | NATS client (WebSocket) | specification.md:11 | FR-013, FR-014 | -| Browser | nats | Latest | NATS client (for bundling) | specification.md:11 | FR-013, FR-014 | -| Python | nats-py | Latest | NATS client | specification.md:11 | FR-013, FR-014 | +| Browser | - | - | Transport-agnostic (caller provides) | specification.md:11 | FR-013, FR-014 | | Python | aiohttp | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 | | Python | pyarrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 | -| Dart | nats | Latest | NATS client | specification.md:11 | FR-013, FR-014 | | Dart | http | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 | | Dart | uuid | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 | | Dart | dart-arrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 | -| Rust | nats | Latest | NATS client | specification.md:11 | FR-013, FR-014 | | Rust | serde | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 | | Rust | serde_json | Latest | JSON handling | specification.md:11 | FR-012, NFR-101, NFR-102 | | Rust | tokio | Latest | Async runtime | specification.md:11 | FR-013, FR-014 | diff --git a/docs/requirements.md b/docs/requirements.md index f6b9198..fc20b68 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -11,7 +11,7 @@ ### 1.1 Business Goal -msghandler is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS. +msghandler is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over the transport layer. ### 1.2 User Stories (with acceptance criteria) @@ -19,13 +19,13 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless |-------|----------|---------------------| | **As a Julia developer**, I want to send text messages to JavaScript/Dart applications that lives on a server and also on a browser | P1 | Text messages are serialized, encoded, and received correctly across platforms | | **As a Python developer**, I want to send tabular data to Julia/Dart applications | P1 | DataFrame exchange works with both Arrow IPC and JSON formats | -| **As a JavaScript developer**, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS | +| **As a JavaScript developer**, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via the transport layer | | **As a Dart developer**, I want to send text messages to other platforms | P1 | Text messages are serialized, encoded, and received correctly across platforms | | **As a Dart developer**, I want to send dictionary data to other platforms | P1 | JSON-serializable data is exchanged correctly | | **As a Dart developer**, I want to send tabular data (List) to other platforms | P1 | JSON table format exchange works with Arrow IPC on desktop | -| **As a Dart developer**, I want to send large files (>0.5MB) | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS | +| **As a Dart developer**, I want to send large files (>0.5MB) | P1 | Large files are automatically uploaded to file server and URLs are sent via the transport layer | | **As a MicroPython developer**, I want to send sensor data with minimal memory usage | P1 | Direct transport works for payloads <100KB on memory-constrained devices | -| **As a Rust developer**, I want to send and receive messages with type-safe APIs | P1 | Rust implementation uses serde for serialization, tokio for async, and nats-io for NATS connectivity | +| **As a Rust developer**, I want to send and receive messages with type-safe APIs | P1 | Rust implementation uses serde for serialization, tokio for async, and transport-agnostic client for connectivity | | **As a developer**, I want to send mixed-content messages (text + image + file) | P1 | msghandler accepts list of (dataname, data, type) tuples and handles each payload appropriately | | **As a developer**, I want to receive multi-payload messages | P1 | msghandler returns payloads as list of tuples with correct types preserved | | **As a developer**, I want to use Plik as the file server | P2 | Plik one-shot upload mode is supported with upload ID and token handling | @@ -59,37 +59,32 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless | File server integration | Plik one-shot upload and custom HTTP server support | | Reliability features | Exponential backoff retry and correlation ID propagation | | Message serialization | Converts data types to binary format (Base64, JSON, Arrow IPC) | -| NATS communication | Publishing and subscription via NATS subjects | +| Transport communication | Publishing and subscription via message broker (NATS, MQTT, WebSocket, etc.) | ### 2.2 Out of Scope | Feature | Reason | |---------|--------| -| NATS JetStream support | Core NATS sufficient for current use cases | +| Advanced transport features | Basic transport sufficient for current use cases | | Message compression | Compression adds complexity without clear benefit | | Message encryption | Payload encryption is application-layer concern | -| Persistent message queues | NATS request-reply pattern sufficient | -| Advanced routing rules | Simple NATS subject matching sufficient | +| Persistent message queues | Transport request-reply pattern sufficient | +| Advanced routing rules | Simple topic matching sufficient | ### 2.3 Dependencies | Platform | Package | Version | |----------|---------|---------| -| Julia | NATS.jl | Latest stable | | Julia | JSON.jl | Latest stable | | Julia | Arrow.jl | Latest stable | | Julia | HTTP.jl | Latest stable | | Julia | UUIDs.jl | Latest stable | -| Node.js | nats | Latest stable | | Node.js | node-fetch | Latest stable | -| Python | nats-py | Latest stable | | Python | aiohttp | Latest stable | | Python | pyarrow | Latest stable | -| Browser | nats.ws | Latest stable | -| Dart | nats | Latest stable | +| Browser | - | Transport-agnostic (caller provides) | | Dart | http | Latest stable | | Dart | uuid | Latest stable | -| Rust | nats | Latest stable | | Rust | serde | Latest stable | | Rust | serde_json | Latest stable | | Rust | tokio | Latest stable | @@ -100,7 +95,7 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless | Platform | Minimum Version | Notes | |----------|-----------------|-------| | Julia | 1.7+ | Arrow.jl required for arrowtable support | -| Node.js | 16+ | nats.js required, Arrow IPC supported | +| Node.js | 16+ | Transport client required, Arrow IPC supported | | Python | 3.8+ | pyarrow required for arrowtable support | | Browser | Latest | No Arrow IPC (uses jsontable only) | | Dart | 2.17+ | Supports Desktop (Dart SDK), Flutter (Dart SDK), and Web (Dart SDK) | @@ -115,8 +110,8 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless |----|-------------|-------------| | **FR-001** | Cross-platform text messaging | System shall allow users to send text messages between Julia, JavaScript, Python, and MicroPython applications | | **FR-002** | Cross-platform tabular data | System shall support DataFrame exchange between Julia and Python applications using Arrow IPC format | -| **FR-003** | Large file handling | System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via NATS | -| **FR-004** | Direct transport for small payloads | System shall send payloads <0.5MB directly via NATS without file server upload | +| **FR-003** | Large file handling | System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via transport | +| **FR-004** | Direct transport for small payloads | System shall send payloads <0.5MB directly via transport without file server upload | | **FR-005** | MicroPython support | System shall support payloads <100KB on MicroPython devices using direct transport | | **FR-006** | Multi-payload messages | System shall accept and process lists of (dataname, data, type) tuples | | **FR-007** | Payload type preservation | System shall preserve payload types when returning multi-payload messages | @@ -125,8 +120,8 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless | **FR-010** | Exponential backoff retry | System shall implement exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) for file server download failures | | **FR-011** | Correlation ID propagation | System shall propagate correlation IDs through all message processing steps | | **FR-012** | Message serialization | System shall serialize data types using Base64, JSON, or Arrow IPC encoding | -| **FR-013** | NATS publishing | System shall return JSON string representation for caller to publish to NATS subjects (caller is responsible for actual NATS publish) | -| **FR-014** | NATS subscription | System shall receive and process NATS messages by accepting JSON string from NATS payload | +| **FR-013** | Transport publishing | System shall return JSON string representation for caller to publish via transport layer (caller is responsible for actual transport publish) | +| **FR-014** | Transport subscription | System shall receive and process messages by accepting JSON string from transport payload | --- @@ -138,10 +133,10 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless |----|-------------|---------------|-------------| | **NFR-101** | Message serialization overhead | <50ms for 10KB payload | Benchmark tests | | **NFR-102** | Message deserialization overhead | <50ms for 10KB payload | Benchmark tests | -| **NFR-103** | NATS connection establishment | <100ms | Connection pool benchmarks | +| **NFR-103** | Transport connection establishment | <100ms | Connection pool benchmarks | | **NFR-104** | File upload latency | <1s for 0.5MB file | Integration tests | | **NFR-105** | File download latency | <1s for 0.5MB file | Integration tests | -| **NFR-106** | Concurrent connections | Support 100+ simultaneous NATS connections | Scale testing | +| **NFR-106** | Concurrent connections | Support 100+ simultaneous transport connections | Scale testing | | **NFR-107** | Message throughput | Handle 1000+ messages/second per instance | Load testing | | **NFR-108** | File server scalability | Support horizontal scaling of file server backend | Architecture review | @@ -149,16 +144,16 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless | ID | Requirement | Specification | |----|-------------|---------------| -| **NFR-201** | Message delivery | At-least-once delivery semantics via NATS | +| **NFR-201** | Message delivery | At-least-once delivery semantics via transport | | **NFR-202** | File server availability | Graceful degradation when file server is unavailable | -| **NFR-203** | Connection recovery | Auto-reconnect on NATS connection failure | +| **NFR-203** | Connection recovery | Auto-reconnect on transport connection failure | ### 4.3 Privacy & Security | ID | Requirement | Specification | |----|-------------|---------------| | **NFR-301** | Payload integrity | SHA-256 checksum support via metadata | -| **NFR-302** | Transport security | TLS support for NATS connections | +| **NFR-302** | Transport security | TLS support for transport connections | | **NFR-303** | File server security | Authentication token for file uploads | ### 4.4 Observability & Telemetry @@ -234,11 +229,11 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless | Platform | Maximum | Notes | |----------|---------|-------| -| Desktop | Unlimited | Limited by NATS server configuration | -| Dart Desktop | Unlimited | Limited by NATS server configuration | -| Dart Flutter | Unlimited | Limited by NATS server configuration | -| Dart Web | Unlimited | Limited by NATS server configuration | -| Rust | Unlimited | Limited by NATS server configuration | +| Desktop | Unlimited | Limited by transport server configuration | +| Dart Desktop | Unlimited | Limited by transport server configuration | +| Dart Flutter | Unlimited | Limited by transport server configuration | +| Dart Web | Unlimited | Limited by transport server configuration | +| Rust | Unlimited | Limited by transport server configuration | | MicroPython | 50KB | Hard limit due to 256KB-1MB memory | --- @@ -252,7 +247,7 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless | `correlation_id` | String (UUID) | Track message flow across systems | | `msg_id` | String (UUID) | Unique message identifier | | `timestamp` | String (ISO 8601) | Message publication timestamp | -| `send_to` | String | NATS subject to publish to | +| `send_to` | String | Topic/subject to publish to | | `msg_purpose` | String | ACK, NACK, updateStatus, shutdown, chat | | `sender_name` | String | Sender application name | | `sender_id` | String (UUID) | Sender unique identifier | @@ -260,7 +255,7 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless | `receiver_id` | String (UUID) | Receiver unique identifier (empty = broadcast) | | `reply_to` | String | Topic for reply messages | | `reply_to_msg_id` | String | Message ID being replied to | -| `broker_url` | String | NATS server URL | +| `broker_url` | String | Broker URL for the transport layer | | `metadata` | Dict | Message-level metadata | | `payloads` | Array | List of payload objects | @@ -289,14 +284,14 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless | `Failed to upload` | File server error | Throw error | | `Failed to fetch` | File server unavailable | Retry with exponential backoff | | `Unknown transport` | Invalid transport type | Throw error | -| `NATS connection failed` | NATS unavailable | Throw error | +| `Transport connection failed` | Transport/broker unavailable | Throw error | ### 9.2 Exception Handling | Scenario | Handler | |----------|---------| | File server unavailable | Retry up to 5 times with exponential backoff | -| NATS publish failure | Connection auto-reconnect | +| Transport publish failure | Handled by caller | | Deserialization error | Log correlation ID and throw error | | Memory overflow (MicroPython) | Reject payloads >50KB | @@ -350,7 +345,7 @@ function smartsend( )::Tuple{msg_envelope_v1, String} where {T1<:Any} ``` -**Note**: NATS publishing is the caller's responsibility. `smartsend` returns `(env::msg_envelope_v1, env_json_str::String)`. +**Note**: Publishing via the transport layer is the caller's responsibility. `smartsend` returns `(env::msg_envelope_v1, env_json_str::String)`. ### 11.2 smartreceive Signature @@ -364,7 +359,9 @@ function smartreceive( )::JSON.Object{String, Any} ``` -**Note**: Pass `String(nats_msg.payload)` from NATS subscription to `smartreceive`. +**Note**: Pass the payload string from the transport subscription to `smartreceive`. The input is the JSON string payload from the transport message, not the transport message object directly. + +**Note**: Pass the payload from the transport subscription to `smartreceive`. --- @@ -374,7 +371,7 @@ function smartreceive( | Component | Minimum | Notes | |-----------|---------|-------| -| NATS Server | 1 instance | Single node for development | +| Message Broker | 1 instance | Single node for development | | File Server | 1 instance | HTTP server for large payloads | | Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) | | Client Memory | 256KB | MicroPython devices | @@ -383,7 +380,7 @@ function smartreceive( | Variable | Default | Description | |----------|---------|-------------| -| `NATS_URL` | `nats://localhost:4222` | NATS server URL | +| `BROKER_URL` | `ws://localhost:4222` | Message broker URL | | `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL | | `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) | @@ -409,6 +406,10 @@ function smartreceive( | Date | Version | Changes | |------|---------|---------| +| 2026-05-15 | 1.3.0 | Made transport layer agnostic | +| - | - | Removed all NATS-specific dependencies and references | +| - | - | Updated all NATS references to generic "transport layer"/"message broker" | +| - | - | Removed NATS client packages from dependencies tables | | 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | | - | - | Fixed smartsend signature: removed is_publish, NATS_connection; added sender_name | | - | - | Fixed smartreceive signature: takes msg_json_str::String instead of msg::NATS.Msg | diff --git a/docs/specification.md b/docs/specification.md index f3b2231..2b32c91 100644 --- a/docs/specification.md +++ b/docs/specification.md @@ -10,7 +10,7 @@ ## 1. Technical Contract Overview -This document defines the **technical contract** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus. +This document defines the **technical contract** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer. This specification serves as the single source of truth for: - **Inputs**: What data structures are accepted by `smartsend()` @@ -28,7 +28,7 @@ This specification serves as the single source of truth for: | Section 5 (Enumerations) | FR-003, FR-004, FR-006, NFR-101 | Enumerations for transport and encoding | | Section 6 (Transport Protocols) | FR-003, FR-004, NFR-104, NFR-105 | Direct and link transport protocols | | Section 7 (Size Thresholds) | FR-004, FR-005, NFR-104, NFR-105 | Size thresholds for transport selection | -| Section 8 (NATS Subject Convention) | FR-013, FR-014 | NATS subject naming patterns | +| Section 8 (Topic Convention) | FR-013, FR-014 | Topic/subject naming patterns | | Section 9 (Error Handling) | FR-010, FR-011, NFR-201, NFR-202, NFR-203 | Error codes and exception handling | | Section 10 (Serialization Rules) | FR-001, FR-002, FR-003, FR-012, NFR-101, NFR-102 | Serialization and encoding rules | | Section 11 (API Contract) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Function signatures for all platforms | @@ -95,7 +95,7 @@ This specification serves as the single source of truth for: | `correlation_id` | `string` | Yes | UUID v4 format | Track message flow across distributed systems | FR-011, NFR-401 | | `msg_id` | `string` | Yes | UUID v4 format | Unique identifier for this specific message | FR-012, NFR-401 | | `timestamp` | `string` | Yes | ISO 8601 UTC | Message publication timestamp | FR-012, NFR-401 | -| `send_to` | `string` | Yes | Non-empty string | NATS subject/topic to publish the message to | FR-013 | +| `send_to` | `string` | Yes | Non-empty string | Topic/subject to publish the message to | FR-013 | | `msg_purpose` | `string` | Yes | Enum | Purpose of the message | FR-013 | | `sender_name` | `string` | Yes | Non-empty string | Name of the sender application | FR-013 | | `sender_id` | `string` | Yes | UUID v4 format | Unique identifier for the sender | FR-013 | @@ -103,7 +103,7 @@ This specification serves as the single source of truth for: | `receiver_id` | `string` | Yes | Any string | UUID of the receiver (empty = broadcast) | FR-013 | | `reply_to` | `string` | Yes | Any string | Topic where receiver should reply | FR-013 | | `reply_to_msg_id` | `string` | Yes | Any string | Message ID this message is replying to | FR-013 | -| `broker_url` | `string` | Yes | Valid URL | NATS broker URL | FR-013 | +| `broker_url` | `string` | Yes | Valid URL | Broker URL for the transport layer | FR-013 | | `metadata` | `object` | No | Any JSON object | Message-level metadata | NFR-401 | | `payloads` | `array` | Yes | Non-empty array | List of payload objects | FR-012, FR-013 | @@ -240,7 +240,7 @@ await smartsend("/agent/v1/process", data) | Value | Description | Data Format | Use Case | |-------|-------------|-------------|----------| -| `direct` | Payload sent directly via NATS | Base64-encoded string | Payloads < size_threshold | +| `direct` | Payload sent directly via the transport layer | Base64-encoded string | Payloads < size_threshold | | `link` | Payload uploaded to file server | HTTP URL | Payloads ≥ size_threshold | ### `encoding` Enum @@ -312,7 +312,7 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa --- -## NATS Subject Convention +## Topic Convention ### Subject Naming Pattern @@ -357,7 +357,7 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa | `INVALID_TRANSPORT` | 400 | Unsupported transport type | Use `direct` or `link` | FR-003, FR-004, FR-006 | | `UPLOAD_FAILED` | 500 | File server upload failed | Retry or use direct transport | FR-008, FR-009 | | `DOWNLOAD_FAILED` | 503 | File server download failed | Retry with exponential backoff | FR-010, FR-011, NFR-201, NFR-202 | -| `NATS_CONNECTION_FAILED` | 503 | NATS connection failed | Check NATS server availability | FR-013, FR-014, NFR-201, NFR-203 | +| `TRANSPORT_CONNECTION_FAILED` | 503 | Transport connection failed | Check broker/server availability | FR-013, FR-014, NFR-201, NFR-203 | | `DESERIALIZATION_ERROR` | 500 | Payload deserialization failed | Check payload_type matches data | FR-001, FR-002, FR-003, FR-012 | | `SIZE_EXCEEDED` | 413 | Payload exceeds maximum size | Split payload or use link transport | FR-003, FR-004, FR-005, NFR-104, NFR-105 | @@ -366,7 +366,7 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa | Scenario | Handler | Retry Policy | Requirement ID | |----------|---------|--------------|----------------| | File server unavailable | Retry up to 5 times | Exponential backoff (100ms → 5000ms) | FR-010, NFR-202 | -| NATS publish failure | Connection auto-reconnect | TCP-level reconnection | FR-013, FR-014, NFR-201, NFR-203 | +| Transport publish failure | Handled by caller | Caller's retry logic | FR-013, FR-014, NFR-201, NFR-203 | | Deserialization error | Log correlation ID and throw | No retry (data corruption) | FR-001, FR-002, FR-003, FR-012, NFR-401 | | Memory overflow (MicroPython) | Reject payloads >50KB | No retry (client-side check) | FR-005, NFR-106 | @@ -435,7 +435,7 @@ function smartsend( )::Tuple{msg_envelope_v1, String} where {T1<:Any} ``` -**Note**: NATS publishing is the caller's responsibility. Returns `(env::msg_envelope_v1, env_json_str::String)`. +**Note**: Publishing via the transport layer is the caller's responsibility. Returns `(env::msg_envelope_v1, env_json_str::String)`. #### Python @@ -443,7 +443,7 @@ function smartsend( async def smartsend( subject: str, data: List[Tuple[str, Any, str]], - broker_url: str = "nats://localhost:4222", + broker_url: str = DEFAULT_BROKER_URL, fileserver_url: str = "http://localhost:8080", fileserver_upload_handler: Callable = plik_oneshot_upload, size_threshold: int = 500_000, @@ -459,7 +459,7 @@ async def smartsend( ) -> Tuple[Dict, str]: ``` -**Note**: NATS publishing is the caller's responsibility. +**Note**: Publishing via the transport layer is the caller's responsibility. #### JavaScript (Node.js) @@ -485,7 +485,7 @@ async function smartsend( ): Promise<[Object, string]>; ``` -**Note**: NATS publishing is the caller's responsibility. +**Note**: Publishing via the transport layer is the caller's responsibility. #### JavaScript (Browser) @@ -511,7 +511,7 @@ async function smartsend( ): Promise<[Object, string]>; ``` -**Note**: NATS publishing is the caller's responsibility. +**Note**: Publishing via the transport layer is the caller's responsibility. #### MicroPython @@ -524,7 +524,7 @@ def smartsend( ) -> Tuple[Dict, str]: ``` -**Note**: NATS publishing is the caller's responsibility. +**Note**: Publishing via the transport layer is the caller's responsibility. #### Dart (Desktop/Flutter) @@ -532,10 +532,10 @@ def smartsend( Future<[Map, String]> smartsend( String subject, List> data, { - String brokerUrl = 'nats://localhost:4222', - String fileserverUrl = 'http://localhost:8080', + String brokerUrl = DEFAULT_BROKER_URL, + String fileserverUrl = DEFAULT_FILESERVER_URL, Function? fileserverUploadHandler, - int sizeThreshold = 500000, + int sizeThreshold = DEFAULT_SIZE_THRESHOLD, String? correlationId, String msgPurpose = 'chat', String senderName = 'msghandler', @@ -547,9 +547,8 @@ Future<[Map, String]> smartsend( String? senderId, }) async { // Returns [envelope, jsonString] - // NATS publishing is caller's responsibility + // Publishing via transport layer is caller's responsibility } -``` #### Dart Web @@ -557,7 +556,7 @@ Future<[Map, String]> smartsend( Future<[Map, String]> smartsend( String subject, List> data, { - String brokerUrl = 'nats://localhost:4222', + String brokerUrl = DEFAULT_BROKER_URL, String fileserverUrl = 'http://localhost:8080', Function? fileserverUploadHandler, int sizeThreshold = 500000, @@ -572,7 +571,7 @@ Future<[Map, String]> smartsend( String? senderId, }) async { // Returns [envelope, jsonString] - // NATS publishing is caller's responsibility + // Publishing via transport layer is caller's responsibility } ``` @@ -635,7 +634,7 @@ pub struct MsgEnvelopeV1 { } ``` -**Note**: NATS publishing is the caller's responsibility. Returns `Result<(MsgEnvelopeV1, String), msghandlerError>`. Uses `serde` for JSON serialization. +**Note**: Publishing via the transport layer is the caller's responsibility. Returns `Result<(MsgEnvelopeV1, String), msghandlerError>`. Uses `serde` for JSON serialization. ### `smartreceive` Function Signature @@ -643,7 +642,7 @@ pub struct MsgEnvelopeV1 { ```julia function smartreceive( - msg_json_str::String; # Pass String(nats_msg.payload) from NATS subscription + msg_json_str::String; # Pass payload from transport subscription fileserver_download_handler::Function = _fetch_with_backoff, max_retries::Int = 5, base_delay::Int = 100, @@ -651,13 +650,13 @@ function smartreceive( )::JSON.Object{String, Any} ``` -**Note**: Input is JSON string from NATS message payload, not NATS.Msg directly. +**Note**: Input is the JSON string payload from the transport subscription, not the transport message object directly. #### Python ```python async def smartreceive( - msg_json_str: str, # JSON string from NATS message payload + msg_json_str: str, # JSON string from transport message payload fileserver_download_handler: Callable = fetch_with_backoff, max_retries: int = 5, base_delay: int = 100, @@ -665,13 +664,13 @@ async def smartreceive( ) -> Dict[str, Any]: ``` -**Note**: Input is JSON string from NATS message payload. +**Note**: Input is the JSON string payload from the transport message. #### JavaScript (Node.js) ```typescript async function smartreceive( - msg_json_str: string, // JSON string from NATS message payload + msg_json_str: string, // JSON string from transport message payload options?: { fileserver_download_handler?: Function; max_retries?: number; @@ -685,7 +684,7 @@ async function smartreceive( ```typescript async function smartreceive( - msg_json_str: string, // JSON string from NATS message payload + msg_json_str: string, // JSON string from transport message payload options?: { fileserver_download_handler?: Function; max_retries?: number; @@ -695,7 +694,7 @@ async function smartreceive( ): Promise; ``` -**Note**: Input is JSON string from NATS message payload. +**Note**: Input is the JSON string payload from the transport message. #### MicroPython @@ -703,13 +702,13 @@ async function smartreceive( def smartreceive(msg_json_str: str, **kwargs) -> Dict[str, Any]: ``` -**Note**: Input is JSON string from NATS message payload. +**Note**: Input is the JSON string payload from the transport message. #### Dart (Desktop/Flutter) ```dart Future> smartreceive( - Map msg_json_str, // JSON object from NATS message payload + Map msg_json_str, // JSON object from transport message payload { Function? fileserverDownloadHandler, int maxRetries = 5, @@ -724,7 +723,7 @@ Future> smartreceive( ```dart Future> smartreceive( - Map msg_json_str, // JSON object from NATS message payload + Map msg_json_str, // JSON object from transport message payload { Function? fileserverDownloadHandler, int maxRetries = 5, @@ -739,7 +738,7 @@ Future> smartreceive( ```rust pub async fn smartreceive( - msg_json_str: &str, // JSON string from NATS message payload + msg_json_str: &str, // JSON string from transport message payload options: &SmartreceiveOptions, ) -> Result @@ -752,7 +751,7 @@ pub struct SmartreceiveOptions { } ``` -**Note**: Input is JSON string from NATS message payload. Returns `Result`. +**Note**: Input is the JSON string payload from the transport message. Returns `Result`. --- @@ -897,7 +896,7 @@ function fileserver_download_handler( |------|----------|----------|-------| | [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | Ground truth implementation | | [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | Server-side JavaScript | -| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | Client-side rendering | +| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | Client-side rendering | | [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | Desktop Python | | [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | Desktop/Flutter/Web | | [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe | Uses tokio + serde + arrow2 | @@ -910,7 +909,7 @@ The browser implementation ([`src/msghandler_csr.js`](../src/msghandler_csr.js)) | Constraint | Reason | Workaround | |------------|--------|------------| | No Apache Arrow IPC | Browser-incompatible dependency | Use `jsontable` for tabular data | -| WebSocket NATS only | Browser cannot use TCP directly | Use `ws://` or `wss://` broker URLs | +| WebSocket only | Browser cannot use TCP directly | Use `ws://` or `wss://` broker URLs | | Fetch API for HTTP | Browser fetch() API only | Compatible with Plik and other HTTP servers | ### Payload Type Availability by Platform @@ -941,7 +940,7 @@ flowchart TD D --> F[Build envelope with metadata] E --> F F --> G[Convert envelope to JSON string] - G --> H[Publish to NATS subject] + G --> H[Publish to topic via transport] H --> I[Return envelope and JSON string to caller] style A fill:#f9f9f9,stroke:#333 @@ -954,7 +953,7 @@ flowchart TD ```mermaid flowchart TD - A[NATS message arrives] --> B[Parse JSON envelope] + A[Transport message arrives] --> B[Parse JSON envelope] B --> C[For each payload: Check transport type] C -->|transport == direct| D[Direct Transport: Extract Base64] C -->|transport == link| E[Link Transport: Fetch from URL] @@ -1046,23 +1045,17 @@ flowchart TD | Platform | Package | Version | Purpose | |----------|---------|---------|---------| -| Julia | NATS.jl | Latest | NATS client | | Julia | JSON.jl | Latest | JSON serialization | | Julia | Arrow.jl | Latest | Arrow IPC support | | Julia | HTTP.jl | Latest | HTTP file server | | Julia | UUIDs.jl | Latest | UUID generation | -| Node.js | nats | Latest | NATS client (TCP) | | Node.js | node-fetch | Latest | HTTP file server | -| Browser | nats.ws | Latest | NATS client (WebSocket) | -| Browser | nats | Latest | NATS client (for bundling) | -| Python | nats-py | Latest | NATS client | +| Browser | - | - | Transport-agnostic (caller provides) | | Python | aiohttp | Latest | HTTP file server | | Python | pyarrow | Latest | Arrow IPC support | -| Dart | nats | Latest | NATS client | | Dart | http | Latest | HTTP file server | | Dart | uuid | Latest | UUID generation | | Dart | dart-arrow | Latest | Arrow IPC support (Desktop/Flutter) | -| Rust | nats | Latest | NATS client | | Rust | serde | Latest | JSON serialization | | Rust | serde_json | Latest | JSON handling | | Rust | tokio | Latest | Async runtime | @@ -1084,6 +1077,10 @@ flowchart TD | Date | Version | Changes | |------|---------|---------| +| 2026-05-15 | 1.3.0 | Made transport layer agnostic | +| - | - | Removed all NATS-specific dependencies (NATS.jl, nats, nats-py, nats.ws) | +| - | - | Updated docs to reference generic message broker/transport | +| - | - | broker_url is now metadata only, not used for active connections | | 2026-03-15 | 1.1.0 | Browser connection management | | - | - | Added NATSClient class with keepAlive support | | - | - | Added NATSConnectionPool for connection reuse | @@ -1119,7 +1116,7 @@ flowchart TD |------|----------|----------|--------------------------| | [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 | -| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 | | [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe | FR-001 through FR-014, NFR-101 through NFR-405 | @@ -1129,23 +1126,17 @@ flowchart TD | Platform | Package | Version | Purpose | Requirements Traceability | |----------|---------|---------|---------|--------------------------| -| Julia | NATS.jl | Latest | NATS client | FR-013, FR-014, NFR-201 | | Julia | JSON.jl | Latest | JSON serialization | FR-012, NFR-101, NFR-102 | | Julia | Arrow.jl | Latest | Arrow IPC support | FR-002, FR-012 | | Julia | HTTP.jl | Latest | HTTP file server | FR-008, FR-009 | | Julia | UUIDs.jl | Latest | UUID generation | FR-011, NFR-401 | -| Node.js | nats | Latest | NATS client (TCP) | FR-013, FR-014 | | Node.js | node-fetch | Latest | HTTP file server | FR-008, FR-009 | -| Browser | nats.ws | Latest | NATS client (WebSocket) | FR-013, FR-014 | -| Browser | nats | Latest | NATS client (for bundling) | FR-013, FR-014 | -| Python | nats-py | Latest | NATS client | FR-013, FR-014 | +| Browser | - | - | Transport-agnostic (caller provides) | FR-013, FR-014 | | Python | aiohttp | Latest | HTTP file server | FR-008, FR-009 | | Python | pyarrow | Latest | Arrow IPC support | FR-002, FR-012 | -| Dart | nats | Latest | NATS client | FR-013, FR-014 | | Dart | http | Latest | HTTP file server | FR-008, FR-009 | | Dart | uuid | Latest | UUID generation | FR-011, NFR-401 | | Dart | dart-arrow | Latest | Arrow IPC support | FR-002, FR-012 | -| Rust | nats | Latest | NATS client | FR-013, FR-014 | | Rust | serde | Latest | JSON serialization | FR-012, NFR-101, NFR-102 | | Rust | serde_json | Latest | JSON handling | FR-012, NFR-101, NFR-102 | | Rust | tokio | Latest | Async runtime | FR-013, FR-014 | @@ -1160,6 +1151,10 @@ flowchart TD | Date | Version | Changes | Requirement ID(s) | |------|---------|---------|-------------------| +| 2026-05-15 | 1.3.0 | Made transport layer agnostic | All | +| - | - | Removed NATS-specific dependencies and references from all docs | All | +| - | - | Updated all NATS references to generic "transport layer"/"message broker" | All | +| - | - | Removed NATS client packages from dependencies tables | All | | 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All | | - | - | Updated smartsend signatures: removed is_publish, nats_connection; added sender_name | FR-001 through FR-014 | | - | - | Updated smartreceive signatures: takes msg_json_str::String instead of msg | FR-001 through FR-014 | @@ -1200,7 +1195,7 @@ flowchart TD "send_to": { "type": "string", "minLength": 1, - "description": "NATS subject to publish to" + "description": "Topic/subject to publish to" }, "msg_purpose": { "type": "string", @@ -1236,8 +1231,7 @@ flowchart TD }, "broker_url": { "type": "string", - "pattern": "^nats://[^\\s]+$", - "description": "NATS broker URL" + "description": "Broker URL for the transport layer" }, "metadata": { "type": "object", @@ -1302,14 +1296,14 @@ flowchart TD } ``` -### B. AsyncAPI Specification (NATS) +### B. AsyncAPI Specification ```yaml asyncapi: '2.6.0' info: title: msghandler API version: '1.0.0' - description: Cross-platform bi-directional data bridge using NATS + description: Cross-platform bi-directional data bridge using a message broker contact: name: msghandler Team url: https://github.com/your-org/msghandler @@ -1331,12 +1325,12 @@ channels: schema: type: string publish: - summary: Publish message to NATS + summary: Publish message to transport operationId: publishMessage message: $ref: '#/components/message' subscribe: - summary: Subscribe to NATS messages + summary: Subscribe to messages from transport operationId: subscribeMessage message: $ref: '#/components/message' diff --git a/docs/walkthrough.md b/docs/walkthrough.md index 335fad8..a489997 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -9,7 +9,7 @@ ## 1. Executive Summary -This document provides the **end-to-end trace** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus. +This document provides the **end-to-end trace** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer. This walkthrough serves as the primary onboarding guide for new developers and explains: - **User scenarios** - Real-world use cases from developer perspective @@ -51,7 +51,7 @@ flowchart TB S3["Size Check"] S4["Transport Selection"] S5["Build Envelope"] - S6["Publish to NATS"] + S6["Publish to transport"] S1 --> S2 S2 --> S3 @@ -62,7 +62,7 @@ flowchart TB subgraph Receiver["Receiver (smartreceive)"] direction LR - R1["Subscribe to NATS"] + R1["Subscribe via transport"] R2["Parse Envelope"] R3["Check Transport"] R4["Deserialize Data"] @@ -99,7 +99,7 @@ flowchart TB | Principle | Description | Rationale | |-----------|-------------|-----------| -| **Claim-Check Pattern** | Large payloads uploaded to HTTP server, URL sent via NATS | NATS has message size limits; avoids NATS overflow | +| **Claim-Check Pattern** | Large payloads uploaded to HTTP server, URL sent via transport | Transport has message size limits; avoids overflow | | **Automatic Transport Selection** | Direct (< threshold) vs Link (≥ threshold) based on size | Optimizes memory vs network I/O trade-off | | **Cross-Platform API** | Consistent `smartsend()`/`smartreceive()` across all platforms | Simplifies developer experience | | **Exponential Backoff** | Retry downloads with increasing delays | Handles transient failures gracefully | @@ -148,7 +148,7 @@ For each payload, msghandler determines transport: **Rationale**: - Direct transport is faster for small payloads (no file server round-trip) -- Link transport is used when payload ≥ 0.5MB (avoids NATS size limits) +- Link transport is used when payload ≥ 0.5MB (avoids transport size limits) #### Step 3: Serialization and Encoding @@ -213,25 +213,26 @@ msghandler builds the message envelope: - **reply_to**: Tells backend where to send response - **payloads array**: Contains all data with metadata for proper handling -#### Step 5: Publish to NATS (Caller's Responsibility) +#### Step 5: Publish to Transport (Caller's Responsibility) ```javascript -// NATS publishing is the caller's responsibility -const conn = await NATS.connect({ servers: "ws://localhost:4222" }); -await conn.publish("/agent/wine/api/v1/prompt", msgJson); +// Publishing via the transport layer is the caller's responsibility +// Example with any transport (NATS, MQTT, WebSocket, etc.) +// const conn = await transportClient.connect({ servers: "ws://localhost:4222" }); +// await conn.publish("/agent/wine/api/v1/prompt", msgJson); ``` **Rationale**: -- NATS provides low-latency message delivery +- The transport layer provides message delivery (NATS, MQTT, WebSocket, etc.) - JSON format ensures cross-platform compatibility -- `smartsend()` returns `(env, msgJson)` - caller handles publishing +- `smartsend()` returns `(env, msgJson)` - caller handles publishing via their chosen transport #### Step 6: Julia Backend Receives Message ```julia # Julia backend -nats_msg = NATS.subscription.next() # Get message from NATS -env = smartreceive(String(nats_msg.payload)) +transport_msg = transport_subscription.next() # Get message from transport +env = smartreceive(String(transport_msg.payload)) # env["payloads"] is now: # [ @@ -302,7 +303,7 @@ const [env, msgJson] = await msghandler.smartsend( **Rationale**: - Link transport used for large payloads - File server handles large file upload -- NATS only sends URL (small message) +- Transport only sends URL (small message) #### Step 3: File Server Upload @@ -356,8 +357,8 @@ const response = await plikOneshotUpload( ```julia # Julia backend -nats_msg = NATS.subscription.next() -env = smartreceive(String(nats_msg.payload)) +transport_msg = transport_subscription.next() +env = smartreceive(String(transport_msg.payload)) # msghandler automatically: # 1. Extracts URL from payload @@ -396,7 +397,7 @@ df = pd.DataFrame({ env, msg_json = await smartsend( "/agent/wine/api/v1/analyze", [("data", df, "arrowtable")], - broker_url="nats://localhost:4222", + broker_url=DEFAULT_BROKER_URL, receiver_name="agent-backend" ) ``` @@ -429,8 +430,8 @@ arrow_bytes = buf.getvalue() ```julia # Julia backend -nats_msg = NATS.subscription.next() -env = smartreceive(String(nats_msg.payload)) +transport_msg = transport_subscription.next() +env = smartreceive(String(transport_msg.payload)) # env["payloads"][1] is now: # ("data", DataFrame with id, name, score columns, "arrowtable") @@ -479,7 +480,7 @@ use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64}; #[tokio::main] async fn main() { - let conn = nats::connect("nats://localhost:4222").unwrap(); + let conn = transport_client::connect("ws://localhost:4222").unwrap(); // Subscribe and receive messages let mut sub = conn.subscribe("/agent/wine/api/v1/analyze").unwrap(); @@ -520,7 +521,7 @@ async fn main() { **Rationale**: - **serde serialization**: Automatic JSON deserialization to `MsgEnvelopeV1` -- **tokio runtime**: Efficient async I/O for NATS and HTTP operations +- **tokio runtime**: Efficient async I/O for transport and HTTP operations - **smartreceive deserialization**: Payload data is deserialized and stored as strings in `payload.data` - **Type dispatch**: `payload_type` field determines how to interpret the `data` string @@ -548,14 +549,14 @@ let (envelope, json_str) = smartsend( ), ], &SmartsendOptions { - broker_url: "nats://localhost:4222".to_string(), + broker_url: DEFAULT_BROKER_URL.to_string(), reply_to: "/python/worker/v1/results".to_string(), msg_purpose: "chat".to_string(), ..Default::default() }, ).await?; -// Caller publishes to NATS +// Caller publishes via transport conn.publish("/agent/wine/api/v1/results", &json_str)?; ``` @@ -569,7 +570,7 @@ conn.publish("/agent/wine/api/v1/results", &json_str)?; ```python # Python backend receives Rust response -env = await smartreceive(str(nats_msg.payload)) +env = await smartreceive(str(transport_msg.payload)) # env["payloads"][0] is now: # ("results", arrow_table_data, "arrowtable") @@ -598,9 +599,9 @@ let (envelope, json_str) = smartsend( ), ], &SmartsendOptions { - broker_url: "nats://localhost:4222".to_string(), - fileserver_url: "http://localhost:8080".to_string(), - size_threshold: 500_000, // 0.5MB triggers link transport + broker_url: DEFAULT_BROKER_URL.to_string(), + fileserver_url: DEFAULT_FILESERVER_URL.to_string(), + size_threshold: DEFAULT_SIZE_THRESHOLD, // threshold triggers link transport ..Default::default() }, ).await?; @@ -637,7 +638,7 @@ sensor_data = { env, msg_json = smartsend( "/sensor/device/v1/readings", [("data", sensor_data, "dictionary")], - broker_url="nats://localhost:4222", + broker_url=DEFAULT_BROKER_URL, size_threshold=100000 # 100KB for MicroPython ) ``` @@ -658,15 +659,15 @@ payload_b64 = base64.b64encode(json_bytes).decode('ascii') **Rationale**: - JSON format for human-readable data -- Base64 for NATS compatibility +- Base64 for transport compatibility - UTF-8 for text encoding #### Step 3: Python Backend Receives ```python # Python backend -nats_msg = await nats_consumer.next() -env = await smartreceive(str(nats_msg.payload)) +transport_msg = await transport_consumer.next() +env = await smartreceive(str(transport_msg.payload)) # env["payloads"][0] is now: # ("data", {"temperature": 25.5, "humidity": 60.0, ...}, "dictionary") @@ -708,14 +709,14 @@ const [env, msgJson] = await msghandler.smartsend( **Rationale**: - Empty `receiver_name` = broadcast to all subscribers - Chat messages often include text + images -- NATS wildcard subscriptions route to correct recipients +- Transport wildcard subscriptions route to correct recipients #### Step 2: Python Backend Receives ```python # Python (Backend) -nats_msg = await nats_consumer.next() -env = await smartreceive(str(nats_msg.payload)) +transport_msg = await transport_consumer.next() +env = await smartreceive(str(transport_msg.payload)) # env["payloads"] is now: # [ @@ -733,8 +734,8 @@ env = await smartreceive(str(nats_msg.payload)) ```julia # Julia (Backend) -nats_msg = NATS.subscription.next() -env = smartreceive(String(nats_msg.payload)) +transport_msg = transport_subscription.next() +env = smartreceive(String(transport_msg.payload)) # env["payloads"] is now: # [ @@ -795,7 +796,7 @@ await msghandler.smartsend( | File server unavailable | `UPLOAD_FAILED` | Fall back to direct transport or smaller payloads | | File server download fails | `DOWNLOAD_FAILED` | Retry with exponential backoff | | Payload type mismatch | `DESERIALIZATION_ERROR` | Validate payload_type matches data | -| NATS connection lost | `NATS_CONNECTION_FAILED` | NATS client auto-reconnects | +| Transport connection lost | `TRANSPORT_CONNECTION_FAILED` | Transport client auto-reconnects | ### Error Response Format @@ -828,14 +829,14 @@ correlation_id = string(uuid4()) # Use throughout the flow log_trace(correlation_id, "Starting smartsend") log_trace(correlation_id, "Serialized payload size: 100 bytes") -log_trace(correlation_id, "Published to NATS") +log_trace(correlation_id, "Published to transport") ``` **Log Format**: ``` [2026-03-13T16:30:00.000Z] [Correlation: abc123...] Starting smartsend [2026-03-13T16:30:00.001Z] [Correlation: abc123...] Serialized payload size: 100 bytes -[2026-03-13T16:30:00.002Z] [Correlation: abc123...] Published to NATS +[2026-03-13T16:30:00.002Z] [Correlation: abc123...] Published to transport ``` --- @@ -846,7 +847,7 @@ log_trace(correlation_id, "Published to NATS") | Strategy | Description | When to Use | |----------|-------------|-------------| -| Pre-create NATS connection | Reuse connection for multiple sends | High-throughput scenarios | +| Pre-create transport connection | Reuse connection for multiple sends | High-throughput scenarios | | Adjust size threshold | Increase threshold if file server slow | File server bottleneck | | Use direct transport | Avoid file server for small payloads | Low latency requirements | @@ -868,7 +869,7 @@ log_trace(correlation_id, "Published to NATS") | Component | Minimum | Notes | |-----------|---------|-------| -| NATS Server | 1 instance | Single node for development | +| Message Broker | 1 instance | Single node for development | | File Server | 1 instance | HTTP server for large payloads | | Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) | | Client Memory | 256KB | MicroPython devices | @@ -877,7 +878,7 @@ log_trace(correlation_id, "Published to NATS") | Variable | Default | Description | |----------|---------|-------------| -| `NATS_URL` | `nats://localhost:4222` | NATS server URL | +| `BROKER_URL` | `ws://localhost:4222` | Message broker URL | | `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL | | `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) | @@ -911,7 +912,7 @@ log_trace(correlation_id, "Published to NATS") |------|----------|----------|---------------------------| | [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) | | [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) | -| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | specification.md:2-19 (all sections) | +| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | specification.md:2-19 (all sections) | | [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) | | [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) | | [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe, file upload helpers | specification.md:2-19 (all sections) | @@ -923,6 +924,10 @@ log_trace(correlation_id, "Published to NATS") | Date | Version | Changes | Specification Reference | |------|---------|---------|------------------------| +| 2026-05-15 | 1.5.0 | Made transport layer agnostic | All sections | +| - | - | Removed all NATS-specific references from walkthrough | All sections | +| - | - | Updated code examples to use transport-agnostic patterns | All sections | +| - | - | Updated diagrams to remove NATS-specific labels | All sections | | 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections | | - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 | | - | - | Added `plik_upload_file` convenience function documentation | specification.md:13 | @@ -932,8 +937,8 @@ log_trace(correlation_id, "Published to NATS") | - | - | Added Rust user scenario (User Scenario 4) | specification.md:11 (Rust API) | | - | - | Updated scenario numbering (MicroPython → Scenario 5, Cross-Platform → Scenario 6) | All sections | | 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All sections | -| - | - | Updated smartreceive calls to use String(nats_msg.payload) pattern | All sections | -| - | - | Removed NATSClient.publish() calls (caller responsible for NATS publishing) | All sections | +| - | - | Updated smartreceive calls to use transport payload pattern | All sections | +| - | - | Removed NATSClient.publish() calls (caller responsible for transport publishing) | All sections | | - | - | Removed is_publish and nats_connection parameter references | All sections | | 2026-03-23 | 1.0.0 | Updated to ASG Framework walkthrough guidelines | All sections | | 2026-03-13 | 1.0.0 | Initial walkthrough documentation | specification.md:2-19 (all sections) |