ton/msghandler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

update docs

Browse Source
This commit is contained in:
narawat
2026-05-15 13:25:48 +07:00
parent df9012e0eb
commit d0910ccc3f
4 changed files with 206 additions and 205 deletions
Download Patch File Download Diff File Expand all files Collapse all files

75
docs/requirements.md
View File

@@ -11,7 +11,7 @@
### 1.1 Business Goal
msghandler is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS.
msghandler is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over the transport layer.
### 1.2 User Stories (with acceptance criteria)
@@ -19,13 +19,13 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
|-------|----------|---------------------|
| **As a Julia developer**, I want to send text messages to JavaScript/Dart applications that lives on a server and also on a browser | P1 | Text messages are serialized, encoded, and received correctly across platforms |
| **As a Python developer**, I want to send tabular data to Julia/Dart applications | P1 | DataFrame exchange works with both Arrow IPC and JSON formats |
| **As a JavaScript developer**, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS |
| **As a JavaScript developer**, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via the transport layer |
| **As a Dart developer**, I want to send text messages to other platforms | P1 | Text messages are serialized, encoded, and received correctly across platforms |
| **As a Dart developer**, I want to send dictionary data to other platforms | P1 | JSON-serializable data is exchanged correctly |
| **As a Dart developer**, I want to send tabular data (List<Map>) to other platforms | P1 | JSON table format exchange works with Arrow IPC on desktop |
| **As a Dart developer**, I want to send large files (>0.5MB) | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS |
| **As a Dart developer**, I want to send large files (>0.5MB) | P1 | Large files are automatically uploaded to file server and URLs are sent via the transport layer |
| **As a MicroPython developer**, I want to send sensor data with minimal memory usage | P1 | Direct transport works for payloads <100KB on memory-constrained devices |
| **As a Rust developer**, I want to send and receive messages with type-safe APIs | P1 | Rust implementation uses serde for serialization, tokio for async, and nats-io for NATS connectivity |
| **As a Rust developer**, I want to send and receive messages with type-safe APIs | P1 | Rust implementation uses serde for serialization, tokio for async, and transport-agnostic client for connectivity |
| **As a developer**, I want to send mixed-content messages (text + image + file) | P1 | msghandler accepts list of (dataname, data, type) tuples and handles each payload appropriately |
| **As a developer**, I want to receive multi-payload messages | P1 | msghandler returns payloads as list of tuples with correct types preserved |
| **As a developer**, I want to use Plik as the file server | P2 | Plik one-shot upload mode is supported with upload ID and token handling |
@@ -59,37 +59,32 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| File server integration | Plik one-shot upload and custom HTTP server support |
| Reliability features | Exponential backoff retry and correlation ID propagation |
| Message serialization | Converts data types to binary format (Base64, JSON, Arrow IPC) |
| NATS communication | Publishing and subscription via NATS subjects |
| Transport communication | Publishing and subscription via message broker (NATS, MQTT, WebSocket, etc.) |
### 2.2 Out of Scope
| Feature | Reason |
|---------|--------|
| NATS JetStream support | Core NATS sufficient for current use cases |
| Advanced transport features | Basic transport sufficient for current use cases |
| Message compression | Compression adds complexity without clear benefit |
| Message encryption | Payload encryption is application-layer concern |
| Persistent message queues | NATS request-reply pattern sufficient |
| Advanced routing rules | Simple NATS subject matching sufficient |
| Persistent message queues | Transport request-reply pattern sufficient |
| Advanced routing rules | Simple topic matching sufficient |
### 2.3 Dependencies
| Platform | Package | Version |
|----------|---------|---------|
| Julia | NATS.jl | Latest stable |
| Julia | JSON.jl | Latest stable |
| Julia | Arrow.jl | Latest stable |
| Julia | HTTP.jl | Latest stable |
| Julia | UUIDs.jl | Latest stable |
| Node.js | nats | Latest stable |
| Node.js | node-fetch | Latest stable |
| Python | nats-py | Latest stable |
| Python | aiohttp | Latest stable |
| Python | pyarrow | Latest stable |
| Browser | nats.ws | Latest stable |
| Dart | nats | Latest stable |
| Browser | - | Transport-agnostic (caller provides) |
| Dart | http | Latest stable |
| Dart | uuid | Latest stable |
| Rust | nats | Latest stable |
| Rust | serde | Latest stable |
| Rust | serde_json | Latest stable |
| Rust | tokio | Latest stable |
@@ -100,7 +95,7 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| Platform | Minimum Version | Notes |
|----------|-----------------|-------|
| Julia | 1.7+ | Arrow.jl required for arrowtable support |
| Node.js | 16+ | nats.js required, Arrow IPC supported |
| Node.js | 16+ | Transport client required, Arrow IPC supported |
| Python | 3.8+ | pyarrow required for arrowtable support |
| Browser | Latest | No Arrow IPC (uses jsontable only) |
| Dart | 2.17+ | Supports Desktop (Dart SDK), Flutter (Dart SDK), and Web (Dart SDK) |
@@ -115,8 +110,8 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
|----|-------------|-------------|
| **FR-001** | Cross-platform text messaging | System shall allow users to send text messages between Julia, JavaScript, Python, and MicroPython applications |
| **FR-002** | Cross-platform tabular data | System shall support DataFrame exchange between Julia and Python applications using Arrow IPC format |
| **FR-003** | Large file handling | System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via NATS |
| **FR-004** | Direct transport for small payloads | System shall send payloads <0.5MB directly via NATS without file server upload |
| **FR-003** | Large file handling | System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via transport |
| **FR-004** | Direct transport for small payloads | System shall send payloads <0.5MB directly via transport without file server upload |
| **FR-005** | MicroPython support | System shall support payloads <100KB on MicroPython devices using direct transport |
| **FR-006** | Multi-payload messages | System shall accept and process lists of (dataname, data, type) tuples |
| **FR-007** | Payload type preservation | System shall preserve payload types when returning multi-payload messages |
@@ -125,8 +120,8 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| **FR-010** | Exponential backoff retry | System shall implement exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) for file server download failures |
| **FR-011** | Correlation ID propagation | System shall propagate correlation IDs through all message processing steps |
| **FR-012** | Message serialization | System shall serialize data types using Base64, JSON, or Arrow IPC encoding |
| **FR-013** | NATS publishing | System shall return JSON string representation for caller to publish to NATS subjects (caller is responsible for actual NATS publish) |
| **FR-014** | NATS subscription | System shall receive and process NATS messages by accepting JSON string from NATS payload |
| **FR-013** | Transport publishing | System shall return JSON string representation for caller to publish via transport layer (caller is responsible for actual transport publish) |
| **FR-014** | Transport subscription | System shall receive and process messages by accepting JSON string from transport payload |
---
@@ -138,10 +133,10 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
|----|-------------|---------------|-------------|
| **NFR-101** | Message serialization overhead | <50ms for 10KB payload | Benchmark tests |
| **NFR-102** | Message deserialization overhead | <50ms for 10KB payload | Benchmark tests |
| **NFR-103** | NATS connection establishment | <100ms | Connection pool benchmarks |
| **NFR-103** | Transport connection establishment | <100ms | Connection pool benchmarks |
| **NFR-104** | File upload latency | <1s for 0.5MB file | Integration tests |
| **NFR-105** | File download latency | <1s for 0.5MB file | Integration tests |
| **NFR-106** | Concurrent connections | Support 100+ simultaneous NATS connections | Scale testing |
| **NFR-106** | Concurrent connections | Support 100+ simultaneous transport connections | Scale testing |
| **NFR-107** | Message throughput | Handle 1000+ messages/second per instance | Load testing |
| **NFR-108** | File server scalability | Support horizontal scaling of file server backend | Architecture review |
@@ -149,16 +144,16 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| ID | Requirement | Specification |
|----|-------------|---------------|
| **NFR-201** | Message delivery | At-least-once delivery semantics via NATS |
| **NFR-201** | Message delivery | At-least-once delivery semantics via transport |
| **NFR-202** | File server availability | Graceful degradation when file server is unavailable |
| **NFR-203** | Connection recovery | Auto-reconnect on NATS connection failure |
| **NFR-203** | Connection recovery | Auto-reconnect on transport connection failure |
### 4.3 Privacy & Security
| ID | Requirement | Specification |
|----|-------------|---------------|
| **NFR-301** | Payload integrity | SHA-256 checksum support via metadata |
| **NFR-302** | Transport security | TLS support for NATS connections |
| **NFR-302** | Transport security | TLS support for transport connections |
| **NFR-303** | File server security | Authentication token for file uploads |
### 4.4 Observability & Telemetry
@@ -234,11 +229,11 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| Platform | Maximum | Notes |
|----------|---------|-------|
| Desktop | Unlimited | Limited by NATS server configuration |
| Dart Desktop | Unlimited | Limited by NATS server configuration |
| Dart Flutter | Unlimited | Limited by NATS server configuration |
| Dart Web | Unlimited | Limited by NATS server configuration |
| Rust | Unlimited | Limited by NATS server configuration |
| Desktop | Unlimited | Limited by transport server configuration |
| Dart Desktop | Unlimited | Limited by transport server configuration |
| Dart Flutter | Unlimited | Limited by transport server configuration |
| Dart Web | Unlimited | Limited by transport server configuration |
| Rust | Unlimited | Limited by transport server configuration |
| MicroPython | 50KB | Hard limit due to 256KB-1MB memory |
---
@@ -252,7 +247,7 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| `correlation_id` | String (UUID) | Track message flow across systems |
| `msg_id` | String (UUID) | Unique message identifier |
| `timestamp` | String (ISO 8601) | Message publication timestamp |
| `send_to` | String | NATS subject to publish to |
| `send_to` | String | Topic/subject to publish to |
| `msg_purpose` | String | ACK, NACK, updateStatus, shutdown, chat |
| `sender_name` | String | Sender application name |
| `sender_id` | String (UUID) | Sender unique identifier |
@@ -260,7 +255,7 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| `receiver_id` | String (UUID) | Receiver unique identifier (empty = broadcast) |
| `reply_to` | String | Topic for reply messages |
| `reply_to_msg_id` | String | Message ID being replied to |
| `broker_url` | String | NATS server URL |
| `broker_url` | String | Broker URL for the transport layer |
| `metadata` | Dict | Message-level metadata |
| `payloads` | Array | List of payload objects |
@@ -289,14 +284,14 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
| `Failed to upload` | File server error | Throw error |
| `Failed to fetch` | File server unavailable | Retry with exponential backoff |
| `Unknown transport` | Invalid transport type | Throw error |
| `NATS connection failed` | NATS unavailable | Throw error |
| `Transport connection failed` | Transport/broker unavailable | Throw error |
### 9.2 Exception Handling
| Scenario | Handler |
|----------|---------|
| File server unavailable | Retry up to 5 times with exponential backoff |
| NATS publish failure | Connection auto-reconnect |
| Transport publish failure | Handled by caller |
| Deserialization error | Log correlation ID and throw error |
| Memory overflow (MicroPython) | Reject payloads >50KB |
@@ -350,7 +345,7 @@ function smartsend(
)::Tuple{msg_envelope_v1, String} where {T1<:Any}
```
**Note**: NATS publishing is the caller's responsibility. `smartsend` returns `(env::msg_envelope_v1, env_json_str::String)`.
**Note**: Publishing via the transport layer is the caller's responsibility. `smartsend` returns `(env::msg_envelope_v1, env_json_str::String)`.
### 11.2 smartreceive Signature
@@ -364,7 +359,9 @@ function smartreceive(
)::JSON.Object{String, Any}
```
**Note**: Pass `String(nats_msg.payload)` from NATS subscription to `smartreceive`.
**Note**: Pass the payload string from the transport subscription to `smartreceive`. The input is the JSON string payload from the transport message, not the transport message object directly.
**Note**: Pass the payload from the transport subscription to `smartreceive`.
---
@@ -374,7 +371,7 @@ function smartreceive(
| Component | Minimum | Notes |
|-----------|---------|-------|
| NATS Server | 1 instance | Single node for development |
| Message Broker | 1 instance | Single node for development |
| File Server | 1 instance | HTTP server for large payloads |
| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices |
@@ -383,7 +380,7 @@ function smartreceive(
| Variable | Default | Description |
|----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL |
| `BROKER_URL` | `ws://localhost:4222` | Message broker URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) |
@@ -409,6 +406,10 @@ function smartreceive(
| Date | Version | Changes |
|------|---------|---------|
| 2026-05-15 | 1.3.0 | Made transport layer agnostic |
| - | - | Removed all NATS-specific dependencies and references |
| - | - | Updated all NATS references to generic "transport layer"/"message broker" |
| - | - | Removed NATS client packages from dependencies tables |
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) |
| - | - | Fixed smartsend signature: removed is_publish, NATS_connection; added sender_name |
| - | - | Fixed smartreceive signature: takes msg_json_str::String instead of msg::NATS.Msg |