ton/msghandler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

update docs

Browse Source
This commit is contained in:
narawat
2026-05-15 13:25:48 +07:00
parent df9012e0eb
commit d0910ccc3f
4 changed files with 206 additions and 205 deletions
Download Patch File Download Diff File Expand all files Collapse all files

127
docs/architecture.md
View File

@@ -10,7 +10,7 @@
## 1. Executive Summary
This document defines the **blueprint** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus.
This document defines the **blueprint** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer.
This architecture document serves as the single source of truth for:
- **System Structure**: How components fit together and interact
@@ -47,26 +47,25 @@ This architecture document serves as the single source of truth for:
```mermaid
flowchart TD
subgraph "External Systems"
NATS_Server[NATS Server]
subgraph "External Systems"
Message_Broker[Message Broker<br/>NATS/MQTT/WebSocket/Custom]
File_Server[HTTP File Server<br/>Plik/AWS S3/Custom]
end
subgraph "Client Applications"
Julia_App[Julia Application]
JS_App[JavaScript Application<br/>Node.js/Browser]
Python_App[Python Application<br/>Desktop]
Dart_App[Dart Application<br/>Desktop/Flutter/Web]
Rust_App[Rust Application<br/>Server/Desktop]
MicroPython_App[MicroPython Device]
Julia_App[Julia Application]
JS_App[JavaScript Application<br/>Node.js/Browser]
Python_App[Python Application<br/>Desktop]
Dart_App[Dart Application<br/>Desktop/Flutter/Web]
Rust_App[Rust Application<br/>Server/Desktop]
MicroPython_App[MicroPython Device]
end
Julia_App -->|NATS| NATS_Server
JS_App -->|NATS| NATS_Server
Python_App -->|NATS| NATS_Server
Dart_App -->|NATS| NATS_Server
Rust_App -->|NATS| NATS_Server
MicroPython_App -->|NATS| NATS_Server
Julia_App -->|Transport| Message_Broker
JS_App -->|Transport| Message_Broker
Python_App -->|Transport| Message_Broker
Dart_App -->|Transport| Message_Broker
Rust_App -->|Transport| Message_Broker
MicroPython_App -->|Transport| Message_Broker
Julia_App -->|HTTP| File_Server
JS_App -->|HTTP| File_Server
@@ -75,7 +74,7 @@ flowchart TD
Rust_App -->|HTTP| File_Server
MicroPython_App -->|HTTP| File_Server
style NATS_Server fill:#fff3e0,stroke:#f57c00
style Message_Broker fill:#fff3e0,stroke:#f57c00
style File_Server fill:#f3e5f5,stroke:#9c27b4
style Julia_App fill:#e8f5e9,stroke:#4caf50
style JS_App fill:#e3f2fd,stroke:#2196f3
@@ -98,14 +97,14 @@ flowchart TD
MicroPython_Module[MicroPython msghandler Module]
end
Julia_Module --> NATS_Client
JS_Module --> NATS_Client
Python_Module --> NATS_Client
Dart_Module --> NATS_Client
Rust_Module --> NATS_Client
MicroPython_Module --> NATS_Client
Julia_Module --> Transport_Client
JS_Module --> Transport_Client
Python_Module --> Transport_Client
Dart_Module --> Transport_Client
Rust_Module --> Transport_Client
MicroPython_Module --> Transport_Client
NATS_Client --> NATS_Broker
Transport_Client --> Message_Broker
Julia_Module --> File_Client
JS_Module --> File_Client
@@ -122,7 +121,7 @@ flowchart TD
style Dart_Module fill:#fff0f6,stroke:#e91e63
style Rust_Module fill:#dea584,stroke:#e65100
style MicroPython_Module fill:#fce4ec,stroke:#e91e63
style NATS_Broker fill:#fff3e0,stroke:#f57c00
style Message_Broker fill:#fff3e0,stroke:#f57c00
style File_Server fill:#f3e5f5,stroke:#9c27b4
```
@@ -174,8 +173,8 @@ flowchart TD
| Component | Purpose | Platform Support |
|-----------|---------|------------------|
| **smartsend** | Send data via NATS with automatic transport selection, returns (envelope, json_string) for caller to publish | All |
| **smartreceive** | Receive and process NATS messages from JSON string | All |
| **smartsend** | Send data with automatic transport selection, returns (envelope, json_string) for caller to publish via transport | All |
| **smartreceive** | Receive and process messages from JSON string | All |
| **_serialize_data** | Serialize data according to payload type | All |
| **_deserialize_data** | Deserialize bytes to native data types | All |
| **envelope_to_json** | Convert msg_envelope_v1 struct to JSON string | All |
@@ -224,7 +223,7 @@ struct msg_envelope_v1
msg_id::String # UUID v4 for this message
timestamp::String # ISO 8601 UTC timestamp
send_to::String # NATS subject to publish to
send_to::String # Topic/subject to publish to
msg_purpose::String # ACK, NACK, updateStatus, shutdown, chat
sender_name::String # Sender application name
sender_id::String # UUID v4 of sender
@@ -233,7 +232,7 @@ struct msg_envelope_v1
reply_to::String # Topic for reply messages
reply_to_msg_id::String # Message ID being replied to
broker_url::String # NATS broker URL
broker_url::String # Broker URL for the transport layer
metadata::Dict{String, Any} # Message-level metadata
payloads::Vector{msg_payload_v1} # List of payloads
@@ -436,13 +435,13 @@ JavaScript uses async/await for non-blocking I/O:
#### Node.js Implementation (msghandler_ssr.js)
- **TCP NATS connections**: Uses `nats://` or `tls://` URLs
- **Transport connections**: Uses broker URLs (e.g., `nats://`, `mqtt://`, `ws://`)
- **Apache Arrow IPC**: Full support via `apache-arrow`
- **Buffer for binary data**: Native Node.js Buffer handling
#### Browser Implementation (msghandler_csr.js)
- **WebSocket NATS connections**: Uses `ws://` or `wss://` URLs via `nats.ws`
- **WebSocket connections**: Uses `ws://` or `wss://` URLs (transport-agnostic)
- **No Apache Arrow**: Uses `jsontable` for tabular data only
- **Uint8Array for binary data**: Browser-compatible binary handling
- **Web Crypto API**: UUID generation via `crypto.getRandomValues()`
@@ -474,7 +473,7 @@ Dart uses classes for stateful operations with async/await:
- **Async/await**: I/O operations
- **dart-arrow**: Arrow IPC support (Desktop/Flutter only)
- **HTTP package**: HTTP file server communication
- **nats package**: NATS client with WebSocket support (Dart Web)
- **Transport package**: Transport client with WebSocket support (Dart Web)
```dart
class msghandler {
@@ -484,7 +483,7 @@ class msghandler {
final String fileserverUrl;
msghandler({
this.brokerUrl = 'nats://localhost:4222',
this.brokerUrl = DEFAULT_BROKER_URL,
this.fileserverUrl = 'http://localhost:8080',
});
}
@@ -492,19 +491,19 @@ class msghandler {
#### Dart Desktop (Dart SDK)
- **TCP NATS connections**: Uses `nats://` or `tls://` URLs
- **Transport connections**: Uses broker URLs (e.g., `nats://`, `mqtt://`)
- **Apache Arrow IPC**: Full support via `dart-arrow`
- **Uint8List for binary data**: Native Dart binary handling
#### Dart Flutter (Dart SDK)
- **TCP NATS connections**: Uses `nats://` or `tls://` URLs
- **Transport connections**: Uses broker URLs (e.g., `nats://`, `mqtt://`)
- **Apache Arrow IPC**: Full support via `dart-arrow`
- **Uint8List for binary data**: Native Dart binary handling
#### Dart Web (Dart SDK)
- **WebSocket NATS connections**: Uses `ws://` or `wss://` URLs via `nats` package
- **WebSocket connections**: Uses `ws://` or `wss://` URLs (transport-agnostic)
- **No Apache Arrow**: Uses `jsontable` for tabular data only
- **Uint8List for binary data**: Browser-compatible binary handling
- **Fetch API**: HTTP file server communication via `http` package
@@ -516,7 +515,7 @@ Browser JavaScript has specific constraints due to security and compatibility:
- **Async/await**: Native async/await support
- **No Apache Arrow**: Arrow IPC not available in browsers
- **JSON table only**: Use "jsontable" for tabular data
- **WebSocket NATS**: Uses nats.ws for browser-compatible NATS connections
- **WebSocket transport**: Uses transport client for browser-compatible connections
- **Fetch API**: HTTP file server communication via fetch
### MicroPython Architecture
@@ -540,7 +539,7 @@ Rust leverages compile-time type safety and async runtimes:
- **Type-safe payloads**: Rust enum discriminates between `Text`, `Dictionary`, `ArrowTable`, `Binary`, etc.
- **serde serialization**: Automatic JSON deserialization via `#[derive(Serialize, Deserialize)]`
- **tokio runtime**: Efficient async I/O for NATS connections and HTTP file server operations
- **tokio runtime**: Efficient async I/O for transport connections and HTTP file server operations
- **arrow2 integration**: Native Arrow IPC deserialization without intermediate format conversion
- **reqwest**: High-performance HTTP client with built-in TLS and connection pooling
- **Zero-copy patterns**: `Vec<u8>` passed directly to avoid unnecessary memory copies
@@ -572,8 +571,8 @@ pub struct SmartsendOptions {
// ... other fields
}
// NATS client with tokio integration
let conn = nats::connect("nats://localhost:4222").await?;
// Transport client with tokio integration
let conn = transport_client::connect(DEFAULT_BROKER_URL).await?;
// Subscribe and process messages
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze")?;
@@ -599,7 +598,7 @@ for msg in sub.messages() {
| Component | Scaling Strategy |
|-----------|------------------|
| **NATS Server** | Cluster deployment with multiple nodes |
| **Message Broker** | Cluster deployment with multiple nodes |
| **File Server** | Load balancer + multiple instances |
| **Client Applications** | Deploy multiple instances behind load balancer |
@@ -607,7 +606,7 @@ for msg in sub.messages() {
| Component | Scaling Strategy |
|-----------|------------------|
| **NATS Server** | Increase memory, CPU, disk I/O |
| **Message Broker** | Increase memory, CPU, disk I/O |
| **File Server** | Increase memory, CPU, disk capacity |
| **Client Applications** | Increase heap size (Python/JS) |
@@ -617,7 +616,7 @@ for msg in sub.messages() {
|--------|--------|-------|
| Message serialization overhead | <50ms | For 10KB payload |
| Message deserialization overhead | <50ms | For 10KB payload |
| NATS connection establishment | <100ms | Connection pool recommended |
| Transport connection establishment | <100ms | Connection pool recommended |
| File upload latency | <1s | For 0.5MB file |
| File download latency | <1s | For 0.5MB file |
@@ -625,16 +624,16 @@ for msg in sub.messages() {
## Failure Modes and Recovery
### NATS Connection Failure
### Transport Connection Failure
**Scenario**: NATS server unavailable
**Scenario**: Message broker unavailable
**Handler**:
- Connection auto-reconnect via TCP-level reconnection
- Connection auto-reconnect via transport-level reconnection
- Retry with exponential backoff for publish operations
**Recovery**:
- NATS client automatically attempts reconnection
- Transport client automatically attempts reconnection
- Application can check connection status before publishing
### File Server Unavailable
@@ -697,7 +696,7 @@ for msg in sub.messages() {
**Rationale**:
- Simplifies JSON serialization (all data is string-compatible)
- Increases payload size by ~33%, but NATS can handle this
- Increases payload size by ~33%, but transport can handle this
- Alternative would be binary payload support (more complex)
### Decision 3: Multiple Platform Implementations
@@ -730,7 +729,7 @@ for msg in sub.messages() {
| Component | Minimum | Notes |
|-----------|---------|-------|
| NATS Server | 1 instance | Single node for development |
| Message Broker | 1 instance | Single node for development |
| File Server | 1 instance | HTTP server for large payloads |
| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices |
@@ -739,7 +738,7 @@ for msg in sub.messages() {
| Variable | Default | Description |
|----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL |
| `BROKER_URL` | `ws://localhost:4222` | Message broker URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) |
@@ -748,15 +747,15 @@ for msg in sub.messages() {
```mermaid
flowchart TD
subgraph "Docker Network"
NATS_Container[NATS Server]
Broker_Container[Message Broker]
FileServer_Container[Plik File Server]
App_Container[Application Container]
end
App_Container -->|NATS| NATS_Container
App_Container -->|Transport| Broker_Container
App_Container -->|HTTP| FileServer_Container
style NATS_Container fill:#fff3e0,stroke:#f57c00
style Broker_Container fill:#fff3e0,stroke:#f57c00
style FileServer_Container fill:#f3e5f5,stroke:#9c27b4
style App_Container fill:#e3f2fd,stroke:#2196f3
```
@@ -775,11 +774,12 @@ flowchart TD
### Transport Security
**Mechanism**: TLS support for NATS connections
**Mechanism**: TLS support for transport connections
**Implementation**:
- Use `nats://` URL for plain text
- Use `tls://` URL for TLS-encrypted connections
- Use `ws://` or `wss://` for WebSocket connections
### File Server Security
@@ -835,6 +835,11 @@ flowchart TD
| Date | Version | Changes |
|------|---------|---------|
| 2026-05-15 | 1.5.0 | Made transport layer agnostic | All sections |
| - | - | Removed all NATS-specific references from architecture docs | All sections |
| - | - | Updated diagrams to use generic "Message Broker" instead of "NATS Server" | All sections |
| - | - | Updated code examples to use transport-agnostic patterns | All sections |
| - | - | Removed NATS client packages from external dependencies | All sections |
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections |
| - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
| - | - | Added `plik_upload_file` convenience function to component table | specification.md:13 |
@@ -849,6 +854,7 @@ flowchart TD
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) |
| - | - | Removed publish_message component (commented out in source) |
| - | - | Removed NATSClient and NATSConnectionPool classes (not in ground truth) |
| - | - | Updated smartsend to return JSON for caller to publish via transport |
| - | - | Updated component diagram to match actual module structure |
| - | - | Updated data flow to show smartsend returns JSON for caller to publish |
| - | - | Fixed SIZE_THRESHOLD default to 500,000 bytes |
@@ -856,6 +862,7 @@ flowchart TD
| - | - | Added NATSClient with keepAlive support |
| - | - | Added NATSConnectionPool for connection reuse |
| - | - | Added publishMessage function with closeConnection option |
| (Historical - pre-transport-agnostic refactor) | | |
| 2026-03-13 | 1.0.0 | Initial architecture documentation |
---
@@ -880,7 +887,7 @@ flowchart TD
|------|----------|----------|---------------------------|-------------------|
| [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe, file upload helpers | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
@@ -889,24 +896,18 @@ flowchart TD
### 16.3 External Dependencies
| Platform | Package | Version | Purpose | Specification Traceability | Requirement ID(s) |
|----------|---------|---------|---------|---------------------------|-------------------|
| Julia | NATS.jl | Latest | NATS client | specification.md:11 | FR-013, FR-014, NFR-201 |
|----------|---------|---------|---------|--------------------------|-------------------|
| Julia | JSON.jl | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Julia | Arrow.jl | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Julia | HTTP.jl | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Julia | UUIDs.jl | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 |
| Node.js | nats | Latest | NATS client (TCP) | specification.md:11 | FR-013, FR-014 |
| Node.js | node-fetch | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Browser | nats.ws | Latest | NATS client (WebSocket) | specification.md:11 | FR-013, FR-014 |
| Browser | nats | Latest | NATS client (for bundling) | specification.md:11 | FR-013, FR-014 |
| Python | nats-py | Latest | NATS client | specification.md:11 | FR-013, FR-014 |
| Browser | - | - | Transport-agnostic (caller provides) | specification.md:11 | FR-013, FR-014 |
| Python | aiohttp | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Python | pyarrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Dart | nats | Latest | NATS client | specification.md:11 | FR-013, FR-014 |
| Dart | http | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Dart | uuid | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 |
| Dart | dart-arrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Rust | nats | Latest | NATS client | specification.md:11 | FR-013, FR-014 |
| Rust | serde | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Rust | serde_json | Latest | JSON handling | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Rust | tokio | Latest | Async runtime | specification.md:11 | FR-013, FR-014 |