ton/msghandler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

update docs

Browse Source
This commit is contained in:
narawat
2026-05-15 13:25:48 +07:00
parent df9012e0eb
commit d0910ccc3f
4 changed files with 206 additions and 205 deletions
Download Patch File Download Diff File Expand all files Collapse all files

114
docs/specification.md
View File

@@ -10,7 +10,7 @@
## 1. Technical Contract Overview
This document defines the **technical contract** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus.
This document defines the **technical contract** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer.
This specification serves as the single source of truth for:
- **Inputs**: What data structures are accepted by `smartsend()`
@@ -28,7 +28,7 @@ This specification serves as the single source of truth for:
| Section 5 (Enumerations) | FR-003, FR-004, FR-006, NFR-101 | Enumerations for transport and encoding |
| Section 6 (Transport Protocols) | FR-003, FR-004, NFR-104, NFR-105 | Direct and link transport protocols |
| Section 7 (Size Thresholds) | FR-004, FR-005, NFR-104, NFR-105 | Size thresholds for transport selection |
| Section 8 (NATS Subject Convention) | FR-013, FR-014 | NATS subject naming patterns |
| Section 8 (Topic Convention) | FR-013, FR-014 | Topic/subject naming patterns |
| Section 9 (Error Handling) | FR-010, FR-011, NFR-201, NFR-202, NFR-203 | Error codes and exception handling |
| Section 10 (Serialization Rules) | FR-001, FR-002, FR-003, FR-012, NFR-101, NFR-102 | Serialization and encoding rules |
| Section 11 (API Contract) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Function signatures for all platforms |
@@ -95,7 +95,7 @@ This specification serves as the single source of truth for:
| `correlation_id` | `string` | Yes | UUID v4 format | Track message flow across distributed systems | FR-011, NFR-401 |
| `msg_id` | `string` | Yes | UUID v4 format | Unique identifier for this specific message | FR-012, NFR-401 |
| `timestamp` | `string` | Yes | ISO 8601 UTC | Message publication timestamp | FR-012, NFR-401 |
| `send_to` | `string` | Yes | Non-empty string | NATS subject/topic to publish the message to | FR-013 |
| `send_to` | `string` | Yes | Non-empty string | Topic/subject to publish the message to | FR-013 |
| `msg_purpose` | `string` | Yes | Enum | Purpose of the message | FR-013 |
| `sender_name` | `string` | Yes | Non-empty string | Name of the sender application | FR-013 |
| `sender_id` | `string` | Yes | UUID v4 format | Unique identifier for the sender | FR-013 |
@@ -103,7 +103,7 @@ This specification serves as the single source of truth for:
| `receiver_id` | `string` | Yes | Any string | UUID of the receiver (empty = broadcast) | FR-013 |
| `reply_to` | `string` | Yes | Any string | Topic where receiver should reply | FR-013 |
| `reply_to_msg_id` | `string` | Yes | Any string | Message ID this message is replying to | FR-013 |
| `broker_url` | `string` | Yes | Valid URL | NATS broker URL | FR-013 |
| `broker_url` | `string` | Yes | Valid URL | Broker URL for the transport layer | FR-013 |
| `metadata` | `object` | No | Any JSON object | Message-level metadata | NFR-401 |
| `payloads` | `array` | Yes | Non-empty array | List of payload objects | FR-012, FR-013 |
@@ -240,7 +240,7 @@ await smartsend("/agent/v1/process", data)
| Value | Description | Data Format | Use Case |
|-------|-------------|-------------|----------|
| `direct` | Payload sent directly via NATS | Base64-encoded string | Payloads < size_threshold |
| `direct` | Payload sent directly via the transport layer | Base64-encoded string | Payloads < size_threshold |
| `link` | Payload uploaded to file server | HTTP URL | Payloads ≥ size_threshold |
### `encoding` Enum
@@ -312,7 +312,7 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa
---
## NATS Subject Convention
## Topic Convention
### Subject Naming Pattern
@@ -357,7 +357,7 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa
| `INVALID_TRANSPORT` | 400 | Unsupported transport type | Use `direct` or `link` | FR-003, FR-004, FR-006 |
| `UPLOAD_FAILED` | 500 | File server upload failed | Retry or use direct transport | FR-008, FR-009 |
| `DOWNLOAD_FAILED` | 503 | File server download failed | Retry with exponential backoff | FR-010, FR-011, NFR-201, NFR-202 |
| `NATS_CONNECTION_FAILED` | 503 | NATS connection failed | Check NATS server availability | FR-013, FR-014, NFR-201, NFR-203 |
| `TRANSPORT_CONNECTION_FAILED` | 503 | Transport connection failed | Check broker/server availability | FR-013, FR-014, NFR-201, NFR-203 |
| `DESERIALIZATION_ERROR` | 500 | Payload deserialization failed | Check payload_type matches data | FR-001, FR-002, FR-003, FR-012 |
| `SIZE_EXCEEDED` | 413 | Payload exceeds maximum size | Split payload or use link transport | FR-003, FR-004, FR-005, NFR-104, NFR-105 |
@@ -366,7 +366,7 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa
| Scenario | Handler | Retry Policy | Requirement ID |
|----------|---------|--------------|----------------|
| File server unavailable | Retry up to 5 times | Exponential backoff (100ms → 5000ms) | FR-010, NFR-202 |
| NATS publish failure | Connection auto-reconnect | TCP-level reconnection | FR-013, FR-014, NFR-201, NFR-203 |
| Transport publish failure | Handled by caller | Caller's retry logic | FR-013, FR-014, NFR-201, NFR-203 |
| Deserialization error | Log correlation ID and throw | No retry (data corruption) | FR-001, FR-002, FR-003, FR-012, NFR-401 |
| Memory overflow (MicroPython) | Reject payloads >50KB | No retry (client-side check) | FR-005, NFR-106 |
@@ -435,7 +435,7 @@ function smartsend(
)::Tuple{msg_envelope_v1, String} where {T1<:Any}
```
**Note**: NATS publishing is the caller's responsibility. Returns `(env::msg_envelope_v1, env_json_str::String)`.
**Note**: Publishing via the transport layer is the caller's responsibility. Returns `(env::msg_envelope_v1, env_json_str::String)`.
#### Python
@@ -443,7 +443,7 @@ function smartsend(
async def smartsend(
subject: str,
data: List[Tuple[str, Any, str]],
broker_url: str = "nats://localhost:4222",
broker_url: str = DEFAULT_BROKER_URL,
fileserver_url: str = "http://localhost:8080",
fileserver_upload_handler: Callable = plik_oneshot_upload,
size_threshold: int = 500_000,
@@ -459,7 +459,7 @@ async def smartsend(
) -> Tuple[Dict, str]:
```
**Note**: NATS publishing is the caller's responsibility.
**Note**: Publishing via the transport layer is the caller's responsibility.
#### JavaScript (Node.js)
@@ -485,7 +485,7 @@ async function smartsend(
): Promise<[Object, string]>;
```
**Note**: NATS publishing is the caller's responsibility.
**Note**: Publishing via the transport layer is the caller's responsibility.
#### JavaScript (Browser)
@@ -511,7 +511,7 @@ async function smartsend(
): Promise<[Object, string]>;
```
**Note**: NATS publishing is the caller's responsibility.
**Note**: Publishing via the transport layer is the caller's responsibility.
#### MicroPython
@@ -524,7 +524,7 @@ def smartsend(
) -> Tuple[Dict, str]:
```
**Note**: NATS publishing is the caller's responsibility.
**Note**: Publishing via the transport layer is the caller's responsibility.
#### Dart (Desktop/Flutter)
@@ -532,10 +532,10 @@ def smartsend(
Future<[Map<String, dynamic>, String]> smartsend(
String subject,
List<List<dynamic>> data, {
String brokerUrl = 'nats://localhost:4222',
String fileserverUrl = 'http://localhost:8080',
String brokerUrl = DEFAULT_BROKER_URL,
String fileserverUrl = DEFAULT_FILESERVER_URL,
Function? fileserverUploadHandler,
int sizeThreshold = 500000,
int sizeThreshold = DEFAULT_SIZE_THRESHOLD,
String? correlationId,
String msgPurpose = 'chat',
String senderName = 'msghandler',
@@ -547,9 +547,8 @@ Future<[Map<String, dynamic>, String]> smartsend(
String? senderId,
}) async {
// Returns [envelope, jsonString]
// NATS publishing is caller's responsibility
// Publishing via transport layer is caller's responsibility
}
```
#### Dart Web
@@ -557,7 +556,7 @@ Future<[Map<String, dynamic>, String]> smartsend(
Future<[Map<String, dynamic>, String]> smartsend(
String subject,
List<List<dynamic>> data, {
String brokerUrl = 'nats://localhost:4222',
String brokerUrl = DEFAULT_BROKER_URL,
String fileserverUrl = 'http://localhost:8080',
Function? fileserverUploadHandler,
int sizeThreshold = 500000,
@@ -572,7 +571,7 @@ Future<[Map<String, dynamic>, String]> smartsend(
String? senderId,
}) async {
// Returns [envelope, jsonString]
// NATS publishing is caller's responsibility
// Publishing via transport layer is caller's responsibility
}
```
@@ -635,7 +634,7 @@ pub struct MsgEnvelopeV1 {
}
```
**Note**: NATS publishing is the caller's responsibility. Returns `Result<(MsgEnvelopeV1, String), msghandlerError>`. Uses `serde` for JSON serialization.
**Note**: Publishing via the transport layer is the caller's responsibility. Returns `Result<(MsgEnvelopeV1, String), msghandlerError>`. Uses `serde` for JSON serialization.
### `smartreceive` Function Signature
@@ -643,7 +642,7 @@ pub struct MsgEnvelopeV1 {
```julia
function smartreceive(
msg_json_str::String; # Pass String(nats_msg.payload) from NATS subscription
msg_json_str::String; # Pass payload from transport subscription
fileserver_download_handler::Function = _fetch_with_backoff,
max_retries::Int = 5,
base_delay::Int = 100,
@@ -651,13 +650,13 @@ function smartreceive(
)::JSON.Object{String, Any}
```
**Note**: Input is JSON string from NATS message payload, not NATS.Msg directly.
**Note**: Input is the JSON string payload from the transport subscription, not the transport message object directly.
#### Python
```python
async def smartreceive(
msg_json_str: str, # JSON string from NATS message payload
msg_json_str: str, # JSON string from transport message payload
fileserver_download_handler: Callable = fetch_with_backoff,
max_retries: int = 5,
base_delay: int = 100,
@@ -665,13 +664,13 @@ async def smartreceive(
) -> Dict[str, Any]:
```
**Note**: Input is JSON string from NATS message payload.
**Note**: Input is the JSON string payload from the transport message.
#### JavaScript (Node.js)
```typescript
async function smartreceive(
msg_json_str: string, // JSON string from NATS message payload
msg_json_str: string, // JSON string from transport message payload
options?: {
fileserver_download_handler?: Function;
max_retries?: number;
@@ -685,7 +684,7 @@ async function smartreceive(
```typescript
async function smartreceive(
msg_json_str: string, // JSON string from NATS message payload
msg_json_str: string, // JSON string from transport message payload
options?: {
fileserver_download_handler?: Function;
max_retries?: number;
@@ -695,7 +694,7 @@ async function smartreceive(
): Promise<Object>;
```
**Note**: Input is JSON string from NATS message payload.
**Note**: Input is the JSON string payload from the transport message.
#### MicroPython
@@ -703,13 +702,13 @@ async function smartreceive(
def smartreceive(msg_json_str: str, **kwargs) -> Dict[str, Any]:
```
**Note**: Input is JSON string from NATS message payload.
**Note**: Input is the JSON string payload from the transport message.
#### Dart (Desktop/Flutter)
```dart
Future<Map<String, dynamic>> smartreceive(
Map<String, dynamic> msg_json_str, // JSON object from NATS message payload
Map<String, dynamic> msg_json_str, // JSON object from transport message payload
{
Function? fileserverDownloadHandler,
int maxRetries = 5,
@@ -724,7 +723,7 @@ Future<Map<String, dynamic>> smartreceive(
```dart
Future<Map<String, dynamic>> smartreceive(
Map<String, dynamic> msg_json_str, // JSON object from NATS message payload
Map<String, dynamic> msg_json_str, // JSON object from transport message payload
{
Function? fileserverDownloadHandler,
int maxRetries = 5,
@@ -739,7 +738,7 @@ Future<Map<String, dynamic>> smartreceive(
```rust
pub async fn smartreceive(
msg_json_str: &str, // JSON string from NATS message payload
msg_json_str: &str, // JSON string from transport message payload
options: &SmartreceiveOptions,
) -> Result<MsgEnvelopeV1, msghandlerError>
@@ -752,7 +751,7 @@ pub struct SmartreceiveOptions {
}
```
**Note**: Input is JSON string from NATS message payload. Returns `Result<MsgEnvelopeV1, msghandlerError>`.
**Note**: Input is the JSON string payload from the transport message. Returns `Result<MsgEnvelopeV1, msghandlerError>`.
---
@@ -897,7 +896,7 @@ function fileserver_download_handler(
|------|----------|----------|-------|
| [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | Ground truth implementation |
| [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | Server-side JavaScript |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | Client-side rendering |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | Client-side rendering |
| [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | Desktop Python |
| [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | Desktop/Flutter/Web |
| [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe | Uses tokio + serde + arrow2 |
@@ -910,7 +909,7 @@ The browser implementation ([`src/msghandler_csr.js`](../src/msghandler_csr.js))
| Constraint | Reason | Workaround |
|------------|--------|------------|
| No Apache Arrow IPC | Browser-incompatible dependency | Use `jsontable` for tabular data |
| WebSocket NATS only | Browser cannot use TCP directly | Use `ws://` or `wss://` broker URLs |
| WebSocket only | Browser cannot use TCP directly | Use `ws://` or `wss://` broker URLs |
| Fetch API for HTTP | Browser fetch() API only | Compatible with Plik and other HTTP servers |
### Payload Type Availability by Platform
@@ -941,7 +940,7 @@ flowchart TD
D --> F[Build envelope with metadata]
E --> F
F --> G[Convert envelope to JSON string]
G --> H[Publish to NATS subject]
G --> H[Publish to topic via transport]
H --> I[Return envelope and JSON string to caller]
style A fill:#f9f9f9,stroke:#333
@@ -954,7 +953,7 @@ flowchart TD
```mermaid
flowchart TD
A[NATS message arrives] --> B[Parse JSON envelope]
A[Transport message arrives] --> B[Parse JSON envelope]
B --> C[For each payload: Check transport type]
C -->|transport == direct| D[Direct Transport: Extract Base64]
C -->|transport == link| E[Link Transport: Fetch from URL]
@@ -1046,23 +1045,17 @@ flowchart TD
| Platform | Package | Version | Purpose |
|----------|---------|---------|---------|
| Julia | NATS.jl | Latest | NATS client |
| Julia | JSON.jl | Latest | JSON serialization |
| Julia | Arrow.jl | Latest | Arrow IPC support |
| Julia | HTTP.jl | Latest | HTTP file server |
| Julia | UUIDs.jl | Latest | UUID generation |
| Node.js | nats | Latest | NATS client (TCP) |
| Node.js | node-fetch | Latest | HTTP file server |
| Browser | nats.ws | Latest | NATS client (WebSocket) |
| Browser | nats | Latest | NATS client (for bundling) |
| Python | nats-py | Latest | NATS client |
| Browser | - | - | Transport-agnostic (caller provides) |
| Python | aiohttp | Latest | HTTP file server |
| Python | pyarrow | Latest | Arrow IPC support |
| Dart | nats | Latest | NATS client |
| Dart | http | Latest | HTTP file server |
| Dart | uuid | Latest | UUID generation |
| Dart | dart-arrow | Latest | Arrow IPC support (Desktop/Flutter) |
| Rust | nats | Latest | NATS client |
| Rust | serde | Latest | JSON serialization |
| Rust | serde_json | Latest | JSON handling |
| Rust | tokio | Latest | Async runtime |
@@ -1084,6 +1077,10 @@ flowchart TD
| Date | Version | Changes |
|------|---------|---------|
| 2026-05-15 | 1.3.0 | Made transport layer agnostic |
| - | - | Removed all NATS-specific dependencies (NATS.jl, nats, nats-py, nats.ws) |
| - | - | Updated docs to reference generic message broker/transport |
| - | - | broker_url is now metadata only, not used for active connections |
| 2026-03-15 | 1.1.0 | Browser connection management |
| - | - | Added NATSClient class with keepAlive support |
| - | - | Added NATSConnectionPool for connection reuse |
@@ -1119,7 +1116,7 @@ flowchart TD
|------|----------|----------|--------------------------|
| [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe | FR-001 through FR-014, NFR-101 through NFR-405 |
@@ -1129,23 +1126,17 @@ flowchart TD
| Platform | Package | Version | Purpose | Requirements Traceability |
|----------|---------|---------|---------|--------------------------|
| Julia | NATS.jl | Latest | NATS client | FR-013, FR-014, NFR-201 |
| Julia | JSON.jl | Latest | JSON serialization | FR-012, NFR-101, NFR-102 |
| Julia | Arrow.jl | Latest | Arrow IPC support | FR-002, FR-012 |
| Julia | HTTP.jl | Latest | HTTP file server | FR-008, FR-009 |
| Julia | UUIDs.jl | Latest | UUID generation | FR-011, NFR-401 |
| Node.js | nats | Latest | NATS client (TCP) | FR-013, FR-014 |
| Node.js | node-fetch | Latest | HTTP file server | FR-008, FR-009 |
| Browser | nats.ws | Latest | NATS client (WebSocket) | FR-013, FR-014 |
| Browser | nats | Latest | NATS client (for bundling) | FR-013, FR-014 |
| Python | nats-py | Latest | NATS client | FR-013, FR-014 |
| Browser | - | - | Transport-agnostic (caller provides) | FR-013, FR-014 |
| Python | aiohttp | Latest | HTTP file server | FR-008, FR-009 |
| Python | pyarrow | Latest | Arrow IPC support | FR-002, FR-012 |
| Dart | nats | Latest | NATS client | FR-013, FR-014 |
| Dart | http | Latest | HTTP file server | FR-008, FR-009 |
| Dart | uuid | Latest | UUID generation | FR-011, NFR-401 |
| Dart | dart-arrow | Latest | Arrow IPC support | FR-002, FR-012 |
| Rust | nats | Latest | NATS client | FR-013, FR-014 |
| Rust | serde | Latest | JSON serialization | FR-012, NFR-101, NFR-102 |
| Rust | serde_json | Latest | JSON handling | FR-012, NFR-101, NFR-102 |
| Rust | tokio | Latest | Async runtime | FR-013, FR-014 |
@@ -1160,6 +1151,10 @@ flowchart TD
| Date | Version | Changes | Requirement ID(s) |
|------|---------|---------|-------------------|
| 2026-05-15 | 1.3.0 | Made transport layer agnostic | All |
| - | - | Removed NATS-specific dependencies and references from all docs | All |
| - | - | Updated all NATS references to generic "transport layer"/"message broker" | All |
| - | - | Removed NATS client packages from dependencies tables | All |
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All |
| - | - | Updated smartsend signatures: removed is_publish, nats_connection; added sender_name | FR-001 through FR-014 |
| - | - | Updated smartreceive signatures: takes msg_json_str::String instead of msg | FR-001 through FR-014 |
@@ -1200,7 +1195,7 @@ flowchart TD
"send_to": {
"type": "string",
"minLength": 1,
"description": "NATS subject to publish to"
"description": "Topic/subject to publish to"
},
"msg_purpose": {
"type": "string",
@@ -1236,8 +1231,7 @@ flowchart TD
},
"broker_url": {
"type": "string",
"pattern": "^nats://[^\\s]+$",
"description": "NATS broker URL"
"description": "Broker URL for the transport layer"
},
"metadata": {
"type": "object",
@@ -1302,14 +1296,14 @@ flowchart TD
}
```
### B. AsyncAPI Specification (NATS)
### B. AsyncAPI Specification
```yaml
asyncapi: '2.6.0'
info:
title: msghandler API
version: '1.0.0'
description: Cross-platform bi-directional data bridge using NATS
description: Cross-platform bi-directional data bridge using a message broker
contact:
name: msghandler Team
url: https://github.com/your-org/msghandler
@@ -1331,12 +1325,12 @@ channels:
schema:
type: string
publish:
summary: Publish message to NATS
summary: Publish message to transport
operationId: publishMessage
message:
$ref: '#/components/message'
subscribe:
summary: Subscribe to NATS messages
summary: Subscribe to messages from transport
operationId: subscribeMessage
message:
$ref: '#/components/message'