ton/msghandler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

v1.0.0 #1

Merged
ton merged 14 commits from v1.0.0 into main 2026-05-22 22:16:17 +00:00
Conversation 0 Commits 14 Files Changed 26 +367 -169
1 changed files with 367 additions and 169 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit e93fe9c5c7 - Show all commits

530
docs/walkthrough.md
View File

@@ -1,6 +1,6 @@
# Walkthrough: msghandler # Walkthrough: msghandler
**Version**: 1.5.0 **Version**: 1.6.0
**Date**: 2026-05-22 **Date**: 2026-05-22
**Status**: Active **Status**: Active
**Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl) **Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl)
@@ -16,21 +16,23 @@ This walkthrough serves as the primary onboarding guide for new developers and e
- **User scenarios** - Real-world use cases from developer perspective - **User scenarios** - Real-world use cases from developer perspective
- **Why steps are sequenced** - The rationale behind architectural decisions - **Why steps are sequenced** - The rationale behind architectural decisions
- **What could go wrong** - Common failure scenarios and recovery strategies - **What could go wrong** - Common failure scenarios and recovery strategies
- **Transport integration** - Complete end-to-end flow with publish/subscribe patterns
### 1.1 Specification Traceability ### 1.1 Specification Traceability
| Walkthrough Section | Specification Reference | Requirement ID(s) | Solution Design Ref(s) | Description | | Walkthrough Section | Specification Reference | Requirement ID(s) | Solution Design Ref(s) | Description |
|---------------------|-------------------------|-------------------|------------------------|-------------| |---------------------|-------------------------|-------------------|------------------------|-------------|
| Section 2 (Big Picture) | specification.md:2, specification.md:15 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-002, SD-005, SD-006 | End-to-end system flow diagrams | | Section 2 (Big Picture) | specification.md:2, specification.md:15 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-002, SD-005, SD-006 | End-to-end system flow diagrams and transport pattern |
| Section 3 (Chat Scenario) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | FR-001, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-004, SD-005, SD-006 | Chat webapp ↔ Julia backend with mixed payloads | | Section 3 (Chat Scenario) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | FR-001, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-004, SD-005, SD-006 | Chat webapp ↔ Julia backend with mixed payloads |
| Section 4 (Large File) | specification.md:6, specification.md:7 | FR-003, FR-004, FR-008, FR-009, FR-010, NFR-104, NFR-105 | SD-001, SD-002, SD-003, SD-007 | Large file transfer with link transport | | Section 4 (Large File) | specification.md:6, specification.md:7 | FR-003, FR-004, FR-008, FR-009, FR-010, NFR-104, NFR-105 | SD-001, SD-002, SD-003, SD-007 | Large file transfer with link transport |
| Section 5 (Tabular Data) | specification.md:5, specification.md:10 | FR-002, FR-012, NFR-101, NFR-102 | SD-004, SD-005 | Arrow IPC tabular data exchange | | Section 5 (Tabular Data) | specification.md:5, specification.md:10 | FR-002, FR-012, NFR-101, NFR-102 | SD-004, SD-005 | Arrow IPC tabular data exchange |
| Section 6 (MicroPython) | specification.md:13, specification.md:17 | FR-005, FR-006, FR-012, NFR-106 | SD-002, SD-004 | Memory-constrained device communication | | Section 6 (MicroPython) | specification.md:13, specification.md:17 | FR-005, FR-006, FR-012, NFR-106 | SD-002, SD-004 | Memory-constrained device communication |
| Section 7 (Cross-Platform) | specification.md:3, specification.md:4, specification.md:5, specification.md:11 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-002, SD-004, SD-005, SD-006 | Multi-platform chat application | | Section 7 (Cross-Platform) | specification.md:3, specification.md:4, specification.md:5, specification.md:11 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-002, SD-004, SD-005, SD-006 | Multi-platform chat application |
| Section 8 (Error Handling) | specification.md:9 | FR-008, FR-009, FR-010, NFR-201, NFR-202, NFR-203 | SD-003, SD-007 | Common error scenarios and recovery | | Section 10 (Error Handling) | specification.md:9 | FR-008, FR-009, FR-010, NFR-201, NFR-202, NFR-203 | SD-003, SD-007 | Common error scenarios and recovery |
| Section 9 (Debugging) | specification.md:4, specification.md:11 | FR-011, NFR-401, NFR-403 | SD-008 | Correlation ID tracking | | Section 11 (Debugging) | specification.md:4, specification.md:11 | FR-011, NFR-401, NFR-403 | SD-008 | Correlation ID tracking |
| Section 10 (Performance) | specification.md:7, specification.md:13 | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 | SD-001, SD-002, SD-006 | Optimization strategies | | Section 12 (Performance) | specification.md:7, specification.md:13 | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 | SD-001, SD-002, SD-006 | Optimization strategies |
| Section 11 (Deployment) | specification.md:12, specification.md:18 | FR-013, FR-014, NFR-201, NFR-203 | SD-006 | Infrastructure requirements | | Section 13 (Deployment) | specification.md:12, specification.md:18 | FR-013, FR-014, NFR-201, NFR-203 | SD-006 | Infrastructure requirements |
| Section 9 (Transport Layer) | specification.md:3, specification.md:5 | FR-013, FR-014 | SD-006 | Complete end-to-end transport integration examples |
--- ---
@@ -38,6 +40,47 @@ This walkthrough serves as the primary onboarding guide for new developers and e
msghandler implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB): msghandler implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB):
### 2.0 Transport Layer Pattern
**Critical**: msghandler is **transport-agnostic**. The caller is responsible for:
1. **Sending side**: Call `smartpack()` → receive `(envelope, json_string)` → publish JSON string via transport
2. **Receiving side**: Subscribe to transport → receive JSON string → call `smartunpack()`
```mermaid
flowchart LR
subgraph msghandler["msghandler Module"]
direction TB
S1["smartpack(data)"]
S2["Returns (envelope, json_str)"]
R1["smartunpack(json_str)"]
R2["Returns payloads"]
end
subgraph Transport["Transport Layer (Caller's Responsibility)"]
direction TB
PUBLISH["Publish JSON string"]
SUBSCRIBE["Subscribe to messages"]
end
S1 --> S2
S2 --> PUBLISH
SUBSCRIBE --> R1
R1 --> R2
PUBLISH --> SUBSCRIBE
style msghandler fill:#b3e5fc,stroke:#0288d1
style Transport fill:#ffe0b2,stroke:#f57c00
```
**Supported Transports**:
- NATS (WebSocket/TCP)
- MQTT (WebSocket/TCP)
- WebSocket (direct HTTP)
- HTTP (polling/long-polling)
**Key Principle**: msghandler only handles **serialization/deserialization**. Transport is handled by caller.
```mermaid ```mermaid
flowchart TB flowchart TB
subgraph msghandler["msghandler Module"] subgraph msghandler["msghandler Module"]
@@ -92,7 +135,7 @@ flowchart TB
style FileServer fill:#ffe0b2,stroke:#f57c00 style FileServer fill:#ffe0b2,stroke:#f57c00
``` ```
### 2.1 Key Design Principles ### 2.2 Key Design Principles
| Principle | Description | Rationale | | Principle | Description | Rationale |
|-----------|-------------|-----------| |-----------|-------------|-----------|
@@ -100,6 +143,7 @@ flowchart TB
| **Automatic Transport Selection** | Direct (< threshold) vs Link (≥ threshold) based on size | Optimizes memory vs network I/O trade-off | | **Automatic Transport Selection** | Direct (< threshold) vs Link (≥ threshold) based on size | Optimizes memory vs network I/O trade-off |
| **Cross-Platform API** | Consistent `smartpack()`/`smartunpack()` across all platforms | Simplifies developer experience | | **Cross-Platform API** | Consistent `smartpack()`/`smartunpack()` across all platforms | Simplifies developer experience |
| **Exponential Backoff** | Retry downloads with increasing delays | Handles transient failures gracefully | | **Exponential Backoff** | Retry downloads with increasing delays | Handles transient failures gracefully |
| **Transport Agnostic** | Caller handles transport (NATS/MQTT/WebSocket) | No vendor lock-in; works with any broker |
--- ---
@@ -109,68 +153,7 @@ flowchart TB
A JavaScript chat webapp wants to send mixed payloads (text message + user avatar image) to a Julia backend, and receive mixed payloads (text response + AI-generated image) back. A JavaScript chat webapp wants to send mixed payloads (text message + user avatar image) to a Julia backend, and receive mixed payloads (text response + AI-generated image) back.
### Complete End-to-End Round-Trip Flow ### Step-by-Step Flow
msghandler implements a **transport-agnostic** messaging pattern. The library handles serialization and envelope building, while the caller is responsible for publishing/subscribing via their chosen transport (NATS, MQTT, WebSocket, HTTP, etc.).
```
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ END-TO-END ROUND-TRIP FLOW │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
SENDER (JavaScript Webapp) TRANSPORT LAYER RECEIVER (Julia Backend)
┌─────────────────────┐ ┌───────────────────┐ ┌─────────────────────┐
│ │ │ │ │ │
│ 1. Prepare data │ │ │ │ │
│ [(msg, data, │ │ │ │ │
│ type)] │ │ │ │ │
│ │ │ │ │ │
│ 2. Call smartpack() │──────────────────────────────────────>│ │ │ │
│ Returns (env, │ │ JSON Message │ │ │
│ json_str) │ │ (json_str) │ │ │
│ │ │ │ │ │
│ 3. Publish via │ │ │ │ │
│ transport: │──────────────────────────────────────>│ (NATS/MQTT/ │────────────────────────>│ 4. Subscribe via │
│ MY_TRANSPORT. │ │ WebSocket, etc.) │ │ transport: │
│ publish(subject, │ │ │ │ msg = │
│ msgJson) │ │ │ │ SUBSCRIBE() │
│ │ │ │ │ │
│ │ │ │ │ 5. Call │
│ │ │ │ │ smartunpack( │
│ │ │ │ │ String( │
│ │ │ │ │ msg.payload) │
│ │ │ │ │ │
│ │ │ │ │ 6. Receive │
│ │ │ │ │ payloads: │
│ │ │ │ │ [(msg, data, │
│ │ │ │ │ type), ...] │
│ │ │ │ │ │
└─────────────────────┘ └───────────────────┘ └─────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ JULIA BACKEND RESPONSE (Reverse Flow) │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
RECEIVER (Julia Backend) TRANSPORT LAYER SENDER (JavaScript Webapp)
┌─────────────────────┐ ┌───────────────────┐ ┌─────────────────────┐
│ │ │ │ │ │
│ 7. Process data │ │ │ │ │
│ (AI inference) │ │ │ │ │
│ │ │ │ │ │
│ 8. Call smartpack() │──────────────────────────────────────>│ │ │ │
│ Returns (env, │ │ JSON Message │ │ │
│ json_str) │ │ (json_str) │ │ │
│ │ │ │ │ │
│ 9. Publish reply │ │ │ │ │
│ via transport: │──────────────────────────────────────>│ (NATS/MQTT/ │────────────────────────>│ 10. Subscribe via │
│ MY_TRANSPORT. │ │ WebSocket, etc.) │ │ reply_to topic │
│ publish(replyTo, │ │ │ │ │
│ msgJson) │ │ │ │ │
│ │ │ │ │ │
└─────────────────────┘ └───────────────────┘ └─────────────────────┘
```
### Step-by-Step Flow (Sender Perspective)
#### 3.1 Step 1: JavaScript Webapp Sends Mixed Payloads #### 3.1 Step 1: JavaScript Webapp Sends Mixed Payloads
@@ -195,6 +178,14 @@ const [env, msgJson] = await msghandler.smartpack(
- **Why text first?** Text is smaller, sent via direct transport (fast, no file server needed) - **Why text first?** Text is smaller, sent via direct transport (fast, no file server needed)
- **Why image second?** Images may trigger link transport if >0.5MB - **Why image second?** Images may trigger link transport if >0.5MB
#### 3.1b Step 1b: JavaScript Publishes via Transport
```javascript
// Publish the JSON string via WebSocket (or NATS/MQTT)
const conn = await transportClient.connect({ servers: "ws://localhost:4222" });
await conn.publish("/agent/wine/api/v1/prompt", msgJson);
```
#### 3.2 Step 2: Transport Selection #### 3.2 Step 2: Transport Selection
For each payload, msghandler determines transport: For each payload, msghandler determines transport:
@@ -271,63 +262,28 @@ msghandler builds the message envelope:
- **reply_to**: Tells backend where to send response - **reply_to**: Tells backend where to send response
- **payloads array**: Contains all data with metadata for proper handling - **payloads array**: Contains all data with metadata for proper handling
#### 3.5 Step 5: Publish to Transport (Caller's Responsibility) #### 3.5 Step 5: Publish JSON String via Transport
**Choose your transport** (replace `MY_TRANSPORT` with your actual library):
```javascript ```javascript
// Example 1: NATS (Node.js) // JavaScript: Publish via WebSocket (NATS/MQTT/HTTP work similarly)
import { connect } from 'nats'; const conn = await transportClient.connect({ servers: "ws://localhost:4222" });
const nats = connect({ servers: 'ws://localhost:4222' }); await conn.publish("/agent/wine/api/v1/prompt", msgJson);
await nats.publish("/agent/wine/api/v1/prompt", msgJson);
// Example 2: MQTT (Node.js)
import * as mqtt from 'mqtt';
const client = mqtt.connect('ws://localhost:1883');
client.publish("/agent/wine/api/v1/prompt", msgJson);
// Example 3: WebSocket (Browser)
const ws = new WebSocket('ws://localhost:4222/ws');
ws.send(msgJson);
// Example 4: Custom HTTP POST
fetch('http://localhost:8000/publish', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: msgJson
});
``` ```
**Why caller responsibility?** #### 3.6 Step 6: Julia Backend Receives and Unpacks
- **Transport agnostic**: msghandler supports NATS, MQTT, WebSocket, HTTP, or any custom transport
- **Connection reuse**: Callers can manage connection pools efficiently
- **Flexibility**: No library lock-in; use whatever transport best fits your stack
#### 3.6 Step 6: Julia Backend Receives Message
```julia ```julia
# Julia backend # Julia backend
# Choose your transport (replace MY_TRANSPORT with your actual library) transport_msg = transport_subscription.next() # Get message from transport
env = smartunpack(String(transport_msg.payload))
# Example: NATS subscription # env["payloads"] is now:
using NATS # [
conn = NATS.connect("nats.yiem.cc") # ("msg", "Hello! I'm Ton.", "text"),
NATS.subscribe(conn, "/agent/wine/api/v1/prompt") do msg # ("avatar", binary_data, "image")
env = smartunpack(String(msg.payload)) # ]
# env["payloads"] is now:
# [
# ("msg", "Hello! I'm Ton.", "text"),
# ("avatar", binary_data, "image")
# ]
end
``` ```
**Rationale**:
- `smartunpack()` handles both transport types automatically
- Deserialization is type-aware based on `payload_type`
- Returns consistent tuple format regardless of transport
#### 3.7 Step 7: Julia Backend Sends Response #### 3.7 Step 7: Julia Backend Sends Response
```julia ```julia
@@ -342,13 +298,9 @@ env, msg_json = smartpack(
("generated_image", generated_image, "image") ("generated_image", generated_image, "image")
], ],
reply_to = "/chat/user/v1/message", reply_to = "/chat/user/v1/message",
reply_to_msg_id = env["msg_id"] reply_to_msg_id = msg["msg_id"]
); );
publish(transport_conn, "/agent/wine/api/v1/response", msg_json);
# Publish response via transport (caller's responsibility)
# Example: NATS
using NATS
NATS.publish(conn, "/agent/wine/api/v1/response", msg_json)
``` ```
**Rationale**: **Rationale**:
@@ -381,6 +333,14 @@ const [env, msgJson] = await msghandler.smartpack(
); );
``` ```
#### 4.1b Step 1b: JavaScript Publishes via Transport
```javascript
// Publish the JSON string via WebSocket (or NATS/MQTT)
const conn = await transportClient.connect({ servers: "ws://localhost:4222" });
await conn.publish("/agent/wine/api/v1/process", msgJson);
```
#### 4.2 Step 2: Transport Selection (Link) #### 4.2 Step 2: Transport Selection (Link)
| Payload | Size | Transport | Reason | | Payload | Size | Transport | Reason |
@@ -440,7 +400,7 @@ const response = await plikOneshotUpload(
- `transport: "link"` signals URL-based download - `transport: "link"` signals URL-based download
- `encoding: "none"` indicates no additional encoding - `encoding: "none"` indicates no additional encoding
#### 4.5 Step 5: Julia Backend Receives and Downloads #### 4.5 Step 5: Julia Backend Receives, Downloads, and Unpacks
```julia ```julia
# Julia backend # Julia backend
@@ -448,8 +408,8 @@ transport_msg = transport_subscription.next();
env = smartunpack(String(transport_msg.payload)); env = smartunpack(String(transport_msg.payload));
# msghandler automatically: # msghandler automatically:
# 1. Extracts URL from payload # 1. Extracts URL from payload (link transport)
# 2. Downloads with exponential backoff # 2. Downloads file with exponential backoff
# 3. Deserializes to binary data # 3. Deserializes to binary data
``` ```
@@ -494,6 +454,13 @@ env, msg_json = await smartpack(
- Arrow IPC format preserves data types - Arrow IPC format preserves data types
- Much faster than JSON serialization - Much faster than JSON serialization
#### 5.1b Step 1b: Python Publishes via Transport
```python
# Publish the JSON string via WebSocket/NATS/MQTT
await transport_publisher.publish("/agent/wine/api/v1/analyze", msg_json)
```
#### 5.2 Step 2: Serialization to Arrow IPC #### 5.2 Step 2: Serialization to Arrow IPC
```python ```python
@@ -529,7 +496,7 @@ env = smartunpack(String(transport_msg.payload));
- DataFrame returned with correct types - DataFrame returned with correct types
- No manual parsing needed - No manual parsing needed
#### 5.4 Step 4: Julia Sends Results #### 5.4 Step 4: Julia Sends Results (Complete Round-Trip)
```julia ```julia
# Julia backend # Julia backend
@@ -541,6 +508,7 @@ env, msg_json = smartpack(
[("results", results, "arrowtable")], [("results", results, "arrowtable")],
reply_to = "/python/worker/v1/results" reply_to = "/python/worker/v1/results"
); );
publish(transport_conn, "/agent/wine/api/v1/results", msg_json);
``` ```
**Rationale**: **Rationale**:
@@ -643,7 +611,7 @@ let (envelope, json_str) = smartpack(
}, },
).await?; ).await?;
// Caller publishes via transport // Caller publishes via transport (WebSocket/MQTT/NATS)
conn.publish("/agent/wine/api/v1/results", &json_str)?; conn.publish("/agent/wine/api/v1/results", &json_str)?;
``` ```
@@ -692,6 +660,9 @@ let (envelope, json_str) = smartpack(
..Default::default() ..Default::default()
}, },
).await?; ).await?;
// Caller publishes via transport
conn.publish("/agent/wine/api/v1/upload", &json_str)?;
``` ```
**Rationale**: **Rationale**:
@@ -735,6 +706,13 @@ env, msg_json = smartpack(
- Smaller threshold (100KB) for memory constraints - Smaller threshold (100KB) for memory constraints
- Direct transport only (no file server support) - Direct transport only (no file server support)
#### 7.1b Step 1b: MicroPython Publishes via Transport
```python
# Publish the JSON string via MQTT (common on IoT devices)
mqtt_client.publish("/sensor/device/v1/readings", msg_json)
```
#### 7.2 Step 2: Serialization #### 7.2 Step 2: Serialization
```python ```python
@@ -765,6 +743,40 @@ env = await smartunpack(str(transport_msg.payload));
- Dictionary returned directly - Dictionary returned directly
- No Arrow support (memory constraints) - No Arrow support (memory constraints)
#### 7.4 Step 4: Complete MicroPython End-to-End Flow
```mermaid
flowchart LR
subgraph MicroPython["MicroPython Device"]
MPY1["sensor_data = {temp, humidity}"]
MPY2["smartpack() → JSON"]
MPY3["publish via MQTT"]
end
subgraph Transport["MQTT Broker"]
direction TB
MQTT["broker.local:1883"]
end
subgraph Python["Python Backend"]
PY1["subscribe via MQTT"]
PY2["smartunpack(JSON)"]
PY3["get dictionary"]
end
MPY1 --> MPY2
MPY2 --> MPY3
MPY3 --> MQTT
MQTT --> PY1
PY1 --> PY2
PY2 --> PY3
```
**Key Points**:
- MicroPython limited to `text` and `dictionary` types (memory constraints)
- Direct transport only (no file server support)
- Size threshold: 100KB (vs 500KB for desktop)
--- ---
## 8. User Scenario 6: Cross-Platform Chat with Mixed Payloads ## 8. User Scenario 6: Cross-Platform Chat with Mixed Payloads
@@ -798,6 +810,14 @@ const [env, msgJson] = await msghandler.smartpack(
- Chat messages often include text + images - Chat messages often include text + images
- Transport wildcard subscriptions route to correct recipients - Transport wildcard subscriptions route to correct recipients
#### 8.1b Step 1b: JavaScript Publishes via Transport
```javascript
// Publish via WebSocket/NATS/MQTT
const conn = await transportClient.connect({ servers: "ws://localhost:4222" });
await conn.publish("/chat/user/v1/message", msgJson);
```
#### 8.2 Step 2: Python Backend Receives #### 8.2 Step 2: Python Backend Receives
```python ```python
@@ -836,47 +856,203 @@ env = smartunpack(String(transport_msg.payload));
- Same function signature across platforms - Same function signature across platforms
- Type information enables proper deserialization - Type information enables proper deserialization
#### 8.4 Step 4: All Platforms Reply #### 8.4 Step 4: Complete End-to-End Flow
Each platform can reply using the same API: ```mermaid
flowchart TB
subgraph Sender["Sender (JavaScript)"]
direction TB
JS1["Create data tuples"]
JS2["smartpack() → JSON string"]
JS3["Publish via WebSocket"]
end
```python subgraph Transport["Transport Layer"]
# Python reply direction TB
await smartpack( BROKER["Message Broker"]
"/chat/user/v1/reply", end
[("response", "Nice!", "text")],
reply_to="/chat/user/v1/message"
);
```
```julia subgraph Receiver["Receiver (Julia/Python)"]
# Julia reply direction TB
smartpack( REC1["Subscribe via WebSocket"]
"/chat/user/v1/reply", REC2["smartunpack(JSON string)"]
[("response", "Nice!", "text")], REC3["Get payload tuples"]
reply_to="/chat/user/v1/message" end
);
```
```javascript JS1 --> JS2
// JavaScript reply JS2 --> JS3
await msghandler.smartpack( JS3 --> BROKER
"/chat/user/v1/reply", BROKER --> REC1
[["response", "Nice!", "text"]], REC1 --> REC2
{ reply_to: "/chat/user/v1/message" } REC2 --> REC3
);
``` ```
**Rationale**: **Rationale**:
- Same API across platforms - `smartpack()` handles serialization, transport selection, and envelope building
- Consistent behavior - Caller publishes JSON string via WebSocket/NATS/MQTT
- Easy to maintain parity - `smartunpack()` handles transport detection, file downloads (if link), and deserialization
- Same API across all platforms ensures consistency
#### 8.6 Step 6: Complete Cross-Platform Flow
```mermaid
flowchart TB
subgraph JS["JavaScript (Frontend)"]
J1["Create message tuples"]
J2["smartpack()"]
J3["Publish via WebSocket"]
J4["Subscribe for replies"]
end
subgraph Python["Python (Backend)"]
P1["Subscribe via NATS"]
P2["smartunpack()"]
P3["Process data"]
P4["smartpack()"]
P5["Publish reply via NATS"]
end
subgraph Julia["Julia (Backend)"]
L1["Subscribe via MQTT"]
L2["smartunpack()"]
L3["Process data"]
L4["smartpack()"]
L5["Publish reply via MQTT"]
end
subgraph Transport["Message Broker (NATS/MQTT/WebSocket)"]
direction TB
NATS["NATS: ws://localhost:4222"]
MQTT["MQTT: broker.local:1883"]
end
J2 --> J3
J3 --> NATS
NATS --> P1
P1 --> P2
P2 --> P3
P3 --> P4
P4 --> P5
P5 --> MQTT
MQTT --> L1
L1 --> L2
L2 --> L3
L3 --> L4
L4 --> L5
L5 --> NATS
NATS --> J4
style JS fill:#e3f2fd,stroke:#1976d2
style Python fill:#fff3e0,stroke:#f57c00
style Julia fill:#e8f5e9,stroke:#388e3c
style Transport fill:#f3e5f5,stroke:#7b1fa2
```
**Key Points**:
- Same `smartpack()`/`smartunpack()` API across all platforms
- JSON wire format ensures compatibility
- Each platform uses their preferred transport protocol
--- ---
## 9. Error Handling ## 9. Transport Layer Integration
### 9.1 Common Error Scenarios msghandler is **transport-agnostic** - it only handles serialization/deserialization. The caller is responsible for:
### 9.1 Publishing Flow (Sender)
1. Call `smartpack()` → gets `(envelope, json_string)`
2. Publish JSON string via transport (WebSocket/MQTT/NATS/etc.)
3. Transport delivers message to subscriber
### 9.2 Subscribing Flow (Receiver)
1. Subscribe to transport topic
2. Receive JSON string from transport
3. Call `smartunpack(json_string)` → gets payloads array
### 9.3 Complete End-to-End Example
#### JavaScript → Julia Complete Flow
```mermaid
flowchart LR
subgraph JavaScript["JavaScript Webapp"]
J1["Create tuples"]
J2["smartpack()"]
J3["Publish via WebSocket"]
end
subgraph Transport["NATS Broker"]
direction TB
NATS["ws://localhost:4222"]
end
subgraph Julia["Julia Backend"]
J4["Subscribe via WebSocket"]
J5["smartunpack()"]
J6["Process payloads"]
end
J1 --> J2
J2 --> J3
J3 --> NATS
NATS --> J4
J4 --> J5
J5 --> J6
```
```javascript
// JavaScript - Sender
const [env, msgJson] = await msghandler.smartpack(
"/agent/wine/api/v1/prompt",
[["msg", "Hello!", "text"]],
{ broker_url: "ws://localhost:4222" }
);
const conn = await nats.connect({ servers: "ws://localhost:4222" });
await conn.publish("/agent/wine/api/v1/prompt", msgJson);
```
```julia
# Julia - Receiver
transport_msg = transport_subscription.next()
env = smartunpack(String(transport_msg.payload))
# env["payloads"] is now: [("msg", "Hello!", "text")]
```
#### Transport Examples (Single Platform)
```javascript
const conn = await nats.connect({ servers: "ws://localhost:4222" });
await conn.publish(topic, json_string);
```
#### MQTT (Python)
```python
await mqtt_client.publish(topic, json_string)
```
#### WebSocket (Browser)
```javascript
const ws = new WebSocket("ws://localhost:4222");
ws.send(json_string);
```
#### Rust (Tokio)
```rust
conn.publish(topic, &json_str)?;
```
### 9.4 Important Notes
- **JSON format**: All messages use JSON string format for cross-platform compatibility
- **Caller responsibility**: Transport publishing/subscription is always the caller's code
- **No vendor lock-in**: Works with any message broker that supports your platform
---
## 10. Error Handling
### 10.1 Common Error Scenarios
| Scenario | Error | Recovery | | Scenario | Error | Recovery |
|----------|-------|----------| |----------|-------|----------|
@@ -885,7 +1061,7 @@ await msghandler.smartpack(
| Payload type mismatch | `DESERIALIZATION_ERROR` | Validate payload_type matches data | | Payload type mismatch | `DESERIALIZATION_ERROR` | Validate payload_type matches data |
| Transport connection lost | `TRANSPORT_CONNECTION_FAILED` | Transport client auto-reconnects | | Transport connection lost | `TRANSPORT_CONNECTION_FAILED` | Transport client auto-reconnects |
### 9.2 Error Response Format ### 10.2 Error Response Format
```json ```json
{ {
@@ -903,9 +1079,9 @@ await msghandler.smartpack(
--- ---
## 10. Debugging and Tracing ## 11. Debugging and Tracing
### 10.1 Correlation ID Tracking ### 11.1 Correlation ID Tracking
Every message includes a `correlation_id`: Every message includes a `correlation_id`:
@@ -935,9 +1111,9 @@ log_trace(correlation_id, "Published to transport");
--- ---
## 11. Performance Considerations ## 12. Performance Considerations
### 11.1 Optimization Strategies ### 12.1 Optimization Strategies
| Strategy | Description | When to Use | | Strategy | Description | When to Use |
|----------|-------------|-------------| |----------|-------------|-------------|
@@ -945,7 +1121,7 @@ log_trace(correlation_id, "Published to transport");
| Adjust size threshold | Increase threshold if file server slow | File server bottleneck | | Adjust size threshold | Increase threshold if file server slow | File server bottleneck |
| Use direct transport | Avoid file server for small payloads | Low latency requirements | | Use direct transport | Avoid file server for small payloads | Low latency requirements |
### 11.2 Size Threshold by Platform ### 12.2 Size Threshold by Platform
| Platform | Threshold | Notes | | Platform | Threshold | Notes |
|----------|-----------|-------| |----------|-----------|-------|
@@ -957,7 +1133,7 @@ log_trace(correlation_id, "Published to transport");
--- ---
## 12. Deployment Considerations ## 13. Deployment Considerations
### 12.1 Minimum Infrastructure ### 12.1 Minimum Infrastructure
@@ -1006,7 +1182,7 @@ log_trace(correlation_id, "Published to transport");
--- ---
## 15. Gap-Check Validation ## 14. Gap-Check Validation
| Stage Transition | Gap-Check Question | Status | | Stage Transition | Gap-Check Question | Status |
|------------------|-------------------|--------| |------------------|-------------------|--------|
@@ -1017,6 +1193,27 @@ log_trace(correlation_id, "Published to transport");
--- ---
## 15. Complete End-to-End Flow Summary
The msghandler transport-agnostic pattern works as follows:
```
Sender: Transport Layer: Receiver:
smartpack() Publish JSON string Subscribe
↓ ↓ ↓
(envelope, json_str) ────────────→ JSON string smartunpack()
↓
payloads array
```
**Key Points**:
- `smartpack()` handles serialization, transport selection, and envelope building
- Caller publishes JSON string via WebSocket/NATS/MQTT
- `smartunpack()` handles transport detection, file downloads (if link), and deserialization
- Same API across all platforms ensures consistency
---
## 16. References ## 16. References
### 16.1 Documentation Artifacts ### 16.1 Documentation Artifacts
@@ -1049,6 +1246,7 @@ log_trace(correlation_id, "Published to transport");
| Date | Version | Changes | Specification Reference | Solution Design Reference | | Date | Version | Changes | Specification Reference | Solution Design Reference |
|------|---------|---------|------------------------|--------------------------| |------|---------|---------|------------------------|--------------------------|
| 2026-05-22 | 1.6.0 | Added complete transport integration examples with end-to-end flow diagrams | Section 9 | SD-006 |
| 2026-05-22 | 1.5.0 | Updated to ASG Framework v8 pillars - aligned with specification and solution-design | All sections | All SD-XXX | | 2026-05-22 | 1.5.0 | Updated to ASG Framework v8 pillars - aligned with specification and solution-design | All sections | All SD-XXX |
| 2026-05-15 | 1.4.0 | Made transport layer agnostic | All sections | SD-001 through SD-008 | | 2026-05-15 | 1.4.0 | Made transport layer agnostic | All sections | SD-001 through SD-008 |
| 2026-05-14 | 1.3.0 | Updated Rust API to reflect `smartunpack` deserialization changes | All sections | SD-001 through SD-008 | | 2026-05-14 | 1.3.0 | Updated Rust API to reflect `smartunpack` deserialization changes | All sections | SD-001 through SD-008 |
@@ -1070,7 +1268,7 @@ log_trace(correlation_id, "Published to transport");
| Requirements | ✅ Complete | requirements.md: FR-001 through FR-014, NFR-101 through NFR-405 | | Requirements | ✅ Complete | requirements.md: FR-001 through FR-014, NFR-101 through NFR-405 |
| Solution Design | ✅ Complete | solution-design.md: SD-001 through SD-008 | | Solution Design | ✅ Complete | solution-design.md: SD-001 through SD-008 |
| Specification | ✅ Complete | specification.md: Section 2-19 | | Specification | ✅ Complete | specification.md: Section 2-19 |
| Walkthrough | ✅ Complete | walkthrough.md: Sections 2-17 | | Walkthrough | ✅ Complete | walkthrough.md: Sections 2-18 (includes Transport Layer Integration) |
| Implementation Plan | ⏳ Pending | implementation-plan.md | | Implementation Plan | ⏳ Pending | implementation-plan.md |
| Validation | ⏳ Pending | validation.md | | Validation | ⏳ Pending | validation.md |
| Runbook | ⏳ Pending | runbook.md | | Runbook | ⏳ Pending | runbook.md |