v1.0.0 #1
@@ -1,6 +1,6 @@
|
||||
# Walkthrough: msghandler
|
||||
|
||||
**Version**: 1.5.0
|
||||
**Version**: 1.6.0
|
||||
**Date**: 2026-05-22
|
||||
**Status**: Active
|
||||
**Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl)
|
||||
@@ -16,21 +16,23 @@ This walkthrough serves as the primary onboarding guide for new developers and e
|
||||
- **User scenarios** - Real-world use cases from developer perspective
|
||||
- **Why steps are sequenced** - The rationale behind architectural decisions
|
||||
- **What could go wrong** - Common failure scenarios and recovery strategies
|
||||
- **Transport integration** - Complete end-to-end flow with publish/subscribe patterns
|
||||
|
||||
### 1.1 Specification Traceability
|
||||
|
||||
| Walkthrough Section | Specification Reference | Requirement ID(s) | Solution Design Ref(s) | Description |
|
||||
|---------------------|-------------------------|-------------------|------------------------|-------------|
|
||||
| Section 2 (Big Picture) | specification.md:2, specification.md:15 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-002, SD-005, SD-006 | End-to-end system flow diagrams |
|
||||
| Section 2 (Big Picture) | specification.md:2, specification.md:15 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-002, SD-005, SD-006 | End-to-end system flow diagrams and transport pattern |
|
||||
| Section 3 (Chat Scenario) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | FR-001, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-004, SD-005, SD-006 | Chat webapp ↔ Julia backend with mixed payloads |
|
||||
| Section 4 (Large File) | specification.md:6, specification.md:7 | FR-003, FR-004, FR-008, FR-009, FR-010, NFR-104, NFR-105 | SD-001, SD-002, SD-003, SD-007 | Large file transfer with link transport |
|
||||
| Section 5 (Tabular Data) | specification.md:5, specification.md:10 | FR-002, FR-012, NFR-101, NFR-102 | SD-004, SD-005 | Arrow IPC tabular data exchange |
|
||||
| Section 6 (MicroPython) | specification.md:13, specification.md:17 | FR-005, FR-006, FR-012, NFR-106 | SD-002, SD-004 | Memory-constrained device communication |
|
||||
| Section 7 (Cross-Platform) | specification.md:3, specification.md:4, specification.md:5, specification.md:11 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-002, SD-004, SD-005, SD-006 | Multi-platform chat application |
|
||||
| Section 8 (Error Handling) | specification.md:9 | FR-008, FR-009, FR-010, NFR-201, NFR-202, NFR-203 | SD-003, SD-007 | Common error scenarios and recovery |
|
||||
| Section 9 (Debugging) | specification.md:4, specification.md:11 | FR-011, NFR-401, NFR-403 | SD-008 | Correlation ID tracking |
|
||||
| Section 10 (Performance) | specification.md:7, specification.md:13 | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 | SD-001, SD-002, SD-006 | Optimization strategies |
|
||||
| Section 11 (Deployment) | specification.md:12, specification.md:18 | FR-013, FR-014, NFR-201, NFR-203 | SD-006 | Infrastructure requirements |
|
||||
| Section 10 (Error Handling) | specification.md:9 | FR-008, FR-009, FR-010, NFR-201, NFR-202, NFR-203 | SD-003, SD-007 | Common error scenarios and recovery |
|
||||
| Section 11 (Debugging) | specification.md:4, specification.md:11 | FR-011, NFR-401, NFR-403 | SD-008 | Correlation ID tracking |
|
||||
| Section 12 (Performance) | specification.md:7, specification.md:13 | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 | SD-001, SD-002, SD-006 | Optimization strategies |
|
||||
| Section 13 (Deployment) | specification.md:12, specification.md:18 | FR-013, FR-014, NFR-201, NFR-203 | SD-006 | Infrastructure requirements |
|
||||
| Section 9 (Transport Layer) | specification.md:3, specification.md:5 | FR-013, FR-014 | SD-006 | Complete end-to-end transport integration examples |
|
||||
|
||||
---
|
||||
|
||||
@@ -38,6 +40,47 @@ This walkthrough serves as the primary onboarding guide for new developers and e
|
||||
|
||||
msghandler implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB):
|
||||
|
||||
### 2.0 Transport Layer Pattern
|
||||
|
||||
**Critical**: msghandler is **transport-agnostic**. The caller is responsible for:
|
||||
|
||||
1. **Sending side**: Call `smartpack()` → receive `(envelope, json_string)` → publish JSON string via transport
|
||||
2. **Receiving side**: Subscribe to transport → receive JSON string → call `smartunpack()`
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph msghandler["msghandler Module"]
|
||||
direction TB
|
||||
S1["smartpack(data)"]
|
||||
S2["Returns (envelope, json_str)"]
|
||||
R1["smartunpack(json_str)"]
|
||||
R2["Returns payloads"]
|
||||
end
|
||||
|
||||
subgraph Transport["Transport Layer (Caller's Responsibility)"]
|
||||
direction TB
|
||||
PUBLISH["Publish JSON string"]
|
||||
SUBSCRIBE["Subscribe to messages"]
|
||||
end
|
||||
|
||||
S1 --> S2
|
||||
S2 --> PUBLISH
|
||||
SUBSCRIBE --> R1
|
||||
R1 --> R2
|
||||
PUBLISH --> SUBSCRIBE
|
||||
|
||||
style msghandler fill:#b3e5fc,stroke:#0288d1
|
||||
style Transport fill:#ffe0b2,stroke:#f57c00
|
||||
```
|
||||
|
||||
**Supported Transports**:
|
||||
- NATS (WebSocket/TCP)
|
||||
- MQTT (WebSocket/TCP)
|
||||
- WebSocket (direct HTTP)
|
||||
- HTTP (polling/long-polling)
|
||||
|
||||
**Key Principle**: msghandler only handles **serialization/deserialization**. Transport is handled by caller.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph msghandler["msghandler Module"]
|
||||
@@ -92,7 +135,7 @@ flowchart TB
|
||||
style FileServer fill:#ffe0b2,stroke:#f57c00
|
||||
```
|
||||
|
||||
### 2.1 Key Design Principles
|
||||
### 2.2 Key Design Principles
|
||||
|
||||
| Principle | Description | Rationale |
|
||||
|-----------|-------------|-----------|
|
||||
@@ -100,6 +143,7 @@ flowchart TB
|
||||
| **Automatic Transport Selection** | Direct (< threshold) vs Link (≥ threshold) based on size | Optimizes memory vs network I/O trade-off |
|
||||
| **Cross-Platform API** | Consistent `smartpack()`/`smartunpack()` across all platforms | Simplifies developer experience |
|
||||
| **Exponential Backoff** | Retry downloads with increasing delays | Handles transient failures gracefully |
|
||||
| **Transport Agnostic** | Caller handles transport (NATS/MQTT/WebSocket) | No vendor lock-in; works with any broker |
|
||||
|
||||
---
|
||||
|
||||
@@ -109,68 +153,7 @@ flowchart TB
|
||||
|
||||
A JavaScript chat webapp wants to send mixed payloads (text message + user avatar image) to a Julia backend, and receive mixed payloads (text response + AI-generated image) back.
|
||||
|
||||
### Complete End-to-End Round-Trip Flow
|
||||
|
||||
msghandler implements a **transport-agnostic** messaging pattern. The library handles serialization and envelope building, while the caller is responsible for publishing/subscribing via their chosen transport (NATS, MQTT, WebSocket, HTTP, etc.).
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ END-TO-END ROUND-TRIP FLOW │
|
||||
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
SENDER (JavaScript Webapp) TRANSPORT LAYER RECEIVER (Julia Backend)
|
||||
┌─────────────────────┐ ┌───────────────────┐ ┌─────────────────────┐
|
||||
│ │ │ │ │ │
|
||||
│ 1. Prepare data │ │ │ │ │
|
||||
│ [(msg, data, │ │ │ │ │
|
||||
│ type)] │ │ │ │ │
|
||||
│ │ │ │ │ │
|
||||
│ 2. Call smartpack() │──────────────────────────────────────>│ │ │ │
|
||||
│ Returns (env, │ │ JSON Message │ │ │
|
||||
│ json_str) │ │ (json_str) │ │ │
|
||||
│ │ │ │ │ │
|
||||
│ 3. Publish via │ │ │ │ │
|
||||
│ transport: │──────────────────────────────────────>│ (NATS/MQTT/ │────────────────────────>│ 4. Subscribe via │
|
||||
│ MY_TRANSPORT. │ │ WebSocket, etc.) │ │ transport: │
|
||||
│ publish(subject, │ │ │ │ msg = │
|
||||
│ msgJson) │ │ │ │ SUBSCRIBE() │
|
||||
│ │ │ │ │ │
|
||||
│ │ │ │ │ 5. Call │
|
||||
│ │ │ │ │ smartunpack( │
|
||||
│ │ │ │ │ String( │
|
||||
│ │ │ │ │ msg.payload) │
|
||||
│ │ │ │ │ │
|
||||
│ │ │ │ │ 6. Receive │
|
||||
│ │ │ │ │ payloads: │
|
||||
│ │ │ │ │ [(msg, data, │
|
||||
│ │ │ │ │ type), ...] │
|
||||
│ │ │ │ │ │
|
||||
└─────────────────────┘ └───────────────────┘ └─────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ JULIA BACKEND RESPONSE (Reverse Flow) │
|
||||
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
RECEIVER (Julia Backend) TRANSPORT LAYER SENDER (JavaScript Webapp)
|
||||
┌─────────────────────┐ ┌───────────────────┐ ┌─────────────────────┐
|
||||
│ │ │ │ │ │
|
||||
│ 7. Process data │ │ │ │ │
|
||||
│ (AI inference) │ │ │ │ │
|
||||
│ │ │ │ │ │
|
||||
│ 8. Call smartpack() │──────────────────────────────────────>│ │ │ │
|
||||
│ Returns (env, │ │ JSON Message │ │ │
|
||||
│ json_str) │ │ (json_str) │ │ │
|
||||
│ │ │ │ │ │
|
||||
│ 9. Publish reply │ │ │ │ │
|
||||
│ via transport: │──────────────────────────────────────>│ (NATS/MQTT/ │────────────────────────>│ 10. Subscribe via │
|
||||
│ MY_TRANSPORT. │ │ WebSocket, etc.) │ │ reply_to topic │
|
||||
│ publish(replyTo, │ │ │ │ │
|
||||
│ msgJson) │ │ │ │ │
|
||||
│ │ │ │ │ │
|
||||
└─────────────────────┘ └───────────────────┘ └─────────────────────┘
|
||||
```
|
||||
|
||||
### Step-by-Step Flow (Sender Perspective)
|
||||
### Step-by-Step Flow
|
||||
|
||||
#### 3.1 Step 1: JavaScript Webapp Sends Mixed Payloads
|
||||
|
||||
@@ -195,6 +178,14 @@ const [env, msgJson] = await msghandler.smartpack(
|
||||
- **Why text first?** Text is smaller, sent via direct transport (fast, no file server needed)
|
||||
- **Why image second?** Images may trigger link transport if >0.5MB
|
||||
|
||||
#### 3.1b Step 1b: JavaScript Publishes via Transport
|
||||
|
||||
```javascript
|
||||
// Publish the JSON string via WebSocket (or NATS/MQTT)
|
||||
const conn = await transportClient.connect({ servers: "ws://localhost:4222" });
|
||||
await conn.publish("/agent/wine/api/v1/prompt", msgJson);
|
||||
```
|
||||
|
||||
#### 3.2 Step 2: Transport Selection
|
||||
|
||||
For each payload, msghandler determines transport:
|
||||
@@ -271,63 +262,28 @@ msghandler builds the message envelope:
|
||||
- **reply_to**: Tells backend where to send response
|
||||
- **payloads array**: Contains all data with metadata for proper handling
|
||||
|
||||
#### 3.5 Step 5: Publish to Transport (Caller's Responsibility)
|
||||
|
||||
**Choose your transport** (replace `MY_TRANSPORT` with your actual library):
|
||||
#### 3.5 Step 5: Publish JSON String via Transport
|
||||
|
||||
```javascript
|
||||
// Example 1: NATS (Node.js)
|
||||
import { connect } from 'nats';
|
||||
const nats = connect({ servers: 'ws://localhost:4222' });
|
||||
await nats.publish("/agent/wine/api/v1/prompt", msgJson);
|
||||
|
||||
// Example 2: MQTT (Node.js)
|
||||
import * as mqtt from 'mqtt';
|
||||
const client = mqtt.connect('ws://localhost:1883');
|
||||
client.publish("/agent/wine/api/v1/prompt", msgJson);
|
||||
|
||||
// Example 3: WebSocket (Browser)
|
||||
const ws = new WebSocket('ws://localhost:4222/ws');
|
||||
ws.send(msgJson);
|
||||
|
||||
// Example 4: Custom HTTP POST
|
||||
fetch('http://localhost:8000/publish', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: msgJson
|
||||
});
|
||||
// JavaScript: Publish via WebSocket (NATS/MQTT/HTTP work similarly)
|
||||
const conn = await transportClient.connect({ servers: "ws://localhost:4222" });
|
||||
await conn.publish("/agent/wine/api/v1/prompt", msgJson);
|
||||
```
|
||||
|
||||
**Why caller responsibility?**
|
||||
- **Transport agnostic**: msghandler supports NATS, MQTT, WebSocket, HTTP, or any custom transport
|
||||
- **Connection reuse**: Callers can manage connection pools efficiently
|
||||
- **Flexibility**: No library lock-in; use whatever transport best fits your stack
|
||||
|
||||
#### 3.6 Step 6: Julia Backend Receives Message
|
||||
#### 3.6 Step 6: Julia Backend Receives and Unpacks
|
||||
|
||||
```julia
|
||||
# Julia backend
|
||||
# Choose your transport (replace MY_TRANSPORT with your actual library)
|
||||
transport_msg = transport_subscription.next() # Get message from transport
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
|
||||
# Example: NATS subscription
|
||||
using NATS
|
||||
conn = NATS.connect("nats.yiem.cc")
|
||||
NATS.subscribe(conn, "/agent/wine/api/v1/prompt") do msg
|
||||
env = smartunpack(String(msg.payload))
|
||||
|
||||
# env["payloads"] is now:
|
||||
# [
|
||||
# ("msg", "Hello! I'm Ton.", "text"),
|
||||
# ("avatar", binary_data, "image")
|
||||
# ]
|
||||
end
|
||||
# env["payloads"] is now:
|
||||
# [
|
||||
# ("msg", "Hello! I'm Ton.", "text"),
|
||||
# ("avatar", binary_data, "image")
|
||||
# ]
|
||||
```
|
||||
|
||||
**Rationale**:
|
||||
- `smartunpack()` handles both transport types automatically
|
||||
- Deserialization is type-aware based on `payload_type`
|
||||
- Returns consistent tuple format regardless of transport
|
||||
|
||||
#### 3.7 Step 7: Julia Backend Sends Response
|
||||
|
||||
```julia
|
||||
@@ -342,13 +298,9 @@ env, msg_json = smartpack(
|
||||
("generated_image", generated_image, "image")
|
||||
],
|
||||
reply_to = "/chat/user/v1/message",
|
||||
reply_to_msg_id = env["msg_id"]
|
||||
reply_to_msg_id = msg["msg_id"]
|
||||
);
|
||||
|
||||
# Publish response via transport (caller's responsibility)
|
||||
# Example: NATS
|
||||
using NATS
|
||||
NATS.publish(conn, "/agent/wine/api/v1/response", msg_json)
|
||||
publish(transport_conn, "/agent/wine/api/v1/response", msg_json);
|
||||
```
|
||||
|
||||
**Rationale**:
|
||||
@@ -381,6 +333,14 @@ const [env, msgJson] = await msghandler.smartpack(
|
||||
);
|
||||
```
|
||||
|
||||
#### 4.1b Step 1b: JavaScript Publishes via Transport
|
||||
|
||||
```javascript
|
||||
// Publish the JSON string via WebSocket (or NATS/MQTT)
|
||||
const conn = await transportClient.connect({ servers: "ws://localhost:4222" });
|
||||
await conn.publish("/agent/wine/api/v1/process", msgJson);
|
||||
```
|
||||
|
||||
#### 4.2 Step 2: Transport Selection (Link)
|
||||
|
||||
| Payload | Size | Transport | Reason |
|
||||
@@ -440,7 +400,7 @@ const response = await plikOneshotUpload(
|
||||
- `transport: "link"` signals URL-based download
|
||||
- `encoding: "none"` indicates no additional encoding
|
||||
|
||||
#### 4.5 Step 5: Julia Backend Receives and Downloads
|
||||
#### 4.5 Step 5: Julia Backend Receives, Downloads, and Unpacks
|
||||
|
||||
```julia
|
||||
# Julia backend
|
||||
@@ -448,8 +408,8 @@ transport_msg = transport_subscription.next();
|
||||
env = smartunpack(String(transport_msg.payload));
|
||||
|
||||
# msghandler automatically:
|
||||
# 1. Extracts URL from payload
|
||||
# 2. Downloads with exponential backoff
|
||||
# 1. Extracts URL from payload (link transport)
|
||||
# 2. Downloads file with exponential backoff
|
||||
# 3. Deserializes to binary data
|
||||
```
|
||||
|
||||
@@ -494,6 +454,13 @@ env, msg_json = await smartpack(
|
||||
- Arrow IPC format preserves data types
|
||||
- Much faster than JSON serialization
|
||||
|
||||
#### 5.1b Step 1b: Python Publishes via Transport
|
||||
|
||||
```python
|
||||
# Publish the JSON string via WebSocket/NATS/MQTT
|
||||
await transport_publisher.publish("/agent/wine/api/v1/analyze", msg_json)
|
||||
```
|
||||
|
||||
#### 5.2 Step 2: Serialization to Arrow IPC
|
||||
|
||||
```python
|
||||
@@ -529,7 +496,7 @@ env = smartunpack(String(transport_msg.payload));
|
||||
- DataFrame returned with correct types
|
||||
- No manual parsing needed
|
||||
|
||||
#### 5.4 Step 4: Julia Sends Results
|
||||
#### 5.4 Step 4: Julia Sends Results (Complete Round-Trip)
|
||||
|
||||
```julia
|
||||
# Julia backend
|
||||
@@ -541,6 +508,7 @@ env, msg_json = smartpack(
|
||||
[("results", results, "arrowtable")],
|
||||
reply_to = "/python/worker/v1/results"
|
||||
);
|
||||
publish(transport_conn, "/agent/wine/api/v1/results", msg_json);
|
||||
```
|
||||
|
||||
**Rationale**:
|
||||
@@ -643,7 +611,7 @@ let (envelope, json_str) = smartpack(
|
||||
},
|
||||
).await?;
|
||||
|
||||
// Caller publishes via transport
|
||||
// Caller publishes via transport (WebSocket/MQTT/NATS)
|
||||
conn.publish("/agent/wine/api/v1/results", &json_str)?;
|
||||
```
|
||||
|
||||
@@ -692,6 +660,9 @@ let (envelope, json_str) = smartpack(
|
||||
..Default::default()
|
||||
},
|
||||
).await?;
|
||||
|
||||
// Caller publishes via transport
|
||||
conn.publish("/agent/wine/api/v1/upload", &json_str)?;
|
||||
```
|
||||
|
||||
**Rationale**:
|
||||
@@ -735,6 +706,13 @@ env, msg_json = smartpack(
|
||||
- Smaller threshold (100KB) for memory constraints
|
||||
- Direct transport only (no file server support)
|
||||
|
||||
#### 7.1b Step 1b: MicroPython Publishes via Transport
|
||||
|
||||
```python
|
||||
# Publish the JSON string via MQTT (common on IoT devices)
|
||||
mqtt_client.publish("/sensor/device/v1/readings", msg_json)
|
||||
```
|
||||
|
||||
#### 7.2 Step 2: Serialization
|
||||
|
||||
```python
|
||||
@@ -765,6 +743,40 @@ env = await smartunpack(str(transport_msg.payload));
|
||||
- Dictionary returned directly
|
||||
- No Arrow support (memory constraints)
|
||||
|
||||
#### 7.4 Step 4: Complete MicroPython End-to-End Flow
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph MicroPython["MicroPython Device"]
|
||||
MPY1["sensor_data = {temp, humidity}"]
|
||||
MPY2["smartpack() → JSON"]
|
||||
MPY3["publish via MQTT"]
|
||||
end
|
||||
|
||||
subgraph Transport["MQTT Broker"]
|
||||
direction TB
|
||||
MQTT["broker.local:1883"]
|
||||
end
|
||||
|
||||
subgraph Python["Python Backend"]
|
||||
PY1["subscribe via MQTT"]
|
||||
PY2["smartunpack(JSON)"]
|
||||
PY3["get dictionary"]
|
||||
end
|
||||
|
||||
MPY1 --> MPY2
|
||||
MPY2 --> MPY3
|
||||
MPY3 --> MQTT
|
||||
MQTT --> PY1
|
||||
PY1 --> PY2
|
||||
PY2 --> PY3
|
||||
```
|
||||
|
||||
**Key Points**:
|
||||
- MicroPython limited to `text` and `dictionary` types (memory constraints)
|
||||
- Direct transport only (no file server support)
|
||||
- Size threshold: 100KB (vs 500KB for desktop)
|
||||
|
||||
---
|
||||
|
||||
## 8. User Scenario 6: Cross-Platform Chat with Mixed Payloads
|
||||
@@ -798,6 +810,14 @@ const [env, msgJson] = await msghandler.smartpack(
|
||||
- Chat messages often include text + images
|
||||
- Transport wildcard subscriptions route to correct recipients
|
||||
|
||||
#### 8.1b Step 1b: JavaScript Publishes via Transport
|
||||
|
||||
```javascript
|
||||
// Publish via WebSocket/NATS/MQTT
|
||||
const conn = await transportClient.connect({ servers: "ws://localhost:4222" });
|
||||
await conn.publish("/chat/user/v1/message", msgJson);
|
||||
```
|
||||
|
||||
#### 8.2 Step 2: Python Backend Receives
|
||||
|
||||
```python
|
||||
@@ -836,47 +856,203 @@ env = smartunpack(String(transport_msg.payload));
|
||||
- Same function signature across platforms
|
||||
- Type information enables proper deserialization
|
||||
|
||||
#### 8.4 Step 4: All Platforms Reply
|
||||
#### 8.4 Step 4: Complete End-to-End Flow
|
||||
|
||||
Each platform can reply using the same API:
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Sender["Sender (JavaScript)"]
|
||||
direction TB
|
||||
JS1["Create data tuples"]
|
||||
JS2["smartpack() → JSON string"]
|
||||
JS3["Publish via WebSocket"]
|
||||
end
|
||||
|
||||
```python
|
||||
# Python reply
|
||||
await smartpack(
|
||||
"/chat/user/v1/reply",
|
||||
[("response", "Nice!", "text")],
|
||||
reply_to="/chat/user/v1/message"
|
||||
);
|
||||
```
|
||||
subgraph Transport["Transport Layer"]
|
||||
direction TB
|
||||
BROKER["Message Broker"]
|
||||
end
|
||||
|
||||
```julia
|
||||
# Julia reply
|
||||
smartpack(
|
||||
"/chat/user/v1/reply",
|
||||
[("response", "Nice!", "text")],
|
||||
reply_to="/chat/user/v1/message"
|
||||
);
|
||||
```
|
||||
subgraph Receiver["Receiver (Julia/Python)"]
|
||||
direction TB
|
||||
REC1["Subscribe via WebSocket"]
|
||||
REC2["smartunpack(JSON string)"]
|
||||
REC3["Get payload tuples"]
|
||||
end
|
||||
|
||||
```javascript
|
||||
// JavaScript reply
|
||||
await msghandler.smartpack(
|
||||
"/chat/user/v1/reply",
|
||||
[["response", "Nice!", "text"]],
|
||||
{ reply_to: "/chat/user/v1/message" }
|
||||
);
|
||||
JS1 --> JS2
|
||||
JS2 --> JS3
|
||||
JS3 --> BROKER
|
||||
BROKER --> REC1
|
||||
REC1 --> REC2
|
||||
REC2 --> REC3
|
||||
```
|
||||
|
||||
**Rationale**:
|
||||
- Same API across platforms
|
||||
- Consistent behavior
|
||||
- Easy to maintain parity
|
||||
- `smartpack()` handles serialization, transport selection, and envelope building
|
||||
- Caller publishes JSON string via WebSocket/NATS/MQTT
|
||||
- `smartunpack()` handles transport detection, file downloads (if link), and deserialization
|
||||
- Same API across all platforms ensures consistency
|
||||
|
||||
#### 8.6 Step 6: Complete Cross-Platform Flow
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph JS["JavaScript (Frontend)"]
|
||||
J1["Create message tuples"]
|
||||
J2["smartpack()"]
|
||||
J3["Publish via WebSocket"]
|
||||
J4["Subscribe for replies"]
|
||||
end
|
||||
|
||||
subgraph Python["Python (Backend)"]
|
||||
P1["Subscribe via NATS"]
|
||||
P2["smartunpack()"]
|
||||
P3["Process data"]
|
||||
P4["smartpack()"]
|
||||
P5["Publish reply via NATS"]
|
||||
end
|
||||
|
||||
subgraph Julia["Julia (Backend)"]
|
||||
L1["Subscribe via MQTT"]
|
||||
L2["smartunpack()"]
|
||||
L3["Process data"]
|
||||
L4["smartpack()"]
|
||||
L5["Publish reply via MQTT"]
|
||||
end
|
||||
|
||||
subgraph Transport["Message Broker (NATS/MQTT/WebSocket)"]
|
||||
direction TB
|
||||
NATS["NATS: ws://localhost:4222"]
|
||||
MQTT["MQTT: broker.local:1883"]
|
||||
end
|
||||
|
||||
J2 --> J3
|
||||
J3 --> NATS
|
||||
NATS --> P1
|
||||
P1 --> P2
|
||||
P2 --> P3
|
||||
P3 --> P4
|
||||
P4 --> P5
|
||||
P5 --> MQTT
|
||||
MQTT --> L1
|
||||
L1 --> L2
|
||||
L2 --> L3
|
||||
L3 --> L4
|
||||
L4 --> L5
|
||||
L5 --> NATS
|
||||
NATS --> J4
|
||||
|
||||
style JS fill:#e3f2fd,stroke:#1976d2
|
||||
style Python fill:#fff3e0,stroke:#f57c00
|
||||
style Julia fill:#e8f5e9,stroke:#388e3c
|
||||
style Transport fill:#f3e5f5,stroke:#7b1fa2
|
||||
```
|
||||
|
||||
**Key Points**:
|
||||
- Same `smartpack()`/`smartunpack()` API across all platforms
|
||||
- JSON wire format ensures compatibility
|
||||
- Each platform uses their preferred transport protocol
|
||||
|
||||
---
|
||||
|
||||
## 9. Error Handling
|
||||
## 9. Transport Layer Integration
|
||||
|
||||
### 9.1 Common Error Scenarios
|
||||
msghandler is **transport-agnostic** - it only handles serialization/deserialization. The caller is responsible for:
|
||||
|
||||
### 9.1 Publishing Flow (Sender)
|
||||
|
||||
1. Call `smartpack()` → gets `(envelope, json_string)`
|
||||
2. Publish JSON string via transport (WebSocket/MQTT/NATS/etc.)
|
||||
3. Transport delivers message to subscriber
|
||||
|
||||
### 9.2 Subscribing Flow (Receiver)
|
||||
|
||||
1. Subscribe to transport topic
|
||||
2. Receive JSON string from transport
|
||||
3. Call `smartunpack(json_string)` → gets payloads array
|
||||
|
||||
### 9.3 Complete End-to-End Example
|
||||
|
||||
#### JavaScript → Julia Complete Flow
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph JavaScript["JavaScript Webapp"]
|
||||
J1["Create tuples"]
|
||||
J2["smartpack()"]
|
||||
J3["Publish via WebSocket"]
|
||||
end
|
||||
|
||||
subgraph Transport["NATS Broker"]
|
||||
direction TB
|
||||
NATS["ws://localhost:4222"]
|
||||
end
|
||||
|
||||
subgraph Julia["Julia Backend"]
|
||||
J4["Subscribe via WebSocket"]
|
||||
J5["smartunpack()"]
|
||||
J6["Process payloads"]
|
||||
end
|
||||
|
||||
J1 --> J2
|
||||
J2 --> J3
|
||||
J3 --> NATS
|
||||
NATS --> J4
|
||||
J4 --> J5
|
||||
J5 --> J6
|
||||
```
|
||||
|
||||
```javascript
|
||||
// JavaScript - Sender
|
||||
const [env, msgJson] = await msghandler.smartpack(
|
||||
"/agent/wine/api/v1/prompt",
|
||||
[["msg", "Hello!", "text"]],
|
||||
{ broker_url: "ws://localhost:4222" }
|
||||
);
|
||||
const conn = await nats.connect({ servers: "ws://localhost:4222" });
|
||||
await conn.publish("/agent/wine/api/v1/prompt", msgJson);
|
||||
```
|
||||
|
||||
```julia
|
||||
# Julia - Receiver
|
||||
transport_msg = transport_subscription.next()
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
# env["payloads"] is now: [("msg", "Hello!", "text")]
|
||||
```
|
||||
|
||||
#### Transport Examples (Single Platform)
|
||||
```javascript
|
||||
const conn = await nats.connect({ servers: "ws://localhost:4222" });
|
||||
await conn.publish(topic, json_string);
|
||||
```
|
||||
|
||||
#### MQTT (Python)
|
||||
```python
|
||||
await mqtt_client.publish(topic, json_string)
|
||||
```
|
||||
|
||||
#### WebSocket (Browser)
|
||||
```javascript
|
||||
const ws = new WebSocket("ws://localhost:4222");
|
||||
ws.send(json_string);
|
||||
```
|
||||
|
||||
#### Rust (Tokio)
|
||||
```rust
|
||||
conn.publish(topic, &json_str)?;
|
||||
```
|
||||
|
||||
### 9.4 Important Notes
|
||||
|
||||
- **JSON format**: All messages use JSON string format for cross-platform compatibility
|
||||
- **Caller responsibility**: Transport publishing/subscription is always the caller's code
|
||||
- **No vendor lock-in**: Works with any message broker that supports your platform
|
||||
|
||||
---
|
||||
|
||||
## 10. Error Handling
|
||||
|
||||
### 10.1 Common Error Scenarios
|
||||
|
||||
| Scenario | Error | Recovery |
|
||||
|----------|-------|----------|
|
||||
@@ -885,7 +1061,7 @@ await msghandler.smartpack(
|
||||
| Payload type mismatch | `DESERIALIZATION_ERROR` | Validate payload_type matches data |
|
||||
| Transport connection lost | `TRANSPORT_CONNECTION_FAILED` | Transport client auto-reconnects |
|
||||
|
||||
### 9.2 Error Response Format
|
||||
### 10.2 Error Response Format
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -903,9 +1079,9 @@ await msghandler.smartpack(
|
||||
|
||||
---
|
||||
|
||||
## 10. Debugging and Tracing
|
||||
## 11. Debugging and Tracing
|
||||
|
||||
### 10.1 Correlation ID Tracking
|
||||
### 11.1 Correlation ID Tracking
|
||||
|
||||
Every message includes a `correlation_id`:
|
||||
|
||||
@@ -935,9 +1111,9 @@ log_trace(correlation_id, "Published to transport");
|
||||
|
||||
---
|
||||
|
||||
## 11. Performance Considerations
|
||||
## 12. Performance Considerations
|
||||
|
||||
### 11.1 Optimization Strategies
|
||||
### 12.1 Optimization Strategies
|
||||
|
||||
| Strategy | Description | When to Use |
|
||||
|----------|-------------|-------------|
|
||||
@@ -945,7 +1121,7 @@ log_trace(correlation_id, "Published to transport");
|
||||
| Adjust size threshold | Increase threshold if file server slow | File server bottleneck |
|
||||
| Use direct transport | Avoid file server for small payloads | Low latency requirements |
|
||||
|
||||
### 11.2 Size Threshold by Platform
|
||||
### 12.2 Size Threshold by Platform
|
||||
|
||||
| Platform | Threshold | Notes |
|
||||
|----------|-----------|-------|
|
||||
@@ -957,7 +1133,7 @@ log_trace(correlation_id, "Published to transport");
|
||||
|
||||
---
|
||||
|
||||
## 12. Deployment Considerations
|
||||
## 13. Deployment Considerations
|
||||
|
||||
### 12.1 Minimum Infrastructure
|
||||
|
||||
@@ -1006,7 +1182,7 @@ log_trace(correlation_id, "Published to transport");
|
||||
|
||||
---
|
||||
|
||||
## 15. Gap-Check Validation
|
||||
## 14. Gap-Check Validation
|
||||
|
||||
| Stage Transition | Gap-Check Question | Status |
|
||||
|------------------|-------------------|--------|
|
||||
@@ -1017,6 +1193,27 @@ log_trace(correlation_id, "Published to transport");
|
||||
|
||||
---
|
||||
|
||||
## 15. Complete End-to-End Flow Summary
|
||||
|
||||
The msghandler transport-agnostic pattern works as follows:
|
||||
|
||||
```
|
||||
Sender: Transport Layer: Receiver:
|
||||
smartpack() Publish JSON string Subscribe
|
||||
↓ ↓ ↓
|
||||
(envelope, json_str) ────────────→ JSON string smartunpack()
|
||||
↓
|
||||
payloads array
|
||||
```
|
||||
|
||||
**Key Points**:
|
||||
- `smartpack()` handles serialization, transport selection, and envelope building
|
||||
- Caller publishes JSON string via WebSocket/NATS/MQTT
|
||||
- `smartunpack()` handles transport detection, file downloads (if link), and deserialization
|
||||
- Same API across all platforms ensures consistency
|
||||
|
||||
---
|
||||
|
||||
## 16. References
|
||||
|
||||
### 16.1 Documentation Artifacts
|
||||
@@ -1049,6 +1246,7 @@ log_trace(correlation_id, "Published to transport");
|
||||
|
||||
| Date | Version | Changes | Specification Reference | Solution Design Reference |
|
||||
|------|---------|---------|------------------------|--------------------------|
|
||||
| 2026-05-22 | 1.6.0 | Added complete transport integration examples with end-to-end flow diagrams | Section 9 | SD-006 |
|
||||
| 2026-05-22 | 1.5.0 | Updated to ASG Framework v8 pillars - aligned with specification and solution-design | All sections | All SD-XXX |
|
||||
| 2026-05-15 | 1.4.0 | Made transport layer agnostic | All sections | SD-001 through SD-008 |
|
||||
| 2026-05-14 | 1.3.0 | Updated Rust API to reflect `smartunpack` deserialization changes | All sections | SD-001 through SD-008 |
|
||||
@@ -1070,7 +1268,7 @@ log_trace(correlation_id, "Published to transport");
|
||||
| Requirements | ✅ Complete | requirements.md: FR-001 through FR-014, NFR-101 through NFR-405 |
|
||||
| Solution Design | ✅ Complete | solution-design.md: SD-001 through SD-008 |
|
||||
| Specification | ✅ Complete | specification.md: Section 2-19 |
|
||||
| Walkthrough | ✅ Complete | walkthrough.md: Sections 2-17 |
|
||||
| Walkthrough | ✅ Complete | walkthrough.md: Sections 2-18 (includes Transport Layer Integration) |
|
||||
| Implementation Plan | ⏳ Pending | implementation-plan.md |
|
||||
| Validation | ⏳ Pending | validation.md |
|
||||
| Runbook | ⏳ Pending | runbook.md |
|
||||
|
||||
Reference in New Issue
Block a user