From e93fe9c5c7769c0f52ebf6038216efe982f4bdab Mon Sep 17 00:00:00 2001 From: narawat Date: Fri, 22 May 2026 19:42:34 +0700 Subject: [PATCH] update docs --- docs/walkthrough.md | 536 ++++++++++++++++++++++++++++++-------------- 1 file changed, 367 insertions(+), 169 deletions(-) diff --git a/docs/walkthrough.md b/docs/walkthrough.md index 11f2716..fbd4d52 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -1,6 +1,6 @@ # Walkthrough: msghandler -**Version**: 1.5.0 +**Version**: 1.6.0 **Date**: 2026-05-22 **Status**: Active **Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl) @@ -16,21 +16,23 @@ This walkthrough serves as the primary onboarding guide for new developers and e - **User scenarios** - Real-world use cases from developer perspective - **Why steps are sequenced** - The rationale behind architectural decisions - **What could go wrong** - Common failure scenarios and recovery strategies +- **Transport integration** - Complete end-to-end flow with publish/subscribe patterns ### 1.1 Specification Traceability | Walkthrough Section | Specification Reference | Requirement ID(s) | Solution Design Ref(s) | Description | |---------------------|-------------------------|-------------------|------------------------|-------------| -| Section 2 (Big Picture) | specification.md:2, specification.md:15 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-002, SD-005, SD-006 | End-to-end system flow diagrams | +| Section 2 (Big Picture) | specification.md:2, specification.md:15 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-002, SD-005, SD-006 | End-to-end system flow diagrams and transport pattern | | Section 3 (Chat Scenario) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | FR-001, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-004, SD-005, SD-006 | Chat webapp ↔ Julia backend with mixed payloads | | Section 4 (Large File) | specification.md:6, specification.md:7 | FR-003, FR-004, FR-008, FR-009, FR-010, NFR-104, NFR-105 | SD-001, SD-002, SD-003, SD-007 | Large file transfer with link transport | | Section 5 (Tabular Data) | specification.md:5, specification.md:10 | FR-002, FR-012, NFR-101, NFR-102 | SD-004, SD-005 | Arrow IPC tabular data exchange | | Section 6 (MicroPython) | specification.md:13, specification.md:17 | FR-005, FR-006, FR-012, NFR-106 | SD-002, SD-004 | Memory-constrained device communication | | Section 7 (Cross-Platform) | specification.md:3, specification.md:4, specification.md:5, specification.md:11 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-002, SD-004, SD-005, SD-006 | Multi-platform chat application | -| Section 8 (Error Handling) | specification.md:9 | FR-008, FR-009, FR-010, NFR-201, NFR-202, NFR-203 | SD-003, SD-007 | Common error scenarios and recovery | -| Section 9 (Debugging) | specification.md:4, specification.md:11 | FR-011, NFR-401, NFR-403 | SD-008 | Correlation ID tracking | -| Section 10 (Performance) | specification.md:7, specification.md:13 | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 | SD-001, SD-002, SD-006 | Optimization strategies | -| Section 11 (Deployment) | specification.md:12, specification.md:18 | FR-013, FR-014, NFR-201, NFR-203 | SD-006 | Infrastructure requirements | +| Section 10 (Error Handling) | specification.md:9 | FR-008, FR-009, FR-010, NFR-201, NFR-202, NFR-203 | SD-003, SD-007 | Common error scenarios and recovery | +| Section 11 (Debugging) | specification.md:4, specification.md:11 | FR-011, NFR-401, NFR-403 | SD-008 | Correlation ID tracking | +| Section 12 (Performance) | specification.md:7, specification.md:13 | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 | SD-001, SD-002, SD-006 | Optimization strategies | +| Section 13 (Deployment) | specification.md:12, specification.md:18 | FR-013, FR-014, NFR-201, NFR-203 | SD-006 | Infrastructure requirements | +| Section 9 (Transport Layer) | specification.md:3, specification.md:5 | FR-013, FR-014 | SD-006 | Complete end-to-end transport integration examples | --- @@ -38,6 +40,47 @@ This walkthrough serves as the primary onboarding guide for new developers and e msghandler implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB): +### 2.0 Transport Layer Pattern + +**Critical**: msghandler is **transport-agnostic**. The caller is responsible for: + +1. **Sending side**: Call `smartpack()` → receive `(envelope, json_string)` → publish JSON string via transport +2. **Receiving side**: Subscribe to transport → receive JSON string → call `smartunpack()` + +```mermaid +flowchart LR + subgraph msghandler["msghandler Module"] + direction TB + S1["smartpack(data)"] + S2["Returns (envelope, json_str)"] + R1["smartunpack(json_str)"] + R2["Returns payloads"] + end + + subgraph Transport["Transport Layer (Caller's Responsibility)"] + direction TB + PUBLISH["Publish JSON string"] + SUBSCRIBE["Subscribe to messages"] + end + + S1 --> S2 + S2 --> PUBLISH + SUBSCRIBE --> R1 + R1 --> R2 + PUBLISH --> SUBSCRIBE + + style msghandler fill:#b3e5fc,stroke:#0288d1 + style Transport fill:#ffe0b2,stroke:#f57c00 +``` + +**Supported Transports**: +- NATS (WebSocket/TCP) +- MQTT (WebSocket/TCP) +- WebSocket (direct HTTP) +- HTTP (polling/long-polling) + +**Key Principle**: msghandler only handles **serialization/deserialization**. Transport is handled by caller. + ```mermaid flowchart TB subgraph msghandler["msghandler Module"] @@ -92,7 +135,7 @@ flowchart TB style FileServer fill:#ffe0b2,stroke:#f57c00 ``` -### 2.1 Key Design Principles +### 2.2 Key Design Principles | Principle | Description | Rationale | |-----------|-------------|-----------| @@ -100,6 +143,7 @@ flowchart TB | **Automatic Transport Selection** | Direct (< threshold) vs Link (≥ threshold) based on size | Optimizes memory vs network I/O trade-off | | **Cross-Platform API** | Consistent `smartpack()`/`smartunpack()` across all platforms | Simplifies developer experience | | **Exponential Backoff** | Retry downloads with increasing delays | Handles transient failures gracefully | +| **Transport Agnostic** | Caller handles transport (NATS/MQTT/WebSocket) | No vendor lock-in; works with any broker | --- @@ -109,68 +153,7 @@ flowchart TB A JavaScript chat webapp wants to send mixed payloads (text message + user avatar image) to a Julia backend, and receive mixed payloads (text response + AI-generated image) back. -### Complete End-to-End Round-Trip Flow - -msghandler implements a **transport-agnostic** messaging pattern. The library handles serialization and envelope building, while the caller is responsible for publishing/subscribing via their chosen transport (NATS, MQTT, WebSocket, HTTP, etc.). - -``` -┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ -│ END-TO-END ROUND-TRIP FLOW │ -└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ - -SENDER (JavaScript Webapp) TRANSPORT LAYER RECEIVER (Julia Backend) -┌─────────────────────┐ ┌───────────────────┐ ┌─────────────────────┐ -│ │ │ │ │ │ -│ 1. Prepare data │ │ │ │ │ -│ [(msg, data, │ │ │ │ │ -│ type)] │ │ │ │ │ -│ │ │ │ │ │ -│ 2. Call smartpack() │──────────────────────────────────────>│ │ │ │ -│ Returns (env, │ │ JSON Message │ │ │ -│ json_str) │ │ (json_str) │ │ │ -│ │ │ │ │ │ -│ 3. Publish via │ │ │ │ │ -│ transport: │──────────────────────────────────────>│ (NATS/MQTT/ │────────────────────────>│ 4. Subscribe via │ -│ MY_TRANSPORT. │ │ WebSocket, etc.) │ │ transport: │ -│ publish(subject, │ │ │ │ msg = │ -│ msgJson) │ │ │ │ SUBSCRIBE() │ -│ │ │ │ │ │ -│ │ │ │ │ 5. Call │ -│ │ │ │ │ smartunpack( │ -│ │ │ │ │ String( │ -│ │ │ │ │ msg.payload) │ -│ │ │ │ │ │ -│ │ │ │ │ 6. Receive │ -│ │ │ │ │ payloads: │ -│ │ │ │ │ [(msg, data, │ -│ │ │ │ │ type), ...] │ -│ │ │ │ │ │ -└─────────────────────┘ └───────────────────┘ └─────────────────────┘ - -┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ -│ JULIA BACKEND RESPONSE (Reverse Flow) │ -└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ - -RECEIVER (Julia Backend) TRANSPORT LAYER SENDER (JavaScript Webapp) -┌─────────────────────┐ ┌───────────────────┐ ┌─────────────────────┐ -│ │ │ │ │ │ -│ 7. Process data │ │ │ │ │ -│ (AI inference) │ │ │ │ │ -│ │ │ │ │ │ -│ 8. Call smartpack() │──────────────────────────────────────>│ │ │ │ -│ Returns (env, │ │ JSON Message │ │ │ -│ json_str) │ │ (json_str) │ │ │ -│ │ │ │ │ │ -│ 9. Publish reply │ │ │ │ │ -│ via transport: │──────────────────────────────────────>│ (NATS/MQTT/ │────────────────────────>│ 10. Subscribe via │ -│ MY_TRANSPORT. │ │ WebSocket, etc.) │ │ reply_to topic │ -│ publish(replyTo, │ │ │ │ │ -│ msgJson) │ │ │ │ │ -│ │ │ │ │ │ -└─────────────────────┘ └───────────────────┘ └─────────────────────┘ -``` - -### Step-by-Step Flow (Sender Perspective) +### Step-by-Step Flow #### 3.1 Step 1: JavaScript Webapp Sends Mixed Payloads @@ -195,6 +178,14 @@ const [env, msgJson] = await msghandler.smartpack( - **Why text first?** Text is smaller, sent via direct transport (fast, no file server needed) - **Why image second?** Images may trigger link transport if >0.5MB +#### 3.1b Step 1b: JavaScript Publishes via Transport + +```javascript +// Publish the JSON string via WebSocket (or NATS/MQTT) +const conn = await transportClient.connect({ servers: "ws://localhost:4222" }); +await conn.publish("/agent/wine/api/v1/prompt", msgJson); +``` + #### 3.2 Step 2: Transport Selection For each payload, msghandler determines transport: @@ -271,63 +262,28 @@ msghandler builds the message envelope: - **reply_to**: Tells backend where to send response - **payloads array**: Contains all data with metadata for proper handling -#### 3.5 Step 5: Publish to Transport (Caller's Responsibility) - -**Choose your transport** (replace `MY_TRANSPORT` with your actual library): +#### 3.5 Step 5: Publish JSON String via Transport ```javascript -// Example 1: NATS (Node.js) -import { connect } from 'nats'; -const nats = connect({ servers: 'ws://localhost:4222' }); -await nats.publish("/agent/wine/api/v1/prompt", msgJson); - -// Example 2: MQTT (Node.js) -import * as mqtt from 'mqtt'; -const client = mqtt.connect('ws://localhost:1883'); -client.publish("/agent/wine/api/v1/prompt", msgJson); - -// Example 3: WebSocket (Browser) -const ws = new WebSocket('ws://localhost:4222/ws'); -ws.send(msgJson); - -// Example 4: Custom HTTP POST -fetch('http://localhost:8000/publish', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: msgJson -}); +// JavaScript: Publish via WebSocket (NATS/MQTT/HTTP work similarly) +const conn = await transportClient.connect({ servers: "ws://localhost:4222" }); +await conn.publish("/agent/wine/api/v1/prompt", msgJson); ``` -**Why caller responsibility?** -- **Transport agnostic**: msghandler supports NATS, MQTT, WebSocket, HTTP, or any custom transport -- **Connection reuse**: Callers can manage connection pools efficiently -- **Flexibility**: No library lock-in; use whatever transport best fits your stack - -#### 3.6 Step 6: Julia Backend Receives Message +#### 3.6 Step 6: Julia Backend Receives and Unpacks ```julia # Julia backend -# Choose your transport (replace MY_TRANSPORT with your actual library) +transport_msg = transport_subscription.next() # Get message from transport +env = smartunpack(String(transport_msg.payload)) -# Example: NATS subscription -using NATS -conn = NATS.connect("nats.yiem.cc") -NATS.subscribe(conn, "/agent/wine/api/v1/prompt") do msg - env = smartunpack(String(msg.payload)) - - # env["payloads"] is now: - # [ - # ("msg", "Hello! I'm Ton.", "text"), - # ("avatar", binary_data, "image") - # ] -end +# env["payloads"] is now: +# [ +# ("msg", "Hello! I'm Ton.", "text"), +# ("avatar", binary_data, "image") +# ] ``` -**Rationale**: -- `smartunpack()` handles both transport types automatically -- Deserialization is type-aware based on `payload_type` -- Returns consistent tuple format regardless of transport - #### 3.7 Step 7: Julia Backend Sends Response ```julia @@ -342,13 +298,9 @@ env, msg_json = smartpack( ("generated_image", generated_image, "image") ], reply_to = "/chat/user/v1/message", - reply_to_msg_id = env["msg_id"] + reply_to_msg_id = msg["msg_id"] ); - -# Publish response via transport (caller's responsibility) -# Example: NATS -using NATS -NATS.publish(conn, "/agent/wine/api/v1/response", msg_json) +publish(transport_conn, "/agent/wine/api/v1/response", msg_json); ``` **Rationale**: @@ -381,6 +333,14 @@ const [env, msgJson] = await msghandler.smartpack( ); ``` +#### 4.1b Step 1b: JavaScript Publishes via Transport + +```javascript +// Publish the JSON string via WebSocket (or NATS/MQTT) +const conn = await transportClient.connect({ servers: "ws://localhost:4222" }); +await conn.publish("/agent/wine/api/v1/process", msgJson); +``` + #### 4.2 Step 2: Transport Selection (Link) | Payload | Size | Transport | Reason | @@ -440,7 +400,7 @@ const response = await plikOneshotUpload( - `transport: "link"` signals URL-based download - `encoding: "none"` indicates no additional encoding -#### 4.5 Step 5: Julia Backend Receives and Downloads +#### 4.5 Step 5: Julia Backend Receives, Downloads, and Unpacks ```julia # Julia backend @@ -448,8 +408,8 @@ transport_msg = transport_subscription.next(); env = smartunpack(String(transport_msg.payload)); # msghandler automatically: -# 1. Extracts URL from payload -# 2. Downloads with exponential backoff +# 1. Extracts URL from payload (link transport) +# 2. Downloads file with exponential backoff # 3. Deserializes to binary data ``` @@ -494,6 +454,13 @@ env, msg_json = await smartpack( - Arrow IPC format preserves data types - Much faster than JSON serialization +#### 5.1b Step 1b: Python Publishes via Transport + +```python +# Publish the JSON string via WebSocket/NATS/MQTT +await transport_publisher.publish("/agent/wine/api/v1/analyze", msg_json) +``` + #### 5.2 Step 2: Serialization to Arrow IPC ```python @@ -529,7 +496,7 @@ env = smartunpack(String(transport_msg.payload)); - DataFrame returned with correct types - No manual parsing needed -#### 5.4 Step 4: Julia Sends Results +#### 5.4 Step 4: Julia Sends Results (Complete Round-Trip) ```julia # Julia backend @@ -541,6 +508,7 @@ env, msg_json = smartpack( [("results", results, "arrowtable")], reply_to = "/python/worker/v1/results" ); +publish(transport_conn, "/agent/wine/api/v1/results", msg_json); ``` **Rationale**: @@ -643,7 +611,7 @@ let (envelope, json_str) = smartpack( }, ).await?; -// Caller publishes via transport +// Caller publishes via transport (WebSocket/MQTT/NATS) conn.publish("/agent/wine/api/v1/results", &json_str)?; ``` @@ -692,6 +660,9 @@ let (envelope, json_str) = smartpack( ..Default::default() }, ).await?; + +// Caller publishes via transport +conn.publish("/agent/wine/api/v1/upload", &json_str)?; ``` **Rationale**: @@ -735,6 +706,13 @@ env, msg_json = smartpack( - Smaller threshold (100KB) for memory constraints - Direct transport only (no file server support) +#### 7.1b Step 1b: MicroPython Publishes via Transport + +```python +# Publish the JSON string via MQTT (common on IoT devices) +mqtt_client.publish("/sensor/device/v1/readings", msg_json) +``` + #### 7.2 Step 2: Serialization ```python @@ -765,6 +743,40 @@ env = await smartunpack(str(transport_msg.payload)); - Dictionary returned directly - No Arrow support (memory constraints) +#### 7.4 Step 4: Complete MicroPython End-to-End Flow + +```mermaid +flowchart LR + subgraph MicroPython["MicroPython Device"] + MPY1["sensor_data = {temp, humidity}"] + MPY2["smartpack() → JSON"] + MPY3["publish via MQTT"] + end + + subgraph Transport["MQTT Broker"] + direction TB + MQTT["broker.local:1883"] + end + + subgraph Python["Python Backend"] + PY1["subscribe via MQTT"] + PY2["smartunpack(JSON)"] + PY3["get dictionary"] + end + + MPY1 --> MPY2 + MPY2 --> MPY3 + MPY3 --> MQTT + MQTT --> PY1 + PY1 --> PY2 + PY2 --> PY3 +``` + +**Key Points**: +- MicroPython limited to `text` and `dictionary` types (memory constraints) +- Direct transport only (no file server support) +- Size threshold: 100KB (vs 500KB for desktop) + --- ## 8. User Scenario 6: Cross-Platform Chat with Mixed Payloads @@ -798,6 +810,14 @@ const [env, msgJson] = await msghandler.smartpack( - Chat messages often include text + images - Transport wildcard subscriptions route to correct recipients +#### 8.1b Step 1b: JavaScript Publishes via Transport + +```javascript +// Publish via WebSocket/NATS/MQTT +const conn = await transportClient.connect({ servers: "ws://localhost:4222" }); +await conn.publish("/chat/user/v1/message", msgJson); +``` + #### 8.2 Step 2: Python Backend Receives ```python @@ -836,47 +856,203 @@ env = smartunpack(String(transport_msg.payload)); - Same function signature across platforms - Type information enables proper deserialization -#### 8.4 Step 4: All Platforms Reply +#### 8.4 Step 4: Complete End-to-End Flow -Each platform can reply using the same API: - -```python -# Python reply -await smartpack( - "/chat/user/v1/reply", - [("response", "Nice!", "text")], - reply_to="/chat/user/v1/message" -); -``` - -```julia -# Julia reply -smartpack( - "/chat/user/v1/reply", - [("response", "Nice!", "text")], - reply_to="/chat/user/v1/message" -); -``` - -```javascript -// JavaScript reply -await msghandler.smartpack( - "/chat/user/v1/reply", - [["response", "Nice!", "text"]], - { reply_to: "/chat/user/v1/message" } -); +```mermaid +flowchart TB + subgraph Sender["Sender (JavaScript)"] + direction TB + JS1["Create data tuples"] + JS2["smartpack() → JSON string"] + JS3["Publish via WebSocket"] + end + + subgraph Transport["Transport Layer"] + direction TB + BROKER["Message Broker"] + end + + subgraph Receiver["Receiver (Julia/Python)"] + direction TB + REC1["Subscribe via WebSocket"] + REC2["smartunpack(JSON string)"] + REC3["Get payload tuples"] + end + + JS1 --> JS2 + JS2 --> JS3 + JS3 --> BROKER + BROKER --> REC1 + REC1 --> REC2 + REC2 --> REC3 ``` **Rationale**: -- Same API across platforms -- Consistent behavior -- Easy to maintain parity +- `smartpack()` handles serialization, transport selection, and envelope building +- Caller publishes JSON string via WebSocket/NATS/MQTT +- `smartunpack()` handles transport detection, file downloads (if link), and deserialization +- Same API across all platforms ensures consistency + +#### 8.6 Step 6: Complete Cross-Platform Flow + +```mermaid +flowchart TB + subgraph JS["JavaScript (Frontend)"] + J1["Create message tuples"] + J2["smartpack()"] + J3["Publish via WebSocket"] + J4["Subscribe for replies"] + end + + subgraph Python["Python (Backend)"] + P1["Subscribe via NATS"] + P2["smartunpack()"] + P3["Process data"] + P4["smartpack()"] + P5["Publish reply via NATS"] + end + + subgraph Julia["Julia (Backend)"] + L1["Subscribe via MQTT"] + L2["smartunpack()"] + L3["Process data"] + L4["smartpack()"] + L5["Publish reply via MQTT"] + end + + subgraph Transport["Message Broker (NATS/MQTT/WebSocket)"] + direction TB + NATS["NATS: ws://localhost:4222"] + MQTT["MQTT: broker.local:1883"] + end + + J2 --> J3 + J3 --> NATS + NATS --> P1 + P1 --> P2 + P2 --> P3 + P3 --> P4 + P4 --> P5 + P5 --> MQTT + MQTT --> L1 + L1 --> L2 + L2 --> L3 + L3 --> L4 + L4 --> L5 + L5 --> NATS + NATS --> J4 + + style JS fill:#e3f2fd,stroke:#1976d2 + style Python fill:#fff3e0,stroke:#f57c00 + style Julia fill:#e8f5e9,stroke:#388e3c + style Transport fill:#f3e5f5,stroke:#7b1fa2 +``` + +**Key Points**: +- Same `smartpack()`/`smartunpack()` API across all platforms +- JSON wire format ensures compatibility +- Each platform uses their preferred transport protocol --- -## 9. Error Handling +## 9. Transport Layer Integration -### 9.1 Common Error Scenarios +msghandler is **transport-agnostic** - it only handles serialization/deserialization. The caller is responsible for: + +### 9.1 Publishing Flow (Sender) + +1. Call `smartpack()` → gets `(envelope, json_string)` +2. Publish JSON string via transport (WebSocket/MQTT/NATS/etc.) +3. Transport delivers message to subscriber + +### 9.2 Subscribing Flow (Receiver) + +1. Subscribe to transport topic +2. Receive JSON string from transport +3. Call `smartunpack(json_string)` → gets payloads array + +### 9.3 Complete End-to-End Example + +#### JavaScript → Julia Complete Flow + +```mermaid +flowchart LR + subgraph JavaScript["JavaScript Webapp"] + J1["Create tuples"] + J2["smartpack()"] + J3["Publish via WebSocket"] + end + + subgraph Transport["NATS Broker"] + direction TB + NATS["ws://localhost:4222"] + end + + subgraph Julia["Julia Backend"] + J4["Subscribe via WebSocket"] + J5["smartunpack()"] + J6["Process payloads"] + end + + J1 --> J2 + J2 --> J3 + J3 --> NATS + NATS --> J4 + J4 --> J5 + J5 --> J6 +``` + +```javascript +// JavaScript - Sender +const [env, msgJson] = await msghandler.smartpack( + "/agent/wine/api/v1/prompt", + [["msg", "Hello!", "text"]], + { broker_url: "ws://localhost:4222" } +); +const conn = await nats.connect({ servers: "ws://localhost:4222" }); +await conn.publish("/agent/wine/api/v1/prompt", msgJson); +``` + +```julia +# Julia - Receiver +transport_msg = transport_subscription.next() +env = smartunpack(String(transport_msg.payload)) +# env["payloads"] is now: [("msg", "Hello!", "text")] +``` + +#### Transport Examples (Single Platform) +```javascript +const conn = await nats.connect({ servers: "ws://localhost:4222" }); +await conn.publish(topic, json_string); +``` + +#### MQTT (Python) +```python +await mqtt_client.publish(topic, json_string) +``` + +#### WebSocket (Browser) +```javascript +const ws = new WebSocket("ws://localhost:4222"); +ws.send(json_string); +``` + +#### Rust (Tokio) +```rust +conn.publish(topic, &json_str)?; +``` + +### 9.4 Important Notes + +- **JSON format**: All messages use JSON string format for cross-platform compatibility +- **Caller responsibility**: Transport publishing/subscription is always the caller's code +- **No vendor lock-in**: Works with any message broker that supports your platform + +--- + +## 10. Error Handling + +### 10.1 Common Error Scenarios | Scenario | Error | Recovery | |----------|-------|----------| @@ -885,7 +1061,7 @@ await msghandler.smartpack( | Payload type mismatch | `DESERIALIZATION_ERROR` | Validate payload_type matches data | | Transport connection lost | `TRANSPORT_CONNECTION_FAILED` | Transport client auto-reconnects | -### 9.2 Error Response Format +### 10.2 Error Response Format ```json { @@ -903,9 +1079,9 @@ await msghandler.smartpack( --- -## 10. Debugging and Tracing +## 11. Debugging and Tracing -### 10.1 Correlation ID Tracking +### 11.1 Correlation ID Tracking Every message includes a `correlation_id`: @@ -935,9 +1111,9 @@ log_trace(correlation_id, "Published to transport"); --- -## 11. Performance Considerations +## 12. Performance Considerations -### 11.1 Optimization Strategies +### 12.1 Optimization Strategies | Strategy | Description | When to Use | |----------|-------------|-------------| @@ -945,7 +1121,7 @@ log_trace(correlation_id, "Published to transport"); | Adjust size threshold | Increase threshold if file server slow | File server bottleneck | | Use direct transport | Avoid file server for small payloads | Low latency requirements | -### 11.2 Size Threshold by Platform +### 12.2 Size Threshold by Platform | Platform | Threshold | Notes | |----------|-----------|-------| @@ -957,7 +1133,7 @@ log_trace(correlation_id, "Published to transport"); --- -## 12. Deployment Considerations +## 13. Deployment Considerations ### 12.1 Minimum Infrastructure @@ -1006,7 +1182,7 @@ log_trace(correlation_id, "Published to transport"); --- -## 15. Gap-Check Validation +## 14. Gap-Check Validation | Stage Transition | Gap-Check Question | Status | |------------------|-------------------|--------| @@ -1017,6 +1193,27 @@ log_trace(correlation_id, "Published to transport"); --- +## 15. Complete End-to-End Flow Summary + +The msghandler transport-agnostic pattern works as follows: + +``` +Sender: Transport Layer: Receiver: +smartpack() Publish JSON string Subscribe + ↓ ↓ ↓ +(envelope, json_str) ────────────→ JSON string smartunpack() + ↓ + payloads array +``` + +**Key Points**: +- `smartpack()` handles serialization, transport selection, and envelope building +- Caller publishes JSON string via WebSocket/NATS/MQTT +- `smartunpack()` handles transport detection, file downloads (if link), and deserialization +- Same API across all platforms ensures consistency + +--- + ## 16. References ### 16.1 Documentation Artifacts @@ -1049,6 +1246,7 @@ log_trace(correlation_id, "Published to transport"); | Date | Version | Changes | Specification Reference | Solution Design Reference | |------|---------|---------|------------------------|--------------------------| +| 2026-05-22 | 1.6.0 | Added complete transport integration examples with end-to-end flow diagrams | Section 9 | SD-006 | | 2026-05-22 | 1.5.0 | Updated to ASG Framework v8 pillars - aligned with specification and solution-design | All sections | All SD-XXX | | 2026-05-15 | 1.4.0 | Made transport layer agnostic | All sections | SD-001 through SD-008 | | 2026-05-14 | 1.3.0 | Updated Rust API to reflect `smartunpack` deserialization changes | All sections | SD-001 through SD-008 | @@ -1070,7 +1268,7 @@ log_trace(correlation_id, "Published to transport"); | Requirements | ✅ Complete | requirements.md: FR-001 through FR-014, NFR-101 through NFR-405 | | Solution Design | ✅ Complete | solution-design.md: SD-001 through SD-008 | | Specification | ✅ Complete | specification.md: Section 2-19 | -| Walkthrough | ✅ Complete | walkthrough.md: Sections 2-17 | +| Walkthrough | ✅ Complete | walkthrough.md: Sections 2-18 (includes Transport Layer Integration) | | Implementation Plan | ⏳ Pending | implementation-plan.md | | Validation | ⏳ Pending | validation.md | | Runbook | ⏳ Pending | runbook.md |