ton/msghandler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

update docs

Browse Source
This commit is contained in:
narawat
2026-05-22 10:49:18 +07:00
parent 6171923c05
commit 6da7708e72
1 changed files with 179 additions and 157 deletions
Download Patch File Download Diff File Expand all files Collapse all files

336
docs/walkthrough.md
View File

@@ -1,9 +1,10 @@
# Walkthrough: msghandler # Walkthrough: msghandler
**Version**: 1.4.0 **Version**: 1.5.0
**Date**: 2026-05-14 **Date**: 2026-05-22
**Status**: Active **Status**: Active
**Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl) **Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl)
**ASG Framework Alignment**: v8 pillars - Requirements → Solution Design → Specification → Walkthrough → Implementation Plan → Validation → Runbook
--- ---
@@ -18,24 +19,24 @@ This walkthrough serves as the primary onboarding guide for new developers and e
### 1.1 Specification Traceability ### 1.1 Specification Traceability
| Walkthrough Section | Specification Reference | Requirement ID(s) | Description | | Walkthrough Section | Specification Reference | Requirement ID(s) | Solution Design Ref(s) | Description |
|---------------------|-------------------------|-------------------|-------------| |---------------------|-------------------------|-------------------|------------------------|-------------|
| Section 2 (Big Picture) | specification.md:2, specification.md:15 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | End-to-end system flow diagrams | | Section 2 (Big Picture) | specification.md:2, specification.md:15 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-002, SD-005, SD-006 | End-to-end system flow diagrams |
| Section 3 (Chat Scenario) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | FR-001, FR-006, FR-007, FR-012, FR-013, FR-014 | Chat webapp ↔ Julia backend with mixed payloads | | Section 3 (Chat Scenario) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | FR-001, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-004, SD-005, SD-006 | Chat webapp ↔ Julia backend with mixed payloads |
| Section 4 (Large File) | specification.md:6, specification.md:7 | FR-003, FR-004, FR-008, FR-009, FR-010, NFR-104, NFR-105 | Large file transfer with link transport | | Section 4 (Large File) | specification.md:6, specification.md:7 | FR-003, FR-004, FR-008, FR-009, FR-010, NFR-104, NFR-105 | SD-001, SD-002, SD-003, SD-007 | Large file transfer with link transport |
| Section 5 (Tabular Data) | specification.md:5, specification.md:10 | FR-002, FR-012, NFR-101, NFR-102 | Arrow IPC tabular data exchange | | Section 5 (Tabular Data) | specification.md:5, specification.md:10 | FR-002, FR-012, NFR-101, NFR-102 | SD-004, SD-005 | Arrow IPC tabular data exchange |
| Section 6 (MicroPython) | specification.md:13, specification.md:17 | FR-005, FR-006, FR-012, NFR-106 | Memory-constrained device communication | | Section 6 (MicroPython) | specification.md:13, specification.md:17 | FR-005, FR-006, FR-012, NFR-106 | SD-002, SD-004 | Memory-constrained device communication |
| Section 7 (Cross-Platform) | specification.md:3, specification.md:4, specification.md:5, specification.md:11 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | Multi-platform chat application | | Section 7 (Cross-Platform) | specification.md:3, specification.md:4, specification.md:5, specification.md:11 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | SD-001, SD-002, SD-004, SD-005, SD-006 | Multi-platform chat application |
| Section 8 (Error Handling) | specification.md:9 | FR-008, FR-009, FR-010, NFR-201, NFR-202, NFR-203 | Common error scenarios and recovery | | Section 8 (Error Handling) | specification.md:9 | FR-008, FR-009, FR-010, NFR-201, NFR-202, NFR-203 | SD-003, SD-007 | Common error scenarios and recovery |
| Section 9 (Debugging) | specification.md:4, specification.md:11 | FR-011, NFR-401, NFR-403 | Correlation ID tracking | | Section 9 (Debugging) | specification.md:4, specification.md:11 | FR-011, NFR-401, NFR-403 | SD-008 | Correlation ID tracking |
| Section 10 (Performance) | specification.md:7, specification.md:13 | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 | Optimization strategies | | Section 10 (Performance) | specification.md:7, specification.md:13 | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 | SD-001, SD-002, SD-006 | Optimization strategies |
| Section 11 (Deployment) | specification.md:12, specification.md:18 | FR-013, FR-014, NFR-201, NFR-203 | Infrastructure requirements | | Section 11 (Deployment) | specification.md:12, specification.md:18 | FR-013, FR-014, NFR-201, NFR-203 | SD-006 | Infrastructure requirements |
--- ---
## 2. Overview: The Big Picture ## 2. Overview: The Big Picture
## Overview: The Big Picture ## 2. Overview: The Big Picture
msghandler implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB): msghandler implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB):
@@ -93,9 +94,7 @@ flowchart TB
style FileServer fill:#ffe0b2,stroke:#f57c00 style FileServer fill:#ffe0b2,stroke:#f57c00
``` ```
### Key Design Principles ### 2.1 Key Design Principles
### Key Design Principles
| Principle | Description | Rationale | | Principle | Description | Rationale |
|-----------|-------------|-----------| |-----------|-------------|-----------|
@@ -106,7 +105,7 @@ flowchart TB
--- ---
## User Scenario 1: Chat Webapp ↔ Julia Backend ## 3. User Scenario 1: Chat Webapp ↔ Julia Backend
### Scenario Description ### Scenario Description
@@ -114,7 +113,7 @@ A JavaScript chat webapp wants to send mixed payloads (text message + user avata
### Step-by-Step Flow ### Step-by-Step Flow
#### Step 1: JavaScript Webapp Sends Mixed Payloads #### 3.1 Step 1: JavaScript Webapp Sends Mixed Payloads
```javascript ```javascript
// JavaScript (Browser or Node.js) // JavaScript (Browser or Node.js)
@@ -137,7 +136,7 @@ const [env, msgJson] = await msghandler.smartpack(
- **Why text first?** Text is smaller, sent via direct transport (fast, no file server needed) - **Why text first?** Text is smaller, sent via direct transport (fast, no file server needed)
- **Why image second?** Images may trigger link transport if >0.5MB - **Why image second?** Images may trigger link transport if >0.5MB
#### Step 2: Transport Selection #### 3.2 Step 2: Transport Selection
For each payload, msghandler determines transport: For each payload, msghandler determines transport:
@@ -150,7 +149,7 @@ For each payload, msghandler determines transport:
- Direct transport is faster for small payloads (no file server round-trip) - Direct transport is faster for small payloads (no file server round-trip)
- Link transport is used when payload ≥ 0.5MB (avoids transport size limits) - Link transport is used when payload ≥ 0.5MB (avoids transport size limits)
#### Step 3: Serialization and Encoding #### 3.3 Step 3: Serialization and Encoding
Each payload is serialized: Each payload is serialized:
@@ -164,7 +163,7 @@ Each payload is serialized:
- Images use raw bytes to preserve binary data integrity - Images use raw bytes to preserve binary data integrity
- All payloads encoded as Base64 for JSON compatibility - All payloads encoded as Base64 for JSON compatibility
#### Step 4: Envelope Building #### 3.4 Step 4: Envelope Building
msghandler builds the message envelope: msghandler builds the message envelope:
@@ -213,7 +212,7 @@ msghandler builds the message envelope:
- **reply_to**: Tells backend where to send response - **reply_to**: Tells backend where to send response
- **payloads array**: Contains all data with metadata for proper handling - **payloads array**: Contains all data with metadata for proper handling
#### Step 5: Publish to Transport (Caller's Responsibility) #### 3.5 Step 5: Publish to Transport (Caller's Responsibility)
```javascript ```javascript
// Publishing via the transport layer is the caller's responsibility // Publishing via the transport layer is the caller's responsibility
@@ -227,7 +226,7 @@ msghandler builds the message envelope:
- JSON format ensures cross-platform compatibility - JSON format ensures cross-platform compatibility
- `smartpack()` returns `(env, msgJson)` - caller handles publishing via their chosen transport - `smartpack()` returns `(env, msgJson)` - caller handles publishing via their chosen transport
#### Step 6: Julia Backend Receives Message #### 3.6 Step 6: Julia Backend Receives Message
```julia ```julia
# Julia backend # Julia backend
@@ -246,12 +245,12 @@ env = smartunpack(String(transport_msg.payload))
- Deserialization is type-aware based on `payload_type` - Deserialization is type-aware based on `payload_type`
- Returns consistent tuple format regardless of transport - Returns consistent tuple format regardless of transport
#### Step 7: Julia Backend Sends Response #### 3.7 Step 7: Julia Backend Sends Response
```julia ```julia
# Julia backend processes the message # Julia backend processes the message
response_text = "Hello Ton! I'm the AI assistant." response_text = "Hello Ton! I'm the AI assistant."
generated_image = generate_ai_image(response_text) generated_image = generate_ai_image(response_text);
env, msg_json = smartpack( env, msg_json = smartpack(
"/agent/wine/api/v1/response", "/agent/wine/api/v1/response",
@@ -261,7 +260,7 @@ env, msg_json = smartpack(
], ],
reply_to = "/chat/user/v1/message", reply_to = "/chat/user/v1/message",
reply_to_msg_id = msg["msg_id"] reply_to_msg_id = msg["msg_id"]
) );
``` ```
**Rationale**: **Rationale**:
@@ -271,7 +270,7 @@ env, msg_json = smartpack(
--- ---
## User Scenario 2: Large File Transfer ## 4. User Scenario 2: Large File Transfer
### Scenario Description ### Scenario Description
@@ -279,7 +278,7 @@ A JavaScript webapp wants to upload a large file (10MB) to a Julia backend for p
### Step-by-Step Flow ### Step-by-Step Flow
#### Step 1: JavaScript Webapp Sends Large File #### 4.1 Step 1: JavaScript Webapp Sends Large File
```javascript ```javascript
const [env, msgJson] = await msghandler.smartpack( const [env, msgJson] = await msghandler.smartpack(
@@ -294,7 +293,7 @@ const [env, msgJson] = await msghandler.smartpack(
); );
``` ```
#### Step 2: Transport Selection (Link) #### 4.2 Step 2: Transport Selection (Link)
| Payload | Size | Transport | Reason | | Payload | Size | Transport | Reason |
|---------|------|-----------|--------| |---------|------|-----------|--------|
@@ -305,7 +304,7 @@ const [env, msgJson] = await msghandler.smartpack(
- File server handles large file upload - File server handles large file upload
- Transport only sends URL (small message) - Transport only sends URL (small message)
#### Step 3: File Server Upload #### 4.3 Step 3: File Server Upload
```javascript ```javascript
// msghandler internally calls: // msghandler internally calls:
@@ -329,7 +328,7 @@ const response = await plikOneshotUpload(
- One-shot mode simplifies API - One-shot mode simplifies API
- Returns URL for download - Returns URL for download
#### Step 4: Envelope with Link Transport #### 4.4 Step 4: Envelope with Link Transport
```json ```json
{ {
@@ -353,12 +352,12 @@ const response = await plikOneshotUpload(
- `transport: "link"` signals URL-based download - `transport: "link"` signals URL-based download
- `encoding: "none"` indicates no additional encoding - `encoding: "none"` indicates no additional encoding
#### Step 5: Julia Backend Receives and Downloads #### 4.5 Step 5: Julia Backend Receives and Downloads
```julia ```julia
# Julia backend # Julia backend
transport_msg = transport_subscription.next() transport_msg = transport_subscription.next();
env = smartunpack(String(transport_msg.payload)) env = smartunpack(String(transport_msg.payload));
# msghandler automatically: # msghandler automatically:
# 1. Extracts URL from payload # 1. Extracts URL from payload
@@ -373,7 +372,7 @@ env = smartunpack(String(transport_msg.payload))
--- ---
## User Scenario 3: Tabular Data Exchange ## 5. User Scenario 3: Tabular Data Exchange
### Scenario Description ### Scenario Description
@@ -381,7 +380,7 @@ A Python application sends tabular data (pandas DataFrame) to a Julia backend fo
### Step-by-Step Flow ### Step-by-Step Flow
#### Step 1: Python Sends Tabular Data #### 5.1 Step 1: Python Sends Tabular Data
```python ```python
# Python # Python
@@ -392,14 +391,14 @@ df = pd.DataFrame({
"id": [1, 2, 3], "id": [1, 2, 3],
"name": ["Alice", "Bob", "Charlie"], "name": ["Alice", "Bob", "Charlie"],
"score": [95, 88, 92] "score": [95, 88, 92]
}) });
env, msg_json = await smartpack( env, msg_json = await smartpack(
"/agent/wine/api/v1/analyze", "/agent/wine/api/v1/analyze",
[("data", df, "arrowtable")], [("data", df, "arrowtable")],
broker_url=DEFAULT_BROKER_URL, broker_url=DEFAULT_BROKER_URL,
receiver_name="agent-backend" receiver_name="agent-backend"
) );
``` ```
**Rationale**: **Rationale**:
@@ -407,18 +406,18 @@ env, msg_json = await smartpack(
- Arrow IPC format preserves data types - Arrow IPC format preserves data types
- Much faster than JSON serialization - Much faster than JSON serialization
#### Step 2: Serialization to Arrow IPC #### 5.2 Step 2: Serialization to Arrow IPC
```python ```python
# msghandler internally: # msghandler internally:
import pyarrow as pa import pyarrow as pa
import pyarrow.ipc as ipc import pyarrow.ipc as ipc
table = pa.Table.from_pandas(df) table = pa.Table.from_pandas(df);
buf = io.BytesIO() buf = io.BytesIO();
sink = ipc.new_file(buf, table.schema) sink = ipc.new_file(buf, table.schema);
ipc.write_table(table, sink) ipc.write_table(table, sink);
arrow_bytes = buf.getvalue() arrow_bytes = buf.getvalue();
``` ```
**Rationale**: **Rationale**:
@@ -426,12 +425,12 @@ arrow_bytes = buf.getvalue()
- Binary format is compact - Binary format is compact
- No schema information loss - No schema information loss
#### Step 3: Julia Receives and Deserializes #### 5.3 Step 3: Julia Receives and Deserializes
```julia ```julia
# Julia backend # Julia backend
transport_msg = transport_subscription.next() transport_msg = transport_subscription.next();
env = smartunpack(String(transport_msg.payload)) env = smartunpack(String(transport_msg.payload));
# env["payloads"][1] is now: # env["payloads"][1] is now:
# ("data", DataFrame with id, name, score columns, "arrowtable") # ("data", DataFrame with id, name, score columns, "arrowtable")
@@ -442,18 +441,18 @@ env = smartunpack(String(transport_msg.payload))
- DataFrame returned with correct types - DataFrame returned with correct types
- No manual parsing needed - No manual parsing needed
#### Step 4: Julia Sends Results #### 5.4 Step 4: Julia Sends Results
```julia ```julia
# Julia backend # Julia backend
results = analyze_data(env["payloads"][1][2]) results = analyze_data(env["payloads"][1][2]);
# Send results back # Send results back
env, msg_json = smartpack( env, msg_json = smartpack(
"/agent/wine/api/v1/results", "/agent/wine/api/v1/results",
[("results", results, "arrowtable")], [("results", results, "arrowtable")],
reply_to = "/python/worker/v1/results" reply_to = "/python/worker/v1/results"
) );
``` ```
**Rationale**: **Rationale**:
@@ -463,7 +462,7 @@ env, msg_json = smartpack(
--- ---
## User Scenario 4: Rust Service with Type-Safe API ## 6. User Scenario 4: Rust Service with Type-Safe API
### Scenario Description ### Scenario Description
@@ -471,7 +470,7 @@ A Rust service needs to process messages from a Julia analytics pipeline and sen
### Step-by-Step Flow ### Step-by-Step Flow
#### Step 1: Rust Service Receives Message #### 6.1 Step 1: Rust Service Receives Message
```rust ```rust
// Rust service - using tokio async runtime // Rust service - using tokio async runtime
@@ -525,7 +524,7 @@ async fn main() {
- **smartunpack deserialization**: Payload data is deserialized and stored as strings in `payload.data` - **smartunpack deserialization**: Payload data is deserialized and stored as strings in `payload.data`
- **Type dispatch**: `payload_type` field determines how to interpret the `data` string - **Type dispatch**: `payload_type` field determines how to interpret the `data` string
#### Step 2: Rust Service Sends Processed Results #### 6.2 Step 2: Rust Service Sends Processed Results
```rust ```rust
// Rust service sends results back with mixed payload types // Rust service sends results back with mixed payload types
@@ -566,11 +565,11 @@ conn.publish("/agent/wine/api/v1/results", &json_str)?;
- **Default options**: sensible defaults reduce boilerplate - **Default options**: sensible defaults reduce boilerplate
- **Result<T, E>**: idiomatic Rust error handling - **Result<T, E>**: idiomatic Rust error handling
#### Step 3: Python/Julia Receives Rust Response #### 6.3 Step 3: Python/Julia Receives Rust Response
```python ```python
# Python backend receives Rust response # Python backend receives Rust response
env = await smartunpack(str(transport_msg.payload)) env = await smartunpack(str(transport_msg.payload));
# env["payloads"][0] is now: # env["payloads"][0] is now:
# ("results", arrow_table_data, "arrowtable") # ("results", arrow_table_data, "arrowtable")
@@ -583,7 +582,7 @@ env = await smartunpack(str(transport_msg.payload))
- **Same JSON wire format**: No protocol translation needed - **Same JSON wire format**: No protocol translation needed
- **Type preservation**: Arrow IPC and text types preserved across all platforms - **Type preservation**: Arrow IPC and text types preserved across all platforms
#### Step 4: Large File Transfer from Rust #### 6.4 Step 4: Large File Transfer from Rust
```rust ```rust
// Rust service sends large binary file via link transport // Rust service sends large binary file via link transport
@@ -615,7 +614,7 @@ let (envelope, json_str) = smartpack(
--- ---
## User Scenario 5: MicroPython Device ## 7. User Scenario 5: MicroPython Device
### Scenario Description ### Scenario Description
@@ -623,7 +622,7 @@ A MicroPython sensor device sends sensor readings to a Python backend.
### Step-by-Step Flow ### Step-by-Step Flow
#### Step 1: MicroPython Sends Sensor Data #### 7.1 Step 1: MicroPython Sends Sensor Data
```python ```python
# MicroPython # MicroPython
@@ -633,14 +632,14 @@ sensor_data = {
"temperature": 25.5, "temperature": 25.5,
"humidity": 60.0, "humidity": 60.0,
"pressure": 1013.25 "pressure": 1013.25
} };
env, msg_json = smartpack( env, msg_json = smartpack(
"/sensor/device/v1/readings", "/sensor/device/v1/readings",
[("data", sensor_data, "dictionary")], [("data", sensor_data, "dictionary")],
broker_url=DEFAULT_BROKER_URL, broker_url=DEFAULT_BROKER_URL,
size_threshold=100000 # 100KB for MicroPython size_threshold=100000 # 100KB for MicroPython
) );
``` ```
**Rationale**: **Rationale**:
@@ -648,13 +647,13 @@ env, msg_json = smartpack(
- Smaller threshold (100KB) for memory constraints - Smaller threshold (100KB) for memory constraints
- Direct transport only (no file server support) - Direct transport only (no file server support)
#### Step 2: Serialization #### 7.2 Step 2: Serialization
```python ```python
# msghandler internally: # msghandler internally:
json_str = json.dumps(sensor_data) json_str = json.dumps(sensor_data);
json_bytes = json_str.encode('utf-8') json_bytes = json_str.encode('utf-8');
payload_b64 = base64.b64encode(json_bytes).decode('ascii') payload_b64 = base64.b64encode(json_bytes).decode('ascii');
``` ```
**Rationale**: **Rationale**:
@@ -662,12 +661,12 @@ payload_b64 = base64.b64encode(json_bytes).decode('ascii')
- Base64 for transport compatibility - Base64 for transport compatibility
- UTF-8 for text encoding - UTF-8 for text encoding
#### Step 3: Python Backend Receives #### 7.3 Step 3: Python Backend Receives
```python ```python
# Python backend # Python backend
transport_msg = await transport_consumer.next() transport_msg = await transport_consumer.next();
env = await smartunpack(str(transport_msg.payload)) env = await smartunpack(str(transport_msg.payload));
# env["payloads"][0] is now: # env["payloads"][0] is now:
# ("data", {"temperature": 25.5, "humidity": 60.0, ...}, "dictionary") # ("data", {"temperature": 25.5, "humidity": 60.0, ...}, "dictionary")
@@ -680,7 +679,7 @@ env = await smartunpack(str(transport_msg.payload))
--- ---
## User Scenario 6: Cross-Platform Chat with Mixed Payloads ## 8. User Scenario 6: Cross-Platform Chat with Mixed Payloads
### Scenario Description ### Scenario Description
@@ -688,7 +687,7 @@ Multiple platforms (JavaScript, Python, Julia) communicate in a chat application
### Step-by-Step Flow ### Step-by-Step Flow
#### Step 1: JavaScript Sends Chat Message #### 8.1 Step 1: JavaScript Sends Chat Message
```javascript ```javascript
// JavaScript (Frontend) // JavaScript (Frontend)
@@ -711,12 +710,12 @@ const [env, msgJson] = await msghandler.smartpack(
- Chat messages often include text + images - Chat messages often include text + images
- Transport wildcard subscriptions route to correct recipients - Transport wildcard subscriptions route to correct recipients
#### Step 2: Python Backend Receives #### 8.2 Step 2: Python Backend Receives
```python ```python
# Python (Backend) # Python (Backend)
transport_msg = await transport_consumer.next() transport_msg = await transport_consumer.next();
env = await smartunpack(str(transport_msg.payload)) env = await smartunpack(str(transport_msg.payload));
# env["payloads"] is now: # env["payloads"] is now:
# [ # [
@@ -730,12 +729,12 @@ env = await smartunpack(str(transport_msg.payload))
- Same payload structure regardless of sender - Same payload structure regardless of sender
- Type information preserved - Type information preserved
#### Step 3: Julia Backend Receives #### 8.3 Step 3: Julia Backend Receives
```julia ```julia
# Julia (Backend) # Julia (Backend)
transport_msg = transport_subscription.next() transport_msg = transport_subscription.next();
env = smartunpack(String(transport_msg.payload)) env = smartunpack(String(transport_msg.payload));
# env["payloads"] is now: # env["payloads"] is now:
# [ # [
@@ -749,7 +748,7 @@ env = smartunpack(String(transport_msg.payload))
- Same function signature across platforms - Same function signature across platforms
- Type information enables proper deserialization - Type information enables proper deserialization
#### Step 4: All Platforms Reply #### 8.4 Step 4: All Platforms Reply
Each platform can reply using the same API: Each platform can reply using the same API:
@@ -759,7 +758,7 @@ await smartpack(
"/chat/user/v1/reply", "/chat/user/v1/reply",
[("response", "Nice!", "text")], [("response", "Nice!", "text")],
reply_to="/chat/user/v1/message" reply_to="/chat/user/v1/message"
) );
``` ```
```julia ```julia
@@ -768,7 +767,7 @@ smartpack(
"/chat/user/v1/reply", "/chat/user/v1/reply",
[("response", "Nice!", "text")], [("response", "Nice!", "text")],
reply_to="/chat/user/v1/message" reply_to="/chat/user/v1/message"
) );
``` ```
```javascript ```javascript
@@ -787,9 +786,9 @@ await msghandler.smartpack(
--- ---
## Error Handling ## 9. Error Handling
### Common Error Scenarios ### 9.1 Common Error Scenarios
| Scenario | Error | Recovery | | Scenario | Error | Recovery |
|----------|-------|----------| |----------|-------|----------|
@@ -798,7 +797,7 @@ await msghandler.smartpack(
| Payload type mismatch | `DESERIALIZATION_ERROR` | Validate payload_type matches data | | Payload type mismatch | `DESERIALIZATION_ERROR` | Validate payload_type matches data |
| Transport connection lost | `TRANSPORT_CONNECTION_FAILED` | Transport client auto-reconnects | | Transport connection lost | `TRANSPORT_CONNECTION_FAILED` | Transport client auto-reconnects |
### Error Response Format ### 9.2 Error Response Format
```json ```json
{ {
@@ -816,20 +815,27 @@ await msghandler.smartpack(
--- ---
## Debugging and Tracing ## 10. Debugging and Tracing
### Correlation ID Tracking ### 10.1 Correlation ID Tracking
Every message includes a `correlation_id`: Every message includes a `correlation_id`:
```julia ```julia
# At start of request # At start of request
correlation_id = string(uuid4()) correlation_id = string(uuid4());
# Use throughout the flow # Use throughout the flow
log_trace(correlation_id, "Starting smartpack") log_trace(correlation_id, "Starting smartpack");
log_trace(correlation_id, "Serialized payload size: 100 bytes") log_trace(correlation_id, "Serialized payload size: 100 bytes");
log_trace(correlation_id, "Published to transport") log_trace(correlation_id, "Published to transport");
```
**Log Format**:
```
[2026-03-13T16:30:00.000Z] [Correlation: abc123...] Starting smartpack
[2026-03-13T16:30:00.001Z] [Correlation: abc123...] Serialized payload size: 100 bytes
[2026-03-13T16:30:00.002Z] [Correlation: abc123...] Published to transport
``` ```
**Log Format**: **Log Format**:
@@ -841,9 +847,9 @@ log_trace(correlation_id, "Published to transport")
--- ---
## Performance Considerations ## 11. Performance Considerations
### Optimization Strategies ### 11.1 Optimization Strategies
| Strategy | Description | When to Use | | Strategy | Description | When to Use |
|----------|-------------|-------------| |----------|-------------|-------------|
@@ -851,7 +857,7 @@ log_trace(correlation_id, "Published to transport")
| Adjust size threshold | Increase threshold if file server slow | File server bottleneck | | Adjust size threshold | Increase threshold if file server slow | File server bottleneck |
| Use direct transport | Avoid file server for small payloads | Low latency requirements | | Use direct transport | Avoid file server for small payloads | Low latency requirements |
### Size Threshold by Platform ### 11.2 Size Threshold by Platform
| Platform | Threshold | Notes | | Platform | Threshold | Notes |
|----------|-----------|-------| |----------|-----------|-------|
@@ -863,9 +869,9 @@ log_trace(correlation_id, "Published to transport")
--- ---
## Deployment Considerations ## 12. Deployment Considerations
### Minimum Infrastructure ### 12.1 Minimum Infrastructure
| Component | Minimum | Notes | | Component | Minimum | Notes |
|-----------|---------|-------| |-----------|---------|-------|
@@ -874,7 +880,7 @@ log_trace(correlation_id, "Published to transport")
| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) | | Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices | | Client Memory | 256KB | MicroPython devices |
### Environment Variables ### 12.2 Environment Variables
| Variable | Default | Description | | Variable | Default | Description |
|----------|---------|-------------| |----------|---------|-------------|
@@ -888,64 +894,31 @@ log_trace(correlation_id, "Published to transport")
| Date | Version | Changes | | Date | Version | Changes |
|------|---------|---------| |------|---------|---------|
| 2026-05-22 | 1.5.0 | Updated to ASG Framework v8 pillars - aligned with specification and solution-design |
| - | - | Added solution design traceability (SD-XXX) to specification reference table |
| - | - | Added ASG framework alignment header to document |
| 2026-05-15 | 1.4.0 | Made transport layer agnostic |
| - | - | Removed all NATS-specific references from walkthrough |
| - | - | Updated code examples to use transport-agnostic patterns |
| - | - | Updated diagrams to remove NATS-specific labels |
| 2026-05-14 | 1.3.0 | Updated Rust API to reflect `smartunpack` deserialization changes |
| - | - | `smartunpack` now stores deserialized data in `MsgPayloadV1.data` |
| - | - | Added `plik_upload_file` convenience function documentation |
| - | - | Fixed Rust scenario payload access (data is String, not Payload enum) |
| - | - | Removed `metadata` from link transport examples |
| 2026-05-13 | 1.2.0 | Added Rust support with tokio, serde, and arrow2 |
| - | - | Added Rust user scenario (User Scenario 4) |
| - | - | Updated scenario numbering (MicroPython → Scenario 5, Cross-Platform → Scenario 6) |
| 2026-05-13 | 1.1.0 | Aligned with ground truth implementation (src/msghandler.jl) |
| - | - | Updated smartunpack calls to use transport payload pattern |
| - | - | Removed NATSClient.publish() calls (caller responsible for transport publishing) |
| - | - | Removed is_publish and nats_connection parameter references |
| 2026-03-23 | 1.0.0 | Updated to ASG Framework walkthrough guidelines |
| 2026-03-13 | 1.0.0 | Initial walkthrough documentation | | 2026-03-13 | 1.0.0 | Initial walkthrough documentation |
--- ---
## 12. References ## 15. Gap-Check Validation
### 12.1 Documentation Artifacts
| Document | Purpose | Specification Traceability |
|----------|---------|---------------------------|
| [`docs/requirements.md`](./requirements.md) | Business requirements and user stories | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/specification.md`](./specification.md) | Technical contract for msghandler | specification.md:2-19 (all sections) |
| [`docs/ui-specification.md`](./ui-specification.md) | UI specification for client applications | UI components for data entry and display |
| [`docs/walkthrough.md`](./walkthrough.md) | End-to-end system flow | This document |
| [`docs/architecture.md`](./architecture.md) | System architecture diagrams | Component interaction and data flow |
| [`docs/validation.md`](./validation.md) | CI/CD validation rules | Contract testing and spec compliance |
| [`docs/runbook.md`](./runbook.md) | Operational runbook | Deployment, scaling, and troubleshooting |
### 12.2 Implementation Files
| File | Platform | Features | Specification Traceability |
|------|----------|----------|---------------------------|
| [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) |
| [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | specification.md:2-19 (all sections) |
| [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe, file upload helpers | specification.md:2-19 (all sections) |
| [`src/msghandler_mpy.py`](../src/msghandler_mpy.py) | MicroPython | Limited to direct transport | specification.md:2-19 (all sections) |
---
## 13. Change Log
| Date | Version | Changes | Specification Reference |
|------|---------|---------|------------------------|
| 2026-05-15 | 1.5.0 | Made transport layer agnostic | All sections |
| - | - | Removed all NATS-specific references from walkthrough | All sections |
| - | - | Updated code examples to use transport-agnostic patterns | All sections |
| - | - | Updated diagrams to remove NATS-specific labels | All sections |
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartunpack` deserialization changes | All sections |
| - | - | `smartunpack` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
| - | - | Added `plik_upload_file` convenience function documentation | specification.md:13 |
| - | - | Fixed Rust scenario payload access (data is String, not Payload enum) | All sections |
| - | - | Removed `metadata` from link transport examples | specification.md:3 |
| 2026-05-13 | 1.3.0 | Added Rust support with tokio, serde, and arrow2 | All sections |
| - | - | Added Rust user scenario (User Scenario 4) | specification.md:11 (Rust API) |
| - | - | Updated scenario numbering (MicroPython → Scenario 5, Cross-Platform → Scenario 6) | All sections |
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All sections |
| - | - | Updated smartunpack calls to use transport payload pattern | All sections |
| - | - | Removed NATSClient.publish() calls (caller responsible for transport publishing) | All sections |
| - | - | Removed is_publish and nats_connection parameter references | All sections |
| 2026-03-23 | 1.0.0 | Updated to ASG Framework walkthrough guidelines | All sections |
| 2026-03-13 | 1.0.0 | Initial walkthrough documentation | specification.md:2-19 (all sections) |
---
## 14. Gap-Check Validation
| Stage Transition | Gap-Check Question | Status | | Stage Transition | Gap-Check Question | Status |
|------------------|-------------------|--------| |------------------|-------------------|--------|
@@ -956,15 +929,64 @@ log_trace(correlation_id, "Published to transport")
--- ---
*This walkthrough document is versioned and maintained in git alongside the codebase. All implementations must adhere to this documentation.* ## 16. References
### 16.1 Documentation Artifacts
| Document | Purpose | Specification Traceability | Solution Design Traceability |
|----------|---------|---------------------------|------------------------------|
| [`docs/requirements.md`](./requirements.md) | Business requirements and user stories | FR-001 through FR-014, NFR-101 through NFR-405 | SD-001 through SD-008 |
| [`docs/specification.md`](./specification.md) | Technical contract for msghandler | specification.md:2-19 (all sections) | SD-001 through SD-008 |
| [`docs/ui-specification.md`](./ui-specification.md) | UI specification for client applications | UI components for data entry and display | UI components reference FR-XXX and SD-XXX |
| [`docs/walkthrough.md`](./walkthrough.md) | End-to-end system flow | This document | Full flow validation against SD-XXX |
| [`docs/architecture.md`](./architecture.md) | System architecture diagrams | Component interaction and data flow | Component-to-SD mapping |
| [`docs/validation.md`](./validation.md) | CI/CD validation rules | Contract testing and spec compliance | Validation gates for SD-XXX |
| [`docs/runbook.md`](./runbook.md) | Operational runbook | Deployment, scaling, and troubleshooting | Operation-to-SD mapping |
### 16.2 Implementation Files
| File | Platform | Features | Specification Traceability | Solution Design Traceability |
|------|----------|----------|---------------------------|------------------------------|
| [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | FR-001 through FR-014, NFR-101 through NFR-405 | SD-001 through SD-008 |
| [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 | SD-001 through SD-008 |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | FR-001 through FR-014, NFR-101 through NFR-405 | SD-001 through SD-008 |
| [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 | SD-001 through SD-008 |
| [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 | SD-001 through SD-008 |
| [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe, file upload helpers | FR-001 through FR-014, NFR-101 through NFR-405 | SD-001 through SD-008 |
| [`src/msghandler_mpy.py`](../src/msghandler_mpy.py) | MicroPython | Limited to direct transport | FR-005, FR-006, FR-012 | SD-002, SD-004 |
---
## 17. Change Log
| Date | Version | Changes | Specification Reference | Solution Design Reference |
|------|---------|---------|------------------------|--------------------------|
| 2026-05-22 | 1.5.0 | Updated to ASG Framework v8 pillars - aligned with specification and solution-design | All sections | All SD-XXX |
| 2026-05-15 | 1.4.0 | Made transport layer agnostic | All sections | SD-001 through SD-008 |
| 2026-05-14 | 1.3.0 | Updated Rust API to reflect `smartunpack` deserialization changes | All sections | SD-001 through SD-008 |
| 2026-05-13 | 1.2.0 | Added Rust support with tokio, serde, and arrow2 | specification.md:11 (Rust API) | SD-001 through SD-008 |
| 2026-05-13 | 1.1.0 | Aligned with ground truth implementation (src/msghandler.jl) | All sections | SD-001 through SD-008 |
| 2026-03-23 | 1.0.0 | Updated to ASG Framework walkthrough guidelines | All sections | SD-001 through SD-008 |
| 2026-03-13 | 1.0.0 | Initial walkthrough documentation | specification.md:2-19 (all sections) | SD-001 through SD-008 |
--- ---
*This walkthrough document is versioned and maintained in git alongside the codebase. All implementations must adhere to this documentation.* *This walkthrough document is versioned and maintained in git alongside the codebase. All implementations must adhere to this documentation.*
<tool_call> ---
<function=update_todo_list>
<parameter=todos> ## 18. ASG Framework Validation
[x] Analyze existing documentation (requirements.md, spec.md, architecture.md)
[x] Read all source files in src/ folder | Pillar | Status | Reference |
[x] Write docs/walkthrough.md according to SDD framework with user scenarios |--------|--------|-----------|
| Requirements | ✅ Complete | requirements.md: FR-001 through FR-014, NFR-101 through NFR-405 |
| Solution Design | ✅ Complete | solution-design.md: SD-001 through SD-008 |
| Specification | ✅ Complete | specification.md: Section 2-19 |
| Walkthrough | ✅ Complete | walkthrough.md: Sections 2-17 |
| Implementation Plan | ⏳ Pending | implementation-plan.md |
| Validation | ⏳ Pending | validation.md |
| Runbook | ⏳ Pending | runbook.md |
---
*This walkthrough document is versioned and maintained in git alongside the codebase. All implementations must adhere to this documentation.*