ton/msghandler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

1st commit

Browse Source
This commit is contained in:
narawat
2026-05-15 08:58:30 +07:00
commit 0e24b7d044
28 changed files with 15184 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

965
docs/walkthrough.md Normal file
View File

@@ -0,0 +1,965 @@
# Walkthrough: msghandler
**Version**: 1.4.0
**Date**: 2026-05-14
**Status**: Active
**Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl)
---
## 1. Executive Summary
This document provides the **end-to-end trace** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus.
This walkthrough serves as the primary onboarding guide for new developers and explains:
- **User scenarios** - Real-world use cases from developer perspective
- **Why steps are sequenced** - The rationale behind architectural decisions
- **What could go wrong** - Common failure scenarios and recovery strategies
### 1.1 Specification Traceability
| Walkthrough Section | Specification Reference | Requirement ID(s) | Description |
|---------------------|-------------------------|-------------------|-------------|
| Section 2 (Big Picture) | specification.md:2, specification.md:15 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | End-to-end system flow diagrams |
| Section 3 (Chat Scenario) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | FR-001, FR-006, FR-007, FR-012, FR-013, FR-014 | Chat webapp ↔ Julia backend with mixed payloads |
| Section 4 (Large File) | specification.md:6, specification.md:7 | FR-003, FR-004, FR-008, FR-009, FR-010, NFR-104, NFR-105 | Large file transfer with link transport |
| Section 5 (Tabular Data) | specification.md:5, specification.md:10 | FR-002, FR-012, NFR-101, NFR-102 | Arrow IPC tabular data exchange |
| Section 6 (MicroPython) | specification.md:13, specification.md:17 | FR-005, FR-006, FR-012, NFR-106 | Memory-constrained device communication |
| Section 7 (Cross-Platform) | specification.md:3, specification.md:4, specification.md:5, specification.md:11 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | Multi-platform chat application |
| Section 8 (Error Handling) | specification.md:9 | FR-008, FR-009, FR-010, NFR-201, NFR-202, NFR-203 | Common error scenarios and recovery |
| Section 9 (Debugging) | specification.md:4, specification.md:11 | FR-011, NFR-401, NFR-403 | Correlation ID tracking |
| Section 10 (Performance) | specification.md:7, specification.md:13 | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 | Optimization strategies |
| Section 11 (Deployment) | specification.md:12, specification.md:18 | FR-013, FR-014, NFR-201, NFR-203 | Infrastructure requirements |
---
## 2. Overview: The Big Picture
## Overview: The Big Picture
msghandler implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB):
```mermaid
flowchart TB
subgraph msghandler["msghandler Module"]
direction TB
subgraph Sender["Sender (smartsend)"]
direction LR
S1["Data Tuples<br/>[(dataname, data, type)]"]
S2["Serialize Data"]
S3["Size Check"]
S4["Transport Selection"]
S5["Build Envelope"]
S6["Publish to NATS"]
S1 --> S2
S2 --> S3
S3 --> S4
S4 --> S5
S5 --> S6
end
subgraph Receiver["Receiver (smartreceive)"]
direction LR
R1["Subscribe to NATS"]
R2["Parse Envelope"]
R3["Check Transport"]
R4["Deserialize Data"]
R5["Return Payloads"]
R1 --> R2
R2 --> R3
R3 --> R4
R4 --> R5
end
S6 -.->|Message| R1
end
subgraph FileServer["HTTP File Server (Plik)"]
direction TB
FS1["Upload URL"]
FS2["Download URL"]
S4 -.->|Large Payload| FS1
FS1 -.->|URL| S5
R3 -.->|Fetch URL| FS2
end
style msghandler fill:#e1f5fe,stroke:#0288d1,stroke-width:2px
style Sender fill:#b3e5fc,stroke:#0288d1
style Receiver fill:#b3e5fc,stroke:#0288d1
style FileServer fill:#ffe0b2,stroke:#f57c00
```
### Key Design Principles
### Key Design Principles
| Principle | Description | Rationale |
|-----------|-------------|-----------|
| **Claim-Check Pattern** | Large payloads uploaded to HTTP server, URL sent via NATS | NATS has message size limits; avoids NATS overflow |
| **Automatic Transport Selection** | Direct (< threshold) vs Link (≥ threshold) based on size | Optimizes memory vs network I/O trade-off |
| **Cross-Platform API** | Consistent `smartsend()`/`smartreceive()` across all platforms | Simplifies developer experience |
| **Exponential Backoff** | Retry downloads with increasing delays | Handles transient failures gracefully |
---
## User Scenario 1: Chat Webapp ↔ Julia Backend
### Scenario Description
A JavaScript chat webapp wants to send mixed payloads (text message + user avatar image) to a Julia backend, and receive mixed payloads (text response + AI-generated image) back.
### Step-by-Step Flow
#### Step 1: JavaScript Webapp Sends Mixed Payloads
```javascript
// JavaScript (Browser or Node.js)
const [env, msgJson] = await msghandler.smartsend(
"/agent/wine/api/v1/prompt",
[
["msg", "Hello! I'm Ton.", "text"],
["avatar", avatarImageData, "image"]
],
{
broker_url: "ws://localhost:4222",
receiver_name: "agent-backend",
msg_purpose: "chat"
}
);
```
**Rationale**:
- **Why mixed payloads?** Real chat apps often send both text and images together
- **Why text first?** Text is smaller, sent via direct transport (fast, no file server needed)
- **Why image second?** Images may trigger link transport if >0.5MB
#### Step 2: Transport Selection
For each payload, msghandler determines transport:
| Payload | Size | Transport | Reason |
|---------|------|-----------|--------|
| `"msg"` (text) | ~20 bytes | direct | < 0.5MB threshold |
| `"avatar"` (image) | ~150KB | direct | < 0.5MB threshold |
**Rationale**:
- Direct transport is faster for small payloads (no file server round-trip)
- Link transport is used when payload ≥ 0.5MB (avoids NATS size limits)
#### Step 3: Serialization and Encoding
Each payload is serialized:
| Payload | Type | Serialization | Encoding |
|---------|------|---------------|----------|
| `"msg"` | `text` | UTF-8 bytes | Base64 |
| `"avatar"` | `image` | Raw bytes | Base64 |
**Rationale**:
- Text uses UTF-8 encoding for human-readable data
- Images use raw bytes to preserve binary data integrity
- All payloads encoded as Base64 for JSON compatibility
#### Step 4: Envelope Building
msghandler builds the message envelope:
```json
{
"correlation_id": "a1b2c3d4...",
"msg_id": "e5f6g7h8...",
"timestamp": "2026-03-13T16:30:00.000Z",
"send_to": "/agent/wine/api/v1/prompt",
"msg_purpose": "chat",
"sender_name": "chat-webapp",
"sender_id": "sender-uuid...",
"receiver_name": "agent-backend",
"receiver_id": "",
"reply_to": "/agent/wine/api/v1/response",
"reply_to_msg_id": "",
"broker_url": "ws://localhost:4222",
"metadata": {},
"payloads": [
{
"id": "payload-uuid...",
"dataname": "msg",
"payload_type": "text",
"transport": "direct",
"encoding": "base64",
"size": 20,
"data": "SGVsbG8hIEknIHRlbCB5b3UgSW4gZW5nbGlzaC4=",
"metadata": {"payload_bytes": 20}
},
{
"id": "payload-uuid...",
"dataname": "avatar",
"payload_type": "image",
"transport": "direct",
"encoding": "base64",
"size": 150000,
"data": "iVBORw0KGgoAAAANSUhEUgAA...",
"metadata": {"payload_bytes": 150000}
}
]
}
```
**Rationale**:
- **correlation_id**: Tracks this chat session across all systems
- **reply_to**: Tells backend where to send response
- **payloads array**: Contains all data with metadata for proper handling
#### Step 5: Publish to NATS (Caller's Responsibility)
```javascript
// NATS publishing is the caller's responsibility
const conn = await NATS.connect({ servers: "ws://localhost:4222" });
await conn.publish("/agent/wine/api/v1/prompt", msgJson);
```
**Rationale**:
- NATS provides low-latency message delivery
- JSON format ensures cross-platform compatibility
- `smartsend()` returns `(env, msgJson)` - caller handles publishing
#### Step 6: Julia Backend Receives Message
```julia
# Julia backend
nats_msg = NATS.subscription.next() # Get message from NATS
env = smartreceive(String(nats_msg.payload))
# env["payloads"] is now:
# [
# ("msg", "Hello! I'm Ton.", "text"),
# ("avatar", binary_data, "image")
# ]
```
**Rationale**:
- `smartreceive()` handles both transport types automatically
- Deserialization is type-aware based on `payload_type`
- Returns consistent tuple format regardless of transport
#### Step 7: Julia Backend Sends Response
```julia
# Julia backend processes the message
response_text = "Hello Ton! I'm the AI assistant."
generated_image = generate_ai_image(response_text)
env, msg_json = smartsend(
"/agent/wine/api/v1/response",
[
("response", response_text, "text"),
("generated_image", generated_image, "image")
],
reply_to = "/chat/user/v1/message",
reply_to_msg_id = msg["msg_id"]
)
```
**Rationale**:
- **Mixed response**: Text explanation + AI-generated image
- **reply_to**: Ensures response goes to correct topic
- **reply_to_msg_id**: Links response to original message for tracing
---
## User Scenario 2: Large File Transfer
### Scenario Description
A JavaScript webapp wants to upload a large file (10MB) to a Julia backend for processing.
### Step-by-Step Flow
#### Step 1: JavaScript Webapp Sends Large File
```javascript
const [env, msgJson] = await msghandler.smartsend(
"/agent/wine/api/v1/process",
[
["file", largeFileData, "binary"]
],
{
broker_url: "ws://localhost:4222",
receiver_name: "agent-backend"
}
);
```
#### Step 2: Transport Selection (Link)
| Payload | Size | Transport | Reason |
|---------|------|-----------|--------|
| `"file"` | 10MB | link | ≥ 0.5MB threshold |
**Rationale**:
- Link transport used for large payloads
- File server handles large file upload
- NATS only sends URL (small message)
#### Step 3: File Server Upload
```javascript
// msghandler internally calls:
const response = await plikOneshotUpload(
"http://localhost:8080",
"file",
largeFileData
);
// Response:
// {
// status: 200,
// uploadid: "UPLOAD_ID",
// fileid: "FILE_ID",
// url: "http://localhost:8080/file/UPLOAD_ID/FILE_ID/file"
// }
```
**Rationale**:
- Plik handles multipart upload
- One-shot mode simplifies API
- Returns URL for download
#### Step 4: Envelope with Link Transport
```json
{
"correlation_id": "a1b2c3d4...",
"payloads": [
{
"id": "payload-uuid...",
"dataname": "file",
"payload_type": "binary",
"transport": "link",
"encoding": "none",
"size": 10000000,
"data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/file"
}
]
}
```
**Rationale**:
- `data` field contains URL instead of Base64
- `transport: "link"` signals URL-based download
- `encoding: "none"` indicates no additional encoding
#### Step 5: Julia Backend Receives and Downloads
```julia
# Julia backend
nats_msg = NATS.subscription.next()
env = smartreceive(String(nats_msg.payload))
# msghandler automatically:
# 1. Extracts URL from payload
# 2. Downloads with exponential backoff
# 3. Deserializes to binary data
```
**Rationale**:
- Exponential backoff handles transient failures
- Automatic download simplifies receiver code
- Binary data returned directly
---
## User Scenario 3: Tabular Data Exchange
### Scenario Description
A Python application sends tabular data (pandas DataFrame) to a Julia backend for analysis, and receives processed results back.
### Step-by-Step Flow
#### Step 1: Python Sends Tabular Data
```python
# Python
import pandas as pd
from msghandler import smartsend
df = pd.DataFrame({
"id": [1, 2, 3],
"name": ["Alice", "Bob", "Charlie"],
"score": [95, 88, 92]
})
env, msg_json = await smartsend(
"/agent/wine/api/v1/analyze",
[("data", df, "arrowtable")],
broker_url="nats://localhost:4222",
receiver_name="agent-backend"
)
```
**Rationale**:
- `arrowtable` type for efficient tabular data transfer
- Arrow IPC format preserves data types
- Much faster than JSON serialization
#### Step 2: Serialization to Arrow IPC
```python
# msghandler internally:
import pyarrow as pa
import pyarrow.ipc as ipc
table = pa.Table.from_pandas(df)
buf = io.BytesIO()
sink = ipc.new_file(buf, table.schema)
ipc.write_table(table, sink)
arrow_bytes = buf.getvalue()
```
**Rationale**:
- Arrow IPC preserves column types
- Binary format is compact
- No schema information loss
#### Step 3: Julia Receives and Deserializes
```julia
# Julia backend
nats_msg = NATS.subscription.next()
env = smartreceive(String(nats_msg.payload))
# env["payloads"][1] is now:
# ("data", DataFrame with id, name, score columns, "arrowtable")
```
**Rationale**:
- Arrow.jl reads IPC format directly
- DataFrame returned with correct types
- No manual parsing needed
#### Step 4: Julia Sends Results
```julia
# Julia backend
results = analyze_data(env["payloads"][1][2])
# Send results back
env, msg_json = smartsend(
"/agent/wine/api/v1/results",
[("results", results, "arrowtable")],
reply_to = "/python/worker/v1/results"
)
```
**Rationale**:
- Arrow IPC format for efficient round-trip
- Results preserve DataFrame structure
- Python can deserialize to pandas DataFrame
---
## User Scenario 4: Rust Service with Type-Safe API
### Scenario Description
A Rust service needs to process messages from a Julia analytics pipeline and send typed results back. The Rust implementation leverages compile-time type safety via Rust enums and serde for serialization.
### Step-by-Step Flow
#### Step 1: Rust Service Receives Message
```rust
// Rust service - using tokio async runtime
use msghandler::{smartreceive, MsgEnvelopeV1};
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64};
#[tokio::main]
async fn main() {
let conn = nats::connect("nats://localhost:4222").unwrap();
// Subscribe and receive messages
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze").unwrap();
for msg in sub.messages() {
let envelope = smartreceive(
&String::from_utf8_lossy(&msg.payload),
&Default::default(),
).await.unwrap();
// Access deserialized payloads by type
for payload in &envelope.payloads {
match payload.payload_type.as_str() {
"arrowtable" => {
// Data is base64-encoded Arrow IPC bytes after smartreceive()
let arrow_bytes = BASE64.decode(&payload.data).unwrap();
println!("Received arrowtable payload ({} bytes)", arrow_bytes.len());
},
"text" => {
// Data is the decoded text string
println!("Message: {}", payload.data);
},
"image" | "audio" | "video" | "binary" => {
// Data is base64-encoded binary content
let bytes = BASE64.decode(&payload.data).unwrap();
println!("Received {} bytes of {} data", bytes.len(), payload.payload_type);
},
"dictionary" | "jsontable" => {
// Data is a JSON string
println!("Data: {}", payload.data);
},
_ => println!("Unknown payload type: {}", payload.payload_type),
}
}
}
}
```
**Rationale**:
- **serde serialization**: Automatic JSON deserialization to `MsgEnvelopeV1`
- **tokio runtime**: Efficient async I/O for NATS and HTTP operations
- **smartreceive deserialization**: Payload data is deserialized and stored as strings in `payload.data`
- **Type dispatch**: `payload_type` field determines how to interpret the `data` string
#### Step 2: Rust Service Sends Processed Results
```rust
// Rust service sends results back with mixed payload types
use msghandler::{smartsend, Payload, SmartsendOptions};
let results_df = /* processed Arrow table */;
let result_bytes = /* serialize to Arrow IPC */;
let (envelope, json_str) = smartsend(
"/agent/wine/api/v1/results",
&[
(
"results".to_string(),
Payload::ArrowTable(result_bytes),
"arrowtable".to_string(),
),
(
"summary".to_string(),
Payload::Text("Analysis complete: 1500 rows processed".to_string()),
"text".to_string(),
),
],
&SmartsendOptions {
broker_url: "nats://localhost:4222".to_string(),
reply_to: "/python/worker/v1/results".to_string(),
msg_purpose: "chat".to_string(),
..Default::default()
},
).await?;
// Caller publishes to NATS
conn.publish("/agent/wine/api/v1/results", &json_str)?;
```
**Rationale**:
- **Builder pattern**: `SmartsendOptions` provides clean configuration
- **Enum-based payloads**: Type safety prevents sending incorrect data types
- **Default options**: sensible defaults reduce boilerplate
- **Result<T, E>**: idiomatic Rust error handling
#### Step 3: Python/Julia Receives Rust Response
```python
# Python backend receives Rust response
env = await smartreceive(str(nats_msg.payload))
# env["payloads"][0] is now:
# ("results", arrow_table_data, "arrowtable")
# env["payloads"][1] is now:
# ("summary", "Analysis complete: 1500 rows processed", "text")
```
**Rationale**:
- **Cross-platform parity**: Rust envelope matches other platform envelopes exactly
- **Same JSON wire format**: No protocol translation needed
- **Type preservation**: Arrow IPC and text types preserved across all platforms
#### Step 4: Large File Transfer from Rust
```rust
// Rust service sends large binary file via link transport
let large_file_data: Vec<u8> = std::fs::read("/data/large_dataset.parquet")?;
let (envelope, json_str) = smartsend(
"/agent/wine/api/v1/upload",
&[
(
"dataset".to_string(),
Payload::Binary(large_file_data),
"binary".to_string(),
),
],
&SmartsendOptions {
broker_url: "nats://localhost:4222".to_string(),
fileserver_url: "http://localhost:8080".to_string(),
size_threshold: 500_000, // 0.5MB triggers link transport
..Default::default()
},
).await?;
```
**Rationale**:
- **Automatic transport selection**: Same 0.5MB threshold as other desktop platforms
- **reqwest integration**: Efficient HTTP client for file server upload/download
- **Exponential backoff**: Built-in retry with configurable parameters
- **Zero-copy where possible**: `Vec<u8>` passed directly without intermediate copies
---
## User Scenario 5: MicroPython Device
### Scenario Description
A MicroPython sensor device sends sensor readings to a Python backend.
### Step-by-Step Flow
#### Step 1: MicroPython Sends Sensor Data
```python
# MicroPython
from msghandler import smartsend
sensor_data = {
"temperature": 25.5,
"humidity": 60.0,
"pressure": 1013.25
}
env, msg_json = smartsend(
"/sensor/device/v1/readings",
[("data", sensor_data, "dictionary")],
broker_url="nats://localhost:4222",
size_threshold=100000 # 100KB for MicroPython
)
```
**Rationale**:
- `dictionary` type for JSON-serializable sensor data
- Smaller threshold (100KB) for memory constraints
- Direct transport only (no file server support)
#### Step 2: Serialization
```python
# msghandler internally:
json_str = json.dumps(sensor_data)
json_bytes = json_str.encode('utf-8')
payload_b64 = base64.b64encode(json_bytes).decode('ascii')
```
**Rationale**:
- JSON format for human-readable data
- Base64 for NATS compatibility
- UTF-8 for text encoding
#### Step 3: Python Backend Receives
```python
# Python backend
nats_msg = await nats_consumer.next()
env = await smartreceive(str(nats_msg.payload))
# env["payloads"][0] is now:
# ("data", {"temperature": 25.5, "humidity": 60.0, ...}, "dictionary")
```
**Rationale**:
- JSON deserialization
- Dictionary returned directly
- No Arrow support (memory constraints)
---
## User Scenario 6: Cross-Platform Chat with Mixed Payloads
### Scenario Description
Multiple platforms (JavaScript, Python, Julia) communicate in a chat application with mixed payload types.
### Step-by-Step Flow
#### Step 1: JavaScript Sends Chat Message
```javascript
// JavaScript (Frontend)
const [env, msgJson] = await msghandler.smartsend(
"/chat/user/v1/message",
[
["text", "Check this out!", "text"],
["image", imageData, "image"]
],
{
broker_url: "ws://localhost:4222",
receiver_name: "",
msg_purpose: "chat"
}
);
```
**Rationale**:
- Empty `receiver_name` = broadcast to all subscribers
- Chat messages often include text + images
- NATS wildcard subscriptions route to correct recipients
#### Step 2: Python Backend Receives
```python
# Python (Backend)
nats_msg = await nats_consumer.next()
env = await smartreceive(str(nats_msg.payload))
# env["payloads"] is now:
# [
# ("text", "Check this out!", "text"),
# ("image", binary_data, "image")
# ]
```
**Rationale**:
- Consistent API across platforms
- Same payload structure regardless of sender
- Type information preserved
#### Step 3: Julia Backend Receives
```julia
# Julia (Backend)
nats_msg = NATS.subscription.next()
env = smartreceive(String(nats_msg.payload))
# env["payloads"] is now:
# [
# ("text", "Check this out!", "text"),
# ("image", binary_data, "image")
# ]
```
**Rationale**:
- Cross-platform API parity
- Same function signature across platforms
- Type information enables proper deserialization
#### Step 4: All Platforms Reply
Each platform can reply using the same API:
```python
# Python reply
await smartsend(
"/chat/user/v1/reply",
[("response", "Nice!", "text")],
reply_to="/chat/user/v1/message"
)
```
```julia
# Julia reply
smartsend(
"/chat/user/v1/reply",
[("response", "Nice!", "text")],
reply_to="/chat/user/v1/message"
)
```
```javascript
// JavaScript reply
await msghandler.smartsend(
"/chat/user/v1/reply",
[["response", "Nice!", "text"]],
{ reply_to: "/chat/user/v1/message" }
);
```
**Rationale**:
- Same API across platforms
- Consistent behavior
- Easy to maintain parity
---
## Error Handling
### Common Error Scenarios
| Scenario | Error | Recovery |
|----------|-------|----------|
| File server unavailable | `UPLOAD_FAILED` | Fall back to direct transport or smaller payloads |
| File server download fails | `DOWNLOAD_FAILED` | Retry with exponential backoff |
| Payload type mismatch | `DESERIALIZATION_ERROR` | Validate payload_type matches data |
| NATS connection lost | `NATS_CONNECTION_FAILED` | NATS client auto-reconnects |
### Error Response Format
```json
{
"correlation_id": "abc123...",
"error": {
"code": "DOWNLOAD_FAILED",
"message": "Failed to fetch data after 5 attempts",
"details": {
"url": "http://localhost:8080/file/...",
"correlation_id": "abc123..."
}
}
}
```
---
## Debugging and Tracing
### Correlation ID Tracking
Every message includes a `correlation_id`:
```julia
# At start of request
correlation_id = string(uuid4())
# Use throughout the flow
log_trace(correlation_id, "Starting smartsend")
log_trace(correlation_id, "Serialized payload size: 100 bytes")
log_trace(correlation_id, "Published to NATS")
```
**Log Format**:
```
[2026-03-13T16:30:00.000Z] [Correlation: abc123...] Starting smartsend
[2026-03-13T16:30:00.001Z] [Correlation: abc123...] Serialized payload size: 100 bytes
[2026-03-13T16:30:00.002Z] [Correlation: abc123...] Published to NATS
```
---
## Performance Considerations
### Optimization Strategies
| Strategy | Description | When to Use |
|----------|-------------|-------------|
| Pre-create NATS connection | Reuse connection for multiple sends | High-throughput scenarios |
| Adjust size threshold | Increase threshold if file server slow | File server bottleneck |
| Use direct transport | Avoid file server for small payloads | Low latency requirements |
### Size Threshold by Platform
| Platform | Threshold | Notes |
|----------|-----------|-------|
| Desktop (Julia/JS/Python/Dart) | 500,000 bytes (0.5MB) | Default threshold |
| Dart Desktop | 500,000 bytes (0.5MB) | Default threshold |
| Dart Flutter | 500,000 bytes (0.5MB) | Default threshold |
| Dart Web | 500,000 bytes (0.5MB) | Default threshold |
| MicroPython | 100,000 bytes (100KB) | Lower threshold for memory constraints |
---
## Deployment Considerations
### Minimum Infrastructure
| Component | Minimum | Notes |
|-----------|---------|-------|
| NATS Server | 1 instance | Single node for development |
| File Server | 1 instance | HTTP server for large payloads |
| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices |
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) |
---
## Change Log
| Date | Version | Changes |
|------|---------|---------|
| 2026-03-13 | 1.0.0 | Initial walkthrough documentation |
---
## 12. References
### 12.1 Documentation Artifacts
| Document | Purpose | Specification Traceability |
|----------|---------|---------------------------|
| [`docs/requirements.md`](./requirements.md) | Business requirements and user stories | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/specification.md`](./specification.md) | Technical contract for msghandler | specification.md:2-19 (all sections) |
| [`docs/ui-specification.md`](./ui-specification.md) | UI specification for client applications | UI components for data entry and display |
| [`docs/walkthrough.md`](./walkthrough.md) | End-to-end system flow | This document |
| [`docs/architecture.md`](./architecture.md) | System architecture diagrams | Component interaction and data flow |
| [`docs/validation.md`](./validation.md) | CI/CD validation rules | Contract testing and spec compliance |
| [`docs/runbook.md`](./runbook.md) | Operational runbook | Deployment, scaling, and troubleshooting |
### 12.2 Implementation Files
| File | Platform | Features | Specification Traceability |
|------|----------|----------|---------------------------|
| [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) |
| [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only, WebSocket NATS | specification.md:2-19 (all sections) |
| [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe, file upload helpers | specification.md:2-19 (all sections) |
| [`src/msghandler_mpy.py`](../src/msghandler_mpy.py) | MicroPython | Limited to direct transport | specification.md:2-19 (all sections) |
---
## 13. Change Log
| Date | Version | Changes | Specification Reference |
|------|---------|---------|------------------------|
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections |
| - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
| - | - | Added `plik_upload_file` convenience function documentation | specification.md:13 |
| - | - | Fixed Rust scenario payload access (data is String, not Payload enum) | All sections |
| - | - | Removed `metadata` from link transport examples | specification.md:3 |
| 2026-05-13 | 1.3.0 | Added Rust support with tokio, serde, and arrow2 | All sections |
| - | - | Added Rust user scenario (User Scenario 4) | specification.md:11 (Rust API) |
| - | - | Updated scenario numbering (MicroPython → Scenario 5, Cross-Platform → Scenario 6) | All sections |
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All sections |
| - | - | Updated smartreceive calls to use String(nats_msg.payload) pattern | All sections |
| - | - | Removed NATSClient.publish() calls (caller responsible for NATS publishing) | All sections |
| - | - | Removed is_publish and nats_connection parameter references | All sections |
| 2026-03-23 | 1.0.0 | Updated to ASG Framework walkthrough guidelines | All sections |
| 2026-03-13 | 1.0.0 | Initial walkthrough documentation | specification.md:2-19 (all sections) |
---
## 14. Gap-Check Validation
| Stage Transition | Gap-Check Question | Status |
|------------------|-------------------|--------|
| Requirements → Specification | Does the Specification define all edge cases and conflict scenarios from the Requirements? | ✅ Verified - All FR-XXX requirements have corresponding spec rules |
| Specification → UI Specification | Does the UI Specification expose all the data and states defined in the Specification? | ⏳ Pending - UI spec not yet created |
| UI Specification → Walkthrough | Does the Walkthrough reflect the complete flow including error states and timing? | ⏳ Pending - UI spec not yet created |
| Walkthrough → Architecture | Does the Architecture support the performance and integration requirements defined in the Walkthrough? | ⏳ Pending - Architecture not yet created |
---
*This walkthrough document is versioned and maintained in git alongside the codebase. All implementations must adhere to this documentation.*
---
*This walkthrough document is versioned and maintained in git alongside the codebase. All implementations must adhere to this documentation.*
<tool_call>
<function=update_todo_list>
<parameter=todos>
[x] Analyze existing documentation (requirements.md, spec.md, architecture.md)
[x] Read all source files in src/ folder
[x] Write docs/walkthrough.md according to SDD framework with user scenarios