rename to smartpack n smartunpack
This commit is contained in:
@@ -44,7 +44,7 @@ flowchart TB
|
||||
subgraph msghandler["msghandler Module"]
|
||||
direction TB
|
||||
|
||||
subgraph Sender["Sender (smartsend)"]
|
||||
subgraph Sender["Sender (smartpack)"]
|
||||
direction LR
|
||||
S1["Data Tuples<br/>[(dataname, data, type)]"]
|
||||
S2["Serialize Data"]
|
||||
@@ -60,7 +60,7 @@ flowchart TB
|
||||
S5 --> S6
|
||||
end
|
||||
|
||||
subgraph Receiver["Receiver (smartreceive)"]
|
||||
subgraph Receiver["Receiver (smartunpack)"]
|
||||
direction LR
|
||||
R1["Subscribe via transport"]
|
||||
R2["Parse Envelope"]
|
||||
@@ -101,7 +101,7 @@ flowchart TB
|
||||
|-----------|-------------|-----------|
|
||||
| **Claim-Check Pattern** | Large payloads uploaded to HTTP server, URL sent via transport | Transport has message size limits; avoids overflow |
|
||||
| **Automatic Transport Selection** | Direct (< threshold) vs Link (≥ threshold) based on size | Optimizes memory vs network I/O trade-off |
|
||||
| **Cross-Platform API** | Consistent `smartsend()`/`smartreceive()` across all platforms | Simplifies developer experience |
|
||||
| **Cross-Platform API** | Consistent `smartpack()`/`smartunpack()` across all platforms | Simplifies developer experience |
|
||||
| **Exponential Backoff** | Retry downloads with increasing delays | Handles transient failures gracefully |
|
||||
|
||||
---
|
||||
@@ -118,7 +118,7 @@ A JavaScript chat webapp wants to send mixed payloads (text message + user avata
|
||||
|
||||
```javascript
|
||||
// JavaScript (Browser or Node.js)
|
||||
const [env, msgJson] = await msghandler.smartsend(
|
||||
const [env, msgJson] = await msghandler.smartpack(
|
||||
"/agent/wine/api/v1/prompt",
|
||||
[
|
||||
["msg", "Hello! I'm Ton.", "text"],
|
||||
@@ -225,14 +225,14 @@ msghandler builds the message envelope:
|
||||
**Rationale**:
|
||||
- The transport layer provides message delivery (NATS, MQTT, WebSocket, etc.)
|
||||
- JSON format ensures cross-platform compatibility
|
||||
- `smartsend()` returns `(env, msgJson)` - caller handles publishing via their chosen transport
|
||||
- `smartpack()` returns `(env, msgJson)` - caller handles publishing via their chosen transport
|
||||
|
||||
#### Step 6: Julia Backend Receives Message
|
||||
|
||||
```julia
|
||||
# Julia backend
|
||||
transport_msg = transport_subscription.next() # Get message from transport
|
||||
env = smartreceive(String(transport_msg.payload))
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
|
||||
# env["payloads"] is now:
|
||||
# [
|
||||
@@ -242,7 +242,7 @@ env = smartreceive(String(transport_msg.payload))
|
||||
```
|
||||
|
||||
**Rationale**:
|
||||
- `smartreceive()` handles both transport types automatically
|
||||
- `smartunpack()` handles both transport types automatically
|
||||
- Deserialization is type-aware based on `payload_type`
|
||||
- Returns consistent tuple format regardless of transport
|
||||
|
||||
@@ -253,7 +253,7 @@ env = smartreceive(String(transport_msg.payload))
|
||||
response_text = "Hello Ton! I'm the AI assistant."
|
||||
generated_image = generate_ai_image(response_text)
|
||||
|
||||
env, msg_json = smartsend(
|
||||
env, msg_json = smartpack(
|
||||
"/agent/wine/api/v1/response",
|
||||
[
|
||||
("response", response_text, "text"),
|
||||
@@ -282,7 +282,7 @@ A JavaScript webapp wants to upload a large file (10MB) to a Julia backend for p
|
||||
#### Step 1: JavaScript Webapp Sends Large File
|
||||
|
||||
```javascript
|
||||
const [env, msgJson] = await msghandler.smartsend(
|
||||
const [env, msgJson] = await msghandler.smartpack(
|
||||
"/agent/wine/api/v1/process",
|
||||
[
|
||||
["file", largeFileData, "binary"]
|
||||
@@ -358,7 +358,7 @@ const response = await plikOneshotUpload(
|
||||
```julia
|
||||
# Julia backend
|
||||
transport_msg = transport_subscription.next()
|
||||
env = smartreceive(String(transport_msg.payload))
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
|
||||
# msghandler automatically:
|
||||
# 1. Extracts URL from payload
|
||||
@@ -386,7 +386,7 @@ A Python application sends tabular data (pandas DataFrame) to a Julia backend fo
|
||||
```python
|
||||
# Python
|
||||
import pandas as pd
|
||||
from msghandler import smartsend
|
||||
from msghandler import smartpack
|
||||
|
||||
df = pd.DataFrame({
|
||||
"id": [1, 2, 3],
|
||||
@@ -394,7 +394,7 @@ df = pd.DataFrame({
|
||||
"score": [95, 88, 92]
|
||||
})
|
||||
|
||||
env, msg_json = await smartsend(
|
||||
env, msg_json = await smartpack(
|
||||
"/agent/wine/api/v1/analyze",
|
||||
[("data", df, "arrowtable")],
|
||||
broker_url=DEFAULT_BROKER_URL,
|
||||
@@ -431,7 +431,7 @@ arrow_bytes = buf.getvalue()
|
||||
```julia
|
||||
# Julia backend
|
||||
transport_msg = transport_subscription.next()
|
||||
env = smartreceive(String(transport_msg.payload))
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
|
||||
# env["payloads"][1] is now:
|
||||
# ("data", DataFrame with id, name, score columns, "arrowtable")
|
||||
@@ -449,7 +449,7 @@ env = smartreceive(String(transport_msg.payload))
|
||||
results = analyze_data(env["payloads"][1][2])
|
||||
|
||||
# Send results back
|
||||
env, msg_json = smartsend(
|
||||
env, msg_json = smartpack(
|
||||
"/agent/wine/api/v1/results",
|
||||
[("results", results, "arrowtable")],
|
||||
reply_to = "/python/worker/v1/results"
|
||||
@@ -475,7 +475,7 @@ A Rust service needs to process messages from a Julia analytics pipeline and sen
|
||||
|
||||
```rust
|
||||
// Rust service - using tokio async runtime
|
||||
use msghandler::{smartreceive, MsgEnvelopeV1};
|
||||
use msghandler::{smartunpack, MsgEnvelopeV1};
|
||||
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64};
|
||||
|
||||
#[tokio::main]
|
||||
@@ -486,7 +486,7 @@ async fn main() {
|
||||
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze").unwrap();
|
||||
|
||||
for msg in sub.messages() {
|
||||
let envelope = smartreceive(
|
||||
let envelope = smartunpack(
|
||||
&String::from_utf8_lossy(&msg.payload),
|
||||
&Default::default(),
|
||||
).await.unwrap();
|
||||
@@ -495,7 +495,7 @@ async fn main() {
|
||||
for payload in &envelope.payloads {
|
||||
match payload.payload_type.as_str() {
|
||||
"arrowtable" => {
|
||||
// Data is base64-encoded Arrow IPC bytes after smartreceive()
|
||||
// Data is base64-encoded Arrow IPC bytes after smartunpack()
|
||||
let arrow_bytes = BASE64.decode(&payload.data).unwrap();
|
||||
println!("Received arrowtable payload ({} bytes)", arrow_bytes.len());
|
||||
},
|
||||
@@ -522,19 +522,19 @@ async fn main() {
|
||||
**Rationale**:
|
||||
- **serde serialization**: Automatic JSON deserialization to `MsgEnvelopeV1`
|
||||
- **tokio runtime**: Efficient async I/O for transport and HTTP operations
|
||||
- **smartreceive deserialization**: Payload data is deserialized and stored as strings in `payload.data`
|
||||
- **smartunpack deserialization**: Payload data is deserialized and stored as strings in `payload.data`
|
||||
- **Type dispatch**: `payload_type` field determines how to interpret the `data` string
|
||||
|
||||
#### Step 2: Rust Service Sends Processed Results
|
||||
|
||||
```rust
|
||||
// Rust service sends results back with mixed payload types
|
||||
use msghandler::{smartsend, Payload, SmartsendOptions};
|
||||
use msghandler::{smartpack, Payload, smartpackOptions};
|
||||
|
||||
let results_df = /* processed Arrow table */;
|
||||
let result_bytes = /* serialize to Arrow IPC */;
|
||||
|
||||
let (envelope, json_str) = smartsend(
|
||||
let (envelope, json_str) = smartpack(
|
||||
"/agent/wine/api/v1/results",
|
||||
&[
|
||||
(
|
||||
@@ -548,7 +548,7 @@ let (envelope, json_str) = smartsend(
|
||||
"text".to_string(),
|
||||
),
|
||||
],
|
||||
&SmartsendOptions {
|
||||
&smartpackOptions {
|
||||
broker_url: DEFAULT_BROKER_URL.to_string(),
|
||||
reply_to: "/python/worker/v1/results".to_string(),
|
||||
msg_purpose: "chat".to_string(),
|
||||
@@ -561,7 +561,7 @@ conn.publish("/agent/wine/api/v1/results", &json_str)?;
|
||||
```
|
||||
|
||||
**Rationale**:
|
||||
- **Builder pattern**: `SmartsendOptions` provides clean configuration
|
||||
- **Builder pattern**: `smartpackOptions` provides clean configuration
|
||||
- **Enum-based payloads**: Type safety prevents sending incorrect data types
|
||||
- **Default options**: sensible defaults reduce boilerplate
|
||||
- **Result<T, E>**: idiomatic Rust error handling
|
||||
@@ -570,7 +570,7 @@ conn.publish("/agent/wine/api/v1/results", &json_str)?;
|
||||
|
||||
```python
|
||||
# Python backend receives Rust response
|
||||
env = await smartreceive(str(transport_msg.payload))
|
||||
env = await smartunpack(str(transport_msg.payload))
|
||||
|
||||
# env["payloads"][0] is now:
|
||||
# ("results", arrow_table_data, "arrowtable")
|
||||
@@ -589,7 +589,7 @@ env = await smartreceive(str(transport_msg.payload))
|
||||
// Rust service sends large binary file via link transport
|
||||
let large_file_data: Vec<u8> = std::fs::read("/data/large_dataset.parquet")?;
|
||||
|
||||
let (envelope, json_str) = smartsend(
|
||||
let (envelope, json_str) = smartpack(
|
||||
"/agent/wine/api/v1/upload",
|
||||
&[
|
||||
(
|
||||
@@ -598,7 +598,7 @@ let (envelope, json_str) = smartsend(
|
||||
"binary".to_string(),
|
||||
),
|
||||
],
|
||||
&SmartsendOptions {
|
||||
&smartpackOptions {
|
||||
broker_url: DEFAULT_BROKER_URL.to_string(),
|
||||
fileserver_url: DEFAULT_FILESERVER_URL.to_string(),
|
||||
size_threshold: DEFAULT_SIZE_THRESHOLD, // threshold triggers link transport
|
||||
@@ -627,7 +627,7 @@ A MicroPython sensor device sends sensor readings to a Python backend.
|
||||
|
||||
```python
|
||||
# MicroPython
|
||||
from msghandler import smartsend
|
||||
from msghandler import smartpack
|
||||
|
||||
sensor_data = {
|
||||
"temperature": 25.5,
|
||||
@@ -635,7 +635,7 @@ sensor_data = {
|
||||
"pressure": 1013.25
|
||||
}
|
||||
|
||||
env, msg_json = smartsend(
|
||||
env, msg_json = smartpack(
|
||||
"/sensor/device/v1/readings",
|
||||
[("data", sensor_data, "dictionary")],
|
||||
broker_url=DEFAULT_BROKER_URL,
|
||||
@@ -667,7 +667,7 @@ payload_b64 = base64.b64encode(json_bytes).decode('ascii')
|
||||
```python
|
||||
# Python backend
|
||||
transport_msg = await transport_consumer.next()
|
||||
env = await smartreceive(str(transport_msg.payload))
|
||||
env = await smartunpack(str(transport_msg.payload))
|
||||
|
||||
# env["payloads"][0] is now:
|
||||
# ("data", {"temperature": 25.5, "humidity": 60.0, ...}, "dictionary")
|
||||
@@ -692,7 +692,7 @@ Multiple platforms (JavaScript, Python, Julia) communicate in a chat application
|
||||
|
||||
```javascript
|
||||
// JavaScript (Frontend)
|
||||
const [env, msgJson] = await msghandler.smartsend(
|
||||
const [env, msgJson] = await msghandler.smartpack(
|
||||
"/chat/user/v1/message",
|
||||
[
|
||||
["text", "Check this out!", "text"],
|
||||
@@ -716,7 +716,7 @@ const [env, msgJson] = await msghandler.smartsend(
|
||||
```python
|
||||
# Python (Backend)
|
||||
transport_msg = await transport_consumer.next()
|
||||
env = await smartreceive(str(transport_msg.payload))
|
||||
env = await smartunpack(str(transport_msg.payload))
|
||||
|
||||
# env["payloads"] is now:
|
||||
# [
|
||||
@@ -735,7 +735,7 @@ env = await smartreceive(str(transport_msg.payload))
|
||||
```julia
|
||||
# Julia (Backend)
|
||||
transport_msg = transport_subscription.next()
|
||||
env = smartreceive(String(transport_msg.payload))
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
|
||||
# env["payloads"] is now:
|
||||
# [
|
||||
@@ -755,7 +755,7 @@ Each platform can reply using the same API:
|
||||
|
||||
```python
|
||||
# Python reply
|
||||
await smartsend(
|
||||
await smartpack(
|
||||
"/chat/user/v1/reply",
|
||||
[("response", "Nice!", "text")],
|
||||
reply_to="/chat/user/v1/message"
|
||||
@@ -764,7 +764,7 @@ await smartsend(
|
||||
|
||||
```julia
|
||||
# Julia reply
|
||||
smartsend(
|
||||
smartpack(
|
||||
"/chat/user/v1/reply",
|
||||
[("response", "Nice!", "text")],
|
||||
reply_to="/chat/user/v1/message"
|
||||
@@ -773,7 +773,7 @@ smartsend(
|
||||
|
||||
```javascript
|
||||
// JavaScript reply
|
||||
await msghandler.smartsend(
|
||||
await msghandler.smartpack(
|
||||
"/chat/user/v1/reply",
|
||||
[["response", "Nice!", "text"]],
|
||||
{ reply_to: "/chat/user/v1/message" }
|
||||
@@ -827,14 +827,14 @@ Every message includes a `correlation_id`:
|
||||
correlation_id = string(uuid4())
|
||||
|
||||
# Use throughout the flow
|
||||
log_trace(correlation_id, "Starting smartsend")
|
||||
log_trace(correlation_id, "Starting smartpack")
|
||||
log_trace(correlation_id, "Serialized payload size: 100 bytes")
|
||||
log_trace(correlation_id, "Published to transport")
|
||||
```
|
||||
|
||||
**Log Format**:
|
||||
```
|
||||
[2026-03-13T16:30:00.000Z] [Correlation: abc123...] Starting smartsend
|
||||
[2026-03-13T16:30:00.000Z] [Correlation: abc123...] Starting smartpack
|
||||
[2026-03-13T16:30:00.001Z] [Correlation: abc123...] Serialized payload size: 100 bytes
|
||||
[2026-03-13T16:30:00.002Z] [Correlation: abc123...] Published to transport
|
||||
```
|
||||
@@ -928,8 +928,8 @@ log_trace(correlation_id, "Published to transport")
|
||||
| - | - | Removed all NATS-specific references from walkthrough | All sections |
|
||||
| - | - | Updated code examples to use transport-agnostic patterns | All sections |
|
||||
| - | - | Updated diagrams to remove NATS-specific labels | All sections |
|
||||
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections |
|
||||
| - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
|
||||
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartunpack` deserialization changes | All sections |
|
||||
| - | - | `smartunpack` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
|
||||
| - | - | Added `plik_upload_file` convenience function documentation | specification.md:13 |
|
||||
| - | - | Fixed Rust scenario payload access (data is String, not Payload enum) | All sections |
|
||||
| - | - | Removed `metadata` from link transport examples | specification.md:3 |
|
||||
@@ -937,7 +937,7 @@ log_trace(correlation_id, "Published to transport")
|
||||
| - | - | Added Rust user scenario (User Scenario 4) | specification.md:11 (Rust API) |
|
||||
| - | - | Updated scenario numbering (MicroPython → Scenario 5, Cross-Platform → Scenario 6) | All sections |
|
||||
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All sections |
|
||||
| - | - | Updated smartreceive calls to use transport payload pattern | All sections |
|
||||
| - | - | Updated smartunpack calls to use transport payload pattern | All sections |
|
||||
| - | - | Removed NATSClient.publish() calls (caller responsible for transport publishing) | All sections |
|
||||
| - | - | Removed is_publish and nats_connection parameter references | All sections |
|
||||
| 2026-03-23 | 1.0.0 | Updated to ASG Framework walkthrough guidelines | All sections |
|
||||
|
||||
Reference in New Issue
Block a user