rename to smartpack n smartunpack
This commit is contained in:
@@ -130,8 +130,8 @@ flowchart TD
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph "msghandler Module"
|
||||
SmartSend[smartsend Function]
|
||||
SmartReceive[smartreceive Function]
|
||||
smartpack[smartpack Function]
|
||||
smartunpack[smartunpack Function]
|
||||
|
||||
Serialize[_serialize_data]
|
||||
Deserialize[_deserialize_data]
|
||||
@@ -149,18 +149,18 @@ flowchart TD
|
||||
Envelope[msg_envelope_v1 Struct]
|
||||
end
|
||||
|
||||
SmartSend --> Serialize
|
||||
SmartSend --> EnvelopeToJson
|
||||
SmartSend --> FileServerUpload
|
||||
smartpack --> Serialize
|
||||
smartpack --> EnvelopeToJson
|
||||
smartpack --> FileServerUpload
|
||||
|
||||
SmartReceive --> Deserialize
|
||||
SmartReceive --> FileServerDownload
|
||||
smartunpack --> Deserialize
|
||||
smartunpack --> FileServerDownload
|
||||
|
||||
EnvelopeToJson --> Envelope
|
||||
Serialize --> Payload
|
||||
|
||||
style SmartSend fill:#d1fae5,stroke:#10b981
|
||||
style SmartReceive fill:#d1fae5,stroke:#10b981
|
||||
style smartpack fill:#d1fae5,stroke:#10b981
|
||||
style smartunpack fill:#d1fae5,stroke:#10b981
|
||||
style FileServerUpload fill:#fef3c7,stroke:#f59e0b
|
||||
style FileServerDownload fill:#fef3c7,stroke:#f59e0b
|
||||
```
|
||||
@@ -173,8 +173,8 @@ flowchart TD
|
||||
|
||||
| Component | Purpose | Platform Support |
|
||||
|-----------|---------|------------------|
|
||||
| **smartsend** | Send data with automatic transport selection, returns (envelope, json_string) for caller to publish via transport | All |
|
||||
| **smartreceive** | Receive and process messages from JSON string | All |
|
||||
| **smartpack** | Send data with automatic transport selection, returns (envelope, json_string) for caller to publish via transport | All |
|
||||
| **smartunpack** | Receive and process messages from JSON string | All |
|
||||
| **_serialize_data** | Serialize data according to payload type | All |
|
||||
| **_deserialize_data** | Deserialize bytes to native data types | All |
|
||||
| **envelope_to_json** | Convert msg_envelope_v1 struct to JSON string | All |
|
||||
@@ -187,7 +187,7 @@ flowchart TD
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[User calls smartsend subject data] --> B[Process each payload]
|
||||
A[User calls smartpack subject data] --> B[Process each payload]
|
||||
B --> C{Calculate serialized size}
|
||||
C -->|Size < Threshold| D[Direct Transport]
|
||||
C -->|Size >= Threshold| E[Link Transport]
|
||||
@@ -349,7 +349,7 @@ flowchart TD
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[smartsend called] --> B[Serialize payload]
|
||||
A[smartpack called] --> B[Serialize payload]
|
||||
B --> C[Calculate size]
|
||||
C --> D{Size < Threshold?}
|
||||
|
||||
@@ -560,7 +560,7 @@ pub enum Payload {
|
||||
}
|
||||
|
||||
// Configuration via builder pattern
|
||||
pub struct SmartsendOptions {
|
||||
pub struct smartpackOptions {
|
||||
pub broker_url: String,
|
||||
pub fileserver_url: String,
|
||||
pub fileserver_upload_handler: Option<Arc<dyn FileUploadHandler>>,
|
||||
@@ -577,7 +577,7 @@ let conn = transport_client::connect(DEFAULT_BROKER_URL).await?;
|
||||
// Subscribe and process messages
|
||||
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze")?;
|
||||
for msg in sub.messages() {
|
||||
let envelope = smartreceive(&String::from_utf8_lossy(&msg.payload), &Default::default()).await?;
|
||||
let envelope = smartunpack(&String::from_utf8_lossy(&msg.payload), &Default::default()).await?;
|
||||
// Access deserialized payloads by type
|
||||
for payload in &envelope.payloads {
|
||||
match payload.payload_type.as_str() {
|
||||
@@ -840,11 +840,11 @@ flowchart TD
|
||||
| - | - | Updated diagrams to use generic "Message Broker" instead of "NATS Server" | All sections |
|
||||
| - | - | Updated code examples to use transport-agnostic patterns | All sections |
|
||||
| - | - | Removed NATS client packages from external dependencies | All sections |
|
||||
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections |
|
||||
| - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
|
||||
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartunpack` deserialization changes | All sections |
|
||||
| - | - | `smartunpack` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
|
||||
| - | - | Added `plik_upload_file` convenience function to component table | specification.md:13 |
|
||||
| - | - | Fixed Rust payload access pattern (data is String, not Payload enum) | All sections |
|
||||
| - | - | Fixed `SmartsendOptions.fileserver_upload_handler` type to `Arc<dyn FileUploadHandler>` | specification.md:13 |
|
||||
| - | - | Fixed `smartpackOptions.fileserver_upload_handler` type to `Arc<dyn FileUploadHandler>` | specification.md:13 |
|
||||
| - | - | Removed `metadata` from link transport examples (now `None`/omitted) | specification.md:3 |
|
||||
| - | - | Removed duplicate footer text | All sections |
|
||||
| 2026-05-13 | 1.3.0 | Added Rust support with tokio, serde, and arrow2 | All sections |
|
||||
@@ -854,9 +854,9 @@ flowchart TD
|
||||
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) |
|
||||
| - | - | Removed publish_message component (commented out in source) |
|
||||
| - | - | Removed NATSClient and NATSConnectionPool classes (not in ground truth) |
|
||||
| - | - | Updated smartsend to return JSON for caller to publish via transport |
|
||||
| - | - | Updated smartpack to return JSON for caller to publish via transport |
|
||||
| - | - | Updated component diagram to match actual module structure |
|
||||
| - | - | Updated data flow to show smartsend returns JSON for caller to publish |
|
||||
| - | - | Updated data flow to show smartpack returns JSON for caller to publish |
|
||||
| - | - | Fixed SIZE_THRESHOLD default to 500,000 bytes |
|
||||
| 2026-03-15 | 1.1.0 | JavaScript connection management |
|
||||
| - | - | Added NATSClient with keepAlive support |
|
||||
|
||||
@@ -54,7 +54,7 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
|
||||
|---------|-------------|
|
||||
| Cross-platform interoperability | Seamless data exchange between Julia, JavaScript, Python, Dart, Rust, and MicroPython |
|
||||
| Intelligent transport selection | Direct transport (<0.5MB) vs Link transport (≥0.5MB) based on payload size |
|
||||
| Unified API | Consistent `smartsend()` and `smartreceive()` functions across all platforms |
|
||||
| Unified API | Consistent `smartpack()` and `smartunpack()` functions across all platforms |
|
||||
| Multi-payload support | List of (dataname, data, type) tuples with appropriate handling |
|
||||
| File server integration | Plik one-shot upload and custom HTTP server support |
|
||||
| Reliability features | Exponential backoff retry and correlation ID propagation |
|
||||
@@ -323,10 +323,10 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
|
||||
|
||||
## 11. API Contract
|
||||
|
||||
### 11.1 smartsend Signature
|
||||
### 11.1 smartpack Signature
|
||||
|
||||
```julia
|
||||
function smartsend(
|
||||
function smartpack(
|
||||
subject::String,
|
||||
data::AbstractArray{Tuple{String, T1, String}, 1};
|
||||
broker_url::String = DEFAULT_BROKER_URL,
|
||||
@@ -345,12 +345,12 @@ function smartsend(
|
||||
)::Tuple{msg_envelope_v1, String} where {T1<:Any}
|
||||
```
|
||||
|
||||
**Note**: Publishing via the transport layer is the caller's responsibility. `smartsend` returns `(env::msg_envelope_v1, env_json_str::String)`.
|
||||
**Note**: Publishing via the transport layer is the caller's responsibility. `smartpack` returns `(env::msg_envelope_v1, env_json_str::String)`.
|
||||
|
||||
### 11.2 smartreceive Signature
|
||||
### 11.2 smartunpack Signature
|
||||
|
||||
```julia
|
||||
function smartreceive(
|
||||
function smartunpack(
|
||||
msg_json_str::String;
|
||||
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||
max_retries::Int = 5,
|
||||
@@ -359,9 +359,9 @@ function smartreceive(
|
||||
)::JSON.Object{String, Any}
|
||||
```
|
||||
|
||||
**Note**: Pass the payload string from the transport subscription to `smartreceive`. The input is the JSON string payload from the transport message, not the transport message object directly.
|
||||
**Note**: Pass the payload string from the transport subscription to `smartunpack`. The input is the JSON string payload from the transport message, not the transport message object directly.
|
||||
|
||||
**Note**: Pass the payload from the transport subscription to `smartreceive`.
|
||||
**Note**: Pass the payload from the transport subscription to `smartunpack`.
|
||||
|
||||
---
|
||||
|
||||
@@ -411,8 +411,8 @@ function smartreceive(
|
||||
| - | - | Updated all NATS references to generic "transport layer"/"message broker" |
|
||||
| - | - | Removed NATS client packages from dependencies tables |
|
||||
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) |
|
||||
| - | - | Fixed smartsend signature: removed is_publish, NATS_connection; added sender_name |
|
||||
| - | - | Fixed smartreceive signature: takes msg_json_str::String instead of msg::NATS.Msg |
|
||||
| - | - | Fixed smartpack signature: removed is_publish, NATS_connection; added sender_name |
|
||||
| - | - | Fixed smartunpack signature: takes msg_json_str::String instead of msg::NATS.Msg |
|
||||
| - | - | Fixed size_threshold default from 1,000,000 to 500,000 |
|
||||
| - | - | Updated FR-013/FR-014 to reflect caller responsibility for NATS publishing |
|
||||
| - | - | Updated FR-008/FR-009 to include file path upload overload |
|
||||
|
||||
@@ -13,8 +13,8 @@
|
||||
This document defines the **technical contract** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer.
|
||||
|
||||
This specification serves as the single source of truth for:
|
||||
- **Inputs**: What data structures are accepted by `smartsend()`
|
||||
- **Outputs**: What data structures are returned by `smartreceive()`
|
||||
- **Inputs**: What data structures are accepted by `smartpack()`
|
||||
- **Outputs**: What data structures are returned by `smartunpack()`
|
||||
- **Data Shapes**: Exact field names, types, and constraints
|
||||
- **Error Codes**: Standardized error responses for failure scenarios
|
||||
|
||||
@@ -24,7 +24,7 @@ This specification serves as the single source of truth for:
|
||||
|----------------------|-------------------|-------------|
|
||||
| Section 2 (Message Envelope) | FR-012, FR-013, NFR-101, NFR-102 | Message envelope structure and validation |
|
||||
| Section 3 (Payload Schema) | FR-001, FR-002, FR-003, FR-004, NFR-101, NFR-102 | Payload structure and field definitions |
|
||||
| Section 4 (Payload Format) | FR-006, FR-007 | Tuple format for smartsend() |
|
||||
| Section 4 (Payload Format) | FR-006, FR-007 | Tuple format for smartpack() |
|
||||
| Section 5 (Enumerations) | FR-003, FR-004, FR-006, NFR-101 | Enumerations for transport and encoding |
|
||||
| Section 6 (Transport Protocols) | FR-003, FR-004, NFR-104, NFR-105 | Direct and link transport protocols |
|
||||
| Section 7 (Size Thresholds) | FR-004, FR-005, NFR-104, NFR-105 | Size thresholds for transport selection |
|
||||
@@ -143,9 +143,9 @@ This specification serves as the single source of truth for:
|
||||
|
||||
## Payload Format
|
||||
|
||||
### Tuple Format for `smartsend()`
|
||||
### Tuple Format for `smartpack()`
|
||||
|
||||
The `smartsend()` function accepts data as an array of tuples with the format:
|
||||
The `smartpack()` function accepts data as an array of tuples with the format:
|
||||
|
||||
```
|
||||
("data_name", data, "data_type")
|
||||
@@ -161,17 +161,17 @@ The `smartsend()` function accepts data as an array of tuples with the format:
|
||||
|
||||
```julia
|
||||
# Julia
|
||||
smartsend("/chat/user/v1/message", [("msg", "Hello World", "text")])
|
||||
smartpack("/chat/user/v1/message", [("msg", "Hello World", "text")])
|
||||
```
|
||||
|
||||
```python
|
||||
# Python
|
||||
await smartsend("/chat/user/v1/message", [("msg", "Hello World", "text")])
|
||||
await smartpack("/chat/user/v1/message", [("msg", "Hello World", "text")])
|
||||
```
|
||||
|
||||
```typescript
|
||||
// JavaScript
|
||||
await smartsend("/chat/user/v1/message", [["msg", "Hello World", "text"]]);
|
||||
await smartpack("/chat/user/v1/message", [["msg", "Hello World", "text"]]);
|
||||
```
|
||||
|
||||
### Multiple Payloads Example
|
||||
@@ -182,7 +182,7 @@ data = [
|
||||
("msg", "Hello", "text"),
|
||||
("img", binary_data, "image")
|
||||
]
|
||||
smartsend("/agent/v1/process", data)
|
||||
smartpack("/agent/v1/process", data)
|
||||
```
|
||||
|
||||
```python
|
||||
@@ -191,7 +191,7 @@ data = [
|
||||
("msg", "Hello", "text"),
|
||||
("img", binary_data, "image")
|
||||
]
|
||||
await smartsend("/agent/v1/process", data)
|
||||
await smartpack("/agent/v1/process", data)
|
||||
```
|
||||
|
||||
### Data Type Mapping
|
||||
@@ -411,12 +411,12 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa
|
||||
|
||||
## API Contract
|
||||
|
||||
### `smartsend` Function Signature
|
||||
### `smartpack` Function Signature
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
function smartsend(
|
||||
function smartpack(
|
||||
subject::String,
|
||||
data::AbstractArray{Tuple{String, T1, String}, 1};
|
||||
broker_url::String = DEFAULT_BROKER_URL,
|
||||
@@ -440,7 +440,7 @@ function smartsend(
|
||||
#### Python
|
||||
|
||||
```python
|
||||
async def smartsend(
|
||||
async def smartpack(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
broker_url: str = DEFAULT_BROKER_URL,
|
||||
@@ -464,7 +464,7 @@ async def smartsend(
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```typescript
|
||||
async function smartsend(
|
||||
async function smartpack(
|
||||
subject: string,
|
||||
data: Array<[string, any, string]>,
|
||||
options?: {
|
||||
@@ -490,7 +490,7 @@ async function smartsend(
|
||||
#### JavaScript (Browser)
|
||||
|
||||
```typescript
|
||||
async function smartsend(
|
||||
async function smartpack(
|
||||
subject: string,
|
||||
data: Array<[string, any, string]>,
|
||||
options?: {
|
||||
@@ -516,7 +516,7 @@ async function smartsend(
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
def smartsend(
|
||||
def smartpack(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
size_threshold: int = 100_000, # Lower threshold for memory constraints
|
||||
@@ -529,7 +529,7 @@ def smartsend(
|
||||
#### Dart (Desktop/Flutter)
|
||||
|
||||
```dart
|
||||
Future<[Map<String, dynamic>, String]> smartsend(
|
||||
Future<[Map<String, dynamic>, String]> smartpack(
|
||||
String subject,
|
||||
List<List<dynamic>> data, {
|
||||
String brokerUrl = DEFAULT_BROKER_URL,
|
||||
@@ -553,7 +553,7 @@ Future<[Map<String, dynamic>, String]> smartsend(
|
||||
#### Dart Web
|
||||
|
||||
```dart
|
||||
Future<[Map<String, dynamic>, String]> smartsend(
|
||||
Future<[Map<String, dynamic>, String]> smartpack(
|
||||
String subject,
|
||||
List<List<dynamic>> data, {
|
||||
String brokerUrl = DEFAULT_BROKER_URL,
|
||||
@@ -578,14 +578,14 @@ Future<[Map<String, dynamic>, String]> smartsend(
|
||||
#### Rust
|
||||
|
||||
```rust
|
||||
pub async fn smartsend(
|
||||
pub async fn smartpack(
|
||||
subject: &str,
|
||||
data: &[(String, Payload, String)],
|
||||
options: &SmartsendOptions,
|
||||
options: &smartpackOptions,
|
||||
) -> Result<(MsgEnvelopeV1, String), msghandlerError>
|
||||
|
||||
// SmartsendOptions struct
|
||||
pub struct SmartsendOptions {
|
||||
// smartpackOptions struct
|
||||
pub struct smartpackOptions {
|
||||
pub broker_url: String,
|
||||
pub fileserver_url: String,
|
||||
pub fileserver_upload_handler: Option<UploadHandler>,
|
||||
@@ -636,12 +636,12 @@ pub struct MsgEnvelopeV1 {
|
||||
|
||||
**Note**: Publishing via the transport layer is the caller's responsibility. Returns `Result<(MsgEnvelopeV1, String), msghandlerError>`. Uses `serde` for JSON serialization.
|
||||
|
||||
### `smartreceive` Function Signature
|
||||
### `smartunpack` Function Signature
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
function smartreceive(
|
||||
function smartunpack(
|
||||
msg_json_str::String; # Pass payload from transport subscription
|
||||
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||
max_retries::Int = 5,
|
||||
@@ -655,7 +655,7 @@ function smartreceive(
|
||||
#### Python
|
||||
|
||||
```python
|
||||
async def smartreceive(
|
||||
async def smartunpack(
|
||||
msg_json_str: str, # JSON string from transport message payload
|
||||
fileserver_download_handler: Callable = fetch_with_backoff,
|
||||
max_retries: int = 5,
|
||||
@@ -669,7 +669,7 @@ async def smartreceive(
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```typescript
|
||||
async function smartreceive(
|
||||
async function smartunpack(
|
||||
msg_json_str: string, // JSON string from transport message payload
|
||||
options?: {
|
||||
fileserver_download_handler?: Function;
|
||||
@@ -683,7 +683,7 @@ async function smartreceive(
|
||||
#### JavaScript (Browser)
|
||||
|
||||
```typescript
|
||||
async function smartreceive(
|
||||
async function smartunpack(
|
||||
msg_json_str: string, // JSON string from transport message payload
|
||||
options?: {
|
||||
fileserver_download_handler?: Function;
|
||||
@@ -699,7 +699,7 @@ async function smartreceive(
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
def smartreceive(msg_json_str: str, **kwargs) -> Dict[str, Any]:
|
||||
def smartunpack(msg_json_str: str, **kwargs) -> Dict[str, Any]:
|
||||
```
|
||||
|
||||
**Note**: Input is the JSON string payload from the transport message.
|
||||
@@ -707,7 +707,7 @@ def smartreceive(msg_json_str: str, **kwargs) -> Dict[str, Any]:
|
||||
#### Dart (Desktop/Flutter)
|
||||
|
||||
```dart
|
||||
Future<Map<String, dynamic>> smartreceive(
|
||||
Future<Map<String, dynamic>> smartunpack(
|
||||
Map<String, dynamic> msg_json_str, // JSON object from transport message payload
|
||||
{
|
||||
Function? fileserverDownloadHandler,
|
||||
@@ -722,7 +722,7 @@ Future<Map<String, dynamic>> smartreceive(
|
||||
#### Dart Web
|
||||
|
||||
```dart
|
||||
Future<Map<String, dynamic>> smartreceive(
|
||||
Future<Map<String, dynamic>> smartunpack(
|
||||
Map<String, dynamic> msg_json_str, // JSON object from transport message payload
|
||||
{
|
||||
Function? fileserverDownloadHandler,
|
||||
@@ -737,13 +737,13 @@ Future<Map<String, dynamic>> smartreceive(
|
||||
#### Rust
|
||||
|
||||
```rust
|
||||
pub async fn smartreceive(
|
||||
pub async fn smartunpack(
|
||||
msg_json_str: &str, // JSON string from transport message payload
|
||||
options: &SmartreceiveOptions,
|
||||
options: &smartunpackOptions,
|
||||
) -> Result<MsgEnvelopeV1, msghandlerError>
|
||||
|
||||
// SmartreceiveOptions struct
|
||||
pub struct SmartreceiveOptions {
|
||||
// smartunpackOptions struct
|
||||
pub struct smartunpackOptions {
|
||||
pub fileserver_download_handler: Option<DownloadHandler>,
|
||||
pub max_retries: u32,
|
||||
pub base_delay: u64,
|
||||
@@ -933,7 +933,7 @@ The browser implementation ([`src/msghandler_csr.js`](../src/msghandler_csr.js))
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[User calls smartsend subject data] --> B[Serialize payload according to payload_type]
|
||||
A[User calls smartpack subject data] --> B[Serialize payload according to payload_type]
|
||||
B --> C{Calculate serialized size}
|
||||
C -->|Size < Threshold| D[Direct Transport: Encode as Base64]
|
||||
C -->|Size >= Threshold| E[Link Transport: Upload to file server]
|
||||
@@ -1156,8 +1156,8 @@ flowchart TD
|
||||
| - | - | Updated all NATS references to generic "transport layer"/"message broker" | All |
|
||||
| - | - | Removed NATS client packages from dependencies tables | All |
|
||||
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All |
|
||||
| - | - | Updated smartsend signatures: removed is_publish, nats_connection; added sender_name | FR-001 through FR-014 |
|
||||
| - | - | Updated smartreceive signatures: takes msg_json_str::String instead of msg | FR-001 through FR-014 |
|
||||
| - | - | Updated smartpack signatures: removed is_publish, nats_connection; added sender_name | FR-001 through FR-014 |
|
||||
| - | - | Updated smartunpack signatures: takes msg_json_str::String instead of msg | FR-001 through FR-014 |
|
||||
| - | - | Removed publishMessage function and NATSClient/NATSConnectionPool classes from browser section | FR-013, FR-014 |
|
||||
| - | - | Added plik_oneshot_upload(filepath) overload to file server interface | FR-008, FR-009 |
|
||||
| - | - | Fixed SIZE_THRESHOLD default to 500,000 bytes | FR-003, FR-004 |
|
||||
|
||||
@@ -44,7 +44,7 @@ flowchart TB
|
||||
subgraph msghandler["msghandler Module"]
|
||||
direction TB
|
||||
|
||||
subgraph Sender["Sender (smartsend)"]
|
||||
subgraph Sender["Sender (smartpack)"]
|
||||
direction LR
|
||||
S1["Data Tuples<br/>[(dataname, data, type)]"]
|
||||
S2["Serialize Data"]
|
||||
@@ -60,7 +60,7 @@ flowchart TB
|
||||
S5 --> S6
|
||||
end
|
||||
|
||||
subgraph Receiver["Receiver (smartreceive)"]
|
||||
subgraph Receiver["Receiver (smartunpack)"]
|
||||
direction LR
|
||||
R1["Subscribe via transport"]
|
||||
R2["Parse Envelope"]
|
||||
@@ -101,7 +101,7 @@ flowchart TB
|
||||
|-----------|-------------|-----------|
|
||||
| **Claim-Check Pattern** | Large payloads uploaded to HTTP server, URL sent via transport | Transport has message size limits; avoids overflow |
|
||||
| **Automatic Transport Selection** | Direct (< threshold) vs Link (≥ threshold) based on size | Optimizes memory vs network I/O trade-off |
|
||||
| **Cross-Platform API** | Consistent `smartsend()`/`smartreceive()` across all platforms | Simplifies developer experience |
|
||||
| **Cross-Platform API** | Consistent `smartpack()`/`smartunpack()` across all platforms | Simplifies developer experience |
|
||||
| **Exponential Backoff** | Retry downloads with increasing delays | Handles transient failures gracefully |
|
||||
|
||||
---
|
||||
@@ -118,7 +118,7 @@ A JavaScript chat webapp wants to send mixed payloads (text message + user avata
|
||||
|
||||
```javascript
|
||||
// JavaScript (Browser or Node.js)
|
||||
const [env, msgJson] = await msghandler.smartsend(
|
||||
const [env, msgJson] = await msghandler.smartpack(
|
||||
"/agent/wine/api/v1/prompt",
|
||||
[
|
||||
["msg", "Hello! I'm Ton.", "text"],
|
||||
@@ -225,14 +225,14 @@ msghandler builds the message envelope:
|
||||
**Rationale**:
|
||||
- The transport layer provides message delivery (NATS, MQTT, WebSocket, etc.)
|
||||
- JSON format ensures cross-platform compatibility
|
||||
- `smartsend()` returns `(env, msgJson)` - caller handles publishing via their chosen transport
|
||||
- `smartpack()` returns `(env, msgJson)` - caller handles publishing via their chosen transport
|
||||
|
||||
#### Step 6: Julia Backend Receives Message
|
||||
|
||||
```julia
|
||||
# Julia backend
|
||||
transport_msg = transport_subscription.next() # Get message from transport
|
||||
env = smartreceive(String(transport_msg.payload))
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
|
||||
# env["payloads"] is now:
|
||||
# [
|
||||
@@ -242,7 +242,7 @@ env = smartreceive(String(transport_msg.payload))
|
||||
```
|
||||
|
||||
**Rationale**:
|
||||
- `smartreceive()` handles both transport types automatically
|
||||
- `smartunpack()` handles both transport types automatically
|
||||
- Deserialization is type-aware based on `payload_type`
|
||||
- Returns consistent tuple format regardless of transport
|
||||
|
||||
@@ -253,7 +253,7 @@ env = smartreceive(String(transport_msg.payload))
|
||||
response_text = "Hello Ton! I'm the AI assistant."
|
||||
generated_image = generate_ai_image(response_text)
|
||||
|
||||
env, msg_json = smartsend(
|
||||
env, msg_json = smartpack(
|
||||
"/agent/wine/api/v1/response",
|
||||
[
|
||||
("response", response_text, "text"),
|
||||
@@ -282,7 +282,7 @@ A JavaScript webapp wants to upload a large file (10MB) to a Julia backend for p
|
||||
#### Step 1: JavaScript Webapp Sends Large File
|
||||
|
||||
```javascript
|
||||
const [env, msgJson] = await msghandler.smartsend(
|
||||
const [env, msgJson] = await msghandler.smartpack(
|
||||
"/agent/wine/api/v1/process",
|
||||
[
|
||||
["file", largeFileData, "binary"]
|
||||
@@ -358,7 +358,7 @@ const response = await plikOneshotUpload(
|
||||
```julia
|
||||
# Julia backend
|
||||
transport_msg = transport_subscription.next()
|
||||
env = smartreceive(String(transport_msg.payload))
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
|
||||
# msghandler automatically:
|
||||
# 1. Extracts URL from payload
|
||||
@@ -386,7 +386,7 @@ A Python application sends tabular data (pandas DataFrame) to a Julia backend fo
|
||||
```python
|
||||
# Python
|
||||
import pandas as pd
|
||||
from msghandler import smartsend
|
||||
from msghandler import smartpack
|
||||
|
||||
df = pd.DataFrame({
|
||||
"id": [1, 2, 3],
|
||||
@@ -394,7 +394,7 @@ df = pd.DataFrame({
|
||||
"score": [95, 88, 92]
|
||||
})
|
||||
|
||||
env, msg_json = await smartsend(
|
||||
env, msg_json = await smartpack(
|
||||
"/agent/wine/api/v1/analyze",
|
||||
[("data", df, "arrowtable")],
|
||||
broker_url=DEFAULT_BROKER_URL,
|
||||
@@ -431,7 +431,7 @@ arrow_bytes = buf.getvalue()
|
||||
```julia
|
||||
# Julia backend
|
||||
transport_msg = transport_subscription.next()
|
||||
env = smartreceive(String(transport_msg.payload))
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
|
||||
# env["payloads"][1] is now:
|
||||
# ("data", DataFrame with id, name, score columns, "arrowtable")
|
||||
@@ -449,7 +449,7 @@ env = smartreceive(String(transport_msg.payload))
|
||||
results = analyze_data(env["payloads"][1][2])
|
||||
|
||||
# Send results back
|
||||
env, msg_json = smartsend(
|
||||
env, msg_json = smartpack(
|
||||
"/agent/wine/api/v1/results",
|
||||
[("results", results, "arrowtable")],
|
||||
reply_to = "/python/worker/v1/results"
|
||||
@@ -475,7 +475,7 @@ A Rust service needs to process messages from a Julia analytics pipeline and sen
|
||||
|
||||
```rust
|
||||
// Rust service - using tokio async runtime
|
||||
use msghandler::{smartreceive, MsgEnvelopeV1};
|
||||
use msghandler::{smartunpack, MsgEnvelopeV1};
|
||||
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64};
|
||||
|
||||
#[tokio::main]
|
||||
@@ -486,7 +486,7 @@ async fn main() {
|
||||
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze").unwrap();
|
||||
|
||||
for msg in sub.messages() {
|
||||
let envelope = smartreceive(
|
||||
let envelope = smartunpack(
|
||||
&String::from_utf8_lossy(&msg.payload),
|
||||
&Default::default(),
|
||||
).await.unwrap();
|
||||
@@ -495,7 +495,7 @@ async fn main() {
|
||||
for payload in &envelope.payloads {
|
||||
match payload.payload_type.as_str() {
|
||||
"arrowtable" => {
|
||||
// Data is base64-encoded Arrow IPC bytes after smartreceive()
|
||||
// Data is base64-encoded Arrow IPC bytes after smartunpack()
|
||||
let arrow_bytes = BASE64.decode(&payload.data).unwrap();
|
||||
println!("Received arrowtable payload ({} bytes)", arrow_bytes.len());
|
||||
},
|
||||
@@ -522,19 +522,19 @@ async fn main() {
|
||||
**Rationale**:
|
||||
- **serde serialization**: Automatic JSON deserialization to `MsgEnvelopeV1`
|
||||
- **tokio runtime**: Efficient async I/O for transport and HTTP operations
|
||||
- **smartreceive deserialization**: Payload data is deserialized and stored as strings in `payload.data`
|
||||
- **smartunpack deserialization**: Payload data is deserialized and stored as strings in `payload.data`
|
||||
- **Type dispatch**: `payload_type` field determines how to interpret the `data` string
|
||||
|
||||
#### Step 2: Rust Service Sends Processed Results
|
||||
|
||||
```rust
|
||||
// Rust service sends results back with mixed payload types
|
||||
use msghandler::{smartsend, Payload, SmartsendOptions};
|
||||
use msghandler::{smartpack, Payload, smartpackOptions};
|
||||
|
||||
let results_df = /* processed Arrow table */;
|
||||
let result_bytes = /* serialize to Arrow IPC */;
|
||||
|
||||
let (envelope, json_str) = smartsend(
|
||||
let (envelope, json_str) = smartpack(
|
||||
"/agent/wine/api/v1/results",
|
||||
&[
|
||||
(
|
||||
@@ -548,7 +548,7 @@ let (envelope, json_str) = smartsend(
|
||||
"text".to_string(),
|
||||
),
|
||||
],
|
||||
&SmartsendOptions {
|
||||
&smartpackOptions {
|
||||
broker_url: DEFAULT_BROKER_URL.to_string(),
|
||||
reply_to: "/python/worker/v1/results".to_string(),
|
||||
msg_purpose: "chat".to_string(),
|
||||
@@ -561,7 +561,7 @@ conn.publish("/agent/wine/api/v1/results", &json_str)?;
|
||||
```
|
||||
|
||||
**Rationale**:
|
||||
- **Builder pattern**: `SmartsendOptions` provides clean configuration
|
||||
- **Builder pattern**: `smartpackOptions` provides clean configuration
|
||||
- **Enum-based payloads**: Type safety prevents sending incorrect data types
|
||||
- **Default options**: sensible defaults reduce boilerplate
|
||||
- **Result<T, E>**: idiomatic Rust error handling
|
||||
@@ -570,7 +570,7 @@ conn.publish("/agent/wine/api/v1/results", &json_str)?;
|
||||
|
||||
```python
|
||||
# Python backend receives Rust response
|
||||
env = await smartreceive(str(transport_msg.payload))
|
||||
env = await smartunpack(str(transport_msg.payload))
|
||||
|
||||
# env["payloads"][0] is now:
|
||||
# ("results", arrow_table_data, "arrowtable")
|
||||
@@ -589,7 +589,7 @@ env = await smartreceive(str(transport_msg.payload))
|
||||
// Rust service sends large binary file via link transport
|
||||
let large_file_data: Vec<u8> = std::fs::read("/data/large_dataset.parquet")?;
|
||||
|
||||
let (envelope, json_str) = smartsend(
|
||||
let (envelope, json_str) = smartpack(
|
||||
"/agent/wine/api/v1/upload",
|
||||
&[
|
||||
(
|
||||
@@ -598,7 +598,7 @@ let (envelope, json_str) = smartsend(
|
||||
"binary".to_string(),
|
||||
),
|
||||
],
|
||||
&SmartsendOptions {
|
||||
&smartpackOptions {
|
||||
broker_url: DEFAULT_BROKER_URL.to_string(),
|
||||
fileserver_url: DEFAULT_FILESERVER_URL.to_string(),
|
||||
size_threshold: DEFAULT_SIZE_THRESHOLD, // threshold triggers link transport
|
||||
@@ -627,7 +627,7 @@ A MicroPython sensor device sends sensor readings to a Python backend.
|
||||
|
||||
```python
|
||||
# MicroPython
|
||||
from msghandler import smartsend
|
||||
from msghandler import smartpack
|
||||
|
||||
sensor_data = {
|
||||
"temperature": 25.5,
|
||||
@@ -635,7 +635,7 @@ sensor_data = {
|
||||
"pressure": 1013.25
|
||||
}
|
||||
|
||||
env, msg_json = smartsend(
|
||||
env, msg_json = smartpack(
|
||||
"/sensor/device/v1/readings",
|
||||
[("data", sensor_data, "dictionary")],
|
||||
broker_url=DEFAULT_BROKER_URL,
|
||||
@@ -667,7 +667,7 @@ payload_b64 = base64.b64encode(json_bytes).decode('ascii')
|
||||
```python
|
||||
# Python backend
|
||||
transport_msg = await transport_consumer.next()
|
||||
env = await smartreceive(str(transport_msg.payload))
|
||||
env = await smartunpack(str(transport_msg.payload))
|
||||
|
||||
# env["payloads"][0] is now:
|
||||
# ("data", {"temperature": 25.5, "humidity": 60.0, ...}, "dictionary")
|
||||
@@ -692,7 +692,7 @@ Multiple platforms (JavaScript, Python, Julia) communicate in a chat application
|
||||
|
||||
```javascript
|
||||
// JavaScript (Frontend)
|
||||
const [env, msgJson] = await msghandler.smartsend(
|
||||
const [env, msgJson] = await msghandler.smartpack(
|
||||
"/chat/user/v1/message",
|
||||
[
|
||||
["text", "Check this out!", "text"],
|
||||
@@ -716,7 +716,7 @@ const [env, msgJson] = await msghandler.smartsend(
|
||||
```python
|
||||
# Python (Backend)
|
||||
transport_msg = await transport_consumer.next()
|
||||
env = await smartreceive(str(transport_msg.payload))
|
||||
env = await smartunpack(str(transport_msg.payload))
|
||||
|
||||
# env["payloads"] is now:
|
||||
# [
|
||||
@@ -735,7 +735,7 @@ env = await smartreceive(str(transport_msg.payload))
|
||||
```julia
|
||||
# Julia (Backend)
|
||||
transport_msg = transport_subscription.next()
|
||||
env = smartreceive(String(transport_msg.payload))
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
|
||||
# env["payloads"] is now:
|
||||
# [
|
||||
@@ -755,7 +755,7 @@ Each platform can reply using the same API:
|
||||
|
||||
```python
|
||||
# Python reply
|
||||
await smartsend(
|
||||
await smartpack(
|
||||
"/chat/user/v1/reply",
|
||||
[("response", "Nice!", "text")],
|
||||
reply_to="/chat/user/v1/message"
|
||||
@@ -764,7 +764,7 @@ await smartsend(
|
||||
|
||||
```julia
|
||||
# Julia reply
|
||||
smartsend(
|
||||
smartpack(
|
||||
"/chat/user/v1/reply",
|
||||
[("response", "Nice!", "text")],
|
||||
reply_to="/chat/user/v1/message"
|
||||
@@ -773,7 +773,7 @@ smartsend(
|
||||
|
||||
```javascript
|
||||
// JavaScript reply
|
||||
await msghandler.smartsend(
|
||||
await msghandler.smartpack(
|
||||
"/chat/user/v1/reply",
|
||||
[["response", "Nice!", "text"]],
|
||||
{ reply_to: "/chat/user/v1/message" }
|
||||
@@ -827,14 +827,14 @@ Every message includes a `correlation_id`:
|
||||
correlation_id = string(uuid4())
|
||||
|
||||
# Use throughout the flow
|
||||
log_trace(correlation_id, "Starting smartsend")
|
||||
log_trace(correlation_id, "Starting smartpack")
|
||||
log_trace(correlation_id, "Serialized payload size: 100 bytes")
|
||||
log_trace(correlation_id, "Published to transport")
|
||||
```
|
||||
|
||||
**Log Format**:
|
||||
```
|
||||
[2026-03-13T16:30:00.000Z] [Correlation: abc123...] Starting smartsend
|
||||
[2026-03-13T16:30:00.000Z] [Correlation: abc123...] Starting smartpack
|
||||
[2026-03-13T16:30:00.001Z] [Correlation: abc123...] Serialized payload size: 100 bytes
|
||||
[2026-03-13T16:30:00.002Z] [Correlation: abc123...] Published to transport
|
||||
```
|
||||
@@ -928,8 +928,8 @@ log_trace(correlation_id, "Published to transport")
|
||||
| - | - | Removed all NATS-specific references from walkthrough | All sections |
|
||||
| - | - | Updated code examples to use transport-agnostic patterns | All sections |
|
||||
| - | - | Updated diagrams to remove NATS-specific labels | All sections |
|
||||
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections |
|
||||
| - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
|
||||
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartunpack` deserialization changes | All sections |
|
||||
| - | - | `smartunpack` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
|
||||
| - | - | Added `plik_upload_file` convenience function documentation | specification.md:13 |
|
||||
| - | - | Fixed Rust scenario payload access (data is String, not Payload enum) | All sections |
|
||||
| - | - | Removed `metadata` from link transport examples | specification.md:3 |
|
||||
@@ -937,7 +937,7 @@ log_trace(correlation_id, "Published to transport")
|
||||
| - | - | Added Rust user scenario (User Scenario 4) | specification.md:11 (Rust API) |
|
||||
| - | - | Updated scenario numbering (MicroPython → Scenario 5, Cross-Platform → Scenario 6) | All sections |
|
||||
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All sections |
|
||||
| - | - | Updated smartreceive calls to use transport payload pattern | All sections |
|
||||
| - | - | Updated smartunpack calls to use transport payload pattern | All sections |
|
||||
| - | - | Removed NATSClient.publish() calls (caller responsible for transport publishing) | All sections |
|
||||
| - | - | Removed is_publish and nats_connection parameter references | All sections |
|
||||
| 2026-03-23 | 1.0.0 | Updated to ASG Framework walkthrough guidelines | All sections |
|
||||
|
||||
Reference in New Issue
Block a user