rename to smartpack n smartunpack
This commit is contained in:
33
AI_prompt.md
33
AI_prompt.md
@@ -4,7 +4,7 @@ Scenario 2: The "Deep Dive" Analysis (High Bandwidth)Focus: Large Arrow tables,
|
||||
Scenario 3: Live Audio/Signal Processing (Multimedia & Metadata)Focus: Raw binary, bi-directional streaming, headers for metadata.The Action: The JS client captures a 2-second "chunk" of microphone audio. It needs Julia to perform a Fast Fourier Transform (FFT) or AI transcription.The Flow:JS (Sender): Sends the raw binary WAV/PCM data. It uses transport headers to store the metadata ($fs = 44.1kHz$, $channels = 1$) to keep the payload purely binary.Julia (Receiver): Processes the audio and sends back a JSON result (the transcription) and an Arrow Table (the frequency spectrum data).Project Requirement Met: Bi-directional flow involving mixed media (Audio) and technical results (Arrow).
|
||||
Scenario 4: The "Catch-Up" (Persistence & State Sync)Focus: Message persistence, late-joining consumers, state sync.The Action: Julia is constantly publishing "System Health" updates. The JS dashboard is closed for 10 minutes. When the user re-opens the dashboard, they need to see the last 10 minutes of history.The Flow:Transport (Server): Uses a persistence layer with a Limits retention policy.JS (Consumer): Connects and requests a "Replay" from the last 10 minutes. It receives a mix of direct (small updates) and link (historical snapshots) messages.Project Requirement Met: Temporal decoupling—consumers can receive data that was sent while they were offline.
|
||||
|
||||
Role: Principal Systems Architect & Lead Software Engineer.Objective: Implement a high-performance, bi-directional data bridge between a Julia service and a JavaScript (Node.js) service, using a unified message envelope with Claim-Check pattern for large payloads.⚠️ STRICT ARCHITECTURAL CONSTRAINTS (Non-Negotiable)Transport Strategy (Claim-Check Pattern):Direct Path: If payload is < 1MB, send data directly inside the message envelope (Base64 encoded).Link Path: If payload is > 1MB, upload to a shared HTTP fileserver/store. The message must only contain the metadata and the download URL.Tabular Data Format: * MUST use Apache Arrow IPC Stream for all tables/DataFrames. No CSV or standard JSON-serialization of tables allowed.System Symmetry: * Both services must function as Producers AND Consumers.Modular Elegance: * Implementation must be abstracted into a SmartSend function and a SmartReceive handler. The developer calling these functions should not need to care if the data is going via direct or HTTP link.Technical Stack & Use CasesJulia: Arrow.jl, JSON3.jl, HTTP.jl.Node.js: apache-arrow, native fetch.Scenarios to Support: * Large Data: Sending a 500MB Arrow table from Julia $\rightarrow$ JS.Media: Sending a 5MB WAV file from JS $\rightarrow$ Julia.Signals: Sending small JSON control commands ($< 10KB$) directly in the envelope.Implementation Requirements1. Unified JSON Envelope:Define a schema containing: correlation_id (UUID), type (table/binary/json), transport (direct/link), payload (if direct), and url (if link).2. The Julia Module:Implement SmartSend(subject, data, type): Handles Arrow serialization to an IOBuffer, checks size, and manages HTTP uploads for large blobs.Implement SmartReceive(msg): Parses envelope, handles the HTTP fetch with Exponential Backoff (to avoid race conditions), and restores the DataFrame.Include a basic HTTP.listen server to serve as the temporary storage.3. The JavaScript Module:Implement a symmetric SmartSend using native fetch and apache-arrow.Implement a JetStream P... (line truncated to 2000 chars)
|
||||
Role: Principal Systems Architect & Lead Software Engineer.Objective: Implement a high-performance, bi-directional data bridge between a Julia service and a JavaScript (Node.js) service, using a unified message envelope with Claim-Check pattern for large payloads.⚠️ STRICT ARCHITECTURAL CONSTRAINTS (Non-Negotiable)Transport Strategy (Claim-Check Pattern):Direct Path: If payload is < 1MB, send data directly inside the message envelope (Base64 encoded).Link Path: If payload is > 1MB, upload to a shared HTTP fileserver/store. The message must only contain the metadata and the download URL.Tabular Data Format: * MUST use Apache Arrow IPC Stream for all tables/DataFrames. No CSV or standard JSON-serialization of tables allowed.System Symmetry: * Both services must function as Producers AND Consumers.Modular Elegance: * Implementation must be abstracted into a smartpack function and a smartunpack handler. The developer calling these functions should not need to care if the data is going via direct or HTTP link.Technical Stack & Use CasesJulia: Arrow.jl, JSON3.jl, HTTP.jl.Node.js: apache-arrow, native fetch.Scenarios to Support: * Large Data: Sending a 500MB Arrow table from Julia $\rightarrow$ JS.Media: Sending a 5MB WAV file from JS $\rightarrow$ Julia.Signals: Sending small JSON control commands ($< 10KB$) directly in the envelope.Implementation Requirements1. Unified JSON Envelope:Define a schema containing: correlation_id (UUID), type (table/binary/json), transport (direct/link), payload (if direct), and url (if link).2. The Julia Module:Implement smartpack(subject, data, type): Handles Arrow serialization to an IOBuffer, checks size, and manages HTTP uploads for large blobs.Implement smartunpack(msg): Parses envelope, handles the HTTP fetch with Exponential Backoff (to avoid race conditions), and restores the DataFrame.Include a basic HTTP.listen server to serve as the temporary storage.3. The JavaScript Module:Implement a symmetric smartpack using native fetch and apache-arrow.Implement a JetStream P... (line truncated to 2000 chars)
|
||||
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ Task: Update msghandler.js to reflect recent changes in msghandler.jl and docs
|
||||
Context: msghandler.jl and docs has been updated.
|
||||
Requirements:
|
||||
Source of Truth: Treat the updated msghandler.jl and docs as the definitive source.
|
||||
API Consistency: Ensure the Main Package API (e.g., smartsend(), publish_message()) uses consistent naming across all three supported languages.
|
||||
API Consistency: Ensure the Main Package API (e.g., smartpack(), publish_message()) uses consistent naming across all three supported languages.
|
||||
Ecosystem Variance: Low-level native functions (e.g., connect(), JSON.parse()) should follow the conventions of the specific language ecosystem and do not require cross-language consistency.
|
||||
|
||||
|
||||
@@ -164,7 +164,14 @@ Check the following files:
|
||||
I would like to expand this package (msghandler) to include Rust support.
|
||||
Now help me update Rust implementation of this package at ./src/msghandler.rs.
|
||||
|
||||
|
||||
<!-- ------------------------------------------- 100 ------------------------------------------- -->
|
||||
I updated ./src/msghandler-csr.js. Can you check whether files in ./docs needs to be update?
|
||||
You should check the files sequencially in the following order:
|
||||
1) ./docs/requirements.md
|
||||
2) ./docs/specification.md
|
||||
3) ./docs/architecture.md
|
||||
4) ./docs/walkthrough.md
|
||||
<!-- ------------------------------------------- 100 ------------------------------------------- -->
|
||||
|
||||
|
||||
|
||||
@@ -173,7 +180,7 @@ I want to build a client-side-rendering Dioxus-based chat webapp.
|
||||
Dioxus version 0.7+ should be great.
|
||||
I already populate the current folder for the project.
|
||||
my server REST API endpoint is sommpanion.yiem.cc/agent-fronent/api/v1/chat but I didn't run the server yet. A message format is JSON string.
|
||||
I just placed my custom package for encode and decode message at ./src/msghandler.rs. smartsend() is for encoding and smartreceive() is for decoding.
|
||||
I just placed my custom package for encode and decode message at ./src/msghandler.rs. smartpack() is for encoding and smartunpack() is for decoding.
|
||||
you may also check the file /home/ton/docker-apps/sommpanion/msghandler/docs/walkthrough.md for more info about my package.
|
||||
You can test whether Dioxus webapp can be build using this command "dx bundle --web --release --debug-symbols=false"
|
||||
|
||||
@@ -185,8 +192,7 @@ You can test whether Dioxus webapp can be build using this command "dx bundle --
|
||||
|
||||
|
||||
|
||||
|
||||
I want to build similar webapp.
|
||||
Do you know about ChatGPT chat interface? I want to build similar webapp.
|
||||
My app should be built as client-side-rendering Dioxus-based (version 0.7+).
|
||||
I already build backend server and I intend to communicate with the webapp using json string that encode the following message envelop:
|
||||
{
|
||||
@@ -233,11 +239,14 @@ I already build backend server and I intend to communicate with the webapp using
|
||||
]
|
||||
}
|
||||
---
|
||||
I already have Rust file named msghandler.rs containing the following functions for the webapp to use:
|
||||
- smartsend() to encode the above message envelop into json string.
|
||||
- smartreceive() to decode json string back to message envelop.
|
||||
I already have this Rust module ./src/msghandler.rs containing the following functions for the webapp to use:
|
||||
- smartpack() to encode the above message envelop into json string.
|
||||
- smartunpack() to decode json string back to message envelop.
|
||||
- the msghandler.rs walkthrough is at /home/ton/docker-apps/sommpanion/msghandler/docs/walkthrough.md
|
||||
The backend server REST API endpoint is "myservice.mydomain.com/subservice/api/v1/chat". I didn't run the server yet.
|
||||
I already setup the project structure but you can modify the folder as you see fit. Can you implement the app? Use this command "dx bundle --web --release --debug-symbols=false" to check whether the project can be build.
|
||||
P.S. AI_prompt.md is for me to use. do not read.
|
||||
MQTT will be used as communication channel between the webapp and the backend. MQTT broker is "mqtt.mydomain.com". I didn't run the broker yet.
|
||||
I already setup the project structure. Can you implement the app?
|
||||
To test whether this Dioxus project can be build, you may use this command "dx bundle --web --release --debug-symbols=false"
|
||||
|
||||
P.S. In a Dioxus single-page application (SPA), switching screens can be handled perfectly using standard Rust state matching (often called conditional rendering or state-based routing).
|
||||
|
||||
|
||||
|
||||
@@ -23,9 +23,9 @@ futures = "0.3"
|
||||
tempfile = "3"
|
||||
|
||||
[[example]]
|
||||
name = "smartsend_example"
|
||||
path = "examples/smartsend_example.rs"
|
||||
name = "smartpack_example"
|
||||
path = "examples/smartpack_example.rs"
|
||||
|
||||
[[example]]
|
||||
name = "smartreceive_example"
|
||||
path = "examples/smartreceive_example.rs"
|
||||
name = "smartunpack_example"
|
||||
path = "examples/smartunpack_example.rs"
|
||||
|
||||
84
README.md
84
README.md
@@ -110,7 +110,7 @@ msghandler enables seamless communication across multiple platforms through NATS
|
||||
using msghandler
|
||||
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = smartsend("/chat/room1", data; broker_url="nats://localhost:4222")
|
||||
env, env_json_str = smartpack("/chat/room1", data; broker_url="nats://localhost:4222")
|
||||
println("Message sent!")
|
||||
```
|
||||
|
||||
@@ -120,7 +120,7 @@ println("Message sent!")
|
||||
import msghandler from './src/msghandler_ssr.js';
|
||||
|
||||
const data = [["message", "Hello World", "text"]];
|
||||
const [env, env_json_str] = await msghandler.smartsend(
|
||||
const [env, env_json_str] = await msghandler.smartpack(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222" }
|
||||
@@ -134,7 +134,7 @@ console.log("Message sent!");
|
||||
import msghandler from './src/msghandler_csr.js';
|
||||
|
||||
const data = [["message", "Hello World", "text"]];
|
||||
const [env, env_json_str] = await msghandler.smartsend(
|
||||
const [env, env_json_str] = await msghandler.smartpack(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ broker_url: "ws://localhost:4222" }
|
||||
@@ -145,10 +145,10 @@ console.log("Message sent!");
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from msghandler import smartsend
|
||||
from msghandler import smartpack
|
||||
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = await smartsend(
|
||||
env, env_json_str = await smartpack(
|
||||
"/chat/room1",
|
||||
data,
|
||||
broker_url="nats://localhost:4222"
|
||||
@@ -159,10 +159,10 @@ print("Message sent!")
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
from msghandler import smartsend
|
||||
from msghandler import smartpack
|
||||
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = smartsend(
|
||||
env, env_json_str = smartpack(
|
||||
"/chat/room1",
|
||||
data,
|
||||
broker_url="nats://localhost:4222",
|
||||
@@ -179,12 +179,12 @@ print("Message sent!")
|
||||
|
||||
All platforms use the same input/output format for payloads:
|
||||
|
||||
**Input format for `smartsend`:**
|
||||
**Input format for `smartpack`:**
|
||||
```
|
||||
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||
```
|
||||
|
||||
**Output format for `smartreceive`:**
|
||||
**Output format for `smartunpack`:**
|
||||
```json
|
||||
{
|
||||
"correlation_id": "...",
|
||||
@@ -204,7 +204,7 @@ All platforms use the same input/output format for payloads:
|
||||
}
|
||||
```
|
||||
|
||||
### smartsend
|
||||
### smartpack
|
||||
|
||||
Sends data either directly via NATS or via a fileserver URL, depending on payload size.
|
||||
|
||||
@@ -213,7 +213,7 @@ Sends data either directly via NATS or via a fileserver URL, depending on payloa
|
||||
```julia
|
||||
using msghandler
|
||||
|
||||
env, env_json_str = msghandler.smartsend(
|
||||
env, env_json_str = msghandler.smartpack(
|
||||
subject::String,
|
||||
data::AbstractArray{Tuple{String, Any, String}};
|
||||
broker_url::String = "nats://localhost:4222",
|
||||
@@ -240,7 +240,7 @@ env, env_json_str = msghandler.smartsend(
|
||||
```javascript
|
||||
import msghandler from './src/msghandler_ssr.js';
|
||||
|
||||
const [env, env_json_str] = await msghandler.smartsend(
|
||||
const [env, env_json_str] = await msghandler.smartpack(
|
||||
subject,
|
||||
data, // Array of [dataname, data, type] tuples
|
||||
{
|
||||
@@ -269,7 +269,7 @@ const [env, env_json_str] = await msghandler.smartsend(
|
||||
```javascript
|
||||
import msghandler from './src/msghandler_csr.js';
|
||||
|
||||
const [env, env_json_str] = await msghandler.smartsend(
|
||||
const [env, env_json_str] = await msghandler.smartpack(
|
||||
subject,
|
||||
data,
|
||||
{
|
||||
@@ -298,7 +298,7 @@ const [env, env_json_str] = await msghandler.smartsend(
|
||||
```python
|
||||
from msghandler import msghandler
|
||||
|
||||
env, env_json_str = await msghandler.smartsend(
|
||||
env, env_json_str = await msghandler.smartpack(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
broker_url: str = "nats://localhost:4222",
|
||||
@@ -326,7 +326,7 @@ env, env_json_str = await msghandler.smartsend(
|
||||
from msghandler import msghandler
|
||||
|
||||
# Limited to direct transport (< 100KB threshold)
|
||||
env, env_json_str = msghandler.smartsend(
|
||||
env, env_json_str = msghandler.smartpack(
|
||||
subject,
|
||||
data, # List of (dataname, data, type) tuples
|
||||
broker_url="nats://localhost:4222",
|
||||
@@ -335,7 +335,7 @@ env, env_json_str = msghandler.smartsend(
|
||||
# Returns: Tuple[Dict, str]
|
||||
```
|
||||
|
||||
### smartreceive
|
||||
### smartunpack
|
||||
|
||||
Receives and processes messages from NATS, handling both direct and link transport.
|
||||
|
||||
@@ -344,7 +344,7 @@ Receives and processes messages from NATS, handling both direct and link transpo
|
||||
```julia
|
||||
using msghandler
|
||||
|
||||
env = msghandler.smartreceive(
|
||||
env = msghandler.smartunpack(
|
||||
msg::NATS.Msg;
|
||||
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||
max_retries::Int = 5,
|
||||
@@ -359,7 +359,7 @@ env = msghandler.smartreceive(
|
||||
```javascript
|
||||
import msghandler from './src/msghandler_ssr.js';
|
||||
|
||||
const env = await msghandler.smartreceive(
|
||||
const env = await msghandler.smartunpack(
|
||||
msg,
|
||||
{
|
||||
fileserver_download_handler: msghandler.fetchWithBackoff,
|
||||
@@ -376,7 +376,7 @@ const env = await msghandler.smartreceive(
|
||||
```javascript
|
||||
import msghandler from './src/msghandler_csr.js';
|
||||
|
||||
const env = await msghandler.smartreceive(
|
||||
const env = await msghandler.smartunpack(
|
||||
msg,
|
||||
{
|
||||
fileserver_download_handler: msghandler.fetchWithBackoff,
|
||||
@@ -393,7 +393,7 @@ const env = await msghandler.smartreceive(
|
||||
```python
|
||||
from msghandler import msghandler
|
||||
|
||||
env = await msghandler.smartreceive(
|
||||
env = await msghandler.smartunpack(
|
||||
msg,
|
||||
fileserver_download_handler=fetch_with_backoff,
|
||||
max_retries=5,
|
||||
@@ -408,7 +408,7 @@ env = await msghandler.smartreceive(
|
||||
```python
|
||||
from msghandler import msghandler
|
||||
|
||||
env = msghandler.smartreceive(
|
||||
env = msghandler.smartunpack(
|
||||
msg,
|
||||
fileserver_download_handler=_sync_fileserver_download,
|
||||
max_retries=3,
|
||||
@@ -452,7 +452,7 @@ data = [
|
||||
("large_document", large_file_data, "binary")
|
||||
]
|
||||
|
||||
env, env_json_str = smartsend("/chat/room1", data; fileserver_url="http://localhost:8080")
|
||||
env, env_json_str = smartpack("/chat/room1", data; fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
#### JavaScript (Node.js)
|
||||
@@ -466,7 +466,7 @@ const data = [
|
||||
["large_document", largeFileData, "binary"]
|
||||
];
|
||||
|
||||
const [env, env_json_str] = await msghandler.smartsend(
|
||||
const [env, env_json_str] = await msghandler.smartpack(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ fileserver_url: 'http://localhost:8080' }
|
||||
@@ -484,7 +484,7 @@ const data = [
|
||||
["large_document", largeFileData, "binary"]
|
||||
];
|
||||
|
||||
const [env, env_json_str] = await msghandler.smartsend(
|
||||
const [env, env_json_str] = await msghandler.smartpack(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ broker_url: 'ws://localhost:4222', fileserver_url: 'http://localhost:8080' }
|
||||
@@ -502,7 +502,7 @@ data = [
|
||||
("large_document", large_file_data, "binary")
|
||||
]
|
||||
|
||||
env, env_json_str = await msghandler.smartsend(
|
||||
env, env_json_str = await msghandler.smartpack(
|
||||
"/chat/room1",
|
||||
data,
|
||||
fileserver_url="http://localhost:8080"
|
||||
@@ -525,7 +525,7 @@ config = Dict(
|
||||
)
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env, env_json_str = smartsend("/device/config", data)
|
||||
env, env_json_str = smartpack("/device/config", data)
|
||||
```
|
||||
|
||||
#### JavaScript (Node.js)
|
||||
@@ -539,7 +539,7 @@ const config = {
|
||||
update_interval: 60
|
||||
};
|
||||
|
||||
const [env, env_json_str] = await msghandler.smartsend(
|
||||
const [env, env_json_str] = await msghandler.smartpack(
|
||||
"/device/config",
|
||||
[["config", config, "dictionary"]]
|
||||
);
|
||||
@@ -557,7 +557,7 @@ config = {
|
||||
}
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env, env_json_str = await msghandler.smartsend("/device/config", data)
|
||||
env, env_json_str = await msghandler.smartpack("/device/config", data)
|
||||
```
|
||||
|
||||
### Example 3: Table Data (Arrow IPC)
|
||||
@@ -577,7 +577,7 @@ df = DataFrame(
|
||||
)
|
||||
|
||||
data = [("students", df, "arrowtable")]
|
||||
env, env_json_str = smartsend("/data/analysis", data)
|
||||
env, env_json_str = smartpack("/data/analysis", data)
|
||||
```
|
||||
|
||||
#### JavaScript (Node.js)
|
||||
@@ -591,7 +591,7 @@ const df = [
|
||||
{ id: 3, name: "Charlie", score: 92 }
|
||||
];
|
||||
|
||||
const [env, env_json_str] = await msghandler.smartsend(
|
||||
const [env, env_json_str] = await msghandler.smartpack(
|
||||
"/data/analysis",
|
||||
[["students", df, "arrowtable"]]
|
||||
);
|
||||
@@ -610,7 +610,7 @@ df = pd.DataFrame({
|
||||
})
|
||||
|
||||
data = [("students", df, "arrowtable")]
|
||||
env, env_json_str = await msghandler.smartsend("/data/analysis", data)
|
||||
env, env_json_str = await msghandler.smartpack("/data/analysis", data)
|
||||
```
|
||||
|
||||
#### JavaScript (Browser)
|
||||
@@ -626,7 +626,7 @@ const df = [
|
||||
{ id: 3, name: "Charlie", score: 92 }
|
||||
];
|
||||
|
||||
const [env, env_json_str] = await msghandler.smartsend(
|
||||
const [env, env_json_str] = await msghandler.smartpack(
|
||||
"/data/analysis",
|
||||
[["students", df, "jsontable"]], // Use jsontable for browser
|
||||
{ broker_url: 'ws://localhost:4222' }
|
||||
@@ -643,7 +643,7 @@ Bi-directional communication with reply-to support.
|
||||
using msghandler
|
||||
|
||||
# Requester
|
||||
env, env_json_str = smartsend(
|
||||
env, env_json_str = smartpack(
|
||||
"/device/command",
|
||||
[("command", Dict("action" => "read_sensor"), "dictionary")];
|
||||
broker_url="nats://localhost:4222",
|
||||
@@ -652,9 +652,9 @@ env, env_json_str = smartsend(
|
||||
|
||||
# Receiver (in separate application)
|
||||
msg = NATS.subscription.next()
|
||||
env = smartreceive(msg)
|
||||
env = smartunpack(msg)
|
||||
# Process request and send response
|
||||
response_env, response_json = smartsend(
|
||||
response_env, response_json = smartpack(
|
||||
"/device/response",
|
||||
[("result", Dict("value" => 42), "dictionary")],
|
||||
reply_to="/device/command",
|
||||
@@ -668,7 +668,7 @@ response_env, response_json = smartsend(
|
||||
import msghandler from './src/msghandler_ssr.js';
|
||||
|
||||
// Requester
|
||||
const [env, env_json_str] = await msghandler.smartsend(
|
||||
const [env, env_json_str] = await msghandler.smartpack(
|
||||
"/device/command",
|
||||
[["command", { action: "read_sensor" }, "dictionary"]],
|
||||
{ broker_url: 'nats://localhost:4222', reply_to: '/device/response' }
|
||||
@@ -676,9 +676,9 @@ const [env, env_json_str] = await msghandler.smartsend(
|
||||
|
||||
// Receiver (in separate application)
|
||||
// const msg = await natsConsumer.next();
|
||||
// const env = await msghandler.smartreceive(msg);
|
||||
// const env = await msghandler.smartunpack(msg);
|
||||
// Process request and send response
|
||||
// const response_env, response_json = await msghandler.smartsend(
|
||||
// const response_env, response_json = await msghandler.smartpack(
|
||||
// "/device/response",
|
||||
// [["result", { value: 42 }, "dictionary"]],
|
||||
// { reply_to: '/device/command', reply_to_msg_id: env.msg_id }
|
||||
@@ -691,7 +691,7 @@ const [env, env_json_str] = await msghandler.smartsend(
|
||||
from msghandler import msghandler
|
||||
|
||||
# Requester
|
||||
env, env_json_str = await msghandler.smartsend(
|
||||
env, env_json_str = await msghandler.smartpack(
|
||||
"/device/command",
|
||||
[("command", {"action": "read_sensor"}, "dictionary")],
|
||||
broker_url="nats://localhost:4222",
|
||||
@@ -700,9 +700,9 @@ env, env_json_str = await msghandler.smartsend(
|
||||
|
||||
# Receiver (in separate application)
|
||||
# msg = await nats_consumer.next()
|
||||
# env = await msghandler.smartreceive(msg)
|
||||
# env = await msghandler.smartunpack(msg)
|
||||
# Process request and send response
|
||||
# response_env, response_json = await msghandler.smartsend(
|
||||
# response_env, response_json = await msghandler.smartpack(
|
||||
# "/device/response",
|
||||
# [("result", {"value": 42}, "dictionary")],
|
||||
# reply_to="/device/command",
|
||||
@@ -895,7 +895,7 @@ node build.js
|
||||
import msghandlerCSR from './dist/msghandler-csr-bundle.js';
|
||||
|
||||
// Use the library
|
||||
const [env, envJson] = await msghandlerCSR.smartsend(
|
||||
const [env, envJson] = await msghandlerCSR.smartpack(
|
||||
"/chat/user/v1/message",
|
||||
[["msg", "Hello", "text"]],
|
||||
{ broker_url: "wss://nats.example.com" }
|
||||
|
||||
@@ -130,8 +130,8 @@ flowchart TD
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph "msghandler Module"
|
||||
SmartSend[smartsend Function]
|
||||
SmartReceive[smartreceive Function]
|
||||
smartpack[smartpack Function]
|
||||
smartunpack[smartunpack Function]
|
||||
|
||||
Serialize[_serialize_data]
|
||||
Deserialize[_deserialize_data]
|
||||
@@ -149,18 +149,18 @@ flowchart TD
|
||||
Envelope[msg_envelope_v1 Struct]
|
||||
end
|
||||
|
||||
SmartSend --> Serialize
|
||||
SmartSend --> EnvelopeToJson
|
||||
SmartSend --> FileServerUpload
|
||||
smartpack --> Serialize
|
||||
smartpack --> EnvelopeToJson
|
||||
smartpack --> FileServerUpload
|
||||
|
||||
SmartReceive --> Deserialize
|
||||
SmartReceive --> FileServerDownload
|
||||
smartunpack --> Deserialize
|
||||
smartunpack --> FileServerDownload
|
||||
|
||||
EnvelopeToJson --> Envelope
|
||||
Serialize --> Payload
|
||||
|
||||
style SmartSend fill:#d1fae5,stroke:#10b981
|
||||
style SmartReceive fill:#d1fae5,stroke:#10b981
|
||||
style smartpack fill:#d1fae5,stroke:#10b981
|
||||
style smartunpack fill:#d1fae5,stroke:#10b981
|
||||
style FileServerUpload fill:#fef3c7,stroke:#f59e0b
|
||||
style FileServerDownload fill:#fef3c7,stroke:#f59e0b
|
||||
```
|
||||
@@ -173,8 +173,8 @@ flowchart TD
|
||||
|
||||
| Component | Purpose | Platform Support |
|
||||
|-----------|---------|------------------|
|
||||
| **smartsend** | Send data with automatic transport selection, returns (envelope, json_string) for caller to publish via transport | All |
|
||||
| **smartreceive** | Receive and process messages from JSON string | All |
|
||||
| **smartpack** | Send data with automatic transport selection, returns (envelope, json_string) for caller to publish via transport | All |
|
||||
| **smartunpack** | Receive and process messages from JSON string | All |
|
||||
| **_serialize_data** | Serialize data according to payload type | All |
|
||||
| **_deserialize_data** | Deserialize bytes to native data types | All |
|
||||
| **envelope_to_json** | Convert msg_envelope_v1 struct to JSON string | All |
|
||||
@@ -187,7 +187,7 @@ flowchart TD
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[User calls smartsend subject data] --> B[Process each payload]
|
||||
A[User calls smartpack subject data] --> B[Process each payload]
|
||||
B --> C{Calculate serialized size}
|
||||
C -->|Size < Threshold| D[Direct Transport]
|
||||
C -->|Size >= Threshold| E[Link Transport]
|
||||
@@ -349,7 +349,7 @@ flowchart TD
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[smartsend called] --> B[Serialize payload]
|
||||
A[smartpack called] --> B[Serialize payload]
|
||||
B --> C[Calculate size]
|
||||
C --> D{Size < Threshold?}
|
||||
|
||||
@@ -560,7 +560,7 @@ pub enum Payload {
|
||||
}
|
||||
|
||||
// Configuration via builder pattern
|
||||
pub struct SmartsendOptions {
|
||||
pub struct smartpackOptions {
|
||||
pub broker_url: String,
|
||||
pub fileserver_url: String,
|
||||
pub fileserver_upload_handler: Option<Arc<dyn FileUploadHandler>>,
|
||||
@@ -577,7 +577,7 @@ let conn = transport_client::connect(DEFAULT_BROKER_URL).await?;
|
||||
// Subscribe and process messages
|
||||
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze")?;
|
||||
for msg in sub.messages() {
|
||||
let envelope = smartreceive(&String::from_utf8_lossy(&msg.payload), &Default::default()).await?;
|
||||
let envelope = smartunpack(&String::from_utf8_lossy(&msg.payload), &Default::default()).await?;
|
||||
// Access deserialized payloads by type
|
||||
for payload in &envelope.payloads {
|
||||
match payload.payload_type.as_str() {
|
||||
@@ -840,11 +840,11 @@ flowchart TD
|
||||
| - | - | Updated diagrams to use generic "Message Broker" instead of "NATS Server" | All sections |
|
||||
| - | - | Updated code examples to use transport-agnostic patterns | All sections |
|
||||
| - | - | Removed NATS client packages from external dependencies | All sections |
|
||||
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections |
|
||||
| - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
|
||||
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartunpack` deserialization changes | All sections |
|
||||
| - | - | `smartunpack` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
|
||||
| - | - | Added `plik_upload_file` convenience function to component table | specification.md:13 |
|
||||
| - | - | Fixed Rust payload access pattern (data is String, not Payload enum) | All sections |
|
||||
| - | - | Fixed `SmartsendOptions.fileserver_upload_handler` type to `Arc<dyn FileUploadHandler>` | specification.md:13 |
|
||||
| - | - | Fixed `smartpackOptions.fileserver_upload_handler` type to `Arc<dyn FileUploadHandler>` | specification.md:13 |
|
||||
| - | - | Removed `metadata` from link transport examples (now `None`/omitted) | specification.md:3 |
|
||||
| - | - | Removed duplicate footer text | All sections |
|
||||
| 2026-05-13 | 1.3.0 | Added Rust support with tokio, serde, and arrow2 | All sections |
|
||||
@@ -854,9 +854,9 @@ flowchart TD
|
||||
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) |
|
||||
| - | - | Removed publish_message component (commented out in source) |
|
||||
| - | - | Removed NATSClient and NATSConnectionPool classes (not in ground truth) |
|
||||
| - | - | Updated smartsend to return JSON for caller to publish via transport |
|
||||
| - | - | Updated smartpack to return JSON for caller to publish via transport |
|
||||
| - | - | Updated component diagram to match actual module structure |
|
||||
| - | - | Updated data flow to show smartsend returns JSON for caller to publish |
|
||||
| - | - | Updated data flow to show smartpack returns JSON for caller to publish |
|
||||
| - | - | Fixed SIZE_THRESHOLD default to 500,000 bytes |
|
||||
| 2026-03-15 | 1.1.0 | JavaScript connection management |
|
||||
| - | - | Added NATSClient with keepAlive support |
|
||||
|
||||
@@ -54,7 +54,7 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
|
||||
|---------|-------------|
|
||||
| Cross-platform interoperability | Seamless data exchange between Julia, JavaScript, Python, Dart, Rust, and MicroPython |
|
||||
| Intelligent transport selection | Direct transport (<0.5MB) vs Link transport (≥0.5MB) based on payload size |
|
||||
| Unified API | Consistent `smartsend()` and `smartreceive()` functions across all platforms |
|
||||
| Unified API | Consistent `smartpack()` and `smartunpack()` functions across all platforms |
|
||||
| Multi-payload support | List of (dataname, data, type) tuples with appropriate handling |
|
||||
| File server integration | Plik one-shot upload and custom HTTP server support |
|
||||
| Reliability features | Exponential backoff retry and correlation ID propagation |
|
||||
@@ -323,10 +323,10 @@ msghandler is a cross-platform, bi-directional data bridge that enables seamless
|
||||
|
||||
## 11. API Contract
|
||||
|
||||
### 11.1 smartsend Signature
|
||||
### 11.1 smartpack Signature
|
||||
|
||||
```julia
|
||||
function smartsend(
|
||||
function smartpack(
|
||||
subject::String,
|
||||
data::AbstractArray{Tuple{String, T1, String}, 1};
|
||||
broker_url::String = DEFAULT_BROKER_URL,
|
||||
@@ -345,12 +345,12 @@ function smartsend(
|
||||
)::Tuple{msg_envelope_v1, String} where {T1<:Any}
|
||||
```
|
||||
|
||||
**Note**: Publishing via the transport layer is the caller's responsibility. `smartsend` returns `(env::msg_envelope_v1, env_json_str::String)`.
|
||||
**Note**: Publishing via the transport layer is the caller's responsibility. `smartpack` returns `(env::msg_envelope_v1, env_json_str::String)`.
|
||||
|
||||
### 11.2 smartreceive Signature
|
||||
### 11.2 smartunpack Signature
|
||||
|
||||
```julia
|
||||
function smartreceive(
|
||||
function smartunpack(
|
||||
msg_json_str::String;
|
||||
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||
max_retries::Int = 5,
|
||||
@@ -359,9 +359,9 @@ function smartreceive(
|
||||
)::JSON.Object{String, Any}
|
||||
```
|
||||
|
||||
**Note**: Pass the payload string from the transport subscription to `smartreceive`. The input is the JSON string payload from the transport message, not the transport message object directly.
|
||||
**Note**: Pass the payload string from the transport subscription to `smartunpack`. The input is the JSON string payload from the transport message, not the transport message object directly.
|
||||
|
||||
**Note**: Pass the payload from the transport subscription to `smartreceive`.
|
||||
**Note**: Pass the payload from the transport subscription to `smartunpack`.
|
||||
|
||||
---
|
||||
|
||||
@@ -411,8 +411,8 @@ function smartreceive(
|
||||
| - | - | Updated all NATS references to generic "transport layer"/"message broker" |
|
||||
| - | - | Removed NATS client packages from dependencies tables |
|
||||
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) |
|
||||
| - | - | Fixed smartsend signature: removed is_publish, NATS_connection; added sender_name |
|
||||
| - | - | Fixed smartreceive signature: takes msg_json_str::String instead of msg::NATS.Msg |
|
||||
| - | - | Fixed smartpack signature: removed is_publish, NATS_connection; added sender_name |
|
||||
| - | - | Fixed smartunpack signature: takes msg_json_str::String instead of msg::NATS.Msg |
|
||||
| - | - | Fixed size_threshold default from 1,000,000 to 500,000 |
|
||||
| - | - | Updated FR-013/FR-014 to reflect caller responsibility for NATS publishing |
|
||||
| - | - | Updated FR-008/FR-009 to include file path upload overload |
|
||||
|
||||
@@ -13,8 +13,8 @@
|
||||
This document defines the **technical contract** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer.
|
||||
|
||||
This specification serves as the single source of truth for:
|
||||
- **Inputs**: What data structures are accepted by `smartsend()`
|
||||
- **Outputs**: What data structures are returned by `smartreceive()`
|
||||
- **Inputs**: What data structures are accepted by `smartpack()`
|
||||
- **Outputs**: What data structures are returned by `smartunpack()`
|
||||
- **Data Shapes**: Exact field names, types, and constraints
|
||||
- **Error Codes**: Standardized error responses for failure scenarios
|
||||
|
||||
@@ -24,7 +24,7 @@ This specification serves as the single source of truth for:
|
||||
|----------------------|-------------------|-------------|
|
||||
| Section 2 (Message Envelope) | FR-012, FR-013, NFR-101, NFR-102 | Message envelope structure and validation |
|
||||
| Section 3 (Payload Schema) | FR-001, FR-002, FR-003, FR-004, NFR-101, NFR-102 | Payload structure and field definitions |
|
||||
| Section 4 (Payload Format) | FR-006, FR-007 | Tuple format for smartsend() |
|
||||
| Section 4 (Payload Format) | FR-006, FR-007 | Tuple format for smartpack() |
|
||||
| Section 5 (Enumerations) | FR-003, FR-004, FR-006, NFR-101 | Enumerations for transport and encoding |
|
||||
| Section 6 (Transport Protocols) | FR-003, FR-004, NFR-104, NFR-105 | Direct and link transport protocols |
|
||||
| Section 7 (Size Thresholds) | FR-004, FR-005, NFR-104, NFR-105 | Size thresholds for transport selection |
|
||||
@@ -143,9 +143,9 @@ This specification serves as the single source of truth for:
|
||||
|
||||
## Payload Format
|
||||
|
||||
### Tuple Format for `smartsend()`
|
||||
### Tuple Format for `smartpack()`
|
||||
|
||||
The `smartsend()` function accepts data as an array of tuples with the format:
|
||||
The `smartpack()` function accepts data as an array of tuples with the format:
|
||||
|
||||
```
|
||||
("data_name", data, "data_type")
|
||||
@@ -161,17 +161,17 @@ The `smartsend()` function accepts data as an array of tuples with the format:
|
||||
|
||||
```julia
|
||||
# Julia
|
||||
smartsend("/chat/user/v1/message", [("msg", "Hello World", "text")])
|
||||
smartpack("/chat/user/v1/message", [("msg", "Hello World", "text")])
|
||||
```
|
||||
|
||||
```python
|
||||
# Python
|
||||
await smartsend("/chat/user/v1/message", [("msg", "Hello World", "text")])
|
||||
await smartpack("/chat/user/v1/message", [("msg", "Hello World", "text")])
|
||||
```
|
||||
|
||||
```typescript
|
||||
// JavaScript
|
||||
await smartsend("/chat/user/v1/message", [["msg", "Hello World", "text"]]);
|
||||
await smartpack("/chat/user/v1/message", [["msg", "Hello World", "text"]]);
|
||||
```
|
||||
|
||||
### Multiple Payloads Example
|
||||
@@ -182,7 +182,7 @@ data = [
|
||||
("msg", "Hello", "text"),
|
||||
("img", binary_data, "image")
|
||||
]
|
||||
smartsend("/agent/v1/process", data)
|
||||
smartpack("/agent/v1/process", data)
|
||||
```
|
||||
|
||||
```python
|
||||
@@ -191,7 +191,7 @@ data = [
|
||||
("msg", "Hello", "text"),
|
||||
("img", binary_data, "image")
|
||||
]
|
||||
await smartsend("/agent/v1/process", data)
|
||||
await smartpack("/agent/v1/process", data)
|
||||
```
|
||||
|
||||
### Data Type Mapping
|
||||
@@ -411,12 +411,12 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa
|
||||
|
||||
## API Contract
|
||||
|
||||
### `smartsend` Function Signature
|
||||
### `smartpack` Function Signature
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
function smartsend(
|
||||
function smartpack(
|
||||
subject::String,
|
||||
data::AbstractArray{Tuple{String, T1, String}, 1};
|
||||
broker_url::String = DEFAULT_BROKER_URL,
|
||||
@@ -440,7 +440,7 @@ function smartsend(
|
||||
#### Python
|
||||
|
||||
```python
|
||||
async def smartsend(
|
||||
async def smartpack(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
broker_url: str = DEFAULT_BROKER_URL,
|
||||
@@ -464,7 +464,7 @@ async def smartsend(
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```typescript
|
||||
async function smartsend(
|
||||
async function smartpack(
|
||||
subject: string,
|
||||
data: Array<[string, any, string]>,
|
||||
options?: {
|
||||
@@ -490,7 +490,7 @@ async function smartsend(
|
||||
#### JavaScript (Browser)
|
||||
|
||||
```typescript
|
||||
async function smartsend(
|
||||
async function smartpack(
|
||||
subject: string,
|
||||
data: Array<[string, any, string]>,
|
||||
options?: {
|
||||
@@ -516,7 +516,7 @@ async function smartsend(
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
def smartsend(
|
||||
def smartpack(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
size_threshold: int = 100_000, # Lower threshold for memory constraints
|
||||
@@ -529,7 +529,7 @@ def smartsend(
|
||||
#### Dart (Desktop/Flutter)
|
||||
|
||||
```dart
|
||||
Future<[Map<String, dynamic>, String]> smartsend(
|
||||
Future<[Map<String, dynamic>, String]> smartpack(
|
||||
String subject,
|
||||
List<List<dynamic>> data, {
|
||||
String brokerUrl = DEFAULT_BROKER_URL,
|
||||
@@ -553,7 +553,7 @@ Future<[Map<String, dynamic>, String]> smartsend(
|
||||
#### Dart Web
|
||||
|
||||
```dart
|
||||
Future<[Map<String, dynamic>, String]> smartsend(
|
||||
Future<[Map<String, dynamic>, String]> smartpack(
|
||||
String subject,
|
||||
List<List<dynamic>> data, {
|
||||
String brokerUrl = DEFAULT_BROKER_URL,
|
||||
@@ -578,14 +578,14 @@ Future<[Map<String, dynamic>, String]> smartsend(
|
||||
#### Rust
|
||||
|
||||
```rust
|
||||
pub async fn smartsend(
|
||||
pub async fn smartpack(
|
||||
subject: &str,
|
||||
data: &[(String, Payload, String)],
|
||||
options: &SmartsendOptions,
|
||||
options: &smartpackOptions,
|
||||
) -> Result<(MsgEnvelopeV1, String), msghandlerError>
|
||||
|
||||
// SmartsendOptions struct
|
||||
pub struct SmartsendOptions {
|
||||
// smartpackOptions struct
|
||||
pub struct smartpackOptions {
|
||||
pub broker_url: String,
|
||||
pub fileserver_url: String,
|
||||
pub fileserver_upload_handler: Option<UploadHandler>,
|
||||
@@ -636,12 +636,12 @@ pub struct MsgEnvelopeV1 {
|
||||
|
||||
**Note**: Publishing via the transport layer is the caller's responsibility. Returns `Result<(MsgEnvelopeV1, String), msghandlerError>`. Uses `serde` for JSON serialization.
|
||||
|
||||
### `smartreceive` Function Signature
|
||||
### `smartunpack` Function Signature
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
function smartreceive(
|
||||
function smartunpack(
|
||||
msg_json_str::String; # Pass payload from transport subscription
|
||||
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||
max_retries::Int = 5,
|
||||
@@ -655,7 +655,7 @@ function smartreceive(
|
||||
#### Python
|
||||
|
||||
```python
|
||||
async def smartreceive(
|
||||
async def smartunpack(
|
||||
msg_json_str: str, # JSON string from transport message payload
|
||||
fileserver_download_handler: Callable = fetch_with_backoff,
|
||||
max_retries: int = 5,
|
||||
@@ -669,7 +669,7 @@ async def smartreceive(
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```typescript
|
||||
async function smartreceive(
|
||||
async function smartunpack(
|
||||
msg_json_str: string, // JSON string from transport message payload
|
||||
options?: {
|
||||
fileserver_download_handler?: Function;
|
||||
@@ -683,7 +683,7 @@ async function smartreceive(
|
||||
#### JavaScript (Browser)
|
||||
|
||||
```typescript
|
||||
async function smartreceive(
|
||||
async function smartunpack(
|
||||
msg_json_str: string, // JSON string from transport message payload
|
||||
options?: {
|
||||
fileserver_download_handler?: Function;
|
||||
@@ -699,7 +699,7 @@ async function smartreceive(
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
def smartreceive(msg_json_str: str, **kwargs) -> Dict[str, Any]:
|
||||
def smartunpack(msg_json_str: str, **kwargs) -> Dict[str, Any]:
|
||||
```
|
||||
|
||||
**Note**: Input is the JSON string payload from the transport message.
|
||||
@@ -707,7 +707,7 @@ def smartreceive(msg_json_str: str, **kwargs) -> Dict[str, Any]:
|
||||
#### Dart (Desktop/Flutter)
|
||||
|
||||
```dart
|
||||
Future<Map<String, dynamic>> smartreceive(
|
||||
Future<Map<String, dynamic>> smartunpack(
|
||||
Map<String, dynamic> msg_json_str, // JSON object from transport message payload
|
||||
{
|
||||
Function? fileserverDownloadHandler,
|
||||
@@ -722,7 +722,7 @@ Future<Map<String, dynamic>> smartreceive(
|
||||
#### Dart Web
|
||||
|
||||
```dart
|
||||
Future<Map<String, dynamic>> smartreceive(
|
||||
Future<Map<String, dynamic>> smartunpack(
|
||||
Map<String, dynamic> msg_json_str, // JSON object from transport message payload
|
||||
{
|
||||
Function? fileserverDownloadHandler,
|
||||
@@ -737,13 +737,13 @@ Future<Map<String, dynamic>> smartreceive(
|
||||
#### Rust
|
||||
|
||||
```rust
|
||||
pub async fn smartreceive(
|
||||
pub async fn smartunpack(
|
||||
msg_json_str: &str, // JSON string from transport message payload
|
||||
options: &SmartreceiveOptions,
|
||||
options: &smartunpackOptions,
|
||||
) -> Result<MsgEnvelopeV1, msghandlerError>
|
||||
|
||||
// SmartreceiveOptions struct
|
||||
pub struct SmartreceiveOptions {
|
||||
// smartunpackOptions struct
|
||||
pub struct smartunpackOptions {
|
||||
pub fileserver_download_handler: Option<DownloadHandler>,
|
||||
pub max_retries: u32,
|
||||
pub base_delay: u64,
|
||||
@@ -933,7 +933,7 @@ The browser implementation ([`src/msghandler_csr.js`](../src/msghandler_csr.js))
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[User calls smartsend subject data] --> B[Serialize payload according to payload_type]
|
||||
A[User calls smartpack subject data] --> B[Serialize payload according to payload_type]
|
||||
B --> C{Calculate serialized size}
|
||||
C -->|Size < Threshold| D[Direct Transport: Encode as Base64]
|
||||
C -->|Size >= Threshold| E[Link Transport: Upload to file server]
|
||||
@@ -1156,8 +1156,8 @@ flowchart TD
|
||||
| - | - | Updated all NATS references to generic "transport layer"/"message broker" | All |
|
||||
| - | - | Removed NATS client packages from dependencies tables | All |
|
||||
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All |
|
||||
| - | - | Updated smartsend signatures: removed is_publish, nats_connection; added sender_name | FR-001 through FR-014 |
|
||||
| - | - | Updated smartreceive signatures: takes msg_json_str::String instead of msg | FR-001 through FR-014 |
|
||||
| - | - | Updated smartpack signatures: removed is_publish, nats_connection; added sender_name | FR-001 through FR-014 |
|
||||
| - | - | Updated smartunpack signatures: takes msg_json_str::String instead of msg | FR-001 through FR-014 |
|
||||
| - | - | Removed publishMessage function and NATSClient/NATSConnectionPool classes from browser section | FR-013, FR-014 |
|
||||
| - | - | Added plik_oneshot_upload(filepath) overload to file server interface | FR-008, FR-009 |
|
||||
| - | - | Fixed SIZE_THRESHOLD default to 500,000 bytes | FR-003, FR-004 |
|
||||
|
||||
@@ -44,7 +44,7 @@ flowchart TB
|
||||
subgraph msghandler["msghandler Module"]
|
||||
direction TB
|
||||
|
||||
subgraph Sender["Sender (smartsend)"]
|
||||
subgraph Sender["Sender (smartpack)"]
|
||||
direction LR
|
||||
S1["Data Tuples<br/>[(dataname, data, type)]"]
|
||||
S2["Serialize Data"]
|
||||
@@ -60,7 +60,7 @@ flowchart TB
|
||||
S5 --> S6
|
||||
end
|
||||
|
||||
subgraph Receiver["Receiver (smartreceive)"]
|
||||
subgraph Receiver["Receiver (smartunpack)"]
|
||||
direction LR
|
||||
R1["Subscribe via transport"]
|
||||
R2["Parse Envelope"]
|
||||
@@ -101,7 +101,7 @@ flowchart TB
|
||||
|-----------|-------------|-----------|
|
||||
| **Claim-Check Pattern** | Large payloads uploaded to HTTP server, URL sent via transport | Transport has message size limits; avoids overflow |
|
||||
| **Automatic Transport Selection** | Direct (< threshold) vs Link (≥ threshold) based on size | Optimizes memory vs network I/O trade-off |
|
||||
| **Cross-Platform API** | Consistent `smartsend()`/`smartreceive()` across all platforms | Simplifies developer experience |
|
||||
| **Cross-Platform API** | Consistent `smartpack()`/`smartunpack()` across all platforms | Simplifies developer experience |
|
||||
| **Exponential Backoff** | Retry downloads with increasing delays | Handles transient failures gracefully |
|
||||
|
||||
---
|
||||
@@ -118,7 +118,7 @@ A JavaScript chat webapp wants to send mixed payloads (text message + user avata
|
||||
|
||||
```javascript
|
||||
// JavaScript (Browser or Node.js)
|
||||
const [env, msgJson] = await msghandler.smartsend(
|
||||
const [env, msgJson] = await msghandler.smartpack(
|
||||
"/agent/wine/api/v1/prompt",
|
||||
[
|
||||
["msg", "Hello! I'm Ton.", "text"],
|
||||
@@ -225,14 +225,14 @@ msghandler builds the message envelope:
|
||||
**Rationale**:
|
||||
- The transport layer provides message delivery (NATS, MQTT, WebSocket, etc.)
|
||||
- JSON format ensures cross-platform compatibility
|
||||
- `smartsend()` returns `(env, msgJson)` - caller handles publishing via their chosen transport
|
||||
- `smartpack()` returns `(env, msgJson)` - caller handles publishing via their chosen transport
|
||||
|
||||
#### Step 6: Julia Backend Receives Message
|
||||
|
||||
```julia
|
||||
# Julia backend
|
||||
transport_msg = transport_subscription.next() # Get message from transport
|
||||
env = smartreceive(String(transport_msg.payload))
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
|
||||
# env["payloads"] is now:
|
||||
# [
|
||||
@@ -242,7 +242,7 @@ env = smartreceive(String(transport_msg.payload))
|
||||
```
|
||||
|
||||
**Rationale**:
|
||||
- `smartreceive()` handles both transport types automatically
|
||||
- `smartunpack()` handles both transport types automatically
|
||||
- Deserialization is type-aware based on `payload_type`
|
||||
- Returns consistent tuple format regardless of transport
|
||||
|
||||
@@ -253,7 +253,7 @@ env = smartreceive(String(transport_msg.payload))
|
||||
response_text = "Hello Ton! I'm the AI assistant."
|
||||
generated_image = generate_ai_image(response_text)
|
||||
|
||||
env, msg_json = smartsend(
|
||||
env, msg_json = smartpack(
|
||||
"/agent/wine/api/v1/response",
|
||||
[
|
||||
("response", response_text, "text"),
|
||||
@@ -282,7 +282,7 @@ A JavaScript webapp wants to upload a large file (10MB) to a Julia backend for p
|
||||
#### Step 1: JavaScript Webapp Sends Large File
|
||||
|
||||
```javascript
|
||||
const [env, msgJson] = await msghandler.smartsend(
|
||||
const [env, msgJson] = await msghandler.smartpack(
|
||||
"/agent/wine/api/v1/process",
|
||||
[
|
||||
["file", largeFileData, "binary"]
|
||||
@@ -358,7 +358,7 @@ const response = await plikOneshotUpload(
|
||||
```julia
|
||||
# Julia backend
|
||||
transport_msg = transport_subscription.next()
|
||||
env = smartreceive(String(transport_msg.payload))
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
|
||||
# msghandler automatically:
|
||||
# 1. Extracts URL from payload
|
||||
@@ -386,7 +386,7 @@ A Python application sends tabular data (pandas DataFrame) to a Julia backend fo
|
||||
```python
|
||||
# Python
|
||||
import pandas as pd
|
||||
from msghandler import smartsend
|
||||
from msghandler import smartpack
|
||||
|
||||
df = pd.DataFrame({
|
||||
"id": [1, 2, 3],
|
||||
@@ -394,7 +394,7 @@ df = pd.DataFrame({
|
||||
"score": [95, 88, 92]
|
||||
})
|
||||
|
||||
env, msg_json = await smartsend(
|
||||
env, msg_json = await smartpack(
|
||||
"/agent/wine/api/v1/analyze",
|
||||
[("data", df, "arrowtable")],
|
||||
broker_url=DEFAULT_BROKER_URL,
|
||||
@@ -431,7 +431,7 @@ arrow_bytes = buf.getvalue()
|
||||
```julia
|
||||
# Julia backend
|
||||
transport_msg = transport_subscription.next()
|
||||
env = smartreceive(String(transport_msg.payload))
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
|
||||
# env["payloads"][1] is now:
|
||||
# ("data", DataFrame with id, name, score columns, "arrowtable")
|
||||
@@ -449,7 +449,7 @@ env = smartreceive(String(transport_msg.payload))
|
||||
results = analyze_data(env["payloads"][1][2])
|
||||
|
||||
# Send results back
|
||||
env, msg_json = smartsend(
|
||||
env, msg_json = smartpack(
|
||||
"/agent/wine/api/v1/results",
|
||||
[("results", results, "arrowtable")],
|
||||
reply_to = "/python/worker/v1/results"
|
||||
@@ -475,7 +475,7 @@ A Rust service needs to process messages from a Julia analytics pipeline and sen
|
||||
|
||||
```rust
|
||||
// Rust service - using tokio async runtime
|
||||
use msghandler::{smartreceive, MsgEnvelopeV1};
|
||||
use msghandler::{smartunpack, MsgEnvelopeV1};
|
||||
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64};
|
||||
|
||||
#[tokio::main]
|
||||
@@ -486,7 +486,7 @@ async fn main() {
|
||||
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze").unwrap();
|
||||
|
||||
for msg in sub.messages() {
|
||||
let envelope = smartreceive(
|
||||
let envelope = smartunpack(
|
||||
&String::from_utf8_lossy(&msg.payload),
|
||||
&Default::default(),
|
||||
).await.unwrap();
|
||||
@@ -495,7 +495,7 @@ async fn main() {
|
||||
for payload in &envelope.payloads {
|
||||
match payload.payload_type.as_str() {
|
||||
"arrowtable" => {
|
||||
// Data is base64-encoded Arrow IPC bytes after smartreceive()
|
||||
// Data is base64-encoded Arrow IPC bytes after smartunpack()
|
||||
let arrow_bytes = BASE64.decode(&payload.data).unwrap();
|
||||
println!("Received arrowtable payload ({} bytes)", arrow_bytes.len());
|
||||
},
|
||||
@@ -522,19 +522,19 @@ async fn main() {
|
||||
**Rationale**:
|
||||
- **serde serialization**: Automatic JSON deserialization to `MsgEnvelopeV1`
|
||||
- **tokio runtime**: Efficient async I/O for transport and HTTP operations
|
||||
- **smartreceive deserialization**: Payload data is deserialized and stored as strings in `payload.data`
|
||||
- **smartunpack deserialization**: Payload data is deserialized and stored as strings in `payload.data`
|
||||
- **Type dispatch**: `payload_type` field determines how to interpret the `data` string
|
||||
|
||||
#### Step 2: Rust Service Sends Processed Results
|
||||
|
||||
```rust
|
||||
// Rust service sends results back with mixed payload types
|
||||
use msghandler::{smartsend, Payload, SmartsendOptions};
|
||||
use msghandler::{smartpack, Payload, smartpackOptions};
|
||||
|
||||
let results_df = /* processed Arrow table */;
|
||||
let result_bytes = /* serialize to Arrow IPC */;
|
||||
|
||||
let (envelope, json_str) = smartsend(
|
||||
let (envelope, json_str) = smartpack(
|
||||
"/agent/wine/api/v1/results",
|
||||
&[
|
||||
(
|
||||
@@ -548,7 +548,7 @@ let (envelope, json_str) = smartsend(
|
||||
"text".to_string(),
|
||||
),
|
||||
],
|
||||
&SmartsendOptions {
|
||||
&smartpackOptions {
|
||||
broker_url: DEFAULT_BROKER_URL.to_string(),
|
||||
reply_to: "/python/worker/v1/results".to_string(),
|
||||
msg_purpose: "chat".to_string(),
|
||||
@@ -561,7 +561,7 @@ conn.publish("/agent/wine/api/v1/results", &json_str)?;
|
||||
```
|
||||
|
||||
**Rationale**:
|
||||
- **Builder pattern**: `SmartsendOptions` provides clean configuration
|
||||
- **Builder pattern**: `smartpackOptions` provides clean configuration
|
||||
- **Enum-based payloads**: Type safety prevents sending incorrect data types
|
||||
- **Default options**: sensible defaults reduce boilerplate
|
||||
- **Result<T, E>**: idiomatic Rust error handling
|
||||
@@ -570,7 +570,7 @@ conn.publish("/agent/wine/api/v1/results", &json_str)?;
|
||||
|
||||
```python
|
||||
# Python backend receives Rust response
|
||||
env = await smartreceive(str(transport_msg.payload))
|
||||
env = await smartunpack(str(transport_msg.payload))
|
||||
|
||||
# env["payloads"][0] is now:
|
||||
# ("results", arrow_table_data, "arrowtable")
|
||||
@@ -589,7 +589,7 @@ env = await smartreceive(str(transport_msg.payload))
|
||||
// Rust service sends large binary file via link transport
|
||||
let large_file_data: Vec<u8> = std::fs::read("/data/large_dataset.parquet")?;
|
||||
|
||||
let (envelope, json_str) = smartsend(
|
||||
let (envelope, json_str) = smartpack(
|
||||
"/agent/wine/api/v1/upload",
|
||||
&[
|
||||
(
|
||||
@@ -598,7 +598,7 @@ let (envelope, json_str) = smartsend(
|
||||
"binary".to_string(),
|
||||
),
|
||||
],
|
||||
&SmartsendOptions {
|
||||
&smartpackOptions {
|
||||
broker_url: DEFAULT_BROKER_URL.to_string(),
|
||||
fileserver_url: DEFAULT_FILESERVER_URL.to_string(),
|
||||
size_threshold: DEFAULT_SIZE_THRESHOLD, // threshold triggers link transport
|
||||
@@ -627,7 +627,7 @@ A MicroPython sensor device sends sensor readings to a Python backend.
|
||||
|
||||
```python
|
||||
# MicroPython
|
||||
from msghandler import smartsend
|
||||
from msghandler import smartpack
|
||||
|
||||
sensor_data = {
|
||||
"temperature": 25.5,
|
||||
@@ -635,7 +635,7 @@ sensor_data = {
|
||||
"pressure": 1013.25
|
||||
}
|
||||
|
||||
env, msg_json = smartsend(
|
||||
env, msg_json = smartpack(
|
||||
"/sensor/device/v1/readings",
|
||||
[("data", sensor_data, "dictionary")],
|
||||
broker_url=DEFAULT_BROKER_URL,
|
||||
@@ -667,7 +667,7 @@ payload_b64 = base64.b64encode(json_bytes).decode('ascii')
|
||||
```python
|
||||
# Python backend
|
||||
transport_msg = await transport_consumer.next()
|
||||
env = await smartreceive(str(transport_msg.payload))
|
||||
env = await smartunpack(str(transport_msg.payload))
|
||||
|
||||
# env["payloads"][0] is now:
|
||||
# ("data", {"temperature": 25.5, "humidity": 60.0, ...}, "dictionary")
|
||||
@@ -692,7 +692,7 @@ Multiple platforms (JavaScript, Python, Julia) communicate in a chat application
|
||||
|
||||
```javascript
|
||||
// JavaScript (Frontend)
|
||||
const [env, msgJson] = await msghandler.smartsend(
|
||||
const [env, msgJson] = await msghandler.smartpack(
|
||||
"/chat/user/v1/message",
|
||||
[
|
||||
["text", "Check this out!", "text"],
|
||||
@@ -716,7 +716,7 @@ const [env, msgJson] = await msghandler.smartsend(
|
||||
```python
|
||||
# Python (Backend)
|
||||
transport_msg = await transport_consumer.next()
|
||||
env = await smartreceive(str(transport_msg.payload))
|
||||
env = await smartunpack(str(transport_msg.payload))
|
||||
|
||||
# env["payloads"] is now:
|
||||
# [
|
||||
@@ -735,7 +735,7 @@ env = await smartreceive(str(transport_msg.payload))
|
||||
```julia
|
||||
# Julia (Backend)
|
||||
transport_msg = transport_subscription.next()
|
||||
env = smartreceive(String(transport_msg.payload))
|
||||
env = smartunpack(String(transport_msg.payload))
|
||||
|
||||
# env["payloads"] is now:
|
||||
# [
|
||||
@@ -755,7 +755,7 @@ Each platform can reply using the same API:
|
||||
|
||||
```python
|
||||
# Python reply
|
||||
await smartsend(
|
||||
await smartpack(
|
||||
"/chat/user/v1/reply",
|
||||
[("response", "Nice!", "text")],
|
||||
reply_to="/chat/user/v1/message"
|
||||
@@ -764,7 +764,7 @@ await smartsend(
|
||||
|
||||
```julia
|
||||
# Julia reply
|
||||
smartsend(
|
||||
smartpack(
|
||||
"/chat/user/v1/reply",
|
||||
[("response", "Nice!", "text")],
|
||||
reply_to="/chat/user/v1/message"
|
||||
@@ -773,7 +773,7 @@ smartsend(
|
||||
|
||||
```javascript
|
||||
// JavaScript reply
|
||||
await msghandler.smartsend(
|
||||
await msghandler.smartpack(
|
||||
"/chat/user/v1/reply",
|
||||
[["response", "Nice!", "text"]],
|
||||
{ reply_to: "/chat/user/v1/message" }
|
||||
@@ -827,14 +827,14 @@ Every message includes a `correlation_id`:
|
||||
correlation_id = string(uuid4())
|
||||
|
||||
# Use throughout the flow
|
||||
log_trace(correlation_id, "Starting smartsend")
|
||||
log_trace(correlation_id, "Starting smartpack")
|
||||
log_trace(correlation_id, "Serialized payload size: 100 bytes")
|
||||
log_trace(correlation_id, "Published to transport")
|
||||
```
|
||||
|
||||
**Log Format**:
|
||||
```
|
||||
[2026-03-13T16:30:00.000Z] [Correlation: abc123...] Starting smartsend
|
||||
[2026-03-13T16:30:00.000Z] [Correlation: abc123...] Starting smartpack
|
||||
[2026-03-13T16:30:00.001Z] [Correlation: abc123...] Serialized payload size: 100 bytes
|
||||
[2026-03-13T16:30:00.002Z] [Correlation: abc123...] Published to transport
|
||||
```
|
||||
@@ -928,8 +928,8 @@ log_trace(correlation_id, "Published to transport")
|
||||
| - | - | Removed all NATS-specific references from walkthrough | All sections |
|
||||
| - | - | Updated code examples to use transport-agnostic patterns | All sections |
|
||||
| - | - | Updated diagrams to remove NATS-specific labels | All sections |
|
||||
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartreceive` deserialization changes | All sections |
|
||||
| - | - | `smartreceive` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
|
||||
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartunpack` deserialization changes | All sections |
|
||||
| - | - | `smartunpack` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
|
||||
| - | - | Added `plik_upload_file` convenience function documentation | specification.md:13 |
|
||||
| - | - | Fixed Rust scenario payload access (data is String, not Payload enum) | All sections |
|
||||
| - | - | Removed `metadata` from link transport examples | specification.md:3 |
|
||||
@@ -937,7 +937,7 @@ log_trace(correlation_id, "Published to transport")
|
||||
| - | - | Added Rust user scenario (User Scenario 4) | specification.md:11 (Rust API) |
|
||||
| - | - | Updated scenario numbering (MicroPython → Scenario 5, Cross-Platform → Scenario 6) | All sections |
|
||||
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) | All sections |
|
||||
| - | - | Updated smartreceive calls to use transport payload pattern | All sections |
|
||||
| - | - | Updated smartunpack calls to use transport payload pattern | All sections |
|
||||
| - | - | Removed NATSClient.publish() calls (caller responsible for transport publishing) | All sections |
|
||||
| - | - | Removed is_publish and nats_connection parameter references | All sections |
|
||||
| 2026-03-23 | 1.0.0 | Updated to ASG Framework walkthrough guidelines | All sections |
|
||||
|
||||
12
etc.txt
12
etc.txt
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env julia
|
||||
# Test script for mixed-content message testing
|
||||
# Tests receiving a mix of text, json, table, image, audio, video, and binary data
|
||||
# from Julia serviceA to Julia serviceB using msghandler.jl smartreceive
|
||||
# from Julia serviceA to Julia serviceB using msghandler.jl smartunpack
|
||||
#
|
||||
# This test demonstrates that any combination and any number of mixed content
|
||||
# can be sent and received correctly.
|
||||
@@ -38,9 +38,9 @@ function test_mix_receive()
|
||||
log_trace("Received message on $(msg.subject)")
|
||||
incoming_msg = msg
|
||||
|
||||
# # Use msghandler.smartreceive to handle the data
|
||||
# # API: smartreceive(msg, download_handler; max_retries, base_delay, max_delay)
|
||||
# result = msghandler.smartreceive(
|
||||
# # Use msghandler.smartunpack to handle the data
|
||||
# # API: smartunpack(msg, download_handler; max_retries, base_delay, max_delay)
|
||||
# result = msghandler.smartunpack(
|
||||
# msg;
|
||||
# max_retries = 5,
|
||||
# base_delay = 100,
|
||||
@@ -229,7 +229,7 @@ println("Note: This receiver will wait for messages from the sender.")
|
||||
println("Run test_julia_to_julia_mix_sender.jl first to send test data.")
|
||||
|
||||
# Run receiver
|
||||
println("\ntesting smartreceive for mixed content")
|
||||
println("\ntesting smartunpack for mixed content")
|
||||
incoming_msg = test_mix_receive()
|
||||
|
||||
println("\nTest completed.")
|
||||
@@ -250,7 +250,7 @@ println("\nTest completed.")
|
||||
Check architecture.md. For sending table I want to add JSON in addition to Apache Arrow.
|
||||
Currently I use "table" datatype when sending table data using Arrow. Now table that I want to send using JSON
|
||||
I will use "jsontable" as datatype while sending table using Arrow I will use "arrowtable" as datatype.
|
||||
This will select how smartsend and smartreceive serialize/deserialize the table.
|
||||
This will select how smartpack and smartunpack serialize/deserialize the table.
|
||||
|
||||
Can you help me do this? Save the updated architecture.md into updated_architecture.md file. I will deal with source code later.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
use msghandler::{smartreceive, SmartreceiveOptions};
|
||||
use msghandler::{smartunpack, smartunpackOptions};
|
||||
|
||||
fn main() {
|
||||
// Simulated message JSON (received via any transport)
|
||||
@@ -40,9 +40,9 @@ fn main() {
|
||||
]
|
||||
}"#;
|
||||
|
||||
let options = SmartreceiveOptions::default();
|
||||
let options = smartunpackOptions::default();
|
||||
|
||||
match smartreceive(msg_json_str, &options) {
|
||||
match smartunpack(msg_json_str, &options) {
|
||||
Ok(envelope) => {
|
||||
println!("=== Envelope Received ===");
|
||||
println!("Correlation ID: {}", envelope.correlation_id);
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
use msghandler::{smartsend, Payload, SmartsendOptions};
|
||||
use msghandler::{smartpack, Payload, smartpackOptions};
|
||||
|
||||
fn main() {
|
||||
// Create mixed payload data
|
||||
@@ -24,7 +24,7 @@ fn main() {
|
||||
),
|
||||
];
|
||||
|
||||
let options = SmartsendOptions {
|
||||
let options = smartpackOptions {
|
||||
broker_url: "localhost:4222".to_string(),
|
||||
fileserver_url: "http://localhost:8080".to_string(),
|
||||
msg_purpose: "chat".to_string(),
|
||||
@@ -32,7 +32,7 @@ fn main() {
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
match smartsend("/agent/wine/api/v1/prompt", &payloads, &options) {
|
||||
match smartpack("/agent/wine/api/v1/prompt", &payloads, &options) {
|
||||
Ok((envelope, json_str)) => {
|
||||
println!("=== Envelope Created ===");
|
||||
println!("Correlation ID: {}", envelope.correlation_id);
|
||||
|
||||
@@ -378,13 +378,13 @@ function buildPayload(dataname, payloadType, payloadBytes, transport, data) {
|
||||
*
|
||||
* @example
|
||||
* // Send a single payload
|
||||
* const [env, envJsonStr] = await msghandlerCSR.smartsend(
|
||||
* const [env, envJsonStr] = await msghandlerCSR.smartpack(
|
||||
* "/test",
|
||||
* [["dataname1", data1, "dictionary"]]
|
||||
* );
|
||||
*
|
||||
* // Send multiple payloads (use jsontable instead of arrowtable for browser)
|
||||
* const [env, envJsonStr] = await msghandlerCSR.smartsend(
|
||||
* const [env, envJsonStr] = await msghandlerCSR.smartpack(
|
||||
* "/test",
|
||||
* [
|
||||
* ["dataname1", data1, "dictionary"],
|
||||
@@ -395,7 +395,7 @@ function buildPayload(dataname, payloadType, payloadBytes, transport, data) {
|
||||
* // Publish via your transport (NATS, MQTT, HTTP, etc.)
|
||||
* // await myNatsClient.publish("/test", envJsonStr);
|
||||
*/
|
||||
async function smartsend(subject, data, options = {}) {
|
||||
async function smartpack(subject, data, options = {}) {
|
||||
const {
|
||||
broker_url = DEFAULT_BROKER_URL,
|
||||
fileserver_url = DEFAULT_FILESERVER_URL,
|
||||
@@ -412,20 +412,20 @@ async function smartsend(subject, data, options = {}) {
|
||||
sender_id = uuidv4()
|
||||
} = options;
|
||||
|
||||
logTrace(correlation_id, `Starting smartsend for subject: ${subject}`);
|
||||
logTrace(correlation_id, `smartsend: data array length=${data.length}`);
|
||||
logTrace(correlation_id, `Starting smartpack for subject: ${subject}`);
|
||||
logTrace(correlation_id, `smartpack: data array length=${data.length}`);
|
||||
|
||||
// Debug: Log input data structure
|
||||
for (let i = 0; i < data.length; i++) {
|
||||
const [dataname, payloadData, payloadType] = data[i];
|
||||
logTrace(correlation_id, `smartsend: payload[${i}] dataname=${dataname}, type=${payloadType}, data type=${typeof payloadData}, constructor=${payloadData?.constructor?.name}`);
|
||||
logTrace(correlation_id, `smartpack: payload[${i}] dataname=${dataname}, type=${payloadType}, data type=${typeof payloadData}, constructor=${payloadData?.constructor?.name}`);
|
||||
}
|
||||
|
||||
// Process payloads
|
||||
const payloads = [];
|
||||
for (const [dataname, payloadData, payloadType] of data) {
|
||||
logTrace(correlation_id, `smartsend: Processing payload '${dataname}' type=${payloadType}`);
|
||||
logTrace(correlation_id, `smartsend: payloadData type=${typeof payloadData}, constructor=${payloadData?.constructor?.name}`);
|
||||
logTrace(correlation_id, `smartpack: Processing payload '${dataname}' type=${payloadType}`);
|
||||
logTrace(correlation_id, `smartpack: payloadData type=${typeof payloadData}, constructor=${payloadData?.constructor?.name}`);
|
||||
|
||||
const payloadBytes = await serializeData(payloadData, payloadType);
|
||||
const payloadSize = payloadBytes.byteLength;
|
||||
@@ -502,7 +502,7 @@ async function smartsend(subject, data, options = {}) {
|
||||
*
|
||||
* @example
|
||||
* // Receive from JSON string directly
|
||||
* const env = await msghandlerCSR.smartreceive(jsonString, {
|
||||
* const env = await msghandlerCSR.smartunpack(jsonString, {
|
||||
* fileserver_download_handler: msghandlerCSR.fetchWithBackoff,
|
||||
* max_retries: 5,
|
||||
* base_delay: 100,
|
||||
@@ -510,7 +510,7 @@ async function smartsend(subject, data, options = {}) {
|
||||
* });
|
||||
*
|
||||
* // Receive from transport message object (e.g., NATS, MQTT)
|
||||
* const env = await msghandlerCSR.smartreceive(natsMsg, {
|
||||
* const env = await msghandlerCSR.smartunpack(natsMsg, {
|
||||
* fileserver_download_handler: msghandlerCSR.fetchWithBackoff
|
||||
* });
|
||||
* // env.payloads is an Array of [dataname, data, type] arrays
|
||||
@@ -518,7 +518,7 @@ async function smartsend(subject, data, options = {}) {
|
||||
* console.log(`${dataname}: ${data} (type: ${type})`);
|
||||
* }
|
||||
*/
|
||||
async function smartreceive(msg, options = {}) {
|
||||
async function smartunpack(msg, options = {}) {
|
||||
const {
|
||||
fileserver_download_handler = fetchWithBackoff,
|
||||
max_retries = 5,
|
||||
@@ -542,28 +542,28 @@ async function smartreceive(msg, options = {}) {
|
||||
throw new Error('Invalid message format: expected JSON string or message object');
|
||||
}
|
||||
|
||||
logTrace('smartreceive', `smartreceive: raw payload length=${payload.length}`);
|
||||
logTrace('smartunpack', `smartunpack: raw payload length=${payload.length}`);
|
||||
|
||||
// Debug: Show first 200 chars of payload
|
||||
const payloadPreview = payload.substring(0, 200);
|
||||
logTrace('smartreceive', `smartreceive: payload preview: ${payloadPreview}`);
|
||||
logTrace('smartunpack', `smartunpack: payload preview: ${payloadPreview}`);
|
||||
|
||||
let envJsonObj;
|
||||
try {
|
||||
envJsonObj = JSON.parse(payload);
|
||||
} catch (e) {
|
||||
logTrace('smartreceive', `smartreceive: JSON parse failed: ${e.message}`);
|
||||
logTrace('smartunpack', `smartunpack: JSON parse failed: ${e.message}`);
|
||||
throw e;
|
||||
}
|
||||
|
||||
logTrace(envJsonObj.correlation_id, 'Processing received message');
|
||||
logTrace(envJsonObj.correlation_id, `smartreceive: envelope has ${envJsonObj.payloads.length} payloads`);
|
||||
logTrace(envJsonObj.correlation_id, `smartunpack: envelope has ${envJsonObj.payloads.length} payloads`);
|
||||
|
||||
// Process all payloads in the envelope
|
||||
const payloadsList = [];
|
||||
const numPayloads = envJsonObj.payloads.length;
|
||||
|
||||
logTrace(envJsonObj.correlation_id, `smartreceive: Processing ${numPayloads} payloads`);
|
||||
logTrace(envJsonObj.correlation_id, `smartunpack: Processing ${numPayloads} payloads`);
|
||||
|
||||
for (let i = 0; i < numPayloads; i++) {
|
||||
const payloadObj = envJsonObj.payloads[i];
|
||||
@@ -571,7 +571,7 @@ async function smartreceive(msg, options = {}) {
|
||||
const dataname = payloadObj.dataname;
|
||||
const payloadType = payloadObj.payload_type;
|
||||
|
||||
logTrace(envJsonObj.correlation_id, `smartreceive: Processing payload ${i + 1}/${numPayloads}: dataname=${dataname}, type=${payloadType}, transport=${transport}`);
|
||||
logTrace(envJsonObj.correlation_id, `smartunpack: Processing payload ${i + 1}/${numPayloads}: dataname=${dataname}, type=${payloadType}, transport=${transport}`);
|
||||
|
||||
if (transport === 'direct') {
|
||||
logTrace(envJsonObj.correlation_id, `Direct transport - decoding payload '${dataname}'`);
|
||||
@@ -614,7 +614,7 @@ async function smartreceive(msg, options = {}) {
|
||||
}
|
||||
}
|
||||
|
||||
logTrace(envJsonObj.correlation_id, `smartreceive: Successfully processed all ${payloadsList.length} payloads`);
|
||||
logTrace(envJsonObj.correlation_id, `smartunpack: Successfully processed all ${payloadsList.length} payloads`);
|
||||
envJsonObj.payloads = payloadsList;
|
||||
return envJsonObj;
|
||||
}
|
||||
@@ -625,12 +625,12 @@ const msghandlerCSR = {
|
||||
/**
|
||||
* Send data with automatic transport selection
|
||||
*/
|
||||
smartsend,
|
||||
smartpack,
|
||||
|
||||
/**
|
||||
* Receive and process messages
|
||||
*/
|
||||
smartreceive,
|
||||
smartunpack,
|
||||
|
||||
/**
|
||||
* Upload data to plik server in one-shot mode
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# Bi-Directional Data Bridge - Julia Module
|
||||
# Implements smartsend and smartreceive for message transport
|
||||
# Implements smartpack and smartunpack for message transport
|
||||
# This module provides functionality for sending and receiving data across network boundaries
|
||||
# with support for both direct payload transport and
|
||||
# URL-based transport for larger payloads.
|
||||
@@ -24,10 +24,10 @@
|
||||
#
|
||||
# API Standard:
|
||||
# ```jldoctest
|
||||
# # Input format for smartsend (always a list of tuples with type info)
|
||||
# # Input format for smartpack (always a list of tuples with type info)
|
||||
# [(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||
#
|
||||
# # Output format for smartreceive (always returns a list of tuples)
|
||||
# # Output format for smartunpack (always returns a list of tuples)
|
||||
# [(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||
# ```
|
||||
#
|
||||
@@ -337,7 +337,7 @@ function log_trace(correlation_id::String, message::String)
|
||||
end
|
||||
|
||||
|
||||
""" smartsend - Send data with automatic transport selection, depending on payload size
|
||||
""" smartpack - Send data with automatic transport selection, depending on payload size
|
||||
|
||||
This function intelligently routes data delivery based on payload size relative to a threshold.
|
||||
If the serialized payload is smaller than `size_threshold`, it encodes the data as Base64 and constructs a "direct" msg_payload_v1.
|
||||
@@ -392,23 +392,23 @@ using UUIDs
|
||||
|
||||
# Send a single payload (still wrapped in a list)
|
||||
data = Dict("key" => "value")
|
||||
env, msg_json = smartsend("my.subject", [("dataname1", data, "dictionary")])
|
||||
env, msg_json = smartpack("my.subject", [("dataname1", data, "dictionary")])
|
||||
|
||||
# Send multiple payloads in one message with different types
|
||||
data1 = Dict("key1" => "value1")
|
||||
data2 = rand(10_000) # Small array
|
||||
env, msg_json = smartsend("my.subject", [("dataname1", data1, "dictionary"), ("dataname2", data2, "arrowtable")])
|
||||
env, msg_json = smartpack("my.subject", [("dataname1", data1, "dictionary"), ("dataname2", data2, "arrowtable")])
|
||||
|
||||
# Send a large array using fileserver upload
|
||||
data = rand(10_000_000) # ~80 MB
|
||||
env, msg_json = smartsend("large.data", [("large_arrow_table", data, "arrowtable")])
|
||||
env, msg_json = smartpack("large.data", [("large_arrow_table", data, "arrowtable")])
|
||||
|
||||
# Send jsontable (JSON format)
|
||||
rows = [Dict("id" => 1, "name" => "Alice"), Dict("id" => 2, "name" => "Bob")]
|
||||
env, msg_json = smartsend("json.data", [("users", rows, "jsontable")])
|
||||
env, msg_json = smartpack("json.data", [("users", rows, "jsontable")])
|
||||
|
||||
# Mixed content (e.g., chat with text and image)
|
||||
env, msg_json = smartsend("chat.subject", [
|
||||
env, msg_json = smartpack("chat.subject", [
|
||||
("message_text", "Hello!", "text"),
|
||||
("user_image", image_data, "image"),
|
||||
("audio_clip", audio_data, "audio")
|
||||
@@ -419,8 +419,8 @@ env, msg_json = smartsend("chat.subject", [
|
||||
# my_transport.publish(conn, subject, env_json_str)
|
||||
```
|
||||
"""
|
||||
function smartsend(
|
||||
subject::String, # smartreceive's subject
|
||||
function smartpack(
|
||||
subject::String, # smartunpack's subject
|
||||
data::AbstractArray{Tuple{String, T1, String}, 1}; # List of (dataname, data, type) tuples. Use Tuple{String, Any, String}[] for empty payloads
|
||||
broker_url::String = DEFAULT_BROKER_URL, # Broker URL
|
||||
fileserver_url = DEFAULT_FILESERVER_URL,
|
||||
@@ -446,7 +446,7 @@ function smartsend(
|
||||
)::Tuple{msg_envelope_v1, String} where {T1<:Any}
|
||||
|
||||
# Log start of send operation
|
||||
log_trace(correlation_id, "Starting smartsend for subject: $subject")
|
||||
log_trace(correlation_id, "Starting smartpack for subject: $subject")
|
||||
|
||||
# Process each payload in the list
|
||||
payloads = msg_payload_v1[]
|
||||
@@ -772,7 +772,7 @@ end
|
||||
# end
|
||||
|
||||
|
||||
""" smartreceive - Receive and process messages
|
||||
""" smartunpack - Receive and process messages
|
||||
This function processes incoming messages, handling both direct transport
|
||||
(base64 decoded payloads) and link transport (URL-based payloads).
|
||||
It deserializes the data based on the transport type and returns the result.
|
||||
@@ -801,11 +801,11 @@ A HTTP file server is required along with its download function.
|
||||
```jldoctest
|
||||
# Receive and process message
|
||||
msg_json_str = String(msg.payload)
|
||||
env = smartreceive(msg_json_str; fileserver_download_handler=_fetch_with_backoff, max_retries=5, base_delay=100, max_delay=5000)
|
||||
env = smartunpack(msg_json_str; fileserver_download_handler=_fetch_with_backoff, max_retries=5, base_delay=100, max_delay=5000)
|
||||
# env["payloads"] = [("dataname1", data1, "type1"), ("dataname2", data2, "type2"), ...]
|
||||
```
|
||||
"""
|
||||
function smartreceive(
|
||||
function smartunpack(
|
||||
msg_json_str::String; # get it from String(nats_msg.payload)
|
||||
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||
max_retries::Int = 5,
|
||||
|
||||
@@ -431,13 +431,13 @@ function buildPayload(dataname, payloadType, payloadBytes, transport, data) {
|
||||
*
|
||||
* @example
|
||||
* // Send a single payload
|
||||
* const [env, envJsonStr] = await smartsend(
|
||||
* const [env, envJsonStr] = await smartpack(
|
||||
* "/test",
|
||||
* [["dataname1", data1, "dictionary"]]
|
||||
* );
|
||||
*
|
||||
* // Send multiple payloads
|
||||
* const [env, envJsonStr] = await smartsend(
|
||||
* const [env, envJsonStr] = await smartpack(
|
||||
* "/test",
|
||||
* [
|
||||
* ["dataname1", data1, "dictionary"],
|
||||
@@ -448,7 +448,7 @@ function buildPayload(dataname, payloadType, payloadBytes, transport, data) {
|
||||
* // Publish via your transport (NATS, MQTT, HTTP, etc.)
|
||||
* // await myNatsClient.publish("/test", envJsonStr);
|
||||
*/
|
||||
async function smartsend(subject, data, options = {}) {
|
||||
async function smartpack(subject, data, options = {}) {
|
||||
const {
|
||||
broker_url = DEFAULT_BROKER_URL,
|
||||
fileserver_url = DEFAULT_FILESERVER_URL,
|
||||
@@ -465,20 +465,20 @@ async function smartsend(subject, data, options = {}) {
|
||||
sender_id = uuidv4()
|
||||
} = options;
|
||||
|
||||
logTrace(correlation_id, `Starting smartsend for subject: ${subject}`);
|
||||
logTrace(correlation_id, `smartsend: data array length=${data.length}`);
|
||||
logTrace(correlation_id, `Starting smartpack for subject: ${subject}`);
|
||||
logTrace(correlation_id, `smartpack: data array length=${data.length}`);
|
||||
|
||||
// Debug: Log input data structure
|
||||
for (let i = 0; i < data.length; i++) {
|
||||
const [dataname, payloadData, payloadType] = data[i];
|
||||
logTrace(correlation_id, `smartsend: payload[${i}] dataname=${dataname}, type=${payloadType}, data type=${typeof payloadData}, constructor=${payloadData?.constructor?.name}`);
|
||||
logTrace(correlation_id, `smartpack: payload[${i}] dataname=${dataname}, type=${payloadType}, data type=${typeof payloadData}, constructor=${payloadData?.constructor?.name}`);
|
||||
}
|
||||
|
||||
// Process payloads
|
||||
const payloads = [];
|
||||
for (const [dataname, payloadData, payloadType] of data) {
|
||||
logTrace(correlation_id, `smartsend: Processing payload '${dataname}' type=${payloadType}`);
|
||||
logTrace(correlation_id, `smartsend: payloadData type=${typeof payloadData}, constructor=${payloadData?.constructor?.name}`);
|
||||
logTrace(correlation_id, `smartpack: Processing payload '${dataname}' type=${payloadType}`);
|
||||
logTrace(correlation_id, `smartpack: payloadData type=${typeof payloadData}, constructor=${payloadData?.constructor?.name}`);
|
||||
|
||||
const payloadBytes = await serializeData(payloadData, payloadType);
|
||||
const payloadSize = payloadBytes.byteLength;
|
||||
@@ -552,7 +552,7 @@ async function smartsend(subject, data, options = {}) {
|
||||
*
|
||||
* @example
|
||||
* // Receive from JSON string directly
|
||||
* const env = await smartreceive(jsonString, {
|
||||
* const env = await smartunpack(jsonString, {
|
||||
* fileserver_download_handler: fetchWithBackoff,
|
||||
* max_retries: 5,
|
||||
* base_delay: 100,
|
||||
@@ -560,7 +560,7 @@ async function smartsend(subject, data, options = {}) {
|
||||
* });
|
||||
*
|
||||
* // Receive from transport message object (e.g., NATS, MQTT)
|
||||
* const env = await smartreceive(natsMsg, {
|
||||
* const env = await smartunpack(natsMsg, {
|
||||
* fileserver_download_handler: fetchWithBackoff
|
||||
* });
|
||||
* // env.payloads is an Array of [dataname, data, type] arrays
|
||||
@@ -568,7 +568,7 @@ async function smartsend(subject, data, options = {}) {
|
||||
* console.log(`${dataname}: ${data} (type: ${type})`);
|
||||
* }
|
||||
*/
|
||||
async function smartreceive(msg, options = {}) {
|
||||
async function smartunpack(msg, options = {}) {
|
||||
const {
|
||||
fileserver_download_handler = fetchWithBackoff,
|
||||
max_retries = 5,
|
||||
@@ -592,28 +592,28 @@ async function smartreceive(msg, options = {}) {
|
||||
throw new Error('Invalid message format: expected JSON string or message object');
|
||||
}
|
||||
|
||||
logTrace('smartreceive', `smartreceive: raw payload length=${payload.length}`);
|
||||
logTrace('smartunpack', `smartunpack: raw payload length=${payload.length}`);
|
||||
|
||||
// Debug: Show first 200 chars of payload
|
||||
const payloadPreview = payload.substring(0, 200);
|
||||
logTrace('smartreceive', `smartreceive: payload preview: ${payloadPreview}`);
|
||||
logTrace('smartunpack', `smartunpack: payload preview: ${payloadPreview}`);
|
||||
|
||||
let envJsonObj;
|
||||
try {
|
||||
envJsonObj = JSON.parse(payload);
|
||||
} catch (e) {
|
||||
logTrace('smartreceive', `smartreceive: JSON parse failed: ${e.message}`);
|
||||
logTrace('smartunpack', `smartunpack: JSON parse failed: ${e.message}`);
|
||||
throw e;
|
||||
}
|
||||
|
||||
logTrace(envJsonObj.correlation_id, 'Processing received message');
|
||||
logTrace(envJsonObj.correlation_id, `smartreceive: envelope has ${envJsonObj.payloads.length} payloads`);
|
||||
logTrace(envJsonObj.correlation_id, `smartunpack: envelope has ${envJsonObj.payloads.length} payloads`);
|
||||
|
||||
// Process all payloads in the envelope
|
||||
const payloadsList = [];
|
||||
const numPayloads = envJsonObj.payloads.length;
|
||||
|
||||
logTrace(envJsonObj.correlation_id, `smartreceive: Processing ${numPayloads} payloads`);
|
||||
logTrace(envJsonObj.correlation_id, `smartunpack: Processing ${numPayloads} payloads`);
|
||||
|
||||
for (let i = 0; i < numPayloads; i++) {
|
||||
const payloadObj = envJsonObj.payloads[i];
|
||||
@@ -621,7 +621,7 @@ async function smartreceive(msg, options = {}) {
|
||||
const dataname = payloadObj.dataname;
|
||||
const payloadType = payloadObj.payload_type;
|
||||
|
||||
logTrace(envJsonObj.correlation_id, `smartreceive: Processing payload ${i + 1}/${numPayloads}: dataname=${dataname}, type=${payloadType}, transport=${transport}`);
|
||||
logTrace(envJsonObj.correlation_id, `smartunpack: Processing payload ${i + 1}/${numPayloads}: dataname=${dataname}, type=${payloadType}, transport=${transport}`);
|
||||
|
||||
if (transport === 'direct') {
|
||||
logTrace(envJsonObj.correlation_id, `Direct transport - decoding payload '${dataname}'`);
|
||||
@@ -664,7 +664,7 @@ async function smartreceive(msg, options = {}) {
|
||||
}
|
||||
}
|
||||
|
||||
logTrace(envJsonObj.correlation_id, `smartreceive: Successfully processed all ${payloadsList.length} payloads`);
|
||||
logTrace(envJsonObj.correlation_id, `smartunpack: Successfully processed all ${payloadsList.length} payloads`);
|
||||
envJsonObj.payloads = payloadsList;
|
||||
return envJsonObj;
|
||||
}
|
||||
@@ -675,12 +675,12 @@ const msghandler = {
|
||||
/**
|
||||
* Send data with automatic transport selection
|
||||
*/
|
||||
smartsend,
|
||||
smartpack,
|
||||
|
||||
/**
|
||||
* Receive and process messages
|
||||
*/
|
||||
smartreceive,
|
||||
smartunpack,
|
||||
|
||||
/**
|
||||
* Upload data to plik server in one-shot mode
|
||||
|
||||
@@ -372,7 +372,7 @@ def _build_payload(
|
||||
}
|
||||
|
||||
|
||||
async def smartsend(
|
||||
async def smartpack(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
broker_url: str = DEFAULT_BROKER_URL,
|
||||
@@ -429,7 +429,7 @@ async def smartsend(
|
||||
Example:
|
||||
>>> # Send a single payload (still wrapped in a list)
|
||||
>>> data = {"key": "value"}
|
||||
>>> env, env_json_str = await smartsend(
|
||||
>>> env, env_json_str = await smartpack(
|
||||
... "my.subject",
|
||||
... [("dataname1", data, "dictionary")]
|
||||
... )
|
||||
@@ -444,7 +444,7 @@ async def smartsend(
|
||||
if sender_id is None:
|
||||
sender_id = str(uuid.uuid4())
|
||||
|
||||
log_trace(correlation_id, f"Starting smartsend for subject: {subject}")
|
||||
log_trace(correlation_id, f"Starting smartpack for subject: {subject}")
|
||||
|
||||
# Process payloads
|
||||
payloads = []
|
||||
@@ -494,7 +494,7 @@ async def smartsend(
|
||||
return env, env_json_str
|
||||
|
||||
|
||||
async def smartreceive(
|
||||
async def smartunpack(
|
||||
msg: Any,
|
||||
fileserver_download_handler: Callable = fetch_with_backoff,
|
||||
max_retries: int = 5,
|
||||
@@ -521,10 +521,10 @@ async def smartreceive(
|
||||
|
||||
Example:
|
||||
>>> # Receive from JSON string directly
|
||||
>>> env = await smartreceive(json_string)
|
||||
>>> env = await smartunpack(json_string)
|
||||
>>>
|
||||
>>> # Receive from transport message object (e.g., NATS, MQTT)
|
||||
>>> env = await smartreceive(nats_msg, fileserver_download_handler=fetch_with_backoff)
|
||||
>>> env = await smartunpack(nats_msg, fileserver_download_handler=fetch_with_backoff)
|
||||
>>> # env is a Dict with "payloads" key containing List[Tuple[str, Any, str]]
|
||||
>>> for dataname, data, type_ in env["payloads"]:
|
||||
>>> print(f"{dataname}: {data} (type: {type_})")
|
||||
@@ -623,7 +623,7 @@ class msghandler:
|
||||
self.broker_url = broker_url or self.DEFAULT_BROKER_URL
|
||||
self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL
|
||||
|
||||
async def smartsend(
|
||||
async def smartpack(
|
||||
self,
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
@@ -635,16 +635,16 @@ class msghandler:
|
||||
Args:
|
||||
subject: Subject/topic to send to
|
||||
data: List of (dataname, data, type) tuples
|
||||
**kwargs: Additional options passed to smartsend
|
||||
**kwargs: Additional options passed to smartpack
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str)
|
||||
"""
|
||||
kwargs['broker_url'] = kwargs.get('broker_url', self.broker_url)
|
||||
kwargs['fileserver_url'] = kwargs.get('fileserver_url', self.fileserver_url)
|
||||
return await smartsend(subject, data, **kwargs)
|
||||
return await smartpack(subject, data, **kwargs)
|
||||
|
||||
async def smartreceive(
|
||||
async def smartunpack(
|
||||
self,
|
||||
msg: Any,
|
||||
**kwargs
|
||||
@@ -654,12 +654,12 @@ class msghandler:
|
||||
|
||||
Args:
|
||||
msg: Message to process
|
||||
**kwargs: Additional options passed to smartreceive
|
||||
**kwargs: Additional options passed to smartunpack
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads
|
||||
"""
|
||||
return await smartreceive(msg, **kwargs)
|
||||
return await smartunpack(msg, **kwargs)
|
||||
|
||||
|
||||
# Convenience functions for module-level usage
|
||||
@@ -679,7 +679,7 @@ def send(
|
||||
Returns:
|
||||
Tuple of (env, env_json_str)
|
||||
"""
|
||||
return asyncio.run(smartsend(subject, data, **kwargs))
|
||||
return asyncio.run(smartpack(subject, data, **kwargs))
|
||||
|
||||
|
||||
def receive(
|
||||
@@ -696,12 +696,12 @@ def receive(
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads
|
||||
"""
|
||||
return asyncio.run(smartreceive(msg, **kwargs))
|
||||
return asyncio.run(smartunpack(msg, **kwargs))
|
||||
|
||||
|
||||
__all__ = [
|
||||
'smartsend',
|
||||
'smartreceive',
|
||||
'smartpack',
|
||||
'smartunpack',
|
||||
'plik_oneshot_upload',
|
||||
'fetch_with_backoff',
|
||||
'msghandler',
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// msghandler Rust Module
|
||||
// Cross-platform bi-directional data bridge
|
||||
// Implements smartsend and smartreceive for message transport
|
||||
// Implements smartpack and smartunpack for message transport
|
||||
// with support for both direct payload transport and URL-based transport
|
||||
// for larger payloads using the Claim-Check pattern.
|
||||
//
|
||||
@@ -325,8 +325,8 @@ impl MsgEnvelopeV1 {
|
||||
// Options Structures
|
||||
// ============================================================================
|
||||
|
||||
/// Options for the `smartsend` function
|
||||
pub struct SmartsendOptions {
|
||||
/// Options for the `smartpack` function
|
||||
pub struct smartpackOptions {
|
||||
/// Broker URL
|
||||
pub broker_url: String,
|
||||
/// HTTP file server URL for large payloads
|
||||
@@ -355,9 +355,9 @@ pub struct SmartsendOptions {
|
||||
pub sender_id: String,
|
||||
}
|
||||
|
||||
impl Default for SmartsendOptions {
|
||||
impl Default for smartpackOptions {
|
||||
fn default() -> Self {
|
||||
SmartsendOptions {
|
||||
smartpackOptions {
|
||||
broker_url: DEFAULT_BROKER_URL.to_string(),
|
||||
fileserver_url: DEFAULT_FILESERVER_URL.to_string(),
|
||||
fileserver_upload_handler: None,
|
||||
@@ -375,8 +375,8 @@ impl Default for SmartsendOptions {
|
||||
}
|
||||
}
|
||||
|
||||
/// Options for the `smartreceive` function
|
||||
pub struct SmartreceiveOptions {
|
||||
/// Options for the `smartunpack` function
|
||||
pub struct smartunpackOptions {
|
||||
/// Custom file server download handler (optional, uses exponential backoff by default)
|
||||
pub fileserver_download_handler: Option<Arc<dyn FileDownloadHandler>>,
|
||||
/// Maximum retry attempts for fetching a URL
|
||||
@@ -387,9 +387,9 @@ pub struct SmartreceiveOptions {
|
||||
pub max_delay: u64,
|
||||
}
|
||||
|
||||
impl Default for SmartreceiveOptions {
|
||||
impl Default for smartunpackOptions {
|
||||
fn default() -> Self {
|
||||
SmartreceiveOptions {
|
||||
smartunpackOptions {
|
||||
fileserver_download_handler: None,
|
||||
max_retries: DEFAULT_MAX_RETRIES,
|
||||
base_delay: DEFAULT_BASE_DELAY,
|
||||
@@ -689,7 +689,7 @@ pub fn log_trace(correlation_id: &str, message: &str) {
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Public API: smartsend
|
||||
// Public API: smartpack
|
||||
// ============================================================================
|
||||
|
||||
/// Send data with automatic transport selection.
|
||||
@@ -715,23 +715,23 @@ pub fn log_trace(correlation_id: &str, message: &str) {
|
||||
///
|
||||
/// # Example
|
||||
/// ```no_run
|
||||
/// use msghandler::{smartsend, Payload, SmartsendOptions};
|
||||
/// use msghandler::{smartpack, Payload, smartpackOptions};
|
||||
///
|
||||
/// let (envelope, json_str) = smartsend(
|
||||
/// let (envelope, json_str) = smartpack(
|
||||
/// "/agent/wine/api/v1/prompt",
|
||||
/// &[
|
||||
/// ("msg".to_string(), Payload::Text("Hello!".to_string()), "text".to_string()),
|
||||
/// ("data".to_string(), Payload::Binary(vec![1, 2, 3]), "binary".to_string()),
|
||||
/// ],
|
||||
/// &SmartsendOptions::default(),
|
||||
/// &smartpackOptions::default(),
|
||||
/// ).unwrap();
|
||||
///
|
||||
/// // Caller publishes via their preferred transport
|
||||
/// ```
|
||||
pub fn smartsend(
|
||||
pub fn smartpack(
|
||||
subject: &str,
|
||||
data: &[(String, Payload, String)],
|
||||
options: &SmartsendOptions,
|
||||
options: &smartpackOptions,
|
||||
) -> Result<(MsgEnvelopeV1, String), MsgHandlerError> {
|
||||
let correlation_id = if options.correlation_id.is_empty() {
|
||||
Uuid::new_v4().to_string()
|
||||
@@ -752,7 +752,7 @@ pub fn smartsend(
|
||||
};
|
||||
|
||||
log_trace(&correlation_id, &format!(
|
||||
"Starting smartsend for subject: {}", subject
|
||||
"Starting smartpack for subject: {}", subject
|
||||
));
|
||||
|
||||
let mut payloads: Vec<MsgPayloadV1> = Vec::new();
|
||||
@@ -844,7 +844,7 @@ pub fn smartsend(
|
||||
// ============================================================================
|
||||
|
||||
/// Store deserialized Payload data back into a MsgPayloadV1's data field.
|
||||
/// After smartreceive(), payload.data contains the deserialized content as a string
|
||||
/// After smartunpack(), payload.data contains the deserialized content as a string
|
||||
/// (decoded text, JSON string, or base64 for binary types).
|
||||
fn store_deserialized_data(payload: &MsgPayloadV1, deserialized: &Payload) -> MsgPayloadV1 {
|
||||
let mut p = payload.clone();
|
||||
@@ -861,7 +861,7 @@ fn store_deserialized_data(payload: &MsgPayloadV1, deserialized: &Payload) -> Ms
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Public API: smartreceive
|
||||
// Public API: smartunpack
|
||||
// ============================================================================
|
||||
|
||||
/// Receive and process messages.
|
||||
@@ -880,7 +880,7 @@ fn store_deserialized_data(payload: &MsgPayloadV1, deserialized: &Payload) -> Ms
|
||||
///
|
||||
/// # Example
|
||||
/// ```no_run
|
||||
/// use msghandler::{smartreceive, SmartreceiveOptions};
|
||||
/// use msghandler::{smartunpack, smartunpackOptions};
|
||||
/// use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64};
|
||||
///
|
||||
/// let msg_json_str = r#"{"correlation_id":"abc123","msg_id":"msg-uuid",
|
||||
@@ -893,7 +893,7 @@ fn store_deserialized_data(payload: &MsgPayloadV1, deserialized: &Payload) -> Ms
|
||||
/// "data":"SGVsbG8=","metadata":{"payload_bytes":5}
|
||||
/// }]}"#;
|
||||
///
|
||||
/// let envelope = smartreceive(msg_json_str, &SmartreceiveOptions::default()).unwrap();
|
||||
/// let envelope = smartunpack(msg_json_str, &smartunpackOptions::default()).unwrap();
|
||||
///
|
||||
/// for payload in &envelope.payloads {
|
||||
/// if payload.transport == "direct" {
|
||||
@@ -904,9 +904,9 @@ fn store_deserialized_data(payload: &MsgPayloadV1, deserialized: &Payload) -> Ms
|
||||
/// }
|
||||
/// }
|
||||
/// ```
|
||||
pub fn smartreceive(
|
||||
pub fn smartunpack(
|
||||
msg_json_str: &str,
|
||||
options: &SmartreceiveOptions,
|
||||
options: &smartunpackOptions,
|
||||
) -> Result<MsgEnvelopeV1, MsgHandlerError> {
|
||||
// Parse the JSON envelope
|
||||
let mut env: MsgEnvelopeV1 = serde_json::from_str(msg_json_str)
|
||||
@@ -998,9 +998,9 @@ pub fn smartreceive(
|
||||
pub fn send_text(
|
||||
subject: &str,
|
||||
text: &str,
|
||||
options: &SmartsendOptions,
|
||||
options: &smartpackOptions,
|
||||
) -> Result<(MsgEnvelopeV1, String), MsgHandlerError> {
|
||||
smartsend(
|
||||
smartpack(
|
||||
subject,
|
||||
&[(
|
||||
"text".to_string(),
|
||||
@@ -1015,9 +1015,9 @@ pub fn send_text(
|
||||
pub fn send_dictionary(
|
||||
subject: &str,
|
||||
data: &JsonValue,
|
||||
options: &SmartsendOptions,
|
||||
options: &smartpackOptions,
|
||||
) -> Result<(MsgEnvelopeV1, String), MsgHandlerError> {
|
||||
smartsend(
|
||||
smartpack(
|
||||
subject,
|
||||
&[(
|
||||
"dictionary".to_string(),
|
||||
@@ -1032,9 +1032,9 @@ pub fn send_dictionary(
|
||||
pub fn send_binary(
|
||||
subject: &str,
|
||||
data: &[u8],
|
||||
options: &SmartsendOptions,
|
||||
options: &smartpackOptions,
|
||||
) -> Result<(MsgEnvelopeV1, String), MsgHandlerError> {
|
||||
smartsend(
|
||||
smartpack(
|
||||
subject,
|
||||
&[(
|
||||
"binary".to_string(),
|
||||
@@ -1094,10 +1094,10 @@ pub fn plik_upload_file(
|
||||
|
||||
// All public types are already exported via `pub` on their definitions.
|
||||
// Key types:
|
||||
// - `smartsend`, `smartreceive` - main API functions
|
||||
// - `smartpack`, `smartunpack` - main API functions
|
||||
// - `Payload` - type-safe payload enum
|
||||
// - `MsgEnvelopeV1`, `MsgPayloadV1` - wire format structs
|
||||
// - `SmartsendOptions`, `SmartreceiveOptions` - configuration
|
||||
// - `smartpackOptions`, `smartunpackOptions` - configuration
|
||||
// - `FileUploadHandler`, `FileDownloadHandler` - trait abstractions
|
||||
// - `PlikOneshotUploadHandler`, `BackoffDownloadHandler` - default implementations
|
||||
// - `MsgHandlerError` - error type
|
||||
@@ -1193,12 +1193,12 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn test_default_options() {
|
||||
let opts = SmartsendOptions::default();
|
||||
let opts = smartpackOptions::default();
|
||||
assert_eq!(opts.size_threshold, DEFAULT_SIZE_THRESHOLD);
|
||||
assert_eq!(opts.broker_url, DEFAULT_BROKER_URL);
|
||||
assert_eq!(opts.fileserver_url, DEFAULT_FILESERVER_URL);
|
||||
|
||||
let opts = SmartreceiveOptions::default();
|
||||
let opts = smartunpackOptions::default();
|
||||
assert_eq!(opts.max_retries, DEFAULT_MAX_RETRIES);
|
||||
assert_eq!(opts.base_delay, DEFAULT_BASE_DELAY);
|
||||
assert_eq!(opts.max_delay, DEFAULT_MAX_DELAY);
|
||||
|
||||
@@ -263,7 +263,7 @@ def _publish(subject, message, correlation_id):
|
||||
# Placeholder - actual implementation would publish via preferred transport
|
||||
|
||||
|
||||
def smartsend(subject, data, **kwargs):
|
||||
def smartpack(subject, data, **kwargs):
|
||||
"""
|
||||
Send data with automatic transport selection.
|
||||
|
||||
@@ -306,7 +306,7 @@ def smartsend(subject, data, **kwargs):
|
||||
|
||||
Example:
|
||||
>>> # Send text payload
|
||||
>>> env, env_json_str = smartsend(
|
||||
>>> env, env_json_str = smartpack(
|
||||
... "/chat",
|
||||
... [("message", "Hello!", "text")]
|
||||
... )
|
||||
@@ -330,7 +330,7 @@ def smartsend(subject, data, **kwargs):
|
||||
is_publish = kwargs.get('is_publish', True)
|
||||
fileserver_upload_handler = kwargs.get('fileserver_upload_handler', _sync_fileserver_upload)
|
||||
|
||||
log_trace(correlation_id, f"Starting smartsend for subject: {subject}")
|
||||
log_trace(correlation_id, f"Starting smartpack for subject: {subject}")
|
||||
|
||||
# Process payloads
|
||||
payloads = []
|
||||
@@ -390,7 +390,7 @@ def smartsend(subject, data, **kwargs):
|
||||
return env, env_json_str
|
||||
|
||||
|
||||
def smartreceive(msg, **kwargs):
|
||||
def smartunpack(msg, **kwargs):
|
||||
"""
|
||||
Receive and process messages.
|
||||
|
||||
@@ -414,10 +414,10 @@ def smartreceive(msg, **kwargs):
|
||||
|
||||
Example:
|
||||
>>> # Receive from JSON string
|
||||
>>> env = smartreceive(json_string)
|
||||
>>> env = smartunpack(json_string)
|
||||
>>>
|
||||
>>> # Receive from transport message object
|
||||
>>> env = smartreceive(transport_msg, fileserver_download_handler=_sync_fileserver_download)
|
||||
>>> env = smartunpack(transport_msg, fileserver_download_handler=_sync_fileserver_download)
|
||||
>>> # env is a Dict with "payloads" key containing List[Tuple[str, Any, str]]
|
||||
>>> for dataname, data, type_ in env["payloads"]:
|
||||
... print(f"{dataname}: {data} (type: {type_})")
|
||||
@@ -530,34 +530,34 @@ class msghandler:
|
||||
self.broker_url = broker_url or self.DEFAULT_BROKER_URL
|
||||
self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL
|
||||
|
||||
def smartsend(self, subject, data, **kwargs):
|
||||
def smartpack(self, subject, data, **kwargs):
|
||||
"""
|
||||
Send data.
|
||||
|
||||
Args:
|
||||
subject: Subject/topic to send to
|
||||
data: List of (dataname, data, type) tuples
|
||||
**kwargs: Additional options passed to smartsend
|
||||
**kwargs: Additional options passed to smartpack
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str)
|
||||
"""
|
||||
kwargs['broker_url'] = kwargs.get('broker_url', self.broker_url)
|
||||
kwargs['fileserver_url'] = kwargs.get('fileserver_url', self.fileserver_url)
|
||||
return smartsend(subject, data, **kwargs)
|
||||
return smartpack(subject, data, **kwargs)
|
||||
|
||||
def smartreceive(self, msg, **kwargs):
|
||||
def smartunpack(self, msg, **kwargs):
|
||||
"""
|
||||
Receive and process message.
|
||||
|
||||
Args:
|
||||
msg: Message to process
|
||||
**kwargs: Additional options passed to smartreceive
|
||||
**kwargs: Additional options passed to smartunpack
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads
|
||||
"""
|
||||
return smartreceive(msg, **kwargs)
|
||||
return smartunpack(msg, **kwargs)
|
||||
|
||||
|
||||
# Convenience functions for module-level usage
|
||||
@@ -573,7 +573,7 @@ def send(subject, data, **kwargs):
|
||||
Returns:
|
||||
Tuple of (env, env_json_str)
|
||||
"""
|
||||
return smartsend(subject, data, **kwargs)
|
||||
return smartpack(subject, data, **kwargs)
|
||||
|
||||
|
||||
def receive(msg, **kwargs):
|
||||
@@ -587,12 +587,12 @@ def receive(msg, **kwargs):
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads
|
||||
"""
|
||||
return smartreceive(msg, **kwargs)
|
||||
return smartunpack(msg, **kwargs)
|
||||
|
||||
|
||||
__all__ = [
|
||||
'smartsend',
|
||||
'smartreceive',
|
||||
'smartpack',
|
||||
'smartunpack',
|
||||
'msghandler',
|
||||
'send',
|
||||
'receive',
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/**
|
||||
* JavaScript Mix Payloads Receiver Test
|
||||
* Tests the smartreceive function with mixed payload types
|
||||
* Tests the smartunpack function with mixed payload types
|
||||
*
|
||||
* This test mirrors test_julia_mix_payloads_receiver.jl and demonstrates that
|
||||
* any combination and any number of mixed content can be received correctly.
|
||||
@@ -50,8 +50,8 @@ async function runTest() {
|
||||
console.log(`Received message on ${msg.subject}`);
|
||||
|
||||
try {
|
||||
// Process the message using smartreceive
|
||||
const envelope = await msghandler.smartreceive(msg, {
|
||||
// Process the message using smartunpack
|
||||
const envelope = await msghandler.smartunpack(msg, {
|
||||
fileserver_download_handler: msghandler.fetchWithBackoff,
|
||||
max_retries: 5,
|
||||
base_delay: 100,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/**
|
||||
* JavaScript Mix Payloads Sender Test
|
||||
* Tests the smartsend function with mixed payload types
|
||||
* Tests the smartpack function with mixed payload types
|
||||
*
|
||||
* This test mirrors test_julia_mix_payloads_sender.jl and demonstrates that
|
||||
* any combination and any number of mixed content can be sent correctly.
|
||||
@@ -169,7 +169,7 @@ async function runTest() {
|
||||
try {
|
||||
// Send the message
|
||||
console.log('Sending mixed payloads...\n');
|
||||
const [env, envJsonStr] = await msghandler.smartsend(
|
||||
const [env, envJsonStr] = await msghandler.smartpack(
|
||||
TEST_SUBJECT,
|
||||
payloads,
|
||||
{
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env julia
|
||||
# Test script for mixed-content message testing
|
||||
# Tests receiving a mix of text, json, table, image, audio, video, and binary data
|
||||
# from Julia serviceA to Julia serviceB using msghandler.jl smartreceive
|
||||
# from Julia serviceA to Julia serviceB using msghandler.jl smartunpack
|
||||
#
|
||||
# This test demonstrates that any combination and any number of mixed content
|
||||
# can be sent and received correctly.
|
||||
@@ -36,9 +36,9 @@ function test_mix_receive()
|
||||
NATS.subscribe(conn, SUBJECT) do msg
|
||||
log_trace("Received message on $(msg.subject)")
|
||||
|
||||
# Use msghandler.smartreceive to handle the data
|
||||
# API: smartreceive(msg, download_handler; max_retries, base_delay, max_delay)
|
||||
result = msghandler.smartreceive(
|
||||
# Use msghandler.smartunpack to handle the data
|
||||
# API: smartunpack(msg, download_handler; max_retries, base_delay, max_delay)
|
||||
result = msghandler.smartunpack(
|
||||
msg;
|
||||
max_retries = 5,
|
||||
base_delay = 100,
|
||||
@@ -245,7 +245,7 @@ println("Note: This receiver will wait for messages from the sender.")
|
||||
println("Run test_julia_to_julia_mix_sender.jl first to send test data.")
|
||||
|
||||
# Run receiver
|
||||
println("\ntesting smartreceive for mixed content")
|
||||
println("\ntesting smartunpack for mixed content")
|
||||
test_mix_receive()
|
||||
|
||||
println("\nTest completed.")
|
||||
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env julia
|
||||
# Test script for mixed-content message testing
|
||||
# Tests sending a mix of text, dictionary, arrowtable, jsontable, image, audio, video, and binary data
|
||||
# from Julia serviceA to Julia serviceB using msghandler.jl smartsend
|
||||
# from Julia serviceA to Julia serviceB using msghandler.jl smartpack
|
||||
#
|
||||
# This test demonstrates that any combination and any number of mixed content
|
||||
# can be sent and received correctly.
|
||||
@@ -166,7 +166,7 @@ function create_sample_data()
|
||||
end
|
||||
|
||||
|
||||
# Sender: Send mixed content via smartsend
|
||||
# Sender: Send mixed content via smartpack
|
||||
function test_mix_send()
|
||||
# Create sample data
|
||||
(text_data, dict_data, arrow_table_small, arrow_table_large, json_table_small, json_table_large, audio_data, large_audio_data, video_data, large_video_data, binary_data, large_binary_data) = create_sample_data()
|
||||
@@ -203,8 +203,8 @@ function test_mix_send()
|
||||
("binary_file_large", large_binary_data, "binary")
|
||||
]
|
||||
|
||||
# Use smartsend with mixed content
|
||||
sendinfo = msghandler.smartsend(
|
||||
# Use smartpack with mixed content
|
||||
sendinfo = msghandler.smartpack(
|
||||
SUBJECT,
|
||||
payloads; # List of (dataname, data, type) tuples
|
||||
broker_url = NATS_URL,
|
||||
@@ -251,7 +251,7 @@ println("Starting mixed-content transport test...")
|
||||
println("Correlation ID: $correlation_id")
|
||||
|
||||
# Run sender
|
||||
println("start smartsend for mixed content")
|
||||
println("start smartpack for mixed content")
|
||||
test_mix_send()
|
||||
|
||||
println("\nTest completed.")
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""
|
||||
Python Mix Payloads Sender Test
|
||||
Tests the smartsend function with mixed payload types
|
||||
Tests the smartpack function with mixed payload types
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
@@ -11,7 +11,7 @@ import base64
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from msghandler import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
from msghandler import smartpack, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_SUBJECT = '/test/mix'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
@@ -56,7 +56,7 @@ async def run_test():
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending mixed payloads...')
|
||||
env, env_json_str = await smartsend(
|
||||
env, env_json_str = await smartpack(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
@@ -164,7 +164,7 @@ async def run_test():
|
||||
('audio', bytes([0x46, 0x4C, 0x41, 0x43]), 'audio')
|
||||
]
|
||||
|
||||
chat_env, _ = await smartsend(
|
||||
chat_env, _ = await smartpack(
|
||||
TEST_SUBJECT,
|
||||
chat_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
|
||||
Reference in New Issue
Block a user