Compare commits
26 Commits
v0.4.2
...
7f68d08134
| Author | SHA1 | Date | |
|---|---|---|---|
| 7f68d08134 | |||
| ab20cd896f | |||
| 5a9e93d6e7 | |||
| b51641dc7e | |||
| 45f1257896 | |||
| 3e2b8b1e3a | |||
| 90d81617ef | |||
| 64c62e616b | |||
| 2c340e37c7 | |||
| 7853e94d2e | |||
| 99bf57b154 | |||
| 0fa6eaf95b | |||
| 76f42be740 | |||
| d99dc41be9 | |||
| 263508b8f7 | |||
| 0c2cca30ed | |||
| 46fdf668c6 | |||
| f8a92a45a0 | |||
| cec70e6036 | |||
| f9e08ba628 | |||
| c12a078149 | |||
| dedd803dc3 | |||
| e8e927a491 | |||
| d950bbac23 | |||
| fc8da2ebf5 | |||
| f6e50c405f |
@@ -1,6 +1,6 @@
|
|||||||
name = "NATSBridge"
|
name = "NATSBridge"
|
||||||
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
|
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
|
||||||
version = "0.4.2"
|
version = "0.4.3"
|
||||||
authors = ["narawat <narawat@gmail.com>"]
|
authors = ["narawat <narawat@gmail.com>"]
|
||||||
|
|
||||||
[deps]
|
[deps]
|
||||||
|
|||||||
968
README.md
Normal file
968
README.md
Normal file
@@ -0,0 +1,968 @@
|
|||||||
|
# NATSBridge
|
||||||
|
|
||||||
|
A high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||||
|
|
||||||
|
[](https://opensource.org/licenses/MIT)
|
||||||
|
[](https://nats.io)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [Overview](#overview)
|
||||||
|
- [Features](#features)
|
||||||
|
- [Architecture](#architecture)
|
||||||
|
- [Installation](#installation)
|
||||||
|
- [Quick Start](#quick-start)
|
||||||
|
- [API Reference](#api-reference)
|
||||||
|
- [Payload Types](#payload-types)
|
||||||
|
- [Transport Strategies](#transport-strategies)
|
||||||
|
- [Examples](#examples)
|
||||||
|
- [Testing](#testing)
|
||||||
|
- [License](#license)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
NATSBridge enables seamless communication across Julia, JavaScript, and Python/Micropython applications through NATS, with intelligent transport selection based on payload size:
|
||||||
|
|
||||||
|
| Transport | Payload Size | Method |
|
||||||
|
|-----------|--------------|--------|
|
||||||
|
| **Direct** | < 1MB | Sent directly via NATS (Base64 encoded) |
|
||||||
|
| **Link** | >= 1MB | Uploaded to HTTP file server, URL sent via NATS |
|
||||||
|
|
||||||
|
### Use Cases
|
||||||
|
|
||||||
|
- **Chat Applications**: Text, images, audio, video in a single message
|
||||||
|
- **File Transfer**: Efficient transfer of large files using claim-check pattern
|
||||||
|
- **Streaming Data**: Sensor data, telemetry, and analytics pipelines
|
||||||
|
- **Cross-Platform Communication**: Julia ↔ JavaScript ↔ Python/Micropython
|
||||||
|
- **IoT Devices**: Micropython devices sending data to cloud services
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- ✅ **Bi-directional messaging** between Julia, JavaScript, and Python/Micropython
|
||||||
|
- ✅ **Multi-payload support** - send multiple payloads with different types in one message
|
||||||
|
- ✅ **Automatic transport selection** - direct vs link based on payload size
|
||||||
|
- ✅ **Claim-Check pattern** for payloads > 1MB
|
||||||
|
- ✅ **Apache Arrow IPC** support for tabular data (zero-copy reading)
|
||||||
|
- ✅ **Exponential backoff** for reliable file server downloads
|
||||||
|
- ✅ **Correlation ID tracking** for message tracing
|
||||||
|
- ✅ **Reply-to support** for request-response patterns
|
||||||
|
- ✅ **JetStream support** for message replay and durability
|
||||||
|
- ✅ **Lightweight Micropython implementation** for microcontrollers
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### System Components
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ NATSBridge Architecture │
|
||||||
|
├─────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||||
|
│ │ Julia │ │ JavaScript │ │ Python/ │ │
|
||||||
|
│ │ (NATS.jl) │◄──►│ (nats.js) │◄──►│ Micropython │ │
|
||||||
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||||
|
│ │ │ │ │
|
||||||
|
│ ▼ ▼ ▼ │
|
||||||
|
│ ┌─────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ NATS │ │
|
||||||
|
│ │ (Message Broker) │ │
|
||||||
|
│ └─────────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌──────────────────────┐ │
|
||||||
|
│ │ File Server │ │
|
||||||
|
│ │ (HTTP Upload/Get) │ │
|
||||||
|
│ └──────────────────────┘ │
|
||||||
|
└─────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Message Flow
|
||||||
|
|
||||||
|
1. **Sender** creates a message envelope with payloads
|
||||||
|
2. **NATSBridge** serializes and encodes payloads based on type
|
||||||
|
3. **Transport Decision**: Small payloads go directly to NATS, large payloads are uploaded to file server
|
||||||
|
4. **NATS** routes messages to subscribers
|
||||||
|
5. **Receiver** fetches payloads (from NATS or file server)
|
||||||
|
6. **NATSBridge** deserializes and decodes payloads
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
- **NATS Server** (v2.10+ recommended)
|
||||||
|
- **HTTP File Server** (optional, for payloads > 1MB)
|
||||||
|
|
||||||
|
### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using Pkg
|
||||||
|
Pkg.add("NATS")
|
||||||
|
Pkg.add("https://git.yiem.cc/ton/NATSBridge")
|
||||||
|
```
|
||||||
|
|
||||||
|
### JavaScript
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install nats.js apache-arrow uuid base64-url
|
||||||
|
```
|
||||||
|
|
||||||
|
For Node.js:
|
||||||
|
```javascript
|
||||||
|
const { smartsend, smartreceive } = require('./src/NATSBridge');
|
||||||
|
```
|
||||||
|
|
||||||
|
For browser:
|
||||||
|
```html
|
||||||
|
<script src="./src/NATSBridge.js"></script>
|
||||||
|
<script>
|
||||||
|
// NATSBridge is available as window.NATSBridge
|
||||||
|
</script>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Python/Micropython
|
||||||
|
|
||||||
|
1. Copy [`src/nats_bridge.py`](src/nats_bridge.py) to your device
|
||||||
|
2. Install dependencies:
|
||||||
|
|
||||||
|
**For Python (desktop):**
|
||||||
|
```bash
|
||||||
|
pip install nats-py
|
||||||
|
```
|
||||||
|
|
||||||
|
**For Micropython:**
|
||||||
|
- `urequests` for HTTP requests (built-in for ESP32)
|
||||||
|
- `base64` for base64 encoding (built-in)
|
||||||
|
- `json` for JSON handling (built-in)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Step 1: Start NATS Server
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run -p 4222:4222 nats:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Start HTTP File Server (Optional)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create a directory for file uploads
|
||||||
|
mkdir -p /tmp/fileserver
|
||||||
|
|
||||||
|
# Use Python's built-in server
|
||||||
|
python3 -m http.server 8080 --directory /tmp/fileserver
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Send Your First Message
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
|
# Send a text message
|
||||||
|
data = [("message", "Hello World", "text")]
|
||||||
|
env, env_json_str = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
||||||
|
print("Message sent!")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
// Send a text message
|
||||||
|
const { env, env_json_str } = await smartsend("/chat/room1", [
|
||||||
|
{ dataname: "message", data: "Hello World", type: "text" }
|
||||||
|
], { natsUrl: "nats://localhost:4222" });
|
||||||
|
|
||||||
|
console.log("Message sent!");
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
# Send a text message
|
||||||
|
data = [("message", "Hello World", "text")]
|
||||||
|
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; nats_url="nats://localhost:4222")
|
||||||
|
println("Message sent!")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 4: Receive Messages
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
|
||||||
|
```python
|
||||||
|
import nats
|
||||||
|
import asyncio
|
||||||
|
from nats_bridge import smartreceive
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
SUBJECT = "/chat/room1"
|
||||||
|
NATS_URL = "nats://localhost:4222"
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
# Connect to NATS
|
||||||
|
nc = await nats.connect(NATS_URL)
|
||||||
|
|
||||||
|
# Subscribe to the subject - msg comes from the callback
|
||||||
|
async def message_handler(msg):
|
||||||
|
# Receive and process message
|
||||||
|
env = smartreceive(msg.data)
|
||||||
|
for dataname, data, type in env["payloads"]:
|
||||||
|
print(f"Received {dataname}: {data}")
|
||||||
|
|
||||||
|
sid = await nc.subscribe(SUBJECT, cb=message_handler)
|
||||||
|
await asyncio.sleep(120) # Keep listening
|
||||||
|
await nc.close()
|
||||||
|
|
||||||
|
asyncio.run(main())
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const { smartreceive } = require('./src/NATSBridge');
|
||||||
|
const { connect } = require('nats');
|
||||||
|
|
||||||
|
// Configuration
|
||||||
|
const SUBJECT = "/chat/room1";
|
||||||
|
const NATS_URL = "nats://localhost:4222";
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
// Connect to NATS
|
||||||
|
const nc = await connect({ servers: [NATS_URL] });
|
||||||
|
|
||||||
|
// Subscribe to the subject - msg comes from the async iteration
|
||||||
|
const sub = nc.subscribe(SUBJECT);
|
||||||
|
|
||||||
|
for await (const msg of sub) {
|
||||||
|
// Receive and process message
|
||||||
|
const env = await smartreceive(msg);
|
||||||
|
for (const payload of env.payloads) {
|
||||||
|
console.log(`Received ${payload.dataname}: ${payload.data}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
main();
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using NATS, NATSBridge
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
const SUBJECT = "/chat/room1"
|
||||||
|
const NATS_URL = "nats://localhost:4222"
|
||||||
|
|
||||||
|
# Helper: Log with correlation ID
|
||||||
|
function log_trace(message)
|
||||||
|
timestamp = Dates.now()
|
||||||
|
println("[$timestamp] $message")
|
||||||
|
end
|
||||||
|
|
||||||
|
# Receiver: Listen for messages - msg comes from the callback
|
||||||
|
function test_receive()
|
||||||
|
conn = NATS.connect(NATS_URL)
|
||||||
|
NATS.subscribe(conn, SUBJECT) do msg
|
||||||
|
log_trace("Received message on $(msg.subject)")
|
||||||
|
|
||||||
|
# Receive and process message
|
||||||
|
env, env_json_str = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
|
||||||
|
for (dataname, data, type) in env["payloads"]
|
||||||
|
println("Received $dataname: $data")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# Keep listening for 120 seconds
|
||||||
|
sleep(120)
|
||||||
|
NATS.drain(conn)
|
||||||
|
end
|
||||||
|
|
||||||
|
test_receive()
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## API Reference
|
||||||
|
|
||||||
|
### smartsend
|
||||||
|
|
||||||
|
Sends data either directly via NATS or via a fileserver URL, depending on payload size.
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
|
env, env_json_str = smartsend(
|
||||||
|
subject, # NATS subject to publish to
|
||||||
|
data, # List of (dataname, data, type) tuples
|
||||||
|
nats_url="nats://localhost:4222", # NATS server URL
|
||||||
|
fileserver_url="http://localhost:8080", # File server URL
|
||||||
|
fileserver_upload_handler=plik_oneshot_upload, # Upload handler function
|
||||||
|
size_threshold=1_000_000, # Threshold in bytes (default: 1MB)
|
||||||
|
correlation_id=None, # Optional correlation ID for tracing
|
||||||
|
msg_purpose="chat", # Message purpose
|
||||||
|
sender_name="NATSBridge", # Sender name
|
||||||
|
receiver_name="", # Receiver name (empty = broadcast)
|
||||||
|
receiver_id="", # Receiver UUID (empty = broadcast)
|
||||||
|
reply_to="", # Reply topic
|
||||||
|
reply_to_msg_id="", # Reply message ID
|
||||||
|
is_publish=True # Whether to automatically publish to NATS
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
const { env, env_json_str } = await smartsend(
|
||||||
|
subject, // NATS subject
|
||||||
|
data, // Array of {dataname, data, type}
|
||||||
|
{
|
||||||
|
natsUrl: "nats://localhost:4222",
|
||||||
|
fileserverUrl: "http://localhost:8080",
|
||||||
|
fileserverUploadHandler: customUploadHandler,
|
||||||
|
sizeThreshold: 1_000_000,
|
||||||
|
correlationId: "custom-id",
|
||||||
|
msgPurpose: "chat",
|
||||||
|
senderName: "NATSBridge",
|
||||||
|
receiverName: "",
|
||||||
|
receiverId: "",
|
||||||
|
replyTo: "",
|
||||||
|
replyToMsgId: "",
|
||||||
|
isPublish: true // Whether to automatically publish to NATS
|
||||||
|
}
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
env, env_json_str = NATSBridge.smartsend(
|
||||||
|
subject, # NATS subject
|
||||||
|
data::AbstractArray{Tuple{String, Any, String}}; # List of (dataname, data, type)
|
||||||
|
nats_url::String = "nats://localhost:4222",
|
||||||
|
fileserver_url = "http://localhost:8080",
|
||||||
|
fileserverUploadHandler::Function = plik_oneshot_upload,
|
||||||
|
size_threshold::Int = 1_000_000,
|
||||||
|
correlation_id::Union{String, Nothing} = nothing,
|
||||||
|
msg_purpose::String = "chat",
|
||||||
|
sender_name::String = "NATSBridge",
|
||||||
|
receiver_name::String = "",
|
||||||
|
receiver_id::String = "",
|
||||||
|
reply_to::String = "",
|
||||||
|
reply_to_msg_id::String = "",
|
||||||
|
is_publish::Bool = true # Whether to automatically publish to NATS
|
||||||
|
)
|
||||||
|
# Returns: (msgEnvelope_v1, JSON string)
|
||||||
|
# - env: msgEnvelope_v1 object with all envelope metadata and payloads
|
||||||
|
# - env_json_str: JSON string representation of the envelope for publishing
|
||||||
|
```
|
||||||
|
|
||||||
|
### smartreceive
|
||||||
|
|
||||||
|
Receives and processes messages from NATS, handling both direct and link transport.
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartreceive
|
||||||
|
|
||||||
|
# Note: For nats-py, use msg.data to pass the raw message data
|
||||||
|
env = smartreceive(
|
||||||
|
msg.data, # NATS message data (msg.data for nats-py)
|
||||||
|
fileserver_download_handler=_fetch_with_backoff, # Download handler
|
||||||
|
max_retries=5, # Max retry attempts
|
||||||
|
base_delay=100, # Initial delay in ms
|
||||||
|
max_delay=5000 # Max delay in ms
|
||||||
|
)
|
||||||
|
# Returns: Dict with envelope metadata and 'payloads' field
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const { smartreceive } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
// Note: msg is the NATS message object from subscription
|
||||||
|
const env = await smartreceive(
|
||||||
|
msg, // NATS message (raw object from subscription)
|
||||||
|
{
|
||||||
|
fileserverDownloadHandler: customDownloadHandler,
|
||||||
|
maxRetries: 5,
|
||||||
|
baseDelay: 100,
|
||||||
|
maxDelay: 5000
|
||||||
|
}
|
||||||
|
);
|
||||||
|
// Returns: Object with envelope metadata and payloads array
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
# Note: msg is a NATS.Msg object passed from the subscription callback
|
||||||
|
env, env_json_str = NATSBridge.smartreceive(
|
||||||
|
msg::NATS.Msg;
|
||||||
|
fileserverDownloadHandler::Function = _fetch_with_backoff,
|
||||||
|
max_retries::Int = 5,
|
||||||
|
base_delay::Int = 100,
|
||||||
|
max_delay::Int = 5000
|
||||||
|
)
|
||||||
|
# Returns: Dict with envelope metadata and payloads array
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Payload Types
|
||||||
|
|
||||||
|
| Type | Description | Serialization |
|
||||||
|
|------|-------------|---------------|
|
||||||
|
| `text` | Plain text strings | UTF-8 bytes |
|
||||||
|
| `dictionary` | JSON-serializable dictionaries | JSON |
|
||||||
|
| `table` | Tabular data (DataFrames, arrays) | Apache Arrow IPC |
|
||||||
|
| `image` | Image data (PNG, JPG) | Raw bytes |
|
||||||
|
| `audio` | Audio data (WAV, MP3) | Raw bytes |
|
||||||
|
| `video` | Video data (MP4, AVI) | Raw bytes |
|
||||||
|
| `binary` | Generic binary data | Raw bytes |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Transport Strategies
|
||||||
|
|
||||||
|
### Direct Transport (Payloads < 1MB)
|
||||||
|
|
||||||
|
Small payloads are sent directly via NATS with Base64 encoding.
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
```python
|
||||||
|
data = [("message", "Hello", "text")]
|
||||||
|
smartsend("/topic", data)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
```javascript
|
||||||
|
await smartsend("/topic", [
|
||||||
|
{ dataname: "message", data: "Hello", type: "text" }
|
||||||
|
]);
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
```julia
|
||||||
|
data = [("message", "Hello", "text")]
|
||||||
|
smartsend("/topic", data)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Link Transport (Payloads >= 1MB)
|
||||||
|
|
||||||
|
Large payloads are uploaded to an HTTP file server.
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
```python
|
||||||
|
data = [("file", large_data, "binary")]
|
||||||
|
smartsend("/topic", data, fileserver_url="http://localhost:8080")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
```javascript
|
||||||
|
await smartsend("/topic", [
|
||||||
|
{ dataname: "file", data: largeData, type: "binary" }
|
||||||
|
], { fileserverUrl: "http://localhost:8080" });
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
```julia
|
||||||
|
data = [("file", large_data, "binary")]
|
||||||
|
smartsend("/topic", data; fileserver_url="http://localhost:8080")
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
All examples include code for **Julia**, **JavaScript**, and **Python/Micropython** unless otherwise specified.
|
||||||
|
|
||||||
|
### Example 1: Chat with Mixed Content
|
||||||
|
|
||||||
|
Send text, small image, and large file in one message.
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
|
data = [
|
||||||
|
("message_text", "Hello!", "text"),
|
||||||
|
("user_avatar", image_data, "image"),
|
||||||
|
("large_document", large_file_data, "binary")
|
||||||
|
]
|
||||||
|
|
||||||
|
env, env_json_str = smartsend("/chat/room1", data, fileserver_url="http://localhost:8080")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
```javascript
|
||||||
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
const { env, env_json_str } = await smartsend("/chat/room1", [
|
||||||
|
{ dataname: "message_text", data: "Hello!", type: "text" },
|
||||||
|
{ dataname: "user_avatar", data: image_data, type: "image" },
|
||||||
|
{ dataname: "large_document", data: large_file_data, type: "binary" }
|
||||||
|
], {
|
||||||
|
fileserverUrl: "http://localhost:8080"
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
data = [
|
||||||
|
("message_text", "Hello!", "text"),
|
||||||
|
("user_avatar", image_data, "image"),
|
||||||
|
("large_document", large_file_data, "binary")
|
||||||
|
]
|
||||||
|
|
||||||
|
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; fileserver_url="http://localhost:8080")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example 2: Dictionary Exchange
|
||||||
|
|
||||||
|
Send configuration data between platforms.
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
|
config = {
|
||||||
|
"wifi_ssid": "MyNetwork",
|
||||||
|
"wifi_password": "password123",
|
||||||
|
"update_interval": 60
|
||||||
|
}
|
||||||
|
|
||||||
|
data = [("config", config, "dictionary")]
|
||||||
|
env, env_json_str = smartsend("/device/config", data)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
```javascript
|
||||||
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
const config = {
|
||||||
|
wifi_ssid: "MyNetwork",
|
||||||
|
wifi_password: "password123",
|
||||||
|
update_interval: 60
|
||||||
|
};
|
||||||
|
|
||||||
|
const { env, env_json_str } = await smartsend("/device/config", [
|
||||||
|
{ dataname: "config", data: config, type: "dictionary" }
|
||||||
|
]);
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
config = Dict(
|
||||||
|
"wifi_ssid" => "MyNetwork",
|
||||||
|
"wifi_password" => "password123",
|
||||||
|
"update_interval" => 60
|
||||||
|
)
|
||||||
|
|
||||||
|
data = [("config", config, "dictionary")]
|
||||||
|
env, env_json_str = NATSBridge.smartsend("/device/config", data)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example 3: Table Data (Arrow IPC)
|
||||||
|
|
||||||
|
Send tabular data using Apache Arrow IPC format.
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
```python
|
||||||
|
import pandas as pd
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
|
df = pd.DataFrame({
|
||||||
|
"id": [1, 2, 3],
|
||||||
|
"name": ["Alice", "Bob", "Charlie"],
|
||||||
|
"score": [95, 88, 92]
|
||||||
|
})
|
||||||
|
|
||||||
|
data = [("students", df, "table")]
|
||||||
|
env, env_json_str = smartsend("/data/analysis", data)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
```javascript
|
||||||
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
const tableData = [
|
||||||
|
{ id: 1, name: "Alice", score: 95 },
|
||||||
|
{ id: 2, name: "Bob", score: 88 },
|
||||||
|
{ id: 3, name: "Charlie", score: 92 }
|
||||||
|
];
|
||||||
|
|
||||||
|
const { env, env_json_str } = await smartsend("/data/analysis", [
|
||||||
|
{ dataname: "students", data: tableData, type: "table" }
|
||||||
|
]);
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
using DataFrames
|
||||||
|
|
||||||
|
df = DataFrame(
|
||||||
|
id = [1, 2, 3],
|
||||||
|
name = ["Alice", "Bob", "Charlie"],
|
||||||
|
score = [95, 88, 92]
|
||||||
|
)
|
||||||
|
|
||||||
|
data = [("students", df, "table")]
|
||||||
|
env, env_json_str = NATSBridge.smartsend("/data/analysis", data)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example 4: Request-Response Pattern with Envelope JSON
|
||||||
|
|
||||||
|
Bi-directional communication with reply-to support. The `smartsend` function now returns both the envelope object and a JSON string that can be published directly.
|
||||||
|
|
||||||
|
#### Python/Micropython (Requester)
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
|
env, env_json_str = smartsend(
|
||||||
|
"/device/command",
|
||||||
|
[("command", {"action": "read_sensor"}, "dictionary")],
|
||||||
|
reply_to="/device/response"
|
||||||
|
)
|
||||||
|
# env: msgEnvelope_v1 object
|
||||||
|
# env_json_str: JSON string for publishing to NATS
|
||||||
|
|
||||||
|
# The env_json_str can also be published directly using NATS request-reply pattern
|
||||||
|
# nc.request("/device/command", env_json_str, reply_to="/device/response")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Python/Micropython (Responder)
|
||||||
|
```python
|
||||||
|
import nats
|
||||||
|
import asyncio
|
||||||
|
from nats_bridge import smartreceive, smartsend
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
SUBJECT = "/device/command"
|
||||||
|
REPLY_SUBJECT = "/device/response"
|
||||||
|
NATS_URL = "nats://localhost:4222"
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
nc = await nats.connect(NATS_URL)
|
||||||
|
|
||||||
|
async def message_handler(msg):
|
||||||
|
env = smartreceive(msg.data)
|
||||||
|
for dataname, data, type in env["payloads"]:
|
||||||
|
if data.get("action") == "read_sensor":
|
||||||
|
response = {"sensor_id": "sensor-001", "value": 42.5}
|
||||||
|
smartsend(REPLY_SUBJECT, [("data", response, "dictionary")])
|
||||||
|
|
||||||
|
sid = await nc.subscribe(SUBJECT, cb=message_handler)
|
||||||
|
await asyncio.sleep(120)
|
||||||
|
await nc.close()
|
||||||
|
|
||||||
|
asyncio.run(main())
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript (Requester)
|
||||||
|
```javascript
|
||||||
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
const { env, env_json_str } = await smartsend("/device/command", [
|
||||||
|
{ dataname: "command", data: { action: "read_sensor" }, type: "dictionary" }
|
||||||
|
], {
|
||||||
|
replyTo: "/device/response"
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript (Responder)
|
||||||
|
```javascript
|
||||||
|
const { smartreceive, smartsend } = require('./src/NATSBridge');
|
||||||
|
const { connect } = require('nats');
|
||||||
|
|
||||||
|
// Configuration
|
||||||
|
const SUBJECT = "/device/command";
|
||||||
|
const REPLY_SUBJECT = "/device/response";
|
||||||
|
const NATS_URL = "nats://localhost:4222";
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
const nc = await connect({ servers: [NATS_URL] });
|
||||||
|
|
||||||
|
const sub = nc.subscribe(SUBJECT);
|
||||||
|
|
||||||
|
for await (const msg of sub) {
|
||||||
|
const env = await smartreceive(msg);
|
||||||
|
for (const payload of env.payloads) {
|
||||||
|
if (payload.dataname === "command" && payload.data.action === "read_sensor") {
|
||||||
|
const response = { sensor_id: "sensor-001", value: 42.5 };
|
||||||
|
await smartsend(REPLY_SUBJECT, [
|
||||||
|
{ dataname: "data", data: response, type: "dictionary" }
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
main();
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia (Requester)
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
env, env_json_str = NATSBridge.smartsend(
|
||||||
|
"/device/command",
|
||||||
|
[("command", Dict("action" => "read_sensor"), "dictionary")];
|
||||||
|
reply_to="/device/response"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia (Responder)
|
||||||
|
```julia
|
||||||
|
using NATS, NATSBridge
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
const SUBJECT = "/device/command"
|
||||||
|
const REPLY_SUBJECT = "/device/response"
|
||||||
|
const NATS_URL = "nats://localhost:4222"
|
||||||
|
|
||||||
|
function test_responder()
|
||||||
|
conn = NATS.connect(NATS_URL)
|
||||||
|
NATS.subscribe(conn, SUBJECT) do msg
|
||||||
|
env, env_json_str = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
|
||||||
|
for (dataname, data, type) in env["payloads"]
|
||||||
|
if dataname == "command" && data["action"] == "read_sensor"
|
||||||
|
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
|
||||||
|
smartsend(REPLY_SUBJECT, [("data", response, "dictionary")])
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
sleep(120)
|
||||||
|
NATS.drain(conn)
|
||||||
|
end
|
||||||
|
|
||||||
|
test_responder()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example 5: Micropython IoT Device
|
||||||
|
|
||||||
|
Lightweight Micropython device sending sensor data.
|
||||||
|
|
||||||
|
#### Micropython
|
||||||
|
```python
|
||||||
|
import nats
|
||||||
|
import asyncio
|
||||||
|
from nats_bridge import smartsend, smartreceive
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
SUBJECT = "/device/sensors"
|
||||||
|
NATS_URL = "nats://localhost:4222"
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
nc = await nats.connect(NATS_URL)
|
||||||
|
|
||||||
|
# Send sensor data
|
||||||
|
data = [("temperature", "25.5", "text"), ("humidity", 65, "dictionary")]
|
||||||
|
smartsend("/device/sensors", data, nats_url="nats://localhost:4222")
|
||||||
|
|
||||||
|
# Receive commands - msg comes from the callback
|
||||||
|
async def message_handler(msg):
|
||||||
|
env = smartreceive(msg.data)
|
||||||
|
for dataname, data, type in env["payloads"]:
|
||||||
|
if type == "dictionary" and data.get("action") == "reboot":
|
||||||
|
# Execute reboot
|
||||||
|
pass
|
||||||
|
|
||||||
|
sid = await nc.subscribe(SUBJECT, cb=message_handler)
|
||||||
|
await asyncio.sleep(120)
|
||||||
|
await nc.close()
|
||||||
|
|
||||||
|
asyncio.run(main())
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript (Receiver)
|
||||||
|
```javascript
|
||||||
|
const { smartreceive } = require('./src/NATSBridge');
|
||||||
|
const { connect } = require('nats');
|
||||||
|
|
||||||
|
// Configuration
|
||||||
|
const SUBJECT = "/device/sensors";
|
||||||
|
const NATS_URL = "nats://localhost:4222";
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
const nc = await connect({ servers: [NATS_URL] });
|
||||||
|
|
||||||
|
const sub = nc.subscribe(SUBJECT);
|
||||||
|
|
||||||
|
for await (const msg of sub) {
|
||||||
|
const env = await smartreceive(msg);
|
||||||
|
for (const payload of env.payloads) {
|
||||||
|
if (payload.dataname === "temperature") {
|
||||||
|
console.log(`Temperature: ${payload.data}`);
|
||||||
|
} else if (payload.dataname === "humidity") {
|
||||||
|
console.log(`Humidity: ${payload.data}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
main();
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia (Receiver)
|
||||||
|
```julia
|
||||||
|
using NATS, NATSBridge
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
const SUBJECT = "/device/sensors"
|
||||||
|
const NATS_URL = "nats://localhost:4222"
|
||||||
|
|
||||||
|
function test_receiver()
|
||||||
|
conn = NATS.connect(NATS_URL)
|
||||||
|
NATS.subscribe(conn, SUBJECT) do msg
|
||||||
|
env, env_json_str = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
|
||||||
|
for (dataname, data, type) in env["payloads"]
|
||||||
|
if dataname == "temperature"
|
||||||
|
println("Temperature: $data")
|
||||||
|
elseif dataname == "humidity"
|
||||||
|
println("Humidity: $data")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
sleep(120)
|
||||||
|
NATS.drain(conn)
|
||||||
|
end
|
||||||
|
|
||||||
|
test_receiver()
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Run the test scripts to verify functionality:
|
||||||
|
|
||||||
|
### Python/Micropython
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Basic functionality test
|
||||||
|
python test/test_micropython_basic.py
|
||||||
|
|
||||||
|
# Text message exchange
|
||||||
|
python test/test_micropython_text_sender.py
|
||||||
|
python test/test_micropython_text_receiver.py
|
||||||
|
|
||||||
|
# Dictionary exchange
|
||||||
|
python test/test_micropython_dict_sender.py
|
||||||
|
python test/test_micropython_dict_receiver.py
|
||||||
|
|
||||||
|
# File transfer
|
||||||
|
python test/test_micropython_file_sender.py
|
||||||
|
python test/test_micropython_file_receiver.py
|
||||||
|
|
||||||
|
# Mixed payload types
|
||||||
|
python test/test_micropython_mixed_sender.py
|
||||||
|
python test/test_micropython_mixed_receiver.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### JavaScript
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Text message exchange
|
||||||
|
node test/test_js_text_sender.js
|
||||||
|
node test/test_js_text_receiver.js
|
||||||
|
|
||||||
|
# Dictionary exchange
|
||||||
|
node test/test_js_dict_sender.js
|
||||||
|
node test/test_js_dict_receiver.js
|
||||||
|
|
||||||
|
# File transfer
|
||||||
|
node test/test_js_file_sender.js
|
||||||
|
node test/test_js_file_receiver.js
|
||||||
|
|
||||||
|
# Mixed payload types
|
||||||
|
node test/test_js_mix_payload_sender.js
|
||||||
|
node test/test_js_mix_payloads_receiver.js
|
||||||
|
|
||||||
|
# Table exchange
|
||||||
|
node test/test_js_table_sender.js
|
||||||
|
node test/test_js_table_receiver.js
|
||||||
|
```
|
||||||
|
|
||||||
|
### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
# Text message exchange
|
||||||
|
julia test/test_julia_text_sender.jl
|
||||||
|
julia test/test_julia_text_receiver.jl
|
||||||
|
|
||||||
|
# Dictionary exchange
|
||||||
|
julia test/test_julia_dict_sender.jl
|
||||||
|
julia test/test_julia_dict_receiver.jl
|
||||||
|
|
||||||
|
# File transfer
|
||||||
|
julia test/test_julia_file_sender.jl
|
||||||
|
julia test/test_julia_file_receiver.jl
|
||||||
|
|
||||||
|
# Mixed payload types
|
||||||
|
julia test/test_julia_mix_payloads_sender.jl
|
||||||
|
julia test/test_julia_mix_payloads_receiver.jl
|
||||||
|
|
||||||
|
# Table exchange
|
||||||
|
julia test/test_julia_table_sender.jl
|
||||||
|
julia test/test_julia_table_receiver.jl
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2026 NATSBridge Contributors
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
@@ -17,16 +17,16 @@ The system uses **handler functions** to abstract file server operations, allowi
|
|||||||
|
|
||||||
```julia
|
```julia
|
||||||
# Upload handler - uploads data to file server and returns URL
|
# Upload handler - uploads data to file server and returns URL
|
||||||
# The handler is passed to smartsend as fileserverUploadHandler parameter
|
# The handler is passed to smartsend as fileserver_upload_handler parameter
|
||||||
# It receives: (fileserver_url::String, dataname::String, data::Vector{UInt8})
|
# It receives: (file_server_url::String, dataname::String, data::Vector{UInt8})
|
||||||
# Returns: Dict{String, Any} with keys: "status", "uploadid", "fileid", "url"
|
# Returns: Dict{String, Any} with keys: "status", "uploadid", "fileid", "url"
|
||||||
fileserverUploadHandler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
fileserver_upload_handler(file_server_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
||||||
|
|
||||||
# Download handler - fetches data from file server URL with exponential backoff
|
# Download handler - fetches data from file server URL with exponential backoff
|
||||||
# The handler is passed to smartreceive as fileserverDownloadHandler parameter
|
# The handler is passed to smartreceive as fileserver_download_handler parameter
|
||||||
# It receives: (url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)
|
# It receives: (url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)
|
||||||
# Returns: Vector{UInt8} (the downloaded data)
|
# Returns: Vector{UInt8} (the downloaded data)
|
||||||
fileserverDownloadHandler(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)::Vector{UInt8}
|
fileserver_download_handler(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)::Vector{UInt8}
|
||||||
```
|
```
|
||||||
|
|
||||||
This design allows the system to support multiple file server backends without changing the core messaging logic.
|
This design allows the system to support multiple file server backends without changing the core messaging logic.
|
||||||
@@ -40,21 +40,21 @@ The system uses a **standardized list-of-tuples format** for all payload operati
|
|||||||
# Input format for smartsend (always a list of tuples with type info)
|
# Input format for smartsend (always a list of tuples with type info)
|
||||||
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||||
|
|
||||||
# Output format for smartreceive (returns envelope dictionary with payloads field)
|
# Output format for smartreceive (returns a dictionary with payloads field containing list of tuples)
|
||||||
# Returns: Dict with envelope metadata and payloads field containing list of tuples
|
# Returns: Dict with envelope metadata and payloads field containing Vector{Tuple{String, Any, String}}
|
||||||
# {
|
# {
|
||||||
# "correlationId": "...",
|
# "correlation_id": "...",
|
||||||
# "msgId": "...",
|
# "msg_id": "...",
|
||||||
# "timestamp": "...",
|
# "timestamp": "...",
|
||||||
# "sendTo": "...",
|
# "send_to": "...",
|
||||||
# "msgPurpose": "...",
|
# "msg_purpose": "...",
|
||||||
# "senderName": "...",
|
# "sender_name": "...",
|
||||||
# "senderId": "...",
|
# "sender_id": "...",
|
||||||
# "receiverName": "...",
|
# "receiver_name": "...",
|
||||||
# "receiverId": "...",
|
# "receiver_id": "...",
|
||||||
# "replyTo": "...",
|
# "reply_to": "...",
|
||||||
# "replyToMsgId": "...",
|
# "reply_to_msg_id": "...",
|
||||||
# "brokerURL": "...",
|
# "broker_url": "...",
|
||||||
# "metadata": {...},
|
# "metadata": {...},
|
||||||
# "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
# "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||||
# }
|
# }
|
||||||
@@ -78,17 +78,16 @@ This design allows per-payload type specification, enabling **mixed-content mess
|
|||||||
smartsend(
|
smartsend(
|
||||||
"/test",
|
"/test",
|
||||||
[("dataname1", data1, "dictionary")], # List with one tuple (data, type)
|
[("dataname1", data1, "dictionary")], # List with one tuple (data, type)
|
||||||
nats_url="nats://localhost:4222",
|
broker_url="nats://localhost:4222",
|
||||||
fileserverUploadHandler=plik_oneshot_upload,
|
fileserver_upload_handler=plik_oneshot_upload
|
||||||
metadata=user_provided_envelope_level_metadata
|
|
||||||
)
|
)
|
||||||
|
|
||||||
# Multiple payloads in one message with different types
|
# Multiple payloads in one message with different types
|
||||||
smartsend(
|
smartsend(
|
||||||
"/test",
|
"/test",
|
||||||
[("dataname1", data1, "dictionary"), ("dataname2", data2, "table")],
|
[("dataname1", data1, "dictionary"), ("dataname2", data2, "table")],
|
||||||
nats_url="nats://localhost:4222",
|
broker_url="nats://localhost:4222",
|
||||||
fileserverUploadHandler=plik_oneshot_upload
|
fileserver_upload_handler=plik_oneshot_upload
|
||||||
)
|
)
|
||||||
|
|
||||||
# Mixed content (e.g., chat with text, image, audio)
|
# Mixed content (e.g., chat with text, image, audio)
|
||||||
@@ -99,13 +98,14 @@ smartsend(
|
|||||||
("user_image", image_data, "image"),
|
("user_image", image_data, "image"),
|
||||||
("audio_clip", audio_data, "audio")
|
("audio_clip", audio_data, "audio")
|
||||||
],
|
],
|
||||||
nats_url="nats://localhost:4222"
|
broker_url="nats://localhost:4222"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Receive returns a dictionary envelope with all metadata and deserialized payloads
|
# Receive returns a dictionary envelope with all metadata and deserialized payloads
|
||||||
envelope = smartreceive(msg, fileserverDownloadHandler, max_retries, base_delay, max_delay)
|
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff, max_retries=5, base_delay=100, max_delay=5000)
|
||||||
# envelope["payloads"] = [("dataname1", data1, type1), ("dataname2", data2, type2), ...]
|
# env["payloads"] = [("dataname1", data1, type1), ("dataname2", data2, type2), ...]
|
||||||
# envelope["correlationId"], envelope["msgId"], etc.
|
# env["correlation_id"], env["msg_id"], etc.
|
||||||
|
# env is a dictionary containing envelope metadata and payloads field
|
||||||
```
|
```
|
||||||
|
|
||||||
## Architecture Diagram
|
## Architecture Diagram
|
||||||
@@ -138,48 +138,48 @@ flowchart TD
|
|||||||
|
|
||||||
## System Components
|
## System Components
|
||||||
|
|
||||||
### 1. msgEnvelope_v1 - Message Envelope
|
### 1. msg_envelope_v1 - Message Envelope
|
||||||
|
|
||||||
The `msgEnvelope_v1` structure provides a comprehensive message format for bidirectional communication between Julia, JavaScript, and Python/Micropython applications.
|
The `msg_envelope_v1` structure provides a comprehensive message format for bidirectional communication between Julia, JavaScript, and Python/Micropython applications.
|
||||||
|
|
||||||
**Julia Structure:**
|
**Julia Structure:**
|
||||||
```julia
|
```julia
|
||||||
struct msgEnvelope_v1
|
struct msg_envelope_v1
|
||||||
correlationId::String # Unique identifier to track messages across systems
|
correlation_id::String # Unique identifier to track messages across systems
|
||||||
msgId::String # This message id
|
msg_id::String # This message id
|
||||||
timestamp::String # Message published timestamp
|
timestamp::String # Message published timestamp
|
||||||
|
|
||||||
sendTo::String # Topic/subject the sender sends to
|
send_to::String # Topic/subject the sender sends to
|
||||||
msgPurpose::String # Purpose of this message (ACK | NACK | updateStatus | shutdown | ...)
|
msg_purpose::String # Purpose of this message (ACK | NACK | updateStatus | shutdown | ...)
|
||||||
senderName::String # Sender name (e.g., "agent-wine-web-frontend")
|
sender_name::String # Sender name (e.g., "agent-wine-web-frontend")
|
||||||
senderId::String # Sender id (uuid4)
|
sender_id::String # Sender id (uuid4)
|
||||||
receiverName::String # Message receiver name (e.g., "agent-backend")
|
receiver_name::String # Message receiver name (e.g., "agent-backend")
|
||||||
receiverId::String # Message receiver id (uuid4 or nothing for broadcast)
|
receiver_id::String # Message receiver id (uuid4 or nothing for broadcast)
|
||||||
replyTo::String # Topic to reply to
|
reply_to::String # Topic to reply to
|
||||||
replyToMsgId::String # Message id this message is replying to
|
reply_to_msg_id::String # Message id this message is replying to
|
||||||
brokerURL::String # NATS server address
|
broker_url::String # NATS server address
|
||||||
|
|
||||||
metadata::Dict{String, Any}
|
metadata::Dict{String, Any}
|
||||||
payloads::AbstractArray{msgPayload_v1} # Multiple payloads stored here
|
payloads::Vector{msg_payload_v1} # Multiple payloads stored here
|
||||||
end
|
end
|
||||||
```
|
```
|
||||||
|
|
||||||
**JSON Schema:**
|
**JSON Schema:**
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"correlationId": "uuid-v4-string",
|
"correlation_id": "uuid-v4-string",
|
||||||
"msgId": "uuid-v4-string",
|
"msg_id": "uuid-v4-string",
|
||||||
"timestamp": "2024-01-15T10:30:00Z",
|
"timestamp": "2024-01-15T10:30:00Z",
|
||||||
|
|
||||||
"sendTo": "topic/subject",
|
"send_to": "topic/subject",
|
||||||
"msgPurpose": "ACK | NACK | updateStatus | shutdown | chat",
|
"msg_purpose": "ACK | NACK | updateStatus | shutdown | chat",
|
||||||
"senderName": "agent-wine-web-frontend",
|
"sender_name": "agent-wine-web-frontend",
|
||||||
"senderId": "uuid4",
|
"sender_id": "uuid4",
|
||||||
"receiverName": "agent-backend",
|
"receiver_name": "agent-backend",
|
||||||
"receiverId": "uuid4",
|
"receiver_id": "uuid4",
|
||||||
"replyTo": "topic",
|
"reply_to": "topic",
|
||||||
"replyToMsgId": "uuid4",
|
"reply_to_msg_id": "uuid4",
|
||||||
"brokerURL": "nats://localhost:4222",
|
"broker_url": "nats://localhost:4222",
|
||||||
|
|
||||||
"metadata": {
|
"metadata": {
|
||||||
|
|
||||||
@@ -189,7 +189,7 @@ end
|
|||||||
{
|
{
|
||||||
"id": "uuid4",
|
"id": "uuid4",
|
||||||
"dataname": "login_image",
|
"dataname": "login_image",
|
||||||
"type": "image",
|
"payload_type": "image",
|
||||||
"transport": "direct",
|
"transport": "direct",
|
||||||
"encoding": "base64",
|
"encoding": "base64",
|
||||||
"size": 15433,
|
"size": 15433,
|
||||||
@@ -201,7 +201,7 @@ end
|
|||||||
{
|
{
|
||||||
"id": "uuid4",
|
"id": "uuid4",
|
||||||
"dataname": "large_data",
|
"dataname": "large_data",
|
||||||
"type": "table",
|
"payload_type": "table",
|
||||||
"transport": "link",
|
"transport": "link",
|
||||||
"encoding": "none",
|
"encoding": "none",
|
||||||
"size": 524288,
|
"size": 524288,
|
||||||
@@ -214,16 +214,16 @@ end
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. msgPayload_v1 - Payload Structure
|
### 2. msg_payload_v1 - Payload Structure
|
||||||
|
|
||||||
The `msgPayload_v1` structure provides flexible payload handling for various data types across all supported platforms.
|
The `msg_payload_v1` structure provides flexible payload handling for various data types across all supported platforms.
|
||||||
|
|
||||||
**Julia Structure:**
|
**Julia Structure:**
|
||||||
```julia
|
```julia
|
||||||
struct msgPayload_v1
|
struct msg_payload_v1
|
||||||
id::String # Id of this payload (e.g., "uuid4")
|
id::String # Id of this payload (e.g., "uuid4")
|
||||||
dataname::String # Name of this payload (e.g., "login_image")
|
dataname::String # Name of this payload (e.g., "login_image")
|
||||||
type::String # "text | dictionary | table | image | audio | video | binary"
|
payload_type::String # "text | dictionary | table | image | audio | video | binary"
|
||||||
transport::String # "direct | link"
|
transport::String # "direct | link"
|
||||||
encoding::String # "none | json | base64 | arrow-ipc"
|
encoding::String # "none | json | base64 | arrow-ipc"
|
||||||
size::Integer # Data size in bytes
|
size::Integer # Data size in bytes
|
||||||
@@ -383,13 +383,32 @@ graph TD
|
|||||||
```julia
|
```julia
|
||||||
function smartsend(
|
function smartsend(
|
||||||
subject::String,
|
subject::String,
|
||||||
data::AbstractArray{Tuple{String, Any, String}}; # No standalone type parameter
|
data::AbstractArray{Tuple{String, Any, String}, 1}; # List of (dataname, data, type) tuples
|
||||||
nats_url::String = "nats://localhost:4222",
|
broker_url::String = DEFAULT_BROKER_URL, # NATS server URL
|
||||||
fileserverUploadHandler::Function = plik_oneshot_upload,
|
fileserver_url = DEFAULT_FILESERVER_URL,
|
||||||
size_threshold::Int = 1_000_000 # 1MB
|
fileserver_upload_handler::Function = plik_oneshot_upload,
|
||||||
|
size_threshold::Int = DEFAULT_SIZE_THRESHOLD,
|
||||||
|
correlation_id::Union{String, Nothing} = nothing,
|
||||||
|
msg_purpose::String = "chat",
|
||||||
|
sender_name::String = "NATSBridge",
|
||||||
|
receiver_name::String = "",
|
||||||
|
receiver_id::String = "",
|
||||||
|
reply_to::String = "",
|
||||||
|
reply_to_msg_id::String = "",
|
||||||
|
is_publish::Bool = true # Whether to automatically publish to NATS
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Return Value:**
|
||||||
|
- Returns a tuple `(env, env_json_str)` where:
|
||||||
|
- `env::msg_envelope_v1` - The envelope object containing all metadata and payloads
|
||||||
|
- `env_json_str::String` - JSON string representation of the envelope for publishing
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `is_publish::Bool = true` - When `true` (default), the message is automatically published to NATS. When `false`, the function returns the envelope and JSON string without publishing, allowing manual publishing via NATS request-reply pattern.
|
||||||
|
|
||||||
|
The envelope object can be accessed directly for programmatic use, while the JSON string can be published directly to NATS using the request-reply pattern.
|
||||||
|
|
||||||
**Input Format:**
|
**Input Format:**
|
||||||
- `data::AbstractArray{Tuple{String, Any, String}}` - **Must be a list of (dataname, data, type) tuples**: `[("dataname1", data1, "type1"), ("dataname2", data2, "type2"), ...]`
|
- `data::AbstractArray{Tuple{String, Any, String}}` - **Must be a list of (dataname, data, type) tuples**: `[("dataname1", data1, "type1"), ("dataname2", data2, "type2"), ...]`
|
||||||
- Even for single payloads: `[(dataname1, data1, "type1")]`
|
- Even for single payloads: `[(dataname1, data1, "type1")]`
|
||||||
@@ -406,8 +425,8 @@ function smartsend(
|
|||||||
|
|
||||||
```julia
|
```julia
|
||||||
function smartreceive(
|
function smartreceive(
|
||||||
msg::NATS.Message,
|
msg::NATS.Msg;
|
||||||
fileserverDownloadHandler::Function;
|
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||||
max_retries::Int = 5,
|
max_retries::Int = 5,
|
||||||
base_delay::Int = 100,
|
base_delay::Int = 100,
|
||||||
max_delay::Int = 5000
|
max_delay::Int = 5000
|
||||||
@@ -416,7 +435,7 @@ function smartreceive(
|
|||||||
# Iterate through all payloads
|
# Iterate through all payloads
|
||||||
# For each payload: check transport type
|
# For each payload: check transport type
|
||||||
# If direct: decode Base64 payload
|
# If direct: decode Base64 payload
|
||||||
# If link: fetch from URL with exponential backoff using fileserverDownloadHandler
|
# If link: fetch from URL with exponential backoff using fileserver_download_handler
|
||||||
# Deserialize payload based on type
|
# Deserialize payload based on type
|
||||||
# Return envelope dictionary with all metadata and deserialized payloads
|
# Return envelope dictionary with all metadata and deserialized payloads
|
||||||
end
|
end
|
||||||
@@ -424,7 +443,7 @@ end
|
|||||||
|
|
||||||
**Output Format:**
|
**Output Format:**
|
||||||
- Returns a dictionary (key-value map) containing all envelope fields:
|
- Returns a dictionary (key-value map) containing all envelope fields:
|
||||||
- `correlationId`, `msgId`, `timestamp`, `sendTo`, `msgPurpose`, `senderName`, `senderId`, `receiverName`, `receiverId`, `replyTo`, `replyToMsgId`, `brokerURL`
|
- `correlation_id`, `msg_id`, `timestamp`, `send_to`, `msg_purpose`, `sender_name`, `sender_id`, `receiver_name`, `receiver_id`, `reply_to`, `reply_to_msg_id`, `broker_url`
|
||||||
- `metadata` - Message-level metadata dictionary
|
- `metadata` - Message-level metadata dictionary
|
||||||
- `payloads` - List of dictionaries, each containing deserialized payload data
|
- `payloads` - List of dictionaries, each containing deserialized payload data
|
||||||
|
|
||||||
@@ -434,71 +453,186 @@ end
|
|||||||
3. For each payload:
|
3. For each payload:
|
||||||
- Determine transport type (`direct` or `link`)
|
- Determine transport type (`direct` or `link`)
|
||||||
- If `direct`: decode Base64 data from the message
|
- If `direct`: decode Base64 data from the message
|
||||||
- If `link`: fetch data from URL using exponential backoff (via `fileserverDownloadHandler`)
|
- If `link`: fetch data from URL using exponential backoff (via `fileserver_download_handler`)
|
||||||
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
|
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
|
||||||
4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples
|
4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples
|
||||||
|
|
||||||
**Note:** The `fileserverDownloadHandler` receives `(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)` and returns `Vector{UInt8}`.
|
**Note:** The `fileserver_download_handler` receives `(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)` and returns `Vector{UInt8}`.
|
||||||
|
|
||||||
### JavaScript Implementation
|
### JavaScript Implementation
|
||||||
|
|
||||||
#### Dependencies
|
#### Dependencies
|
||||||
- `nats.js` - Core NATS functionality
|
- `nats.js` - Core NATS functionality
|
||||||
- `apache-arrow` - Arrow IPC serialization
|
- `apache-arrow` - Arrow IPC serialization
|
||||||
- `uuid` - Correlation ID generation
|
- `uuid` - Correlation ID and message ID generation
|
||||||
|
- `base64-arraybuffer` - Base64 encoding/decoding
|
||||||
|
- `node-fetch` or `fetch` - HTTP client for file server
|
||||||
|
|
||||||
#### smartsend Function
|
#### smartsend Function
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
async function smartsend(subject, data, options = {})
|
async function smartsend(
|
||||||
// data format: [(dataname, data, type), ...]
|
subject,
|
||||||
// options object should include:
|
data, // List of (dataname, data, type) tuples: [(dataname1, data1, type1), ...]
|
||||||
// - natsUrl: NATS server URL
|
options = {}
|
||||||
// - fileserverUrl: base URL of the file server
|
)
|
||||||
// - sizeThreshold: threshold in bytes for transport selection
|
|
||||||
// - correlationId: optional correlation ID for tracing
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `broker_url` (String) - NATS server URL (default: `"nats://localhost:4222"`)
|
||||||
|
- `fileserver_url` (String) - Base URL of the file server (default: `"http://localhost:8080"`)
|
||||||
|
- `size_threshold` (Number) - Threshold in bytes for transport selection (default: `1048576` = 1MB)
|
||||||
|
- `correlation_id` (String) - Optional correlation ID for tracing
|
||||||
|
- `msg_purpose` (String) - Purpose of the message (default: `"chat"`)
|
||||||
|
- `sender_name` (String) - Sender name (default: `"NATSBridge"`)
|
||||||
|
- `receiver_name` (String) - Message receiver name (default: `""`)
|
||||||
|
- `receiver_id` (String) - Message receiver ID (default: `""`)
|
||||||
|
- `reply_to` (String) - Topic to reply to (default: `""`)
|
||||||
|
- `reply_to_msg_id` (String) - Message ID this message is replying to (default: `""`)
|
||||||
|
- `fileserver_upload_handler` (Function) - Custom upload handler function
|
||||||
|
|
||||||
|
**Return Value:**
|
||||||
|
- Returns a Promise that resolves to an object containing:
|
||||||
|
- `env` - The envelope object containing all metadata and payloads
|
||||||
|
- `env_json_str` - JSON string representation of the envelope for publishing
|
||||||
|
- `published` - Boolean indicating whether the message was automatically published to NATS
|
||||||
|
|
||||||
**Input Format:**
|
**Input Format:**
|
||||||
- `data` - **Must be a list of (dataname, data, type) tuples**: `[(dataname1, data1, "type1"), (dataname2, data2, "type2"), ...]`
|
- `data` - **Must be a list of (dataname, data, type) tuples**: `[(dataname1, data1, "type1"), (dataname2, data2, "type2"), ...]`
|
||||||
- Even for single payloads: `[(dataname1, data1, "type1")]`
|
- Even for single payloads: `[(dataname1, data1, "type1")]`
|
||||||
- Each payload can have a different type, enabling mixed-content messages
|
- Each payload can have a different type, enabling mixed-content messages
|
||||||
|
- Supported types: `"text"`, `"dictionary"`, `"table"`, `"image"`, `"audio"`, `"video"`, `"binary"`
|
||||||
|
|
||||||
**Flow:**
|
**Flow:**
|
||||||
1. Iterate through the list of (dataname, data, type) tuples
|
1. Generate correlation ID and message ID if not provided
|
||||||
2. For each payload: extract the type from the tuple and serialize accordingly
|
2. Iterate through the list of `(dataname, data, type)` tuples
|
||||||
3. Check payload size
|
3. For each payload:
|
||||||
4. If < threshold: publish directly to NATS
|
- Serialize based on payload type
|
||||||
5. If >= threshold: upload to HTTP server, publish NATS with URL
|
- Check payload size
|
||||||
|
- If < threshold: Base64 encode and include in envelope
|
||||||
|
- If >= threshold: Upload to HTTP server, store URL in envelope
|
||||||
|
4. Publish the JSON envelope to NATS
|
||||||
|
5. Return envelope object and JSON string
|
||||||
|
|
||||||
#### smartreceive Handler
|
#### smartreceive Handler
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
async function smartreceive(msg, options = {})
|
async function smartreceive(msg, options = {})
|
||||||
// options object should include:
|
|
||||||
// - fileserverDownloadHandler: function to fetch data from file server URL
|
|
||||||
// - max_retries: maximum retry attempts for fetching URL
|
|
||||||
// - base_delay: initial delay for exponential backoff in ms
|
|
||||||
// - max_delay: maximum delay for exponential backoff in ms
|
|
||||||
// - correlationId: optional correlation ID for tracing
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `fileserver_download_handler` (Function) - Custom download handler function
|
||||||
|
- `max_retries` (Number) - Maximum retry attempts for fetching URL (default: `5`)
|
||||||
|
- `base_delay` (Number) - Initial delay for exponential backoff in ms (default: `100`)
|
||||||
|
- `max_delay` (Number) - Maximum delay for exponential backoff in ms (default: `5000`)
|
||||||
|
- `correlation_id` (String) - Optional correlation ID for tracing
|
||||||
|
|
||||||
**Output Format:**
|
**Output Format:**
|
||||||
- Returns a dictionary (key-value map) containing all envelope fields:
|
- Returns a Promise that resolves to an object containing all envelope fields:
|
||||||
- `correlationId`, `msgId`, `timestamp`, `sendTo`, `msgPurpose`, `senderName`, `senderId`, `receiverName`, `receiverId`, `replyTo`, `replyToMsgId`, `brokerURL`
|
- `correlation_id`, `msg_id`, `timestamp`, `send_to`, `msg_purpose`, `sender_name`, `sender_id`, `receiver_name`, `receiver_id`, `reply_to`, `reply_to_msg_id`, `broker_url`
|
||||||
- `metadata` - Message-level metadata dictionary
|
- `metadata` - Message-level metadata dictionary
|
||||||
- `payloads` - List of dictionaries, each containing deserialized payload data
|
- `payloads` - List of dictionaries, each containing deserialized payload data with keys: `dataname`, `data`, `payload_type`
|
||||||
|
|
||||||
**Process Flow:**
|
**Process Flow:**
|
||||||
1. Parse the JSON envelope to extract all fields
|
1. Parse the JSON envelope to extract all fields
|
||||||
2. Iterate through each payload in `payloads`
|
2. Iterate through each payload in `payloads` array
|
||||||
3. For each payload:
|
3. For each payload:
|
||||||
- Determine transport type (`direct` or `link`)
|
- Determine transport type (`direct` or `link`)
|
||||||
- If `direct`: decode Base64 data from the message
|
- If `direct`: Base64 decode the data from the message
|
||||||
- If `link`: fetch data from URL using exponential backoff
|
- If `link`: Fetch data from URL using exponential backoff (via `fileserver_download_handler`)
|
||||||
|
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
|
||||||
|
4. Return envelope object with `payloads` field containing list of `(dataname, data, type)` tuples
|
||||||
|
|
||||||
|
**Note:** The `fileserver_download_handler` receives `(url, max_retries, base_delay, max_delay, correlation_id)` and returns `ArrayBuffer` or `Uint8Array`.
|
||||||
|
|
||||||
|
### Python/Micropython Implementation
|
||||||
|
|
||||||
|
#### Dependencies
|
||||||
|
- `nats-python` - Core NATS functionality
|
||||||
|
- `pyarrow` - Arrow IPC serialization
|
||||||
|
- `uuid` - Correlation ID and message ID generation
|
||||||
|
- `base64` - Base64 encoding/decoding
|
||||||
|
- `requests` or `aiohttp` - HTTP client for file server
|
||||||
|
|
||||||
|
#### smartsend Function
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def smartsend(
|
||||||
|
subject: str,
|
||||||
|
data: List[Tuple[str, Any, str]], # List of (dataname, data, type) tuples
|
||||||
|
options: Dict = {}
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `broker_url` (str) - NATS server URL (default: `"nats://localhost:4222"`)
|
||||||
|
- `fileserver_url` (str) - Base URL of the file server (default: `"http://localhost:8080"`)
|
||||||
|
- `size_threshold` (int) - Threshold in bytes for transport selection (default: `1048576` = 1MB)
|
||||||
|
- `correlation_id` (str) - Optional correlation ID for tracing
|
||||||
|
- `msg_purpose` (str) - Purpose of the message (default: `"chat"`)
|
||||||
|
- `sender_name` (str) - Sender name (default: `"NATSBridge"`)
|
||||||
|
- `receiver_name` (str) - Message receiver name (default: `""`)
|
||||||
|
- `receiver_id` (str) - Message receiver ID (default: `""`)
|
||||||
|
- `reply_to` (str) - Topic to reply to (default: `""`)
|
||||||
|
- `reply_to_msg_id` (str) - Message ID this message is replying to (default: `""`)
|
||||||
|
- `fileserver_upload_handler` (Callable) - Custom upload handler function
|
||||||
|
|
||||||
|
**Return Value:**
|
||||||
|
- Returns a tuple `(env, env_json_str)` where:
|
||||||
|
- `env` - The envelope dictionary containing all metadata and payloads
|
||||||
|
- `env_json_str` - JSON string representation of the envelope for publishing
|
||||||
|
|
||||||
|
**Input Format:**
|
||||||
|
- `data` - **Must be a list of (dataname, data, type) tuples**: `[(dataname1, data1, "type1"), (dataname2, data2, "type2"), ...]`
|
||||||
|
- Even for single payloads: `[(dataname1, data1, "type1")]`
|
||||||
|
- Each payload can have a different type, enabling mixed-content messages
|
||||||
|
- Supported types: `"text"`, `"dictionary"`, `"table"`, `"image"`, `"audio"`, `"video"`, `"binary"`
|
||||||
|
|
||||||
|
**Flow:**
|
||||||
|
1. Generate correlation ID and message ID if not provided
|
||||||
|
2. Iterate through the list of `(dataname, data, type)` tuples
|
||||||
|
3. For each payload:
|
||||||
|
- Serialize based on payload type
|
||||||
|
- Check payload size
|
||||||
|
- If < threshold: Base64 encode and include in envelope
|
||||||
|
- If >= threshold: Upload to HTTP server, store URL in envelope
|
||||||
|
4. Publish the JSON envelope to NATS
|
||||||
|
5. Return envelope dictionary and JSON string
|
||||||
|
|
||||||
|
#### smartreceive Handler
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def smartreceive(
|
||||||
|
msg: NATS.Message,
|
||||||
|
options: Dict = {}
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `fileserver_download_handler` (Callable) - Custom download handler function
|
||||||
|
- `max_retries` (int) - Maximum retry attempts for fetching URL (default: `5`)
|
||||||
|
- `base_delay` (int) - Initial delay for exponential backoff in ms (default: `100`)
|
||||||
|
- `max_delay` (int) - Maximum delay for exponential backoff in ms (default: `5000`)
|
||||||
|
- `correlation_id` (str) - Optional correlation ID for tracing
|
||||||
|
|
||||||
|
**Output Format:**
|
||||||
|
- Returns a dictionary containing all envelope fields:
|
||||||
|
- `correlation_id`, `msg_id`, `timestamp`, `send_to`, `msg_purpose`, `sender_name`, `sender_id`, `receiver_name`, `receiver_id`, `reply_to`, `reply_to_msg_id`, `broker_url`
|
||||||
|
- `metadata` - Message-level metadata dictionary
|
||||||
|
- `payloads` - List of tuples, each containing `(dataname, data, payload_type)` with deserialized payload data
|
||||||
|
|
||||||
|
**Process Flow:**
|
||||||
|
1. Parse the JSON envelope to extract all fields
|
||||||
|
2. Iterate through each payload in `payloads` list
|
||||||
|
3. For each payload:
|
||||||
|
- Determine transport type (`direct` or `link`)
|
||||||
|
- If `direct`: Base64 decode the data from the message
|
||||||
|
- If `link`: Fetch data from URL using exponential backoff (via `fileserver_download_handler`)
|
||||||
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
|
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
|
||||||
4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples
|
4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples
|
||||||
|
|
||||||
|
**Note:** The `fileserver_download_handler` receives `(url: str, max_retries: int, base_delay: int, max_delay: int, correlation_id: str)` and returns `bytes`.
|
||||||
|
|
||||||
## Scenario Implementations
|
## Scenario Implementations
|
||||||
|
|
||||||
### Scenario 1: Command & Control (Small Dictionary)
|
### Scenario 1: Command & Control (Small Dictionary)
|
||||||
|
|||||||
@@ -19,49 +19,102 @@ NATSBridge is implemented in three languages, each providing the same API:
|
|||||||
| **JavaScript** | [`src/NATSBridge.js`](../src/NATSBridge.js) | JavaScript implementation for Node.js and browsers |
|
| **JavaScript** | [`src/NATSBridge.js`](../src/NATSBridge.js) | JavaScript implementation for Node.js and browsers |
|
||||||
| **Python/Micropython** | [`src/nats_bridge.py`](../src/nats_bridge.py) | Python implementation for desktop and microcontrollers |
|
| **Python/Micropython** | [`src/nats_bridge.py`](../src/nats_bridge.py) | Python implementation for desktop and microcontrollers |
|
||||||
|
|
||||||
### Multi-Payload Support
|
### File Server Handler Architecture
|
||||||
|
|
||||||
The implementation uses a **standardized list-of-tuples format** for all payload operations. **Even when sending a single payload, the user must wrap it in a list.**
|
The system uses **handler functions** to abstract file server operations, allowing support for different file server implementations (e.g., Plik, AWS S3, custom HTTP server).
|
||||||
|
|
||||||
|
**Handler Function Signatures:**
|
||||||
|
|
||||||
|
```julia
|
||||||
|
# Upload handler - uploads data to file server and returns URL
|
||||||
|
# The handler is passed to smartsend as fileserver_upload_handler parameter
|
||||||
|
# It receives: (file_server_url::String, dataname::String, data::Vector{UInt8})
|
||||||
|
# Returns: Dict{String, Any} with keys: "status", "uploadid", "fileid", "url"
|
||||||
|
fileserver_upload_handler(file_server_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
||||||
|
|
||||||
|
# Download handler - fetches data from file server URL with exponential backoff
|
||||||
|
# The handler is passed to smartreceive as fileserver_download_handler parameter
|
||||||
|
# It receives: (url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)
|
||||||
|
# Returns: Vector{UInt8} (the downloaded data)
|
||||||
|
fileserver_download_handler(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)::Vector{UInt8}
|
||||||
|
```
|
||||||
|
|
||||||
|
This design allows the system to support multiple file server backends without changing the core messaging logic.
|
||||||
|
|
||||||
|
### Multi-Payload Support (Standard API)
|
||||||
|
|
||||||
|
The system uses a **standardized list-of-tuples format** for all payload operations. **Even when sending a single payload, the user must wrap it in a list.**
|
||||||
|
|
||||||
**API Standard:**
|
**API Standard:**
|
||||||
```julia
|
```julia
|
||||||
# Input format for smartsend (always a list of tuples with type info)
|
# Input format for smartsend (always a list of tuples with type info)
|
||||||
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||||
|
|
||||||
# Output format for smartreceive (returns envelope dictionary with payloads field)
|
# Output format for smartreceive (returns a dictionary with payloads field containing list of tuples)
|
||||||
# Returns: Dict with envelope metadata and payloads field containing list of tuples
|
# Returns: Dict with envelope metadata and payloads field containing Vector{Tuple{String, Any, String}}
|
||||||
# {
|
# {
|
||||||
# "correlationId": "...",
|
# "correlation_id": "...",
|
||||||
# "msgId": "...",
|
# "msg_id": "...",
|
||||||
# "timestamp": "...",
|
# "timestamp": "...",
|
||||||
# "sendTo": "...",
|
# "send_to": "...",
|
||||||
# "msgPurpose": "...",
|
# "msg_purpose": "...",
|
||||||
# "senderName": "...",
|
# "sender_name": "...",
|
||||||
# "senderId": "...",
|
# "sender_id": "...",
|
||||||
# "receiverName": "...",
|
# "receiver_name": "...",
|
||||||
# "receiverId": "...",
|
# "receiver_id": "...",
|
||||||
# "replyTo": "...",
|
# "reply_to": "...",
|
||||||
# "replyToMsgId": "...",
|
# "reply_to_msg_id": "...",
|
||||||
# "brokerURL": "...",
|
# "broker_url": "...",
|
||||||
# "metadata": {...},
|
# "metadata": {...},
|
||||||
# "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
# "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||||
# }
|
# }
|
||||||
```
|
```
|
||||||
|
|
||||||
Where `type` can be: `"text"`, `"dictionary"`, `"table"`, `"image"`, `"audio"`, `"video"`, `"binary"`
|
**Supported Types:**
|
||||||
|
- `"text"` - Plain text
|
||||||
|
- `"dictionary"` - JSON-serializable dictionaries (Dict, NamedTuple)
|
||||||
|
- `"table"` - Tabular data (DataFrame, array of structs)
|
||||||
|
- `"image"` - Image data (Bitmap, PNG/JPG bytes)
|
||||||
|
- `"audio"` - Audio data (WAV, MP3 bytes)
|
||||||
|
- `"video"` - Video data (MP4, AVI bytes)
|
||||||
|
- `"binary"` - Generic binary data (Vector{UInt8})
|
||||||
|
|
||||||
|
This design allows per-payload type specification, enabling **mixed-content messages** where different payloads can use different serialization formats in a single message.
|
||||||
|
|
||||||
**Examples:**
|
**Examples:**
|
||||||
```julia
|
```julia
|
||||||
# Single payload - still wrapped in a list (type is required as third element)
|
# Single payload - still wrapped in a list
|
||||||
smartsend("/test", [(dataname1, data1, "text")], ...)
|
smartsend(
|
||||||
|
"/test",
|
||||||
|
[("dataname1", data1, "dictionary")], # List with one tuple (data, type)
|
||||||
|
broker_url="nats://localhost:4222",
|
||||||
|
fileserver_upload_handler=plik_oneshot_upload
|
||||||
|
)
|
||||||
|
|
||||||
# Multiple payloads in one message (each payload has its own type)
|
# Multiple payloads in one message with different types
|
||||||
smartsend("/test", [(dataname1, data1, "dictionary"), (dataname2, data2, "table")], ...)
|
smartsend(
|
||||||
|
"/test",
|
||||||
|
[("dataname1", data1, "dictionary"), ("dataname2", data2, "table")],
|
||||||
|
broker_url="nats://localhost:4222",
|
||||||
|
fileserver_upload_handler=plik_oneshot_upload
|
||||||
|
)
|
||||||
|
|
||||||
|
# Mixed content (e.g., chat with text, image, audio)
|
||||||
|
smartsend(
|
||||||
|
"/chat",
|
||||||
|
[
|
||||||
|
("message_text", "Hello!", "text"),
|
||||||
|
("user_image", image_data, "image"),
|
||||||
|
("audio_clip", audio_data, "audio")
|
||||||
|
],
|
||||||
|
broker_url="nats://localhost:4222"
|
||||||
|
)
|
||||||
|
|
||||||
# Receive returns a dictionary envelope with all metadata and deserialized payloads
|
# Receive returns a dictionary envelope with all metadata and deserialized payloads
|
||||||
envelope = smartreceive(msg, ...)
|
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff, max_retries=5, base_delay=100, max_delay=5000)
|
||||||
# envelope["payloads"] = [(dataname1, data1, "text"), (dataname2, data2, "table"), ...]
|
# env["payloads"] = [("dataname1", data1, type1), ("dataname2", data2, type2), ...]
|
||||||
# envelope["correlationId"], envelope["msgId"], etc.
|
# env["correlation_id"], env["msg_id"], etc.
|
||||||
|
# env is a dictionary containing envelope metadata and payloads field
|
||||||
```
|
```
|
||||||
|
|
||||||
## Cross-Platform Interoperability
|
## Cross-Platform Interoperability
|
||||||
@@ -98,14 +151,14 @@ NATSBridge is designed for seamless communication between Julia, JavaScript, and
|
|||||||
# Julia sender
|
# Julia sender
|
||||||
using NATSBridge
|
using NATSBridge
|
||||||
data = [("message", "Hello from Julia!", "text")]
|
data = [("message", "Hello from Julia!", "text")]
|
||||||
smartsend("/cross_platform", data, nats_url="nats://localhost:4222")
|
smartsend("/cross_platform", data, broker_url="nats://localhost:4222")
|
||||||
```
|
```
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
// JavaScript receiver
|
// JavaScript receiver
|
||||||
const { smartreceive } = require('./src/NATSBridge');
|
const { smartreceive } = require('./src/NATSBridge');
|
||||||
const envelope = await smartreceive(msg);
|
const env = await smartreceive(msg);
|
||||||
// envelope.payloads[0].data === "Hello from Julia!"
|
// env.payloads[0].data === "Hello from Julia!"
|
||||||
```
|
```
|
||||||
|
|
||||||
```python
|
```python
|
||||||
@@ -146,15 +199,31 @@ All three implementations (Julia, JavaScript, Python/Micropython) follow the sam
|
|||||||
└─────────────────┘ └─────────────────┘
|
└─────────────────┘ └─────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
## Files
|
## smartsend Return Value
|
||||||
|
|
||||||
|
The `smartsend` function now returns a tuple containing both the envelope object and the JSON string representation:
|
||||||
|
|
||||||
|
```julia
|
||||||
|
env, env_json_str = smartsend(...)
|
||||||
|
# env::msg_envelope_v1 - The envelope object with all metadata and payloads
|
||||||
|
# env_json_str::String - JSON string for publishing to NATS
|
||||||
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `is_publish::Bool = true` - When `true` (default), the message is automatically published to NATS. When `false`, the function returns the envelope and JSON string without publishing, allowing manual publishing via NATS request-reply pattern.
|
||||||
|
|
||||||
|
This enables two use cases:
|
||||||
|
1. **Programmatic envelope access**: Access envelope fields directly via the `env` object
|
||||||
|
2. **Direct JSON publishing**: Publish the JSON string directly using NATS request-reply pattern
|
||||||
|
|
||||||
### Julia Module: [`src/NATSBridge.jl`](../src/NATSBridge.jl)
|
### Julia Module: [`src/NATSBridge.jl`](../src/NATSBridge.jl)
|
||||||
|
|
||||||
The Julia implementation provides:
|
The Julia implementation provides:
|
||||||
|
|
||||||
- **[`MessageEnvelope`](src/NATSBridge.jl)**: Struct for the unified JSON envelope
|
- **[`msg_envelope_v1`](src/NATSBridge.jl)**: Struct for the unified JSON envelope
|
||||||
- **[`SmartSend()`](src/NATSBridge.jl)**: Handles transport selection based on payload size
|
- **[`msg_payload_v1`](src/NATSBridge.jl)**: Struct for individual payload representation
|
||||||
- **[`SmartReceive()`](src/NATSBridge.jl)**: Handles both direct and link transport
|
- **[`smartsend()`](src/NATSBridge.jl)**: Handles transport selection based on payload size
|
||||||
|
- **[`smartreceive()`](src/NATSBridge.jl)**: Handles both direct and link transport
|
||||||
|
|
||||||
### JavaScript Module: [`src/NATSBridge.js`](../src/NATSBridge.js)
|
### JavaScript Module: [`src/NATSBridge.js`](../src/NATSBridge.js)
|
||||||
|
|
||||||
@@ -247,7 +316,53 @@ node test/scenario3_julia_to_julia.js
|
|||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
### Scenario 0: Basic Multi-Payload Example
|
### Scenario 1: Command & Control (Small Dictionary)
|
||||||
|
|
||||||
|
**Focus:** Sending small dictionary configurations across platforms. This is the simplest use case for command and control scenarios.
|
||||||
|
|
||||||
|
**Julia (Sender/Receiver):**
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
# Subscribe to control subject
|
||||||
|
# Parse JSON envelope
|
||||||
|
# Execute simulation with parameters
|
||||||
|
# Send acknowledgment
|
||||||
|
```
|
||||||
|
|
||||||
|
**JavaScript (Sender/Receiver):**
|
||||||
|
```javascript
|
||||||
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
// Create small dictionary config
|
||||||
|
// Send via smartsend with type="dictionary"
|
||||||
|
const config = {
|
||||||
|
step_size: 0.01,
|
||||||
|
iterations: 1000,
|
||||||
|
threshold: 0.5
|
||||||
|
};
|
||||||
|
|
||||||
|
await smartsend("control", [
|
||||||
|
{ dataname: "config", data: config, type: "dictionary" }
|
||||||
|
]);
|
||||||
|
```
|
||||||
|
|
||||||
|
**Python/Micropython (Sender/Receiver):**
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
|
# Create small dictionary config
|
||||||
|
# Send via smartsend with type="dictionary"
|
||||||
|
config = {
|
||||||
|
"step_size": 0.01,
|
||||||
|
"iterations": 1000,
|
||||||
|
"threshold": 0.5
|
||||||
|
}
|
||||||
|
|
||||||
|
smartsend("control", [("config", config, "dictionary")])
|
||||||
|
```
|
||||||
|
|
||||||
|
### Basic Multi-Payload Example
|
||||||
|
|
||||||
#### Python/Micropython (Sender)
|
#### Python/Micropython (Sender)
|
||||||
```python
|
```python
|
||||||
@@ -262,16 +377,16 @@ smartsend(
|
|||||||
)
|
)
|
||||||
|
|
||||||
# Even single payload must be wrapped in a list with type
|
# Even single payload must be wrapped in a list with type
|
||||||
smartsend("/test", [("single_data", mydata, "dictionary")])
|
smartsend("/test", [("single_data", mydata, "dictionary")], nats_url="nats://localhost:4222")
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Python/Micropython (Receiver)
|
#### Python/Micropython (Receiver)
|
||||||
```python
|
```python
|
||||||
from nats_bridge import smartreceive
|
from nats_bridge import smartreceive
|
||||||
|
|
||||||
# Receive returns a list of (dataname, data, type) tuples
|
# Receive returns a dictionary with envelope metadata and payloads field
|
||||||
payloads = smartreceive(msg)
|
env = smartreceive(msg)
|
||||||
# payloads = [(dataname1, data1, "dictionary"), (dataname2, data2, "table"), ...]
|
# env["payloads"] = [(dataname1, data1, "dictionary"), (dataname2, data2, "table"), ...]
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript (Sender)
|
#### JavaScript (Sender)
|
||||||
@@ -315,18 +430,18 @@ const nc = await connect({ servers: ['nats://localhost:4222'] });
|
|||||||
const sub = nc.subscribe("control");
|
const sub = nc.subscribe("control");
|
||||||
|
|
||||||
for await (const msg of sub) {
|
for await (const msg of sub) {
|
||||||
const envelope = await smartreceive(msg);
|
const env = await smartreceive(msg);
|
||||||
|
|
||||||
// Process the payloads from the envelope
|
// Process the payloads from the envelope
|
||||||
for (const payload of envelope.payloads) {
|
for (const payload of env.payloads) {
|
||||||
const { dataname, data, type } = payload;
|
const { dataname, data, type } = payload;
|
||||||
console.log(`Received ${dataname} of type ${type}`);
|
console.log(`Received ${dataname} of type ${type}`);
|
||||||
console.log(`Data: ${JSON.stringify(data)}`);
|
console.log(`Data: ${JSON.stringify(data)}`);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Also access envelope metadata
|
// Also access envelope metadata
|
||||||
console.log(`Correlation ID: ${envelope.correlationId}`);
|
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||||
console.log(`Message ID: ${envelope.msgId}`);
|
console.log(`Message ID: ${env.msg_id}`);
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -344,19 +459,21 @@ df = DataFrame(
|
|||||||
category = rand(["A", "B", "C"], 10_000_000)
|
category = rand(["A", "B", "C"], 10_000_000)
|
||||||
)
|
)
|
||||||
|
|
||||||
# Send via SmartSend - wrapped in a list (type is part of each tuple)
|
# Send via smartsend - wrapped in a list (type is part of each tuple)
|
||||||
await SmartSend("analysis_results", [("table_data", df, "table")]);
|
env, env_json_str = smartsend("analysis_results", [("table_data", df, "table")], broker_url="nats://localhost:4222")
|
||||||
|
# env: msg_envelope_v1 object with all metadata and payloads
|
||||||
|
# env_json_str: JSON string representation of the envelope for publishing
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript (Receiver)
|
#### JavaScript (Receiver)
|
||||||
```javascript
|
```javascript
|
||||||
const { smartreceive } = require('./src/NATSBridge');
|
const { smartreceive } = require('./src/NATSBridge');
|
||||||
|
|
||||||
const envelope = await smartreceive(msg);
|
const env = await smartreceive(msg);
|
||||||
|
|
||||||
// Use table data from the payloads field
|
// Use table data from the payloads field
|
||||||
// Note: Tables are sent as arrays of objects in JavaScript
|
// Note: Tables are sent as arrays of objects in JavaScript
|
||||||
const table = envelope.payloads;
|
const table = env.payloads;
|
||||||
```
|
```
|
||||||
|
|
||||||
### Scenario 3: Live Binary Processing
|
### Scenario 3: Live Binary Processing
|
||||||
@@ -406,10 +523,10 @@ from nats_bridge import smartreceive
|
|||||||
|
|
||||||
# Receive binary data
|
# Receive binary data
|
||||||
def process_binary(msg):
|
def process_binary(msg):
|
||||||
envelope = smartreceive(msg)
|
env = smartreceive(msg)
|
||||||
|
|
||||||
# Process the binary data from envelope.payloads
|
# Process the binary data from env.payloads
|
||||||
for dataname, data, type in envelope["payloads"]:
|
for dataname, data, type in env["payloads"]:
|
||||||
if type == "binary":
|
if type == "binary":
|
||||||
# data is bytes
|
# data is bytes
|
||||||
print(f"Received binary data: {dataname}, size: {len(data)}")
|
print(f"Received binary data: {dataname}, size: {len(data)}")
|
||||||
@@ -422,10 +539,10 @@ const { smartreceive } = require('./src/NATSBridge');
|
|||||||
|
|
||||||
// Receive binary data
|
// Receive binary data
|
||||||
function process_binary(msg) {
|
function process_binary(msg) {
|
||||||
const envelope = await smartreceive(msg);
|
const env = await smartreceive(msg);
|
||||||
|
|
||||||
// Process the binary data from envelope.payloads
|
// Process the binary data from env.payloads
|
||||||
for (const payload of envelope.payloads) {
|
for (const payload of env.payloads) {
|
||||||
if (payload.type === "binary") {
|
if (payload.type === "binary") {
|
||||||
// data is an ArrayBuffer or Uint8Array
|
// data is an ArrayBuffer or Uint8Array
|
||||||
console.log(`Received binary data: ${payload.dataname}, size: ${payload.data.length}`);
|
console.log(`Received binary data: ${payload.dataname}, size: ${payload.data.length}`);
|
||||||
@@ -444,7 +561,7 @@ using NATSBridge
|
|||||||
function publish_health_status(nats_url)
|
function publish_health_status(nats_url)
|
||||||
# Send status wrapped in a list (type is part of each tuple)
|
# Send status wrapped in a list (type is part of each tuple)
|
||||||
status = Dict("cpu" => rand(), "memory" => rand())
|
status = Dict("cpu" => rand(), "memory" => rand())
|
||||||
smartsend("health", [("status", status, "dictionary")], nats_url=nats_url)
|
smartsend("health", [("status", status, "dictionary")], broker_url=nats_url)
|
||||||
sleep(5) # Every 5 seconds
|
sleep(5) # Every 5 seconds
|
||||||
end
|
end
|
||||||
```
|
```
|
||||||
@@ -466,8 +583,8 @@ const consumer = await js.pullSubscribe("health", {
|
|||||||
|
|
||||||
// Process historical and real-time messages
|
// Process historical and real-time messages
|
||||||
for await (const msg of consumer) {
|
for await (const msg of consumer) {
|
||||||
const envelope = await smartreceive(msg);
|
const env = await smartreceive(msg);
|
||||||
// envelope.payloads contains the list of payloads
|
// env.payloads contains the list of payloads
|
||||||
// Each payload has: dataname, data, type
|
// Each payload has: dataname, data, type
|
||||||
msg.ack();
|
msg.ack();
|
||||||
}
|
}
|
||||||
@@ -484,10 +601,10 @@ import json
|
|||||||
|
|
||||||
# Device configuration handler
|
# Device configuration handler
|
||||||
def handle_device_config(msg):
|
def handle_device_config(msg):
|
||||||
envelope = smartreceive(msg)
|
env = smartreceive(msg)
|
||||||
|
|
||||||
# Process configuration from payloads
|
# Process configuration from payloads
|
||||||
for dataname, data, type in envelope["payloads"]:
|
for dataname, data, type in env["payloads"]:
|
||||||
if type == "dictionary":
|
if type == "dictionary":
|
||||||
print(f"Received configuration: {data}")
|
print(f"Received configuration: {data}")
|
||||||
# Apply configuration to device
|
# Apply configuration to device
|
||||||
@@ -506,7 +623,7 @@ def handle_device_config(msg):
|
|||||||
"device/response",
|
"device/response",
|
||||||
[("config", config, "dictionary")],
|
[("config", config, "dictionary")],
|
||||||
nats_url="nats://localhost:4222",
|
nats_url="nats://localhost:4222",
|
||||||
reply_to=envelope.get("replyTo")
|
reply_to=env.get("reply_to")
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -566,11 +683,11 @@ smartsend(
|
|||||||
const { smartreceive, smartsend } = require('./src/NATSBridge');
|
const { smartreceive, smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
// Receive NATS message with direct transport
|
// Receive NATS message with direct transport
|
||||||
const envelope = await smartreceive(msg);
|
const env = await smartreceive(msg);
|
||||||
|
|
||||||
// Decode Base64 payload (for direct transport)
|
// Decode Base64 payload (for direct transport)
|
||||||
// For tables, data is in envelope.payloads
|
// For tables, data is in env.payloads
|
||||||
const table = envelope.payloads; // Array of objects
|
const table = env.payloads; // Array of objects
|
||||||
|
|
||||||
// User makes selection
|
// User makes selection
|
||||||
const selection = uiComponent.getSelectedOption();
|
const selection = uiComponent.getSelectedOption();
|
||||||
@@ -619,7 +736,7 @@ chat_message = [
|
|||||||
smartsend(
|
smartsend(
|
||||||
"chat.room123",
|
"chat.room123",
|
||||||
chat_message,
|
chat_message,
|
||||||
nats_url="nats://localhost:4222",
|
broker_url="nats://localhost:4222",
|
||||||
msg_purpose="chat",
|
msg_purpose="chat",
|
||||||
reply_to="chat.room123.responses"
|
reply_to="chat.room123.responses"
|
||||||
)
|
)
|
||||||
@@ -667,7 +784,7 @@ await smartsend("chat.room123", message);
|
|||||||
|
|
||||||
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components.
|
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components.
|
||||||
|
|
||||||
**Implementation Note:** The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. See the standard API format in Section 1: `msgEnvelope_v1` supports `AbstractArray{msgPayload_v1}` for multiple payloads.
|
**Implementation Note:** The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. See the standard API format in Section 1: `msg_envelope_v1` supports `Vector{msg_payload_v1}` for multiple payloads.
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
@@ -683,19 +800,19 @@ await smartsend("chat.room123", message);
|
|||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"correlationId": "uuid-v4-string",
|
"correlation_id": "uuid-v4-string",
|
||||||
"msgId": "uuid-v4-string",
|
"msg_id": "uuid-v4-string",
|
||||||
"timestamp": "2024-01-15T10:30:00Z",
|
"timestamp": "2024-01-15T10:30:00Z",
|
||||||
|
|
||||||
"sendTo": "topic/subject",
|
"send_to": "topic/subject",
|
||||||
"msgPurpose": "ACK | NACK | updateStatus | shutdown | chat",
|
"msg_purpose": "ACK | NACK | updateStatus | shutdown | chat",
|
||||||
"senderName": "agent-wine-web-frontend",
|
"sender_name": "agent-wine-web-frontend",
|
||||||
"senderId": "uuid4",
|
"sender_id": "uuid4",
|
||||||
"receiverName": "agent-backend",
|
"receiver_name": "agent-backend",
|
||||||
"receiverId": "uuid4",
|
"receiver_id": "uuid4",
|
||||||
"replyTo": "topic",
|
"reply_to": "topic",
|
||||||
"replyToMsgId": "uuid4",
|
"reply_to_msg_id": "uuid4",
|
||||||
"BrokerURL": "nats://localhost:4222",
|
"broker_url": "nats://localhost:4222",
|
||||||
|
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"content_type": "application/octet-stream",
|
"content_type": "application/octet-stream",
|
||||||
@@ -706,7 +823,7 @@ await smartsend("chat.room123", message);
|
|||||||
{
|
{
|
||||||
"id": "uuid4",
|
"id": "uuid4",
|
||||||
"dataname": "login_image",
|
"dataname": "login_image",
|
||||||
"type": "image",
|
"payload_type": "image",
|
||||||
"transport": "direct",
|
"transport": "direct",
|
||||||
"encoding": "base64",
|
"encoding": "base64",
|
||||||
"size": 15433,
|
"size": 15433,
|
||||||
|
|||||||
@@ -107,10 +107,15 @@ python3 -m http.server 8080 --directory /tmp/fileserver
|
|||||||
```python
|
```python
|
||||||
from nats_bridge import smartsend
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
# Send a text message
|
# Send a text message (is_publish=True by default)
|
||||||
data = [("message", "Hello World", "text")]
|
data = [("message", "Hello World", "text")]
|
||||||
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
env, env_json_str = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
||||||
print("Message sent!")
|
print("Message sent!")
|
||||||
|
|
||||||
|
# Or use is_publish=False to get envelope and JSON without publishing
|
||||||
|
env, env_json_str = smartsend("/chat/room1", data, nats_url="nats://localhost:4222", is_publish=False)
|
||||||
|
# env: MessageEnvelope object
|
||||||
|
# env_json_str: JSON string for publishing to NATS
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript
|
#### JavaScript
|
||||||
@@ -118,12 +123,19 @@ print("Message sent!")
|
|||||||
```javascript
|
```javascript
|
||||||
const { smartsend } = require('./src/NATSBridge');
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
// Send a text message
|
// Send a text message (isPublish=true by default)
|
||||||
await smartsend("/chat/room1", [
|
await smartsend("/chat/room1", [
|
||||||
{ dataname: "message", data: "Hello World", type: "text" }
|
{ dataname: "message", data: "Hello World", type: "text" }
|
||||||
], { natsUrl: "nats://localhost:4222" });
|
], { natsUrl: "nats://localhost:4222" });
|
||||||
|
|
||||||
console.log("Message sent!");
|
console.log("Message sent!");
|
||||||
|
|
||||||
|
// Or use isPublish=false to get envelope and JSON without publishing
|
||||||
|
const { env, env_json_str } = await smartsend("/chat/room1", [
|
||||||
|
{ dataname: "message", data: "Hello World", type: "text" }
|
||||||
|
], { natsUrl: "nats://localhost:4222", isPublish: false });
|
||||||
|
// env: MessageEnvelope object
|
||||||
|
// env_json_str: JSON string for publishing to NATS
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Julia
|
#### Julia
|
||||||
@@ -133,7 +145,9 @@ using NATSBridge
|
|||||||
|
|
||||||
# Send a text message
|
# Send a text message
|
||||||
data = [("message", "Hello World", "text")]
|
data = [("message", "Hello World", "text")]
|
||||||
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
env, env_json_str = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
||||||
|
# env: msgEnvelope_v1 object with all metadata and payloads
|
||||||
|
# env_json_str: JSON string representation of the envelope for publishing
|
||||||
println("Message sent!")
|
println("Message sent!")
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -145,8 +159,8 @@ println("Message sent!")
|
|||||||
from nats_bridge import smartreceive
|
from nats_bridge import smartreceive
|
||||||
|
|
||||||
# Receive and process message
|
# Receive and process message
|
||||||
envelope = smartreceive(msg)
|
env = smartreceive(msg)
|
||||||
for dataname, data, type in envelope["payloads"]:
|
for dataname, data, type in env["payloads"]:
|
||||||
print(f"Received {dataname}: {data}")
|
print(f"Received {dataname}: {data}")
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -156,8 +170,8 @@ for dataname, data, type in envelope["payloads"]:
|
|||||||
const { smartreceive } = require('./src/NATSBridge');
|
const { smartreceive } = require('./src/NATSBridge');
|
||||||
|
|
||||||
// Receive and process message
|
// Receive and process message
|
||||||
const envelope = await smartreceive(msg);
|
const env = await smartreceive(msg);
|
||||||
for (const payload of envelope.payloads) {
|
for (const payload of env.payloads) {
|
||||||
console.log(`Received ${payload.dataname}: ${payload.data}`);
|
console.log(`Received ${payload.dataname}: ${payload.data}`);
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
@@ -168,8 +182,8 @@ for (const payload of envelope.payloads) {
|
|||||||
using NATSBridge
|
using NATSBridge
|
||||||
|
|
||||||
# Receive and process message
|
# Receive and process message
|
||||||
envelope = smartreceive(msg, fileserverDownloadHandler)
|
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff)
|
||||||
for (dataname, data, type) in envelope["payloads"]
|
for (dataname, data, type) in env["payloads"]
|
||||||
println("Received $dataname: $data")
|
println("Received $dataname: $data")
|
||||||
end
|
end
|
||||||
```
|
```
|
||||||
@@ -194,7 +208,7 @@ config = {
|
|||||||
|
|
||||||
# Send as dictionary type
|
# Send as dictionary type
|
||||||
data = [("config", config, "dictionary")]
|
data = [("config", config, "dictionary")]
|
||||||
env = smartsend("/device/config", data, nats_url="nats://localhost:4222")
|
env, env_json_str = smartsend("/device/config", data, nats_url="nats://localhost:4222")
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript
|
#### JavaScript
|
||||||
@@ -208,7 +222,7 @@ const config = {
|
|||||||
update_interval: 60
|
update_interval: 60
|
||||||
};
|
};
|
||||||
|
|
||||||
await smartsend("/device/config", [
|
const { env, env_json_str } = await smartsend("/device/config", [
|
||||||
{ dataname: "config", data: config, type: "dictionary" }
|
{ dataname: "config", data: config, type: "dictionary" }
|
||||||
]);
|
]);
|
||||||
```
|
```
|
||||||
@@ -225,7 +239,7 @@ config = Dict(
|
|||||||
)
|
)
|
||||||
|
|
||||||
data = [("config", config, "dictionary")]
|
data = [("config", config, "dictionary")]
|
||||||
smartsend("/device/config", data)
|
env, env_json_str = smartsend("/device/config", data)
|
||||||
```
|
```
|
||||||
|
|
||||||
### Example 2: Sending Binary Data (Image)
|
### Example 2: Sending Binary Data (Image)
|
||||||
@@ -241,7 +255,7 @@ with open("image.png", "rb") as f:
|
|||||||
|
|
||||||
# Send as binary type
|
# Send as binary type
|
||||||
data = [("user_image", image_data, "binary")]
|
data = [("user_image", image_data, "binary")]
|
||||||
env = smartsend("/chat/image", data, nats_url="nats://localhost:4222")
|
env, env_json_str = smartsend("/chat/image", data, nats_url="nats://localhost:4222")
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript
|
#### JavaScript
|
||||||
@@ -253,7 +267,7 @@ const { smartsend } = require('./src/NATSBridge');
|
|||||||
const fs = require('fs');
|
const fs = require('fs');
|
||||||
const image_data = fs.readFileSync('image.png');
|
const image_data = fs.readFileSync('image.png');
|
||||||
|
|
||||||
await smartsend("/chat/image", [
|
const { env, env_json_str } = await smartsend("/chat/image", [
|
||||||
{ dataname: "user_image", data: image_data, type: "binary" }
|
{ dataname: "user_image", data: image_data, type: "binary" }
|
||||||
]);
|
]);
|
||||||
```
|
```
|
||||||
@@ -267,7 +281,7 @@ using NATSBridge
|
|||||||
image_data = read("image.png")
|
image_data = read("image.png")
|
||||||
|
|
||||||
data = [("user_image", image_data, "binary")]
|
data = [("user_image", image_data, "binary")]
|
||||||
smartsend("/chat/image", data)
|
env, env_json_str = smartsend("/chat/image", data)
|
||||||
```
|
```
|
||||||
|
|
||||||
### Example 3: Request-Response Pattern
|
### Example 3: Request-Response Pattern
|
||||||
@@ -279,13 +293,15 @@ from nats_bridge import smartsend
|
|||||||
|
|
||||||
# Send command with reply-to
|
# Send command with reply-to
|
||||||
data = [("command", {"action": "read_sensor"}, "dictionary")]
|
data = [("command", {"action": "read_sensor"}, "dictionary")]
|
||||||
env = smartsend(
|
env, env_json_str = smartsend(
|
||||||
"/device/command",
|
"/device/command",
|
||||||
data,
|
data,
|
||||||
nats_url="nats://localhost:4222",
|
nats_url="nats://localhost:4222",
|
||||||
reply_to="/device/response",
|
reply_to="/device/response",
|
||||||
reply_to_msg_id="cmd-001"
|
reply_to_msg_id="cmd-001"
|
||||||
)
|
)
|
||||||
|
# env: msgEnvelope_v1 object
|
||||||
|
# env_json_str: JSON string for publishing to NATS
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript (Responder)
|
#### JavaScript (Responder)
|
||||||
@@ -297,10 +313,10 @@ const { smartreceive, smartsend } = require('./src/NATSBridge');
|
|||||||
const sub = nc.subscribe("/device/command");
|
const sub = nc.subscribe("/device/command");
|
||||||
|
|
||||||
for await (const msg of sub) {
|
for await (const msg of sub) {
|
||||||
const envelope = await smartreceive(msg);
|
const env = await smartreceive(msg);
|
||||||
|
|
||||||
// Process command
|
// Process command
|
||||||
for (const payload of envelope.payloads) {
|
for (const payload of env.payloads) {
|
||||||
if (payload.dataname === "command") {
|
if (payload.dataname === "command") {
|
||||||
const command = payload.data;
|
const command = payload.data;
|
||||||
|
|
||||||
@@ -315,8 +331,8 @@ for await (const msg of sub) {
|
|||||||
await smartsend("/device/response", [
|
await smartsend("/device/response", [
|
||||||
{ dataname: "sensor_data", data: response, type: "dictionary" }
|
{ dataname: "sensor_data", data: response, type: "dictionary" }
|
||||||
], {
|
], {
|
||||||
reply_to: envelope.replyTo,
|
reply_to: env.replyTo,
|
||||||
reply_to_msg_id: envelope.msgId
|
reply_to_msg_id: env.msgId
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -342,7 +358,7 @@ import os
|
|||||||
large_data = os.urandom(2_000_000) # 2MB of random data
|
large_data = os.urandom(2_000_000) # 2MB of random data
|
||||||
|
|
||||||
# Send with file server URL
|
# Send with file server URL
|
||||||
env = smartsend(
|
env, env_json_str = smartsend(
|
||||||
"/data/large",
|
"/data/large",
|
||||||
[("large_file", large_data, "binary")],
|
[("large_file", large_data, "binary")],
|
||||||
nats_url="nats://localhost:4222",
|
nats_url="nats://localhost:4222",
|
||||||
@@ -364,7 +380,7 @@ const largeData = new ArrayBuffer(2_000_000);
|
|||||||
const view = new Uint8Array(largeData);
|
const view = new Uint8Array(largeData);
|
||||||
view.fill(42); // Fill with some data
|
view.fill(42); // Fill with some data
|
||||||
|
|
||||||
await smartsend("/data/large", [
|
const { env, env_json_str } = await smartsend("/data/large", [
|
||||||
{ dataname: "large_file", data: largeData, type: "binary" }
|
{ dataname: "large_file", data: largeData, type: "binary" }
|
||||||
], {
|
], {
|
||||||
fileserverUrl: "http://localhost:8080",
|
fileserverUrl: "http://localhost:8080",
|
||||||
@@ -380,7 +396,7 @@ using NATSBridge
|
|||||||
# Create large data (> 1MB)
|
# Create large data (> 1MB)
|
||||||
large_data = rand(UInt8, 2_000_000)
|
large_data = rand(UInt8, 2_000_000)
|
||||||
|
|
||||||
env = smartsend(
|
env, env_json_str = smartsend(
|
||||||
"/data/large",
|
"/data/large",
|
||||||
[("large_file", large_data, "binary")],
|
[("large_file", large_data, "binary")],
|
||||||
fileserver_url="http://localhost:8080"
|
fileserver_url="http://localhost:8080"
|
||||||
@@ -409,7 +425,7 @@ data = [
|
|||||||
("user_avatar", image_data, "image")
|
("user_avatar", image_data, "image")
|
||||||
]
|
]
|
||||||
|
|
||||||
env = smartsend("/chat/mixed", data, nats_url="nats://localhost:4222")
|
env, env_json_str = smartsend("/chat/mixed", data, nats_url="nats://localhost:4222")
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript
|
#### JavaScript
|
||||||
@@ -419,7 +435,7 @@ const { smartsend } = require('./src/NATSBridge');
|
|||||||
|
|
||||||
const fs = require('fs');
|
const fs = require('fs');
|
||||||
|
|
||||||
await smartsend("/chat/mixed", [
|
const { env, env_json_str } = await smartsend("/chat/mixed", [
|
||||||
{
|
{
|
||||||
dataname: "message_text",
|
dataname: "message_text",
|
||||||
data: "Hello with image!",
|
data: "Hello with image!",
|
||||||
@@ -445,7 +461,7 @@ data = [
|
|||||||
("user_avatar", image_data, "image")
|
("user_avatar", image_data, "image")
|
||||||
]
|
]
|
||||||
|
|
||||||
smartsend("/chat/mixed", data)
|
env, env_json_str = smartsend("/chat/mixed", data)
|
||||||
```
|
```
|
||||||
|
|
||||||
### Example 6: Table Data (Arrow IPC)
|
### Example 6: Table Data (Arrow IPC)
|
||||||
@@ -467,7 +483,7 @@ df = pd.DataFrame({
|
|||||||
|
|
||||||
# Send as table type
|
# Send as table type
|
||||||
data = [("students", df, "table")]
|
data = [("students", df, "table")]
|
||||||
env = smartsend("/data/students", data, nats_url="nats://localhost:4222")
|
env, env_json_str = smartsend("/data/students", data, nats_url="nats://localhost:4222")
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Julia
|
#### Julia
|
||||||
@@ -484,7 +500,7 @@ df = DataFrame(
|
|||||||
)
|
)
|
||||||
|
|
||||||
data = [("students", df, "table")]
|
data = [("students", df, "table")]
|
||||||
smartsend("/data/students", data)
|
env, env_json_str = smartsend("/data/students", data)
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -503,7 +519,7 @@ using NATSBridge
|
|||||||
# Send dictionary from Julia to JavaScript
|
# Send dictionary from Julia to JavaScript
|
||||||
config = Dict("step_size" => 0.01, "iterations" => 1000)
|
config = Dict("step_size" => 0.01, "iterations" => 1000)
|
||||||
data = [("config", config, "dictionary")]
|
data = [("config", config, "dictionary")]
|
||||||
smartsend("/analysis/config", data, nats_url="nats://localhost:4222")
|
env, env_json_str = smartsend("/analysis/config", data, nats_url="nats://localhost:4222")
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript Receiver
|
#### JavaScript Receiver
|
||||||
@@ -512,8 +528,8 @@ smartsend("/analysis/config", data, nats_url="nats://localhost:4222")
|
|||||||
const { smartreceive } = require('./src/NATSBridge');
|
const { smartreceive } = require('./src/NATSBridge');
|
||||||
|
|
||||||
// Receive dictionary from Julia
|
// Receive dictionary from Julia
|
||||||
const envelope = await smartreceive(msg);
|
const env = await smartreceive(msg);
|
||||||
for (const payload of envelope.payloads) {
|
for (const payload of env.payloads) {
|
||||||
if (payload.type === "dictionary") {
|
if (payload.type === "dictionary") {
|
||||||
console.log("Received config:", payload.data);
|
console.log("Received config:", payload.data);
|
||||||
// payload.data = { step_size: 0.01, iterations: 1000 }
|
// payload.data = { step_size: 0.01, iterations: 1000 }
|
||||||
@@ -528,7 +544,7 @@ for (const payload of envelope.payloads) {
|
|||||||
```javascript
|
```javascript
|
||||||
const { smartsend } = require('./src/NATSBridge');
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
await smartsend("/data/transfer", [
|
const { env, env_json_str } = await smartsend("/data/transfer", [
|
||||||
{ dataname: "message", data: "Hello from JS!", type: "text" }
|
{ dataname: "message", data: "Hello from JS!", type: "text" }
|
||||||
]);
|
]);
|
||||||
```
|
```
|
||||||
@@ -538,8 +554,8 @@ await smartsend("/data/transfer", [
|
|||||||
```python
|
```python
|
||||||
from nats_bridge import smartreceive
|
from nats_bridge import smartreceive
|
||||||
|
|
||||||
envelope = smartreceive(msg)
|
env = smartreceive(msg)
|
||||||
for dataname, data, type in envelope["payloads"]:
|
for dataname, data, type in env["payloads"]:
|
||||||
if type == "text":
|
if type == "text":
|
||||||
print(f"Received from JS: {data}")
|
print(f"Received from JS: {data}")
|
||||||
```
|
```
|
||||||
@@ -552,7 +568,7 @@ for dataname, data, type in envelope["payloads"]:
|
|||||||
from nats_bridge import smartsend
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
data = [("message", "Hello from Python!", "text")]
|
data = [("message", "Hello from Python!", "text")]
|
||||||
smartsend("/chat/python", data)
|
env, env_json_str = smartsend("/chat/python", data)
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Julia Receiver
|
#### Julia Receiver
|
||||||
@@ -560,8 +576,8 @@ smartsend("/chat/python", data)
|
|||||||
```julia
|
```julia
|
||||||
using NATSBridge
|
using NATSBridge
|
||||||
|
|
||||||
envelope = smartreceive(msg, fileserverDownloadHandler)
|
env = smartreceive(msg, fileserverDownloadHandler)
|
||||||
for (dataname, data, type) in envelope["payloads"]
|
for (dataname, data, type) in env["payloads"]
|
||||||
if type == "text"
|
if type == "text"
|
||||||
println("Received from Python: $data")
|
println("Received from Python: $data")
|
||||||
end
|
end
|
||||||
|
|||||||
@@ -132,7 +132,7 @@ class ChatUI {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
await smartsend(
|
const { env, env_json_str } = await smartsend(
|
||||||
`/chat/${this.currentRoom}`,
|
`/chat/${this.currentRoom}`,
|
||||||
data,
|
data,
|
||||||
{
|
{
|
||||||
@@ -216,15 +216,15 @@ class ChatHandler {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async handleMessage(msg) {
|
async handleMessage(msg) {
|
||||||
const envelope = await smartreceive(msg, {
|
const env = await smartreceive(msg, {
|
||||||
fileserverDownloadHandler: this.downloadFile.bind(this)
|
fileserverDownloadHandler: this.downloadFile.bind(this)
|
||||||
});
|
});
|
||||||
|
|
||||||
// Extract sender info from envelope
|
// Extract sender info from envelope
|
||||||
const sender = envelope.senderName || 'Anonymous';
|
const sender = env.senderName || 'Anonymous';
|
||||||
|
|
||||||
// Process each payload
|
// Process each payload
|
||||||
for (const payload of envelope.payloads) {
|
for (const payload of env.payloads) {
|
||||||
if (payload.type === 'text') {
|
if (payload.type === 'text') {
|
||||||
this.ui.addMessage(sender, payload.data);
|
this.ui.addMessage(sender, payload.data);
|
||||||
} else if (payload.type === 'image') {
|
} else if (payload.type === 'image') {
|
||||||
@@ -304,7 +304,7 @@ class FileUploadService {
|
|||||||
type: 'binary'
|
type: 'binary'
|
||||||
}];
|
}];
|
||||||
|
|
||||||
const envelope = await smartsend(
|
const { env, env_json_str } = await smartsend(
|
||||||
`/files/${recipient}`,
|
`/files/${recipient}`,
|
||||||
data,
|
data,
|
||||||
{
|
{
|
||||||
@@ -314,7 +314,7 @@ class FileUploadService {
|
|||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
return envelope;
|
return env;
|
||||||
}
|
}
|
||||||
|
|
||||||
async uploadLargeFile(filePath, recipient) {
|
async uploadLargeFile(filePath, recipient) {
|
||||||
@@ -356,12 +356,12 @@ class FileDownloadService {
|
|||||||
|
|
||||||
async downloadFile(sender, downloadId) {
|
async downloadFile(sender, downloadId) {
|
||||||
// Subscribe to sender's file channel
|
// Subscribe to sender's file channel
|
||||||
const envelope = await smartreceive(msg, {
|
const env = await smartreceive(msg, {
|
||||||
fileserverDownloadHandler: this.fetchFromUrl.bind(this)
|
fileserverDownloadHandler: this.fetchFromUrl.bind(this)
|
||||||
});
|
});
|
||||||
|
|
||||||
// Process each payload
|
// Process each payload
|
||||||
for (const payload of envelope.payloads) {
|
for (const payload of env.payloads) {
|
||||||
if (payload.type === 'binary') {
|
if (payload.type === 'binary') {
|
||||||
const filePath = `/downloads/${payload.dataname}`;
|
const filePath = `/downloads/${payload.dataname}`;
|
||||||
fs.writeFileSync(filePath, payload.data);
|
fs.writeFileSync(filePath, payload.data);
|
||||||
@@ -422,9 +422,9 @@ async function uploadFile(config) {
|
|||||||
const fileService = new FileUploadService(config.nats_url, config.fileserver_url);
|
const fileService = new FileUploadService(config.nats_url, config.fileserver_url);
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const envelope = await fileService.uploadFile(filePath, recipient);
|
const env = await fileService.uploadFile(filePath, recipient);
|
||||||
console.log('Upload successful!');
|
console.log('Upload successful!');
|
||||||
console.log(`File ID: ${envelope.payloads[0].id}`);
|
console.log(`File ID: ${env.payloads[0].id}`);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error('Upload failed:', error.message);
|
console.error('Upload failed:', error.message);
|
||||||
}
|
}
|
||||||
@@ -514,6 +514,7 @@ class SensorSender:
|
|||||||
|
|
||||||
data = [("reading", reading.to_dict(), "dictionary")]
|
data = [("reading", reading.to_dict(), "dictionary")]
|
||||||
|
|
||||||
|
# Default: is_publish=True (automatically publishes to NATS)
|
||||||
smartsend(
|
smartsend(
|
||||||
f"/sensors/{sensor_id}",
|
f"/sensors/{sensor_id}",
|
||||||
data,
|
data,
|
||||||
@@ -521,6 +522,31 @@ class SensorSender:
|
|||||||
fileserver_url=self.fileserver_url
|
fileserver_url=self.fileserver_url
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def prepare_message_only(self, sensor_id: str, value: float, unit: str):
|
||||||
|
"""Prepare a message without publishing (is_publish=False)."""
|
||||||
|
reading = SensorReading(
|
||||||
|
sensor_id=sensor_id,
|
||||||
|
timestamp=datetime.now().isoformat(),
|
||||||
|
value=value,
|
||||||
|
unit=unit
|
||||||
|
)
|
||||||
|
|
||||||
|
data = [("reading", reading.to_dict(), "dictionary")]
|
||||||
|
|
||||||
|
# With is_publish=False, returns (env, env_json_str) without publishing
|
||||||
|
env, env_json_str = smartsend(
|
||||||
|
f"/sensors/{sensor_id}/prepare",
|
||||||
|
data,
|
||||||
|
nats_url=self.nats_url,
|
||||||
|
fileserver_url=self.fileserver_url,
|
||||||
|
is_publish=False
|
||||||
|
)
|
||||||
|
|
||||||
|
# Now you can publish manually using NATS request-reply pattern
|
||||||
|
# nc.request(subject, env_json_str, reply_to=reply_to_topic)
|
||||||
|
|
||||||
|
return env, env_json_str
|
||||||
|
|
||||||
def send_batch(self, readings: List[SensorReading]):
|
def send_batch(self, readings: List[SensorReading]):
|
||||||
batch = SensorBatch()
|
batch = SensorBatch()
|
||||||
for reading in readings:
|
for reading in readings:
|
||||||
@@ -571,9 +597,9 @@ class SensorReceiver:
|
|||||||
self.fileserver_download_handler = fileserver_download_handler
|
self.fileserver_download_handler = fileserver_download_handler
|
||||||
|
|
||||||
def process_reading(self, msg):
|
def process_reading(self, msg):
|
||||||
envelope = smartreceive(msg, self.fileserver_download_handler)
|
env = smartreceive(msg, self.fileserver_download_handler)
|
||||||
|
|
||||||
for dataname, data, data_type in envelope["payloads"]:
|
for dataname, data, data_type in env["payloads"]:
|
||||||
if data_type == "dictionary":
|
if data_type == "dictionary":
|
||||||
reading = SensorReading(
|
reading = SensorReading(
|
||||||
sensor_id=data["sensor_id"],
|
sensor_id=data["sensor_id"],
|
||||||
@@ -673,10 +699,10 @@ class DeviceBridge:
|
|||||||
# Poll for messages
|
# Poll for messages
|
||||||
msg = self._poll_for_message()
|
msg = self._poll_for_message()
|
||||||
if msg:
|
if msg:
|
||||||
envelope = smartreceive(msg)
|
env = smartreceive(msg)
|
||||||
|
|
||||||
# Process payloads
|
# Process payloads
|
||||||
for dataname, data, data_type in envelope["payloads"]:
|
for dataname, data, data_type in env["payloads"]:
|
||||||
if dataname == "command":
|
if dataname == "command":
|
||||||
callback(data)
|
callback(data)
|
||||||
|
|
||||||
@@ -772,9 +798,9 @@ class DashboardServer:
|
|||||||
|
|
||||||
def receive_selection(self, callback):
|
def receive_selection(self, callback):
|
||||||
def handler(msg):
|
def handler(msg):
|
||||||
envelope = smartreceive(msg)
|
env = smartreceive(msg)
|
||||||
|
|
||||||
for dataname, data, data_type in envelope["payloads"]:
|
for dataname, data, data_type in env["payloads"]:
|
||||||
if data_type == "dictionary":
|
if data_type == "dictionary":
|
||||||
callback(data)
|
callback(data)
|
||||||
|
|
||||||
@@ -807,7 +833,7 @@ class DashboardUI {
|
|||||||
|
|
||||||
async refreshData() {
|
async refreshData() {
|
||||||
// Request fresh data
|
// Request fresh data
|
||||||
await smartsend("/dashboard/request", [
|
const { env, env_json_str } = await smartsend("/dashboard/request", [
|
||||||
{ dataname: "request", data: { type: "refresh" }, type: "dictionary" }
|
{ dataname: "request", data: { type: "refresh" }, type: "dictionary" }
|
||||||
], {
|
], {
|
||||||
fileserverUrl: window.config.fileserver_url
|
fileserverUrl: window.config.fileserver_url
|
||||||
@@ -816,12 +842,12 @@ class DashboardUI {
|
|||||||
|
|
||||||
async fetchData() {
|
async fetchData() {
|
||||||
// Subscribe to data updates
|
// Subscribe to data updates
|
||||||
const envelope = await smartreceive(msg, {
|
const env = await smartreceive(msg, {
|
||||||
fileserverDownloadHandler: this.fetchFromUrl.bind(this)
|
fileserverDownloadHandler: this.fetchFromUrl.bind(this)
|
||||||
});
|
});
|
||||||
|
|
||||||
// Process table data
|
// Process table data
|
||||||
for (const payload of envelope.payloads) {
|
for (const payload of env.payloads) {
|
||||||
if (payload.type === 'table') {
|
if (payload.type === 'table') {
|
||||||
// Deserialize Arrow IPC
|
// Deserialize Arrow IPC
|
||||||
this.data = this.deserializeArrow(payload.data);
|
this.data = this.deserializeArrow(payload.data);
|
||||||
|
|||||||
14
plik_fileserver/docker-compose.yml
Normal file
14
plik_fileserver/docker-compose.yml
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
services:
|
||||||
|
plik:
|
||||||
|
image: rootgg/plik:latest
|
||||||
|
container_name: plik-server
|
||||||
|
restart: unless-stopped
|
||||||
|
ports:
|
||||||
|
- "8080:8080"
|
||||||
|
volumes:
|
||||||
|
# # Mount the config file (created below)
|
||||||
|
# - ./plikd.cfg:/home/plik/server/plikd.cfg
|
||||||
|
# Mount local folder for uploads and database
|
||||||
|
- ./plik-data:/data
|
||||||
|
# Set user to match your host UID to avoid permission issues
|
||||||
|
user: "1000:1000"
|
||||||
@@ -12,10 +12,10 @@
|
|||||||
#
|
#
|
||||||
# ```jldoctest
|
# ```jldoctest
|
||||||
# # Upload handler - uploads data to file server and returns URL
|
# # Upload handler - uploads data to file server and returns URL
|
||||||
# fileserverUploadHandler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
# fileserver_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
||||||
#
|
#
|
||||||
# # Download handler - fetches data from file server URL with exponential backoff
|
# # Download handler - fetches data from file server URL with exponential backoff
|
||||||
# fileserverDownloadHandler(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)::Vector{UInt8}
|
# fileserver_download_handler(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)::Vector{UInt8}
|
||||||
# ```
|
# ```
|
||||||
#
|
#
|
||||||
# Multi-Payload Support (Standard API):
|
# Multi-Payload Support (Standard API):
|
||||||
@@ -35,24 +35,23 @@
|
|||||||
|
|
||||||
module NATSBridge
|
module NATSBridge
|
||||||
|
|
||||||
using Revise
|
|
||||||
using NATS, JSON, Arrow, HTTP, UUIDs, Dates, Base64, PrettyPrinting
|
using NATS, JSON, Arrow, HTTP, UUIDs, Dates, Base64, PrettyPrinting
|
||||||
# ---------------------------------------------- 100 --------------------------------------------- #
|
# ---------------------------------------------- 100 --------------------------------------------- #
|
||||||
|
|
||||||
# Constants
|
# Constants
|
||||||
const DEFAULT_SIZE_THRESHOLD = 1_000_000 # 1MB - threshold for switching from direct to link transport
|
const DEFAULT_SIZE_THRESHOLD = 1_000_000 # 1MB - threshold for switching from direct to link transport
|
||||||
const DEFAULT_NATS_URL = "nats://localhost:4222" # Default NATS server URL
|
const DEFAULT_BROKER_URL = "nats://localhost:4222" # Default NATS server URL
|
||||||
const DEFAULT_FILESERVER_URL = "http://localhost:8080" # Default HTTP file server URL for link transport
|
const DEFAULT_FILESERVER_URL = "http://localhost:8080" # Default HTTP file server URL for link transport
|
||||||
|
|
||||||
|
|
||||||
""" msgPayload_v1 - Internal message payload structure
|
""" msg_payload_v1 - Internal message payload structure
|
||||||
This structure represents a single payload within a NATS message envelope.
|
This structure represents a single payload within a NATS message envelope.
|
||||||
It supports both direct transport (base64-encoded data) and link transport (URL-based).
|
It supports both direct transport (base64-encoded data) and link transport (URL-based).
|
||||||
|
|
||||||
# Arguments:
|
# Arguments:
|
||||||
- `id::String` - Unique identifier for this payload (e.g., "uuid4")
|
- `id::String` - Unique identifier for this payload (e.g., "uuid4")
|
||||||
- `dataname::String` - Name of the payload (e.g., "login_image")
|
- `dataname::String` - Name of the payload (e.g., "login_image")
|
||||||
- `type::String` - Payload type: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
- `payload_type::String` - Payload type: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||||
- `transport::String` - Transport method: "direct" or "link"
|
- `transport::String` - Transport method: "direct" or "link"
|
||||||
- `encoding::String` - Encoding method: "none", "json", "base64", "arrow-ipc"
|
- `encoding::String` - Encoding method: "none", "json", "base64", "arrow-ipc"
|
||||||
- `size::Integer` - Size of the payload in bytes (e.g., 15433)
|
- `size::Integer` - Size of the payload in bytes (e.g., 15433)
|
||||||
@@ -68,14 +67,14 @@ It supports both direct transport (base64-encoded data) and link transport (URL-
|
|||||||
- `metadata::Dict{String, T} = Dict{String, Any}()` - Metadata dictionary
|
- `metadata::Dict{String, T} = Dict{String, Any}()` - Metadata dictionary
|
||||||
|
|
||||||
# Return:
|
# Return:
|
||||||
- A msgPayload_v1 struct instance
|
- A msg_payload_v1 struct instance
|
||||||
|
|
||||||
# Example
|
# Example
|
||||||
```jldoctest
|
```jldoctest
|
||||||
using UUIDs
|
using UUIDs
|
||||||
|
|
||||||
# Create a direct transport payload
|
# Create a direct transport payload
|
||||||
payload = msgPayload_v1(
|
payload = msg_payload_v1(
|
||||||
"Hello World",
|
"Hello World",
|
||||||
"text";
|
"text";
|
||||||
id = string(uuid4()),
|
id = string(uuid4()),
|
||||||
@@ -87,7 +86,7 @@ payload = msgPayload_v1(
|
|||||||
)
|
)
|
||||||
|
|
||||||
# Create a link transport payload
|
# Create a link transport payload
|
||||||
payload = msgPayload_v1(
|
payload = msg_payload_v1(
|
||||||
"http://example.com/file.zip",
|
"http://example.com/file.zip",
|
||||||
"binary";
|
"binary";
|
||||||
id = string(uuid4()),
|
id = string(uuid4()),
|
||||||
@@ -98,21 +97,21 @@ payload = msgPayload_v1(
|
|||||||
)
|
)
|
||||||
```
|
```
|
||||||
"""
|
"""
|
||||||
struct msgPayload_v1
|
struct msg_payload_v1
|
||||||
id::String # id of this payload e.g. "uuid4"
|
id::String # id of this payload e.g. "uuid4"
|
||||||
dataname::String # name of this payload e.g. "login_image"
|
dataname::String # name of this payload e.g. "login_image"
|
||||||
type::String # this payload type. Can be "text | dictionary | table | image | audio | video | binary"
|
payload_type::String # this payload type. Can be "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||||
transport::String # "direct | link"
|
transport::String # transport method: "direct" or "link"
|
||||||
encoding::String # "none | json | base64 | arrow-ipc"
|
encoding::String # encoding method: "none", "json", "base64", "arrow-ipc"
|
||||||
size::Integer # data size in bytes e.g. 15433
|
size::Integer # data size in bytes e.g. 15433
|
||||||
data::Any # payload data in case of direct transport or a URL in case of link
|
data::Any # payload data in case of direct transport or a URL in case of link
|
||||||
metadata::Dict{String, Any} # Dict("checksum" => "sha256_hash", ...) This metadata is for this payload
|
metadata::Dict{String, Any} # Dict("checksum" => "sha256_hash", ...) This metadata is for this payload
|
||||||
end
|
end
|
||||||
|
|
||||||
# constructor
|
# constructor
|
||||||
function msgPayload_v1(
|
function msg_payload_v1(
|
||||||
data::Any,
|
data::Any,
|
||||||
type::String;
|
payload_type::String;
|
||||||
id::String = "",
|
id::String = "",
|
||||||
dataname::String = string(uuid4()),
|
dataname::String = string(uuid4()),
|
||||||
transport::String = "direct",
|
transport::String = "direct",
|
||||||
@@ -120,10 +119,10 @@ function msgPayload_v1(
|
|||||||
size::Integer = 0,
|
size::Integer = 0,
|
||||||
metadata::Dict{String, T} = Dict{String, Any}()
|
metadata::Dict{String, T} = Dict{String, Any}()
|
||||||
) where {T<:Any}
|
) where {T<:Any}
|
||||||
return msgPayload_v1(
|
return msg_payload_v1(
|
||||||
id,
|
id,
|
||||||
dataname,
|
dataname,
|
||||||
type,
|
payload_type,
|
||||||
transport,
|
transport,
|
||||||
encoding,
|
encoding,
|
||||||
size,
|
size,
|
||||||
@@ -133,101 +132,101 @@ function msgPayload_v1(
|
|||||||
end
|
end
|
||||||
|
|
||||||
|
|
||||||
""" msgEnvelope_v1 - Internal message envelope structure
|
""" msg_envelope_v1 - Internal message envelope structure
|
||||||
This structure represents a complete NATS message envelope containing multiple payloads
|
This structure represents a complete NATS message envelope containing multiple payloads
|
||||||
with metadata for routing, tracing, and message context.
|
with metadata for routing, tracing, and message context.
|
||||||
|
|
||||||
# Arguments:
|
# Arguments:
|
||||||
- `sendTo::String` - NATS subject/topic to publish the message to (e.g., "/agent/wine/api/v1/prompt")
|
- `send_to::String` - NATS subject/topic to publish the message to (e.g., "/agent/wine/api/v1/prompt")
|
||||||
- `payloads::AbstractArray{msgPayload_v1}` - List of payloads to include in the message
|
- `payloads::Vector{msg_payload_v1}` - List of payloads to include in the message
|
||||||
|
|
||||||
# Keyword Arguments:
|
# Keyword Arguments:
|
||||||
- `correlationId::String = ""` - Unique identifier to track messages across systems; auto-generated if empty
|
- `correlation_id::String = ""` - Unique identifier to track messages across systems; auto-generated if empty
|
||||||
- `msgId::String = ""` - Unique message identifier; auto-generated if empty
|
- `msg_id::String = ""` - Unique message identifier; auto-generated if empty
|
||||||
- `timestamp::String = string(Dates.now())` - Message publication timestamp
|
- `timestamp::String = string(Dates.now())` - Message publication timestamp
|
||||||
- `msgPurpose::String = ""` - Purpose of the message: "ACK", "NACK", "updateStatus", "shutdown", "chat", etc.
|
- `msg_purpose::String = ""` - Purpose of the message: "ACK", "NACK", "updateStatus", "shutdown", "chat", etc.
|
||||||
- `senderName::String = ""` - Name of the sender (e.g., "agent-wine-web-frontend")
|
- `sender_name::String = ""` - Name of the sender (e.g., "agent-wine-web-frontend")
|
||||||
- `senderId::String = ""` - UUID of the sender; auto-generated if empty
|
- `sender_id::String = ""` - UUID of the sender; auto-generated if empty
|
||||||
- `receiverName::String = ""` - Name of the receiver (empty string means broadcast)
|
- `receiver_name::String = ""` - Name of the receiver (empty string means broadcast)
|
||||||
- `receiverId::String = ""` - UUID of the receiver (empty string means broadcast)
|
- `receiver_id::String = ""` - UUID of the receiver (empty string means broadcast)
|
||||||
- `replyTo::String = ""` - Topic where receiver should reply (empty string if no reply expected)
|
- `reply_to::String = ""` - Topic where receiver should reply (empty string if no reply expected)
|
||||||
- `replyToMsgId::String = ""` - Message ID this message is replying to
|
- `reply_to_msg_id::String = ""` - Message ID this message is replying to
|
||||||
- `brokerURL::String = DEFAULT_NATS_URL` - NATS broker URL
|
- `broker_url::String = DEFAULT_BROKER_URL` - NATS broker URL
|
||||||
- `metadata::Dict{String, Any} = Dict{String, Any}()` - Optional message-level metadata
|
- `metadata::Dict{String, Any} = Dict{String, Any}()` - Optional message-level metadata
|
||||||
|
|
||||||
# Return:
|
# Return:
|
||||||
- A msgEnvelope_v1 struct instance
|
- A msg_envelope_v1 struct instance
|
||||||
|
|
||||||
# Example
|
# Example
|
||||||
```jldoctest
|
```jldoctest
|
||||||
using UUIDs, NATSBridge
|
using UUIDs, NATSBridge
|
||||||
|
|
||||||
# Create payloads for the message
|
# Create payloads for the message
|
||||||
payload1 = msgPayload_v1("Hello", "text"; dataname="message", transport="direct", encoding="base64")
|
payload1 = msg_payload_v1("Hello", "text"; dataname="message", transport="direct", encoding="base64")
|
||||||
payload2 = msgPayload_v1("http://example.com/file.zip", "binary"; dataname="file", transport="link")
|
payload2 = msg_payload_v1("http://example.com/file.zip", "binary"; dataname="file", transport="link")
|
||||||
|
|
||||||
# Create message envelope
|
# Create message envelope
|
||||||
env = msgEnvelope_v1(
|
env = msg_envelope_v1(
|
||||||
"my.subject",
|
"my.subject",
|
||||||
[payload1, payload2];
|
[payload1, payload2];
|
||||||
correlationId = string(uuid4()),
|
correlation_id = string(uuid4()),
|
||||||
msgPurpose = "chat",
|
msg_purpose = "chat",
|
||||||
senderName = "my-app",
|
sender_name = "my-app",
|
||||||
receiverName = "receiver-app",
|
receiver_name = "receiver-app",
|
||||||
replyTo = "reply.subject"
|
reply_to = "reply.subject"
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
"""
|
"""
|
||||||
struct msgEnvelope_v1
|
struct msg_envelope_v1
|
||||||
correlationId::String # Unique identifier to track messages across systems. Many senders can talk about the same topic.
|
correlation_id::String # Unique identifier to track messages across systems. Many senders can talk about the same topic.
|
||||||
msgId::String # this message id
|
msg_id::String # this message id
|
||||||
timestamp::String # message published timestamp. string(Dates.now())
|
timestamp::String # message published timestamp (string(Dates.now()))
|
||||||
|
|
||||||
sendTo::String # topic/subject the sender sends to e.g. "/agent/wine/api/v1/prompt"
|
send_to::String # topic/subject the sender sends to e.g. "/agent/wine/api/v1/prompt"
|
||||||
msgPurpose::String # purpose of this message e.g. "ACK | NACK | updateStatus | shutdown | ..."
|
msg_purpose::String # purpose of this message e.g. "ACK", "NACK", "updateStatus", "shutdown", ...
|
||||||
senderName::String # sender name (String) e.g. "agent-wine-web-frontend"
|
sender_name::String # sender name (String) e.g. "agent-wine-web-frontend"
|
||||||
senderId::String # sender id e.g. uuid4snakecase()
|
sender_id::String # sender id e.g. uuid4()
|
||||||
receiverName::String # msg receiver name (String) e.g. "agent-backend"
|
receiver_name::String # msg receiver name (String) e.g. "agent-backend"
|
||||||
receiverId::String # msg receiver id, nothing means everyone in the topic e.g. uuid4snakecase()
|
receiver_id::String # msg receiver id, nothing means everyone in the topic e.g. uuid4()
|
||||||
|
|
||||||
replyTo::String # sender ask receiver to reply to this topic
|
reply_to::String # sender ask receiver to reply to this topic
|
||||||
replyToMsgId::String # the message id this message is replying to
|
reply_to_msg_id::String # the message id this message is replying to
|
||||||
brokerURL::String # mqtt/NATS server address
|
broker_url::String # NATS server address
|
||||||
|
|
||||||
metadata::Dict{String, Any}
|
metadata::Dict{String, Any}
|
||||||
payloads::AbstractArray{msgPayload_v1} # multiple payload store here
|
payloads::Vector{msg_payload_v1} # multiple payload store here
|
||||||
end
|
end
|
||||||
|
|
||||||
# constructor
|
# constructor
|
||||||
function msgEnvelope_v1(
|
function msg_envelope_v1(
|
||||||
sendTo::String,
|
send_to::String,
|
||||||
payloads::AbstractArray{msgPayload_v1};
|
payloads::Vector{msg_payload_v1};
|
||||||
correlationId::String = "",
|
correlation_id::String = "",
|
||||||
msgId::String = "",
|
msg_id::String = "",
|
||||||
timestamp::String = string(Dates.now()),
|
timestamp::String = string(Dates.now()),
|
||||||
msgPurpose::String = "",
|
msg_purpose::String = "",
|
||||||
senderName::String = "",
|
sender_name::String = "",
|
||||||
senderId::String = "",
|
sender_id::String = "",
|
||||||
receiverName::String = "",
|
receiver_name::String = "",
|
||||||
receiverId::String = "",
|
receiver_id::String = "",
|
||||||
replyTo::String = "",
|
reply_to::String = "",
|
||||||
replyToMsgId::String = "",
|
reply_to_msg_id::String = "",
|
||||||
brokerURL::String = DEFAULT_NATS_URL,
|
broker_url::String = DEFAULT_BROKER_URL,
|
||||||
metadata::Dict{String, Any} = Dict{String, Any}()
|
metadata::Dict{String, Any} = Dict{String, Any}()
|
||||||
)
|
)
|
||||||
return msgEnvelope_v1(
|
return msg_envelope_v1(
|
||||||
correlationId,
|
correlation_id,
|
||||||
msgId,
|
msg_id,
|
||||||
timestamp,
|
timestamp,
|
||||||
sendTo,
|
send_to,
|
||||||
msgPurpose,
|
msg_purpose,
|
||||||
senderName,
|
sender_name,
|
||||||
senderId,
|
sender_id,
|
||||||
receiverName,
|
receiver_name,
|
||||||
receiverId,
|
receiver_id,
|
||||||
replyTo,
|
reply_to,
|
||||||
replyToMsgId,
|
reply_to_msg_id,
|
||||||
brokerURL,
|
broker_url,
|
||||||
metadata,
|
metadata,
|
||||||
payloads
|
payloads
|
||||||
)
|
)
|
||||||
@@ -235,19 +234,19 @@ end
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
""" envelope_to_json - Convert msgEnvelope_v1 to JSON string
|
""" envelope_to_json - Convert msg_envelope_v1 to JSON string
|
||||||
This function converts the msgEnvelope_v1 struct to a JSON string representation,
|
This function converts the msg_envelope_v1 struct to a JSON string representation,
|
||||||
preserving all metadata and payload information for NATS message publishing.
|
preserving all metadata and payload information for NATS message publishing.
|
||||||
|
|
||||||
# Function Workflow:
|
# Function Workflow:
|
||||||
1. Creates a dictionary with envelope metadata (correlationId, msgId, timestamp, etc.)
|
1. Creates a dictionary with envelope metadata (correlation_id, msg_id, timestamp, etc.)
|
||||||
2. Conditionally includes metadata dictionary if not empty
|
2. Conditionally includes metadata dictionary if not empty
|
||||||
3. Iterates through payloads and converts each to JSON-compatible dictionary
|
3. Iterates through payloads and converts each to JSON-compatible dictionary
|
||||||
4. Handles direct transport payloads (Base64 encoding) and link transport payloads (URL)
|
4. Handles direct transport payloads (Base64 encoding) and link transport payloads (URL)
|
||||||
5. Returns final JSON string representation
|
5. Returns final JSON string representation
|
||||||
|
|
||||||
# Arguments:
|
# Arguments:
|
||||||
- `env::msgEnvelope_v1` - The msgEnvelope_v1 struct to convert to JSON
|
- `env::msg_envelope_v1` - The msg_envelope_v1 struct to convert to JSON
|
||||||
|
|
||||||
# Return:
|
# Return:
|
||||||
- `String` - JSON string representation of the envelope
|
- `String` - JSON string representation of the envelope
|
||||||
@@ -257,27 +256,27 @@ preserving all metadata and payload information for NATS message publishing.
|
|||||||
using UUIDs
|
using UUIDs
|
||||||
|
|
||||||
# Create an envelope with payloads
|
# Create an envelope with payloads
|
||||||
payload = msgPayload_v1("Hello", "text"; dataname="msg", transport="direct", encoding="base64")
|
payload = msg_payload_v1("Hello", "text"; dataname="msg", transport="direct", encoding="base64")
|
||||||
env = msgEnvelope_v1("my.subject", [payload])
|
env = msg_envelope_v1("my.subject", [payload])
|
||||||
|
|
||||||
# Convert to JSON for publishing
|
# Convert to JSON for publishing
|
||||||
json_msg = envelope_to_json(env)
|
json_msg = envelope_to_json(env)
|
||||||
```
|
```
|
||||||
"""
|
"""
|
||||||
function envelope_to_json(env::msgEnvelope_v1)
|
function envelope_to_json(env::msg_envelope_v1)
|
||||||
obj = Dict{String, Any}(
|
obj = Dict{String, Any}(
|
||||||
"correlationId" => env.correlationId,
|
"correlation_id" => env.correlation_id,
|
||||||
"msgId" => env.msgId,
|
"msg_id" => env.msg_id,
|
||||||
"timestamp" => env.timestamp,
|
"timestamp" => env.timestamp,
|
||||||
"sendTo" => env.sendTo,
|
"send_to" => env.send_to,
|
||||||
"msgPurpose" => env.msgPurpose,
|
"msg_purpose" => env.msg_purpose,
|
||||||
"senderName" => env.senderName,
|
"sender_name" => env.sender_name,
|
||||||
"senderId" => env.senderId,
|
"sender_id" => env.sender_id,
|
||||||
"receiverName" => env.receiverName,
|
"receiver_name" => env.receiver_name,
|
||||||
"receiverId" => env.receiverId,
|
"receiver_id" => env.receiver_id,
|
||||||
"replyTo" => env.replyTo,
|
"reply_to" => env.reply_to,
|
||||||
"replyToMsgId" => env.replyToMsgId,
|
"reply_to_msg_id" => env.reply_to_msg_id,
|
||||||
"brokerURL" => env.brokerURL
|
"broker_url" => env.broker_url
|
||||||
)
|
)
|
||||||
|
|
||||||
if !isempty(env.metadata) # Only include metadata if it exists and is not empty
|
if !isempty(env.metadata) # Only include metadata if it exists and is not empty
|
||||||
@@ -291,7 +290,7 @@ function envelope_to_json(env::msgEnvelope_v1)
|
|||||||
payload_obj = Dict{String, Any}(
|
payload_obj = Dict{String, Any}(
|
||||||
"id" => payload.id,
|
"id" => payload.id,
|
||||||
"dataname" => payload.dataname,
|
"dataname" => payload.dataname,
|
||||||
"type" => payload.type,
|
"payload_type" => payload.payload_type,
|
||||||
"transport" => payload.transport,
|
"transport" => payload.transport,
|
||||||
"encoding" => payload.encoding,
|
"encoding" => payload.encoding,
|
||||||
"size" => payload.size,
|
"size" => payload.size,
|
||||||
@@ -359,20 +358,22 @@ Each payload can have a different type, enabling mixed-content messages (e.g., c
|
|||||||
1. Iterates through the list of (dataname, data, type) tuples
|
1. Iterates through the list of (dataname, data, type) tuples
|
||||||
2. For each payload: extracts the type from the tuple and serializes accordingly
|
2. For each payload: extracts the type from the tuple and serializes accordingly
|
||||||
3. Compares the serialized size against `size_threshold`
|
3. Compares the serialized size against `size_threshold`
|
||||||
4. For small payloads: encodes as Base64, constructs a "direct" msgPayload_v1
|
4. For small payloads: encodes as Base64, constructs a "direct" msg_payload_v1
|
||||||
5. For large payloads: uploads to the fileserver, constructs a "link" msgPayload_v1 with the URL
|
5. For large payloads: uploads to the fileserver, constructs a "link" msg_payload_v1 with the URL
|
||||||
|
6. Converts envelope to JSON string and optionally publishes to NATS
|
||||||
|
|
||||||
# Arguments:
|
# Arguments:
|
||||||
- `subject::String` - NATS subject to publish the message to
|
- `subject::String` - NATS subject to publish the message to
|
||||||
- `data::AbstractArray{Tuple{String, Any, String}}` - List of (dataname, data, type) tuples to send
|
- `data::AbstractArray{Tuple{String, Any, String}}` - List of (dataname, data, type) tuples to send
|
||||||
- `dataname::String` - Name of the payload
|
- `dataname::String` - Name of the payload
|
||||||
- `data::Any` - The actual data to send
|
- `data::Any` - The actual data to send
|
||||||
- `type::String` - Payload type: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
- `payload_type::String` - Payload type: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||||
- No standalone `type` parameter - type is specified per payload
|
- No standalone `type` parameter - type is specified per payload
|
||||||
|
|
||||||
# Keyword Arguments:
|
# Keyword Arguments:
|
||||||
- `nats_url::String = DEFAULT_NATS_URL` - URL of the NATS server
|
- `broker_url::String = DEFAULT_BROKER_URL` - URL of the NATS server
|
||||||
- `fileserverUploadHandler::Function = plik_oneshot_upload` - Function to handle fileserver uploads (must return Dict with "status", "uploadid", "fileid", "url" keys)
|
- `fileserver_url = DEFAULT_FILESERVER_URL` - URL of the HTTP file server for large payloads
|
||||||
|
- `fileserver_upload_handler::Function = plik_oneshot_upload` - Function to handle fileserver uploads (must return Dict with "status", "uploadid", "fileid", "url" keys)
|
||||||
- `size_threshold::Int = DEFAULT_SIZE_THRESHOLD` - Threshold in bytes separating direct vs link transport
|
- `size_threshold::Int = DEFAULT_SIZE_THRESHOLD` - Threshold in bytes separating direct vs link transport
|
||||||
- `correlation_id::Union{String, Nothing} = nothing` - Optional correlation ID for tracing; if `nothing`, a UUID is generated
|
- `correlation_id::Union{String, Nothing} = nothing` - Optional correlation ID for tracing; if `nothing`, a UUID is generated
|
||||||
- `msg_purpose::String = "chat"` - Purpose of the message: "ACK", "NACK", "updateStatus", "shutdown", "chat", etc.
|
- `msg_purpose::String = "chat"` - Purpose of the message: "ACK", "NACK", "updateStatus", "shutdown", "chat", etc.
|
||||||
@@ -381,9 +382,12 @@ Each payload can have a different type, enabling mixed-content messages (e.g., c
|
|||||||
- `receiver_id::String = ""` - UUID of the receiver (empty string means broadcast)
|
- `receiver_id::String = ""` - UUID of the receiver (empty string means broadcast)
|
||||||
- `reply_to::String = ""` - Topic to reply to (empty string if no reply expected)
|
- `reply_to::String = ""` - Topic to reply to (empty string if no reply expected)
|
||||||
- `reply_to_msg_id::String = ""` - Message ID this message is replying to
|
- `reply_to_msg_id::String = ""` - Message ID this message is replying to
|
||||||
|
- `is_publish::Bool = true` - Whether to automatically publish the message to NATS
|
||||||
|
|
||||||
# Return:
|
# Return:
|
||||||
- A `msgEnvelope_v1` object containing metadata and transport information
|
- A tuple `(env, env_json_str)` where:
|
||||||
|
- `env::msg_envelope_v1` - The envelope object containing all metadata and payloads
|
||||||
|
- `env_json_str::String` - JSON string representation of the envelope for publishing
|
||||||
|
|
||||||
# Example
|
# Example
|
||||||
```jldoctest
|
```jldoctest
|
||||||
@@ -391,31 +395,34 @@ using UUIDs
|
|||||||
|
|
||||||
# Send a single payload (still wrapped in a list)
|
# Send a single payload (still wrapped in a list)
|
||||||
data = Dict("key" => "value")
|
data = Dict("key" => "value")
|
||||||
env = smartsend("my.subject", [("dataname1", data, "dictionary")])
|
env, msg_json = smartsend("my.subject", [("dataname1", data, "dictionary")])
|
||||||
|
|
||||||
# Send multiple payloads in one message with different types
|
# Send multiple payloads in one message with different types
|
||||||
data1 = Dict("key1" => "value1")
|
data1 = Dict("key1" => "value1")
|
||||||
data2 = rand(10_000) # Small array
|
data2 = rand(10_000) # Small array
|
||||||
env = smartsend("my.subject", [("dataname1", data1, "dictionary"), ("dataname2", data2, "table")])
|
env, msg_json = smartsend("my.subject", [("dataname1", data1, "dictionary"), ("dataname2", data2, "table")])
|
||||||
|
|
||||||
# Send a large array using fileserver upload
|
# Send a large array using fileserver upload
|
||||||
data = rand(10_000_000) # ~80 MB
|
data = rand(10_000_000) # ~80 MB
|
||||||
env = smartsend("large.data", [("large_table", data, "table")])
|
env, msg_json = smartsend("large.data", [("large_table", data, "table")])
|
||||||
|
|
||||||
# Mixed content (e.g., chat with text and image)
|
# Mixed content (e.g., chat with text and image)
|
||||||
env = smartsend("chat.subject", [
|
env, msg_json = smartsend("chat.subject", [
|
||||||
("message_text", "Hello!", "text"),
|
("message_text", "Hello!", "text"),
|
||||||
("user_image", image_data, "image"),
|
("user_image", image_data, "image"),
|
||||||
("audio_clip", audio_data, "audio")
|
("audio_clip", audio_data, "audio")
|
||||||
])
|
])
|
||||||
|
|
||||||
|
# Publish the JSON string directly using NATS request-reply pattern
|
||||||
|
# reply = NATS.request(broker_url, subject, env_json_str; reply_to=reply_to_topic)
|
||||||
```
|
```
|
||||||
"""
|
"""
|
||||||
function smartsend(
|
function smartsend(
|
||||||
subject::String, # smartreceive's subject
|
subject::String, # smartreceive's subject
|
||||||
data::AbstractArray{Tuple{String, T1, String}, 1}; # List of (dataname, data, type) tuples
|
data::AbstractArray{Tuple{String, T1, String}, 1}; # List of (dataname, data, type) tuples. Use Tuple{String, Any, String}[] for empty payloads
|
||||||
nats_url::String = DEFAULT_NATS_URL,
|
broker_url::String = DEFAULT_BROKER_URL, # NATS server URL
|
||||||
fileserver_url = DEFAULT_FILESERVER_URL,
|
fileserver_url = DEFAULT_FILESERVER_URL,
|
||||||
fileserverUploadHandler::Function=plik_oneshot_upload, # a function to handle uploading data to specific HTTP fileserver
|
fileserver_upload_handler::Function = plik_oneshot_upload, # a function to handle uploading data to specific HTTP fileserver
|
||||||
size_threshold::Int = DEFAULT_SIZE_THRESHOLD,
|
size_threshold::Int = DEFAULT_SIZE_THRESHOLD,
|
||||||
correlation_id::Union{String, Nothing} = nothing,
|
correlation_id::Union{String, Nothing} = nothing,
|
||||||
msg_purpose::String = "chat",
|
msg_purpose::String = "chat",
|
||||||
@@ -423,7 +430,8 @@ function smartsend(
|
|||||||
receiver_name::String = "",
|
receiver_name::String = "",
|
||||||
receiver_id::String = "",
|
receiver_id::String = "",
|
||||||
reply_to::String = "",
|
reply_to::String = "",
|
||||||
reply_to_msg_id::String = ""
|
reply_to_msg_id::String = "",
|
||||||
|
is_publish::Bool = true # some time the user want to get env and env_json_str from this function without publishing the msg
|
||||||
) where {T1<:Any}
|
) where {T1<:Any}
|
||||||
|
|
||||||
# Generate correlation ID if not provided
|
# Generate correlation ID if not provided
|
||||||
@@ -435,13 +443,13 @@ function smartsend(
|
|||||||
msg_id = string(uuid4())
|
msg_id = string(uuid4())
|
||||||
|
|
||||||
# Process each payload in the list
|
# Process each payload in the list
|
||||||
payloads = msgPayload_v1[]
|
payloads = msg_payload_v1[]
|
||||||
for (dataname, payload_data, payload_type) in data
|
for (dataname, payload_data, payload_type) in data
|
||||||
# Serialize data based on type
|
# Serialize data based on type
|
||||||
payload_bytes = _serialize_data(payload_data, payload_type)
|
payload_bytes = _serialize_data(payload_data, payload_type)
|
||||||
|
|
||||||
payload_size = length(payload_bytes) # Calculate payload size in bytes
|
payload_size = length(payload_bytes) # Calculate payload size in bytes
|
||||||
log_trace(cid, "Serialized payload '$dataname' (type: $payload_type) size: $payload_size bytes") # Log payload size
|
log_trace(cid, "Serialized payload '$dataname' (payload_type: $payload_type) size: $payload_size bytes") # Log payload size
|
||||||
|
|
||||||
# Decision: Direct vs Link
|
# Decision: Direct vs Link
|
||||||
if payload_size < size_threshold # Check if payload is small enough for direct transport
|
if payload_size < size_threshold # Check if payload is small enough for direct transport
|
||||||
@@ -449,8 +457,8 @@ function smartsend(
|
|||||||
payload_b64 = Base64.base64encode(payload_bytes) # Encode bytes as base64 string
|
payload_b64 = Base64.base64encode(payload_bytes) # Encode bytes as base64 string
|
||||||
log_trace(cid, "Using direct transport for $payload_size bytes") # Log transport choice
|
log_trace(cid, "Using direct transport for $payload_size bytes") # Log transport choice
|
||||||
|
|
||||||
# Create msgPayload_v1 for direct transport
|
# Create msg_payload_v1 for direct transport
|
||||||
payload = msgPayload_v1(
|
payload = msg_payload_v1(
|
||||||
payload_b64,
|
payload_b64,
|
||||||
payload_type;
|
payload_type;
|
||||||
id = string(uuid4()),
|
id = string(uuid4()),
|
||||||
@@ -466,7 +474,7 @@ function smartsend(
|
|||||||
log_trace(cid, "Using link transport, uploading to fileserver") # Log link transport choice
|
log_trace(cid, "Using link transport, uploading to fileserver") # Log link transport choice
|
||||||
|
|
||||||
# Upload to HTTP server
|
# Upload to HTTP server
|
||||||
response = fileserverUploadHandler(fileserver_url, dataname, payload_bytes)
|
response = fileserver_upload_handler(fileserver_url, dataname, payload_bytes)
|
||||||
|
|
||||||
if response["status"] != 200 # Check if upload was successful
|
if response["status"] != 200 # Check if upload was successful
|
||||||
error("Failed to upload data to fileserver: $(response["status"])") # Throw error if upload failed
|
error("Failed to upload data to fileserver: $(response["status"])") # Throw error if upload failed
|
||||||
@@ -475,8 +483,8 @@ function smartsend(
|
|||||||
url = response["url"] # URL for the uploaded data
|
url = response["url"] # URL for the uploaded data
|
||||||
log_trace(cid, "Uploaded to URL: $url") # Log successful upload
|
log_trace(cid, "Uploaded to URL: $url") # Log successful upload
|
||||||
|
|
||||||
# Create msgPayload_v1 for link transport
|
# Create msg_payload_v1 for link transport
|
||||||
payload = msgPayload_v1(
|
payload = msg_payload_v1(
|
||||||
url,
|
url,
|
||||||
payload_type;
|
payload_type;
|
||||||
id = string(uuid4()),
|
id = string(uuid4()),
|
||||||
@@ -490,27 +498,29 @@ function smartsend(
|
|||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
# Create msgEnvelope_v1 with all payloads
|
# Create msg_envelope_v1 with all payloads
|
||||||
env = msgEnvelope_v1(
|
env = msg_envelope_v1(
|
||||||
subject,
|
subject,
|
||||||
payloads;
|
payloads;
|
||||||
correlationId = cid,
|
correlation_id = cid,
|
||||||
msgId = msg_id,
|
msg_id = msg_id,
|
||||||
msgPurpose = msg_purpose,
|
msg_purpose = msg_purpose,
|
||||||
senderName = sender_name,
|
sender_name = sender_name,
|
||||||
senderId = string(uuid4()),
|
sender_id = string(uuid4()),
|
||||||
receiverName = receiver_name,
|
receiver_name = receiver_name,
|
||||||
receiverId = receiver_id,
|
receiver_id = receiver_id,
|
||||||
replyTo = reply_to,
|
reply_to = reply_to,
|
||||||
replyToMsgId = reply_to_msg_id,
|
reply_to_msg_id = reply_to_msg_id,
|
||||||
brokerURL = nats_url,
|
broker_url = broker_url,
|
||||||
metadata = Dict{String, Any}(),
|
metadata = Dict{String, Any}(),
|
||||||
)
|
)
|
||||||
|
|
||||||
msg_json = envelope_to_json(env) # Convert envelope to JSON
|
env_json_str = envelope_to_json(env) # Convert envelope to JSON
|
||||||
publish_message(nats_url, subject, msg_json, cid) # Publish message to NATS
|
if is_publish
|
||||||
|
publish_message(broker_url, subject, env_json_str, cid) # Publish message to NATS
|
||||||
|
end
|
||||||
|
|
||||||
return env # Return the envelope for tracking
|
return (env, env_json_str)
|
||||||
end
|
end
|
||||||
|
|
||||||
|
|
||||||
@@ -528,14 +538,14 @@ It supports multiple serialization formats for different data types.
|
|||||||
|
|
||||||
# Arguments:
|
# Arguments:
|
||||||
- `data::Any` - Data to serialize (string for `"text"`, JSON-serializable for `"dictionary"`, table-like for `"table"`, binary for `"image"`, `"audio"`, `"video"`, `"binary"`)
|
- `data::Any` - Data to serialize (string for `"text"`, JSON-serializable for `"dictionary"`, table-like for `"table"`, binary for `"image"`, `"audio"`, `"video"`, `"binary"`)
|
||||||
- `type::String` - Target format: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
- `payload_type::String` - Target format: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||||
|
|
||||||
# Return:
|
# Return:
|
||||||
- `Vector{UInt8}` - Binary representation of the serialized data
|
- `Vector{UInt8}` - Binary representation of the serialized data
|
||||||
|
|
||||||
# Throws:
|
# Throws:
|
||||||
- `Error` if `type` is not one of the supported types
|
- `Error` if `payload_type` is not one of the supported types
|
||||||
- `Error` if `type` is `"image"`, `"audio"`, or `"video"` but `data` is not `Vector{UInt8}`
|
- `Error` if `payload_type` is `"image"`, `"audio"`, or `"video"` but `data` is not `Vector{UInt8}`
|
||||||
|
|
||||||
# Example
|
# Example
|
||||||
```jldoctest
|
```jldoctest
|
||||||
@@ -574,7 +584,7 @@ binary_bytes = _serialize_data(buf, "binary")
|
|||||||
binary_bytes_direct = _serialize_data(UInt8[1, 2, 3], "binary")
|
binary_bytes_direct = _serialize_data(UInt8[1, 2, 3], "binary")
|
||||||
```
|
```
|
||||||
"""
|
"""
|
||||||
function _serialize_data(data::Any, type::String)
|
function _serialize_data(data::Any, payload_type::String)
|
||||||
""" Example on how JSON.jl convert: dictionary -> json string -> json string bytes -> json string -> json object
|
""" Example on how JSON.jl convert: dictionary -> json string -> json string bytes -> json string -> json object
|
||||||
d = Dict(
|
d = Dict(
|
||||||
"name"=>"ton",
|
"name"=>"ton",
|
||||||
@@ -591,40 +601,40 @@ function _serialize_data(data::Any, type::String)
|
|||||||
json_obj = JSON.parse(json_str_2)
|
json_obj = JSON.parse(json_str_2)
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if type == "text" # Text data - convert to UTF-8 bytes
|
if payload_type == "text" # Text data - convert to UTF-8 bytes
|
||||||
if isa(data, String)
|
if isa(data, String)
|
||||||
data_bytes = Vector{UInt8}(data) # Convert string to UTF-8 bytes
|
data_bytes = Vector{UInt8}(data) # Convert string to UTF-8 bytes
|
||||||
return data_bytes
|
return data_bytes
|
||||||
else
|
else
|
||||||
error("Text data must be a String")
|
error("Text data must be a String")
|
||||||
end
|
end
|
||||||
elseif type == "dictionary" # JSON data - serialize directly
|
elseif payload_type == "dictionary" # JSON data - serialize directly
|
||||||
json_str = JSON.json(data) # Convert Julia data to JSON string
|
json_str = JSON.json(data) # Convert Julia data to JSON string
|
||||||
json_str_bytes = Vector{UInt8}(json_str) # Convert JSON string to bytes
|
json_str_bytes = Vector{UInt8}(json_str) # Convert JSON string to bytes
|
||||||
return json_str_bytes
|
return json_str_bytes
|
||||||
elseif type == "table" # Table data - convert to Arrow IPC stream
|
elseif payload_type == "table" # Table data - convert to Arrow IPC stream
|
||||||
io = IOBuffer() # Create in-memory buffer
|
io = IOBuffer() # Create in-memory buffer
|
||||||
Arrow.write(io, data) # Write data as Arrow IPC stream to buffer
|
Arrow.write(io, data) # Write data as Arrow IPC stream to buffer
|
||||||
return take!(io) # Return the buffer contents as bytes
|
return take!(io) # Return the buffer contents as bytes
|
||||||
elseif type == "image" # Image data - treat as binary
|
elseif payload_type == "image" # Image data - treat as binary
|
||||||
if isa(data, Vector{UInt8})
|
if isa(data, Vector{UInt8})
|
||||||
return data # Return binary data directly
|
return data # Return binary data directly
|
||||||
else
|
else
|
||||||
error("Image data must be Vector{UInt8}")
|
error("Image data must be Vector{UInt8}")
|
||||||
end
|
end
|
||||||
elseif type == "audio" # Audio data - treat as binary
|
elseif payload_type == "audio" # Audio data - treat as binary
|
||||||
if isa(data, Vector{UInt8})
|
if isa(data, Vector{UInt8})
|
||||||
return data # Return binary data directly
|
return data # Return binary data directly
|
||||||
else
|
else
|
||||||
error("Audio data must be Vector{UInt8}")
|
error("Audio data must be Vector{UInt8}")
|
||||||
end
|
end
|
||||||
elseif type == "video" # Video data - treat as binary
|
elseif payload_type == "video" # Video data - treat as binary
|
||||||
if isa(data, Vector{UInt8})
|
if isa(data, Vector{UInt8})
|
||||||
return data # Return binary data directly
|
return data # Return binary data directly
|
||||||
else
|
else
|
||||||
error("Video data must be Vector{UInt8}")
|
error("Video data must be Vector{UInt8}")
|
||||||
end
|
end
|
||||||
elseif type == "binary" # Binary data - treat as binary
|
elseif payload_type == "binary" # Binary data - treat as binary
|
||||||
if isa(data, IOBuffer) # Check if data is an IOBuffer
|
if isa(data, IOBuffer) # Check if data is an IOBuffer
|
||||||
return take!(data) # Return buffer contents as bytes
|
return take!(data) # Return buffer contents as bytes
|
||||||
elseif isa(data, Vector{UInt8}) # Check if data is already binary
|
elseif isa(data, Vector{UInt8}) # Check if data is already binary
|
||||||
@@ -633,7 +643,7 @@ function _serialize_data(data::Any, type::String)
|
|||||||
error("Binary data must be binary (Vector{UInt8} or IOBuffer)")
|
error("Binary data must be binary (Vector{UInt8} or IOBuffer)")
|
||||||
end
|
end
|
||||||
else # Unknown type
|
else # Unknown type
|
||||||
error("Unknown type: $type")
|
error("Unknown payload_type: $payload_type")
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
@@ -643,7 +653,7 @@ This internal function publishes a message to a NATS subject with proper
|
|||||||
connection management and logging.
|
connection management and logging.
|
||||||
|
|
||||||
# Arguments:
|
# Arguments:
|
||||||
- `nats_url::String` - NATS server URL (e.g., "nats://localhost:4222")
|
- `broker_url::String` - NATS server URL (e.g., "nats://localhost:4222")
|
||||||
- `subject::String` - NATS subject to publish to (e.g., "/agent/wine/api/v1/prompt")
|
- `subject::String` - NATS subject to publish to (e.g., "/agent/wine/api/v1/prompt")
|
||||||
- `message::String` - JSON message to publish
|
- `message::String` - JSON message to publish
|
||||||
- `correlation_id::String` - Correlation ID for tracing and logging
|
- `correlation_id::String` - Correlation ID for tracing and logging
|
||||||
@@ -656,14 +666,14 @@ connection management and logging.
|
|||||||
using NATS
|
using NATS
|
||||||
|
|
||||||
# Prepare JSON message
|
# Prepare JSON message
|
||||||
message = "{\"correlationId\":\"abc123\",\"payload\":\"test\"}"
|
message = "{\"correlation_id\":\"abc123\",\"payload\":\"test\"}"
|
||||||
|
|
||||||
# Publish to NATS
|
# Publish to NATS
|
||||||
publish_message("nats://localhost:4222", "my.subject", message, "abc123")
|
publish_message("nats://localhost:4222", "my.subject", message, "abc123")
|
||||||
```
|
```
|
||||||
"""
|
"""
|
||||||
function publish_message(nats_url::String, subject::String, message::String, correlation_id::String)
|
function publish_message(broker_url::String, subject::String, message::String, correlation_id::String)
|
||||||
conn = NATS.connect(nats_url) # Create NATS connection
|
conn = NATS.connect(broker_url) # Create NATS connection
|
||||||
try
|
try
|
||||||
NATS.publish(conn, subject, message) # Publish message to NATS
|
NATS.publish(conn, subject, message) # Publish message to NATS
|
||||||
log_trace(correlation_id, "Message published to $subject") # Log successful publish
|
log_trace(correlation_id, "Message published to $subject") # Log successful publish
|
||||||
@@ -690,32 +700,32 @@ A HTTP file server is required along with its download function.
|
|||||||
- `msg::NATS.Msg` - NATS message to process
|
- `msg::NATS.Msg` - NATS message to process
|
||||||
|
|
||||||
# Keyword Arguments:
|
# Keyword Arguments:
|
||||||
- `fileserverDownloadHandler::Function = _fetch_with_backoff` - Function to handle downloading data from file server URLs
|
- `fileserver_download_handler::Function = _fetch_with_backoff` - Function to handle downloading data from file server URLs
|
||||||
- `max_retries::Int = 5` - Maximum retry attempts for fetching URL
|
- `max_retries::Int = 5` - Maximum retry attempts for fetching URL
|
||||||
- `base_delay::Int = 100` - Initial delay for exponential backoff in ms
|
- `base_delay::Int = 100` - Initial delay for exponential backoff in ms
|
||||||
- `max_delay::Int = 5000` - Maximum delay for exponential backoff in ms
|
- `max_delay::Int = 5000` - Maximum delay for exponential backoff in ms
|
||||||
|
|
||||||
# Return:
|
# Return:
|
||||||
- `AbstractArray{Tuple{String, Any, String}}` - List of (dataname, data, type) tuples
|
- `Vector{Tuple{String, Any, String}}` - List of (dataname, data, type) tuples
|
||||||
|
|
||||||
# Example
|
# Example
|
||||||
```jldoctest
|
```jldoctest
|
||||||
# Receive and process message
|
# Receive and process message
|
||||||
msg = nats_message # NATS message
|
msg = nats_message # NATS message
|
||||||
payloads = smartreceive(msg; fileserverDownloadHandler=_fetch_with_backoff, max_retries=5, base_delay=100, max_delay=5000)
|
payloads = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff, max_retries=5, base_delay=100, max_delay=5000)
|
||||||
# payloads = [("dataname1", data1, "type1"), ("dataname2", data2, "type2"), ...]
|
# payloads = [("dataname1", data1, "type1"), ("dataname2", data2, "type2"), ...]
|
||||||
```
|
```
|
||||||
"""
|
"""
|
||||||
function smartreceive(
|
function smartreceive(
|
||||||
msg::NATS.Msg;
|
msg::NATS.Msg;
|
||||||
fileserverDownloadHandler::Function=_fetch_with_backoff,
|
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||||
max_retries::Int = 5,
|
max_retries::Int = 5,
|
||||||
base_delay::Int = 100,
|
base_delay::Int = 100,
|
||||||
max_delay::Int = 5000
|
max_delay::Int = 5000
|
||||||
)
|
)
|
||||||
# Parse the JSON envelope
|
# Parse the JSON envelope
|
||||||
json_data = JSON.parse(String(msg.payload))
|
json_data = JSON.parse(String(msg.payload))
|
||||||
log_trace(json_data["correlationId"], "Processing received message") # Log message processing start
|
log_trace(json_data["correlation_id"], "Processing received message") # Log message processing start
|
||||||
|
|
||||||
# Process all payloads in the envelope
|
# Process all payloads in the envelope
|
||||||
payloads_list = Tuple{String, Any, String}[]
|
payloads_list = Tuple{String, Any, String}[]
|
||||||
@@ -729,7 +739,7 @@ function smartreceive(
|
|||||||
dataname = String(payload["dataname"])
|
dataname = String(payload["dataname"])
|
||||||
|
|
||||||
if transport == "direct" # Direct transport - payload is in the message
|
if transport == "direct" # Direct transport - payload is in the message
|
||||||
log_trace(json_data["correlationId"], "Direct transport - decoding payload '$dataname'") # Log direct transport handling
|
log_trace(json_data["correlation_id"], "Direct transport - decoding payload '$dataname'") # Log direct transport handling
|
||||||
|
|
||||||
# Extract base64 payload from the payload
|
# Extract base64 payload from the payload
|
||||||
payload_b64 = String(payload["data"])
|
payload_b64 = String(payload["data"])
|
||||||
@@ -738,21 +748,21 @@ function smartreceive(
|
|||||||
payload_bytes = Base64.base64decode(payload_b64) # Decode base64 payload to bytes
|
payload_bytes = Base64.base64decode(payload_b64) # Decode base64 payload to bytes
|
||||||
|
|
||||||
# Deserialize based on type
|
# Deserialize based on type
|
||||||
data_type = String(payload["type"])
|
data_type = String(payload["payload_type"])
|
||||||
data = _deserialize_data(payload_bytes, data_type, json_data["correlationId"])
|
data = _deserialize_data(payload_bytes, data_type, json_data["correlation_id"])
|
||||||
|
|
||||||
push!(payloads_list, (dataname, data, data_type))
|
push!(payloads_list, (dataname, data, data_type))
|
||||||
elseif transport == "link" # Link transport - payload is at URL
|
elseif transport == "link" # Link transport - payload is at URL
|
||||||
# Extract download URL from the payload
|
# Extract download URL from the payload
|
||||||
url = String(payload["data"])
|
url = String(payload["data"])
|
||||||
log_trace(json_data["correlationId"], "Link transport - fetching '$dataname' from URL: $url") # Log link transport handling
|
log_trace(json_data["correlation_id"], "Link transport - fetching '$dataname' from URL: $url") # Log link transport handling
|
||||||
|
|
||||||
# Fetch with exponential backoff using the download handler
|
# Fetch with exponential backoff using the download handler
|
||||||
downloaded_data = fileserverDownloadHandler(url, max_retries, base_delay, max_delay, json_data["correlationId"])
|
downloaded_data = fileserver_download_handler(url, max_retries, base_delay, max_delay, json_data["correlation_id"])
|
||||||
|
|
||||||
# Deserialize based on type
|
# Deserialize based on type
|
||||||
data_type = String(payload["type"])
|
data_type = String(payload["payload_type"])
|
||||||
data = _deserialize_data(downloaded_data, data_type, json_data["correlationId"])
|
data = _deserialize_data(downloaded_data, data_type, json_data["correlation_id"])
|
||||||
|
|
||||||
push!(payloads_list, (dataname, data, data_type))
|
push!(payloads_list, (dataname, data, data_type))
|
||||||
else # Unknown transport type
|
else # Unknown transport type
|
||||||
@@ -840,19 +850,19 @@ It handles "text" (string), "dictionary" (JSON deserialization), "table" (Arrow
|
|||||||
|
|
||||||
# Arguments:
|
# Arguments:
|
||||||
- `data::Vector{UInt8}` - Serialized data as bytes
|
- `data::Vector{UInt8}` - Serialized data as bytes
|
||||||
- `type::String` - Data type ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
- `payload_type::String` - Data type ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
||||||
- `correlation_id::String` - Correlation ID for logging
|
- `correlation_id::String` - Correlation ID for logging
|
||||||
|
|
||||||
# Return:
|
# Return:
|
||||||
- Deserialized data (String for "text", DataFrame for "table", JSON data for "dictionary", bytes for "image", "audio", "video", "binary")
|
- Deserialized data (String for "text", DataFrame for "table", JSON data for "dictionary", bytes for "image", "audio", "video", "binary")
|
||||||
|
|
||||||
# Throws:
|
# Throws:
|
||||||
- `Error` if `type` is not one of the supported types
|
- `Error` if `payload_type` is not one of the supported types
|
||||||
|
|
||||||
# Example
|
# Example
|
||||||
```jldoctest
|
```jldoctest
|
||||||
# Text data
|
# Text data
|
||||||
text_bytes = UInt8["Hello World"]
|
text_bytes = Vector{UInt8}("Hello World")
|
||||||
text_data = _deserialize_data(text_bytes, "text", "correlation123")
|
text_data = _deserialize_data(text_bytes, "text", "correlation123")
|
||||||
|
|
||||||
# JSON data
|
# JSON data
|
||||||
@@ -860,34 +870,34 @@ json_bytes = UInt8[123, 34, 110, 97, 109, 101, 34, 58, 34, 65, 108, 105, 99, 101
|
|||||||
json_data = _deserialize_data(json_bytes, "dictionary", "correlation123")
|
json_data = _deserialize_data(json_bytes, "dictionary", "correlation123")
|
||||||
|
|
||||||
# Arrow IPC data (table)
|
# Arrow IPC data (table)
|
||||||
arrow_bytes = UInt8[1, 2, 3] # Arrow IPC bytes
|
arrow_bytes = Vector{UInt8}([1, 2, 3]) # Arrow IPC bytes
|
||||||
table_data = _deserialize_data(arrow_bytes, "table", "correlation123")
|
table_data = _deserialize_data(arrow_bytes, "table", "correlation123")
|
||||||
```
|
```
|
||||||
"""
|
"""
|
||||||
function _deserialize_data(
|
function _deserialize_data(
|
||||||
data::Vector{UInt8},
|
data::Vector{UInt8},
|
||||||
type::String,
|
payload_type::String,
|
||||||
correlation_id::String
|
correlation_id::String
|
||||||
)
|
)
|
||||||
if type == "text" # Text data - convert to string
|
if payload_type == "text" # Text data - convert to string
|
||||||
return String(data) # Convert bytes to string
|
return String(data) # Convert bytes to string
|
||||||
elseif type == "dictionary" # JSON data - deserialize
|
elseif payload_type == "dictionary" # JSON data - deserialize
|
||||||
json_str = String(data) # Convert bytes to string
|
json_str = String(data) # Convert bytes to string
|
||||||
return JSON.parse(json_str) # Parse JSON string to JSON object
|
return JSON.parse(json_str) # Parse JSON string to JSON object
|
||||||
elseif type == "table" # Table data - deserialize Arrow IPC stream
|
elseif payload_type == "table" # Table data - deserialize Arrow IPC stream
|
||||||
io = IOBuffer(data) # Create buffer from bytes
|
io = IOBuffer(data) # Create buffer from bytes
|
||||||
df = Arrow.Table(io) # Read Arrow IPC format from buffer
|
df = Arrow.Table(io) # Read Arrow IPC format from buffer
|
||||||
return df # Return DataFrame
|
return df # Return DataFrame
|
||||||
elseif type == "image" # Image data - return binary
|
elseif payload_type == "image" # Image data - return binary
|
||||||
return data # Return bytes directly
|
return data # Return bytes directly
|
||||||
elseif type == "audio" # Audio data - return binary
|
elseif payload_type == "audio" # Audio data - return binary
|
||||||
return data # Return bytes directly
|
return data # Return bytes directly
|
||||||
elseif type == "video" # Video data - return binary
|
elseif payload_type == "video" # Video data - return binary
|
||||||
return data # Return bytes directly
|
return data # Return bytes directly
|
||||||
elseif type == "binary" # Binary data - return binary
|
elseif payload_type == "binary" # Binary data - return binary
|
||||||
return data # Return bytes directly
|
return data # Return bytes directly
|
||||||
else # Unknown type
|
else # Unknown type
|
||||||
error("Unknown type: $type") # Throw error for unknown type
|
error("Unknown payload_type: $payload_type") # Throw error for unknown type
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
@@ -904,7 +914,7 @@ retrieves an upload ID and token, then uploads the file data as multipart form d
|
|||||||
4. Returns identifiers and download URL for the uploaded file
|
4. Returns identifiers and download URL for the uploaded file
|
||||||
|
|
||||||
# Arguments:
|
# Arguments:
|
||||||
- `fileServerURL::String` - Base URL of the plik server (e.g., `"http://localhost:8080"`)
|
- `file_server_url::String` - Base URL of the plik server (e.g., `"http://localhost:8080"`)
|
||||||
- `filename::String` - Name of the file being uploaded
|
- `filename::String` - Name of the file being uploaded
|
||||||
- `data::Vector{UInt8}` - Raw byte data of the file content
|
- `data::Vector{UInt8}` - Raw byte data of the file content
|
||||||
|
|
||||||
@@ -919,33 +929,33 @@ retrieves an upload ID and token, then uploads the file data as multipart form d
|
|||||||
```jldoctest
|
```jldoctest
|
||||||
using HTTP, JSON
|
using HTTP, JSON
|
||||||
|
|
||||||
fileServerURL = "http://localhost:8080"
|
file_server_url = "http://localhost:8080"
|
||||||
filename = "test.txt"
|
filename = "test.txt"
|
||||||
data = UInt8["hello world"]
|
data = Vector{UInt8}("hello world")
|
||||||
|
|
||||||
# Upload to local plik server
|
# Upload to local plik server
|
||||||
result = plik_oneshot_upload(fileServerURL, filename, data)
|
result = plik_oneshot_upload(file_server_url, filename, data)
|
||||||
|
|
||||||
# Access the result as a Dict
|
# Access the result as a Dict
|
||||||
# result["status"], result["uploadid"], result["fileid"], result["url"]
|
# result["status"], result["uploadid"], result["fileid"], result["url"]
|
||||||
```
|
```
|
||||||
"""
|
"""
|
||||||
function plik_oneshot_upload(fileServerURL::String, filename::String, data::Vector{UInt8})
|
function plik_oneshot_upload(file_server_url::String, filename::String, data::Vector{UInt8})
|
||||||
|
|
||||||
# ----------------------------------------- get upload id ---------------------------------------- #
|
# ----------------------------------------- get upload id ---------------------------------------- #
|
||||||
# Equivalent curl command: curl -X POST -d '{ "OneShot" : true }' http://localhost:8080/upload
|
# Equivalent curl command: curl -X POST -d '{ "OneShot" : true }' http://localhost:8080/upload
|
||||||
url_getUploadID = "$fileServerURL/upload" # URL to get upload ID
|
url_getUploadID = "$file_server_url/upload" # URL to get upload ID
|
||||||
headers = ["Content-Type" => "application/json"]
|
headers = ["Content-Type" => "application/json"]
|
||||||
body = """{ "OneShot" : true }"""
|
body = """{ "OneShot" : true }"""
|
||||||
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
http_response = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
||||||
responseJson = JSON.parse(httpResponse.body)
|
response_json = JSON.parse(http_response.body)
|
||||||
uploadid = responseJson["id"]
|
uploadid = response_json["id"]
|
||||||
uploadtoken = responseJson["uploadToken"]
|
uploadtoken = response_json["uploadToken"]
|
||||||
|
|
||||||
# ------------------------------------------ upload file ----------------------------------------- #
|
# ------------------------------------------ upload file ----------------------------------------- #
|
||||||
# Equivalent curl command: curl -X POST --header "X-UploadToken: UPLOAD_TOKEN" -F "file=@PATH_TO_FILE" http://localhost:8080/file/UPLOAD_ID
|
# Equivalent curl command: curl -X POST --header "X-UploadToken: UPLOAD_TOKEN" -F "file=@PATH_TO_FILE" http://localhost:8080/file/UPLOAD_ID
|
||||||
file_multipart = HTTP.Multipart(filename, IOBuffer(data), "application/octet-stream") # Plik won't accept raw bytes upload
|
file_multipart = HTTP.Multipart(filename, IOBuffer(data), "application/octet-stream") # Plik won't accept raw bytes upload
|
||||||
url_upload = "$fileServerURL/file/$uploadid"
|
url_upload = "$file_server_url/file/$uploadid"
|
||||||
headers = ["X-UploadToken" => uploadtoken]
|
headers = ["X-UploadToken" => uploadtoken]
|
||||||
|
|
||||||
# Create the multipart form data
|
# Create the multipart form data
|
||||||
@@ -954,24 +964,23 @@ function plik_oneshot_upload(fileServerURL::String, filename::String, data::Vect
|
|||||||
))
|
))
|
||||||
|
|
||||||
# Execute the POST request
|
# Execute the POST request
|
||||||
httpResponse = nothing
|
http_response = nothing
|
||||||
try
|
try
|
||||||
httpResponse = HTTP.post(url_upload, headers, form)
|
http_response = HTTP.post(url_upload, headers, form)
|
||||||
responseJson = JSON.parse(httpResponse.body)
|
|
||||||
catch e
|
catch e
|
||||||
@error "Request failed" exception=e
|
@error "Request failed" exception=e
|
||||||
end
|
end
|
||||||
|
response_json = JSON.parse(http_response.body)
|
||||||
fileid = responseJson["id"]
|
fileid = response_json["id"]
|
||||||
|
|
||||||
# url of the uploaded data e.g. "http://192.168.1.20:8080/file/3F62E/4AgGT/test.zip"
|
# url of the uploaded data e.g. "http://192.168.1.20:8080/file/3F62E/4AgGT/test.zip"
|
||||||
url = "$fileServerURL/file/$uploadid/$fileid/$filename"
|
url = "$file_server_url/file/$uploadid/$fileid/$filename"
|
||||||
|
|
||||||
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
return Dict("status" => http_response.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
||||||
end
|
end
|
||||||
|
|
||||||
|
|
||||||
""" plik_oneshot_upload(fileServerURL::String, filepath::String)
|
""" plik_oneshot_upload(file_server_url::String, filepath::String)
|
||||||
This function uploads a file from disk to a plik server in one-shot mode (no upload session).
|
This function uploads a file from disk to a plik server in one-shot mode (no upload session).
|
||||||
It first creates a one-shot upload session by sending a POST request with `{"OneShot": true}`,
|
It first creates a one-shot upload session by sending a POST request with `{"OneShot": true}`,
|
||||||
retrieves an upload ID and token, then uploads the file data as multipart form data using the token.
|
retrieves an upload ID and token, then uploads the file data as multipart form data using the token.
|
||||||
@@ -983,7 +992,7 @@ retrieves an upload ID and token, then uploads the file data as multipart form d
|
|||||||
4. Returns identifiers and download URL for the uploaded file
|
4. Returns identifiers and download URL for the uploaded file
|
||||||
|
|
||||||
# Arguments:
|
# Arguments:
|
||||||
- `fileServerURL::String` - Base URL of the plik server (e.g., `"http://localhost:8080"`)
|
- `file_server_url::String` - Base URL of the plik server (e.g., `"http://localhost:8080"`)
|
||||||
- `filepath::String` - Full path to the local file to upload
|
- `filepath::String` - Full path to the local file to upload
|
||||||
|
|
||||||
# Return:
|
# Return:
|
||||||
@@ -997,59 +1006,58 @@ retrieves an upload ID and token, then uploads the file data as multipart form d
|
|||||||
```jldoctest
|
```jldoctest
|
||||||
using HTTP, JSON
|
using HTTP, JSON
|
||||||
|
|
||||||
fileServerURL = "http://localhost:8080"
|
file_server_url = "http://localhost:8080"
|
||||||
filepath = "./test.zip"
|
filepath = "./test.zip"
|
||||||
|
|
||||||
# Upload to local plik server
|
# Upload to local plik server
|
||||||
result = plik_oneshot_upload(fileServerURL, filepath)
|
result = plik_oneshot_upload(file_server_url, filepath)
|
||||||
|
|
||||||
# Access the result as a Dict
|
# Access the result as a Dict
|
||||||
# result["status"], result["uploadid"], result["fileid"], result["url"]
|
# result["status"], result["uploadid"], result["fileid"], result["url"]
|
||||||
```
|
```
|
||||||
"""
|
"""
|
||||||
function plik_oneshot_upload(fileServerURL::String, filepath::String)
|
function plik_oneshot_upload(file_server_url::String, filepath::String)
|
||||||
|
|
||||||
# ----------------------------------------- get upload id ---------------------------------------- #
|
# ----------------------------------------- get upload id ---------------------------------------- #
|
||||||
# Equivalent curl command: curl -X POST -d '{ "OneShot" : true }' http://localhost:8080/upload
|
# Equivalent curl command: curl -X POST -d '{ "OneShot" : true }' http://localhost:8080/upload
|
||||||
filename = basename(filepath)
|
filename = basename(filepath)
|
||||||
url_getUploadID = "$fileServerURL/upload" # URL to get upload ID
|
url_getUploadID = "$file_server_url/upload" # URL to get upload ID
|
||||||
headers = ["Content-Type" => "application/json"]
|
headers = ["Content-Type" => "application/json"]
|
||||||
body = """{ "OneShot" : true }"""
|
body = """{ "OneShot" : true }"""
|
||||||
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
http_response = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
||||||
responseJson = JSON.parse(httpResponse.body)
|
response_json = JSON.parse(http_response.body)
|
||||||
|
|
||||||
uploadid = responseJson["id"]
|
uploadid = response_json["id"]
|
||||||
uploadtoken = responseJson["uploadToken"]
|
uploadtoken = response_json["uploadToken"]
|
||||||
|
|
||||||
# ------------------------------------------ upload file ----------------------------------------- #
|
# ------------------------------------------ upload file ----------------------------------------- #
|
||||||
# Equivalent curl command: curl -X POST --header "X-UploadToken: UPLOAD_TOKEN" -F "file=@PATH_TO_FILE" http://localhost:8080/file/UPLOAD_ID
|
# Equivalent curl command: curl -X POST --header "X-UploadToken: UPLOAD_TOKEN" -F "file=@PATH_TO_FILE" http://localhost:8080/file/UPLOAD_ID
|
||||||
file_multipart = open(filepath, "r")
|
url_upload = "$file_server_url/file/$uploadid"
|
||||||
url_upload = "$fileServerURL/file/$uploadid"
|
|
||||||
headers = ["X-UploadToken" => uploadtoken]
|
headers = ["X-UploadToken" => uploadtoken]
|
||||||
|
http_response = open(filepath, "r") do file_stream
|
||||||
|
form = HTTP.Form(Dict("file" => file_stream))
|
||||||
|
|
||||||
# Create the multipart form data
|
# Adding status_exception=false prevents 4xx/5xx from triggering 'catch'
|
||||||
form = HTTP.Form(Dict(
|
HTTP.post(url_upload, headers, form; status_exception = false)
|
||||||
"file" => file_multipart
|
|
||||||
))
|
|
||||||
|
|
||||||
# Execute the POST request
|
|
||||||
httpResponse = nothing
|
|
||||||
try
|
|
||||||
httpResponse = HTTP.post(url_upload, headers, form)
|
|
||||||
responseJson = JSON.parse(httpResponse.body)
|
|
||||||
catch e
|
|
||||||
@error "Request failed" exception=e
|
|
||||||
end
|
end
|
||||||
|
|
||||||
fileid = responseJson["id"]
|
if !isnothing(http_response) && http_response.status == 200
|
||||||
|
# Success - response already logged by caller
|
||||||
|
else
|
||||||
|
error("Failed to upload file: server returned status $(http_response.status)")
|
||||||
|
end
|
||||||
|
response_json = JSON.parse(http_response.body)
|
||||||
|
fileid = response_json["id"]
|
||||||
|
|
||||||
# url of the uploaded data e.g. "http://192.168.1.20:8080/file/3F62E/4AgGT/test.zip"
|
# url of the uploaded data e.g. "http://192.168.1.20:8080/file/3F62E/4AgGT/test.zip"
|
||||||
url = "$fileServerURL/file/$uploadid/$fileid/$filename"
|
url = "$file_server_url/file/$uploadid/$fileid/$filename"
|
||||||
|
|
||||||
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
return Dict("status" => http_response.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
||||||
end
|
end
|
||||||
|
|
||||||
|
function _get_payload_bytes(data)
|
||||||
|
@error "didn't implement yet"
|
||||||
|
end
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -460,8 +460,9 @@ async function smartsend(subject, data, options = {}) {
|
|||||||
* @param {string} options.receiverId - UUID of the receiver (default: "")
|
* @param {string} options.receiverId - UUID of the receiver (default: "")
|
||||||
* @param {string} options.replyTo - Topic to reply to (default: "")
|
* @param {string} options.replyTo - Topic to reply to (default: "")
|
||||||
* @param {string} options.replyToMsgId - Message ID this message is replying to (default: "")
|
* @param {string} options.replyToMsgId - Message ID this message is replying to (default: "")
|
||||||
|
* @param {boolean} options.isPublish - Whether to automatically publish the message to NATS (default: true)
|
||||||
*
|
*
|
||||||
* @returns {Promise<MessageEnvelope>} - The envelope for tracking
|
* @returns {Promise<Object>} - An object with { env: MessageEnvelope, env_json_str: string }
|
||||||
*/
|
*/
|
||||||
const {
|
const {
|
||||||
natsUrl = DEFAULT_NATS_URL,
|
natsUrl = DEFAULT_NATS_URL,
|
||||||
@@ -474,7 +475,8 @@ async function smartsend(subject, data, options = {}) {
|
|||||||
receiverName = "",
|
receiverName = "",
|
||||||
receiverId = "",
|
receiverId = "",
|
||||||
replyTo = "",
|
replyTo = "",
|
||||||
replyToMsgId = ""
|
replyToMsgId = "",
|
||||||
|
isPublish = true // Whether to automatically publish the message to NATS
|
||||||
} = options;
|
} = options;
|
||||||
|
|
||||||
log_trace(correlationId, `Starting smartsend for subject: ${subject}`);
|
log_trace(correlationId, `Starting smartsend for subject: ${subject}`);
|
||||||
@@ -556,10 +558,19 @@ async function smartsend(subject, data, options = {}) {
|
|||||||
payloads: payloads
|
payloads: payloads
|
||||||
});
|
});
|
||||||
|
|
||||||
// Publish message to NATS
|
// Convert envelope to JSON string
|
||||||
await publish_message(natsUrl, subject, env.toString(), correlationId);
|
const env_json_str = env.toString();
|
||||||
|
|
||||||
return env;
|
// Publish to NATS if isPublish is true
|
||||||
|
if (isPublish) {
|
||||||
|
await publish_message(natsUrl, subject, env_json_str, correlationId);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Return both envelope and JSON string (tuple-like structure)
|
||||||
|
return {
|
||||||
|
env: env,
|
||||||
|
env_json_str: env_json_str
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
// Helper: Publish message to NATS
|
// Helper: Publish message to NATS
|
||||||
|
|||||||
295
src/README.md
295
src/README.md
@@ -1,295 +0,0 @@
|
|||||||
# NATSBridge
|
|
||||||
|
|
||||||
A high-performance, bi-directional data bridge for **Julia**, **JavaScript**, and **Python/Micropython** using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
NATSBridge enables seamless communication between Julia, JavaScript, and Python/Micropython applications through NATS, with automatic transport selection based on payload size:
|
|
||||||
|
|
||||||
- **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded)
|
|
||||||
- **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL
|
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
- ✅ Bi-directional NATS communication across Julia ↔ JavaScript ↔ Python/Micropython
|
|
||||||
- ✅ Multi-payload support (mixed content in single message)
|
|
||||||
- ✅ Automatic transport selection based on payload size
|
|
||||||
- ✅ File server integration for large payloads
|
|
||||||
- ✅ Exponential backoff for URL fetching
|
|
||||||
- ✅ Correlation ID tracking
|
|
||||||
- ✅ Reply-to support for request-response pattern
|
|
||||||
|
|
||||||
## Supported Payload Types
|
|
||||||
|
|
||||||
| Type | Description |
|
|
||||||
|------|-------------|
|
|
||||||
| `text` | Plain text strings |
|
|
||||||
| `dictionary` | JSON-serializable dictionaries |
|
|
||||||
| `table` | Tabular data (Arrow IPC format) |
|
|
||||||
| `image` | Image data (PNG, JPG bytes) |
|
|
||||||
| `audio` | Audio data (WAV, MP3 bytes) |
|
|
||||||
| `video` | Video data (MP4, AVI bytes) |
|
|
||||||
| `binary` | Generic binary data |
|
|
||||||
|
|
||||||
## Implementation Guides
|
|
||||||
|
|
||||||
### [Julia Implementation](../tutorial_julia.md)
|
|
||||||
|
|
||||||
See the [Julia tutorial](../tutorial_julia.md) for getting started with Julia.
|
|
||||||
|
|
||||||
### [JavaScript Implementation](#javascript-implementation)
|
|
||||||
|
|
||||||
See [`NATSBridge.js`](NATSBridge.js) for the JavaScript implementation.
|
|
||||||
|
|
||||||
### [Python/Micropython Implementation](#pythonmicropython-implementation)
|
|
||||||
|
|
||||||
See [`nats_bridge.py`](nats_bridge.py) for the Python/Micropython implementation.
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
### Julia
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using Pkg
|
|
||||||
Pkg.add("NATS")
|
|
||||||
Pkg.add("Arrow")
|
|
||||||
Pkg.add("JSON3")
|
|
||||||
Pkg.add("HTTP")
|
|
||||||
Pkg.add("UUIDs")
|
|
||||||
Pkg.add("Dates")
|
|
||||||
```
|
|
||||||
|
|
||||||
### JavaScript
|
|
||||||
|
|
||||||
```bash
|
|
||||||
npm install nats.js apache-arrow uuid base64-url
|
|
||||||
```
|
|
||||||
|
|
||||||
### Python/Micropython
|
|
||||||
|
|
||||||
1. Copy `nats_bridge.py` to your device
|
|
||||||
2. Ensure you have the following dependencies:
|
|
||||||
- `urequests` for HTTP requests (Micropython)
|
|
||||||
- `requests` for HTTP requests (Python)
|
|
||||||
- `base64` for base64 encoding
|
|
||||||
- `json` for JSON handling
|
|
||||||
- `socket` for networking (Micropython)
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
### Basic Text Message
|
|
||||||
|
|
||||||
#### Python/Micropython
|
|
||||||
|
|
||||||
```python
|
|
||||||
from nats_bridge import smartsend, smartreceive
|
|
||||||
|
|
||||||
# Sender
|
|
||||||
data = [("message", "Hello World", "text")]
|
|
||||||
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
|
||||||
|
|
||||||
# Receiver
|
|
||||||
payloads = smartreceive(msg)
|
|
||||||
for dataname, data, type in payloads:
|
|
||||||
print("Received {}: {}".format(dataname, data))
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Julia
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
|
|
||||||
# Sender
|
|
||||||
data = [("message", "Hello World", "text")]
|
|
||||||
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
|
||||||
|
|
||||||
# Receiver
|
|
||||||
envelope = smartreceive(msg, fileserverDownloadHandler)
|
|
||||||
# envelope["payloads"] = [("message", "Hello World", "text"), ...]
|
|
||||||
```
|
|
||||||
|
|
||||||
#### JavaScript
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const { smartsend, smartreceive } = require('./src/NATSBridge');
|
|
||||||
|
|
||||||
// Sender
|
|
||||||
await smartsend("/chat/room1", [
|
|
||||||
{ dataname: "message", data: "Hello World", type: "text" }
|
|
||||||
], { natsUrl: "nats://localhost:4222" });
|
|
||||||
|
|
||||||
// Receiver
|
|
||||||
const envelope = await smartreceive(msg);
|
|
||||||
// envelope.payloads = [{ dataname: "message", data: "Hello World", type: "text" }, ...]
|
|
||||||
```
|
|
||||||
|
|
||||||
### Sending JSON Configuration
|
|
||||||
|
|
||||||
#### Python/Micropython
|
|
||||||
|
|
||||||
```python
|
|
||||||
from nats_bridge import smartsend
|
|
||||||
|
|
||||||
config = {
|
|
||||||
"wifi_ssid": "MyNetwork",
|
|
||||||
"wifi_password": "password123",
|
|
||||||
"update_interval": 60
|
|
||||||
}
|
|
||||||
|
|
||||||
data = [("config", config, "dictionary")]
|
|
||||||
env = smartsend("/device/config", data, nats_url="nats://localhost:4222")
|
|
||||||
```
|
|
||||||
|
|
||||||
### Mixed Content (Chat with Text + Image)
|
|
||||||
|
|
||||||
#### Python/Micropython
|
|
||||||
|
|
||||||
```python
|
|
||||||
from nats_bridge import smartsend
|
|
||||||
|
|
||||||
image_data = b"\x89PNG..." # PNG bytes
|
|
||||||
|
|
||||||
data = [
|
|
||||||
("message_text", "Hello with image!", "text"),
|
|
||||||
("user_avatar", image_data, "binary")
|
|
||||||
]
|
|
||||||
|
|
||||||
env = smartsend("/chat/mixed", data, nats_url="nats://localhost:4222")
|
|
||||||
```
|
|
||||||
|
|
||||||
### Request-Response Pattern
|
|
||||||
|
|
||||||
#### Python/Micropython
|
|
||||||
|
|
||||||
```python
|
|
||||||
from nats_bridge import smartsend
|
|
||||||
|
|
||||||
# Send command with reply-to
|
|
||||||
data = [("command", {"action": "read_sensor"}, "dictionary")]
|
|
||||||
env = smartsend(
|
|
||||||
"/device/command",
|
|
||||||
data,
|
|
||||||
nats_url="nats://localhost:4222",
|
|
||||||
reply_to="/device/response",
|
|
||||||
reply_to_msg_id="cmd-001"
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Large Payloads (File Server)
|
|
||||||
|
|
||||||
#### Python/Micropython
|
|
||||||
|
|
||||||
```python
|
|
||||||
from nats_bridge import smartsend
|
|
||||||
|
|
||||||
# Large data (> 1MB)
|
|
||||||
large_data = b"A" * 2000000 # 2MB
|
|
||||||
|
|
||||||
env = smartsend(
|
|
||||||
"/data/large",
|
|
||||||
[("large_file", large_data, "binary")],
|
|
||||||
nats_url="nats://localhost:4222",
|
|
||||||
fileserver_url="http://localhost:8080",
|
|
||||||
size_threshold=1000000 # 1MB threshold
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
## API Reference
|
|
||||||
|
|
||||||
### `smartsend(subject, data, ...)`
|
|
||||||
|
|
||||||
Send data via NATS with automatic transport selection.
|
|
||||||
|
|
||||||
**Arguments:**
|
|
||||||
- `subject` (str): NATS subject to publish to
|
|
||||||
- `data` (list): List of `(dataname, data, type)` tuples
|
|
||||||
- `nats_url` (str): NATS server URL (default: `nats://localhost:4222`)
|
|
||||||
- `fileserver_url` (str): HTTP file server URL (default: `http://localhost:8080`)
|
|
||||||
- `size_threshold` (int): Threshold in bytes (default: 1,000,000)
|
|
||||||
- `correlation_id` (str): Optional correlation ID for tracing
|
|
||||||
- `msg_purpose` (str): Message purpose (default: `"chat"`)
|
|
||||||
- `sender_name` (str): Sender name (default: `"NATSBridge"`)
|
|
||||||
- `receiver_name` (str): Receiver name (default: `""`)
|
|
||||||
- `receiver_id` (str): Receiver ID (default: `""`)
|
|
||||||
- `reply_to` (str): Reply topic (default: `""`)
|
|
||||||
- `reply_to_msg_id` (str): Reply message ID (default: `""`)
|
|
||||||
|
|
||||||
**Returns:** `MessageEnvelope` object
|
|
||||||
|
|
||||||
### `smartreceive(msg, ...)`
|
|
||||||
|
|
||||||
Receive and process NATS messages.
|
|
||||||
|
|
||||||
**Arguments:**
|
|
||||||
- `msg`: NATS message (dict or JSON string)
|
|
||||||
- `fileserver_download_handler` (function): Function to fetch data from URLs
|
|
||||||
- `max_retries` (int): Maximum retry attempts (default: 5)
|
|
||||||
- `base_delay` (int): Initial delay in ms (default: 100)
|
|
||||||
- `max_delay` (int): Maximum delay in ms (default: 5000)
|
|
||||||
|
|
||||||
**Returns:** List of `(dataname, data, type)` tuples
|
|
||||||
|
|
||||||
### `MessageEnvelope`
|
|
||||||
|
|
||||||
Represents a complete NATS message envelope.
|
|
||||||
|
|
||||||
**Attributes:**
|
|
||||||
- `correlation_id`: Unique identifier for tracing
|
|
||||||
- `msg_id`: Unique message identifier
|
|
||||||
- `timestamp`: Message publication timestamp
|
|
||||||
- `send_to`: NATS subject
|
|
||||||
- `msg_purpose`: Message purpose
|
|
||||||
- `sender_name`: Sender name
|
|
||||||
- `sender_id`: Sender UUID
|
|
||||||
- `receiver_name`: Receiver name
|
|
||||||
- `receiver_id`: Receiver UUID
|
|
||||||
- `reply_to`: Reply topic
|
|
||||||
- `reply_to_msg_id`: Reply message ID
|
|
||||||
- `broker_url`: NATS broker URL
|
|
||||||
- `metadata`: Message-level metadata
|
|
||||||
- `payloads`: List of MessagePayload objects
|
|
||||||
|
|
||||||
### `MessagePayload`
|
|
||||||
|
|
||||||
Represents a single payload within a message envelope.
|
|
||||||
|
|
||||||
**Attributes:**
|
|
||||||
- `id`: Unique payload identifier
|
|
||||||
- `dataname`: Name of the payload
|
|
||||||
- `type`: Payload type ("text", "dictionary", etc.)
|
|
||||||
- `transport`: Transport method ("direct" or "link")
|
|
||||||
- `encoding`: Encoding method ("none", "base64", etc.)
|
|
||||||
- `size`: Payload size in bytes
|
|
||||||
- `data`: Payload data (bytes for direct, URL for link)
|
|
||||||
- `metadata`: Payload-level metadata
|
|
||||||
|
|
||||||
## Examples
|
|
||||||
|
|
||||||
See [`examples/micropython_example.py`](../examples/micropython_example.py) for more detailed examples.
|
|
||||||
|
|
||||||
## Testing
|
|
||||||
|
|
||||||
Run the test suite:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Python/Micropython
|
|
||||||
python test/test_micropython_basic.py
|
|
||||||
|
|
||||||
# JavaScript
|
|
||||||
node test/test_js_to_js_text_sender.js
|
|
||||||
node test/test_js_to_js_text_receiver.js
|
|
||||||
|
|
||||||
# Julia
|
|
||||||
julia test/test_julia_to_julia_text_sender.jl
|
|
||||||
julia test/test_julia_to_julia_text_receiver.jl
|
|
||||||
```
|
|
||||||
|
|
||||||
## Requirements
|
|
||||||
|
|
||||||
- **Julia**: NATS server (nats.io), HTTP file server (optional)
|
|
||||||
- **JavaScript**: NATS server (nats.io), HTTP file server (optional)
|
|
||||||
- **Python/Micropython**: NATS server (nats.io), HTTP file server (optional)
|
|
||||||
|
|
||||||
## License
|
|
||||||
|
|
||||||
MIT
|
|
||||||
@@ -437,7 +437,7 @@ def plik_oneshot_upload(file_server_url, filename, data):
|
|||||||
def smartsend(subject, data, nats_url=DEFAULT_NATS_URL, fileserver_url=DEFAULT_FILESERVER_URL,
|
def smartsend(subject, data, nats_url=DEFAULT_NATS_URL, fileserver_url=DEFAULT_FILESERVER_URL,
|
||||||
fileserver_upload_handler=plik_oneshot_upload, size_threshold=DEFAULT_SIZE_THRESHOLD,
|
fileserver_upload_handler=plik_oneshot_upload, size_threshold=DEFAULT_SIZE_THRESHOLD,
|
||||||
correlation_id=None, msg_purpose="chat", sender_name="NATSBridge",
|
correlation_id=None, msg_purpose="chat", sender_name="NATSBridge",
|
||||||
receiver_name="", receiver_id="", reply_to="", reply_to_msg_id=""):
|
receiver_name="", receiver_id="", reply_to="", reply_to_msg_id="", is_publish=True):
|
||||||
"""Send data either directly via NATS or via a fileserver URL, depending on payload size.
|
"""Send data either directly via NATS or via a fileserver URL, depending on payload size.
|
||||||
|
|
||||||
This function intelligently routes data delivery based on payload size relative to a threshold.
|
This function intelligently routes data delivery based on payload size relative to a threshold.
|
||||||
@@ -459,9 +459,12 @@ def smartsend(subject, data, nats_url=DEFAULT_NATS_URL, fileserver_url=DEFAULT_F
|
|||||||
receiver_id: UUID of the receiver
|
receiver_id: UUID of the receiver
|
||||||
reply_to: Topic to reply to
|
reply_to: Topic to reply to
|
||||||
reply_to_msg_id: Message ID this message is replying to
|
reply_to_msg_id: Message ID this message is replying to
|
||||||
|
is_publish: Whether to automatically publish the message to NATS (default: True)
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
MessageEnvelope: The envelope object for tracking
|
tuple: (env, env_json_str) where:
|
||||||
|
- env: MessageEnvelope object with all metadata and payloads
|
||||||
|
- env_json_str: JSON string representation of the envelope for publishing
|
||||||
"""
|
"""
|
||||||
# Generate correlation ID if not provided
|
# Generate correlation ID if not provided
|
||||||
cid = correlation_id if correlation_id else str(uuid.uuid4())
|
cid = correlation_id if correlation_id else str(uuid.uuid4())
|
||||||
@@ -549,13 +552,15 @@ def smartsend(subject, data, nats_url=DEFAULT_NATS_URL, fileserver_url=DEFAULT_F
|
|||||||
|
|
||||||
msg_json = env.to_json()
|
msg_json = env.to_json()
|
||||||
|
|
||||||
# Publish to NATS
|
# Publish to NATS if is_publish is True
|
||||||
|
if is_publish:
|
||||||
nats_conn = NATSConnection(nats_url)
|
nats_conn = NATSConnection(nats_url)
|
||||||
nats_conn.connect()
|
nats_conn.connect()
|
||||||
nats_conn.publish(subject, msg_json)
|
nats_conn.publish(subject, msg_json)
|
||||||
nats_conn.close()
|
nats_conn.close()
|
||||||
|
|
||||||
return env
|
# Return tuple of (envelope, json_string) for both direct and link transport
|
||||||
|
return (env, msg_json)
|
||||||
|
|
||||||
|
|
||||||
def smartreceive(msg, fileserver_download_handler=_fetch_with_backoff, max_retries=5,
|
def smartreceive(msg, fileserver_download_handler=_fetch_with_backoff, max_retries=5,
|
||||||
|
|||||||
@@ -118,7 +118,7 @@ async function test_dict_send() {
|
|||||||
// Use smartsend with dictionary type
|
// Use smartsend with dictionary type
|
||||||
// For small Dictionary: will use direct transport (JSON encoded)
|
// For small Dictionary: will use direct transport (JSON encoded)
|
||||||
// For large Dictionary: will use link transport (uploaded to fileserver)
|
// For large Dictionary: will use link transport (uploaded to fileserver)
|
||||||
const env = await smartsend(
|
const { env, env_json_str } = await smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2],
|
[data1, data2],
|
||||||
{
|
{
|
||||||
@@ -132,7 +132,8 @@ async function test_dict_send() {
|
|||||||
receiverName: "",
|
receiverName: "",
|
||||||
receiverId: "",
|
receiverId: "",
|
||||||
replyTo: "",
|
replyTo: "",
|
||||||
replyToMsgId: ""
|
replyToMsgId: "",
|
||||||
|
isPublish: true // Publish the message to NATS
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -98,7 +98,7 @@ async function test_large_binary_send() {
|
|||||||
|
|
||||||
// Use smartsend with binary type - will automatically use link transport
|
// Use smartsend with binary type - will automatically use link transport
|
||||||
// if file size exceeds the threshold (1MB by default)
|
// if file size exceeds the threshold (1MB by default)
|
||||||
const env = await smartsend(
|
const { env, env_json_str } = await smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2],
|
[data1, data2],
|
||||||
{
|
{
|
||||||
@@ -112,7 +112,8 @@ async function test_large_binary_send() {
|
|||||||
receiverName: "",
|
receiverName: "",
|
||||||
receiverId: "",
|
receiverId: "",
|
||||||
replyTo: "",
|
replyTo: "",
|
||||||
replyToMsgId: ""
|
replyToMsgId: "",
|
||||||
|
isPublish: true // Publish the message to NATS
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -222,7 +222,7 @@ async function test_mix_send() {
|
|||||||
];
|
];
|
||||||
|
|
||||||
// Use smartsend with mixed content
|
// Use smartsend with mixed content
|
||||||
const env = await smartsend(
|
const { env, env_json_str } = await smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
payloads,
|
payloads,
|
||||||
{
|
{
|
||||||
@@ -236,7 +236,8 @@ async function test_mix_send() {
|
|||||||
receiverName: "",
|
receiverName: "",
|
||||||
receiverId: "",
|
receiverId: "",
|
||||||
replyTo: "",
|
replyTo: "",
|
||||||
replyToMsgId: ""
|
replyToMsgId: "",
|
||||||
|
isPublish: true // Publish the message to NATS
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -118,7 +118,7 @@ async function test_table_send() {
|
|||||||
// Use smartsend with table type
|
// Use smartsend with table type
|
||||||
// For small Table: will use direct transport (Arrow IPC encoded)
|
// For small Table: will use direct transport (Arrow IPC encoded)
|
||||||
// For large Table: will use link transport (uploaded to fileserver)
|
// For large Table: will use link transport (uploaded to fileserver)
|
||||||
const env = await smartsend(
|
const { env, env_json_str } = await smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2],
|
[data1, data2],
|
||||||
{
|
{
|
||||||
@@ -132,7 +132,8 @@ async function test_table_send() {
|
|||||||
receiverName: "",
|
receiverName: "",
|
||||||
receiverId: "",
|
receiverId: "",
|
||||||
replyTo: "",
|
replyTo: "",
|
||||||
replyToMsgId: ""
|
replyToMsgId: "",
|
||||||
|
isPublish: true // Publish the message to NATS
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -94,7 +94,7 @@ async function test_text_send() {
|
|||||||
// Use smartsend with text type
|
// Use smartsend with text type
|
||||||
// For small text: will use direct transport (Base64 encoded UTF-8)
|
// For small text: will use direct transport (Base64 encoded UTF-8)
|
||||||
// For large text: will use link transport (uploaded to fileserver)
|
// For large text: will use link transport (uploaded to fileserver)
|
||||||
const env = await smartsend(
|
const { env, env_json_str } = await smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2],
|
[data1, data2],
|
||||||
{
|
{
|
||||||
@@ -108,7 +108,8 @@ async function test_text_send() {
|
|||||||
receiverName: "",
|
receiverName: "",
|
||||||
receiverId: "",
|
receiverId: "",
|
||||||
replyTo: "",
|
replyTo: "",
|
||||||
replyToMsgId: ""
|
replyToMsgId: "",
|
||||||
|
isPublish: true // Publish the message to NATS
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -92,12 +92,12 @@ function test_dict_send()
|
|||||||
# Use smartsend with dictionary type
|
# Use smartsend with dictionary type
|
||||||
# For small Dictionary: will use direct transport (JSON encoded)
|
# For small Dictionary: will use direct transport (JSON encoded)
|
||||||
# For large Dictionary: will use link transport (uploaded to fileserver)
|
# For large Dictionary: will use link transport (uploaded to fileserver)
|
||||||
env = NATSBridge.smartsend(
|
env, env_json_str = NATSBridge.smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2]; # List of (dataname, data, type) tuples
|
[data1, data2]; # List of (dataname, data, type) tuples
|
||||||
nats_url = NATS_URL,
|
broker_url = NATS_URL,
|
||||||
fileserver_url = FILESERVER_URL,
|
fileserver_url = FILESERVER_URL,
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
fileserver_upload_handler = plik_upload_handler,
|
||||||
size_threshold = 1_000_000, # 1MB threshold
|
size_threshold = 1_000_000, # 1MB threshold
|
||||||
correlation_id = correlation_id,
|
correlation_id = correlation_id,
|
||||||
msg_purpose = "chat",
|
msg_purpose = "chat",
|
||||||
@@ -105,7 +105,8 @@ function test_dict_send()
|
|||||||
receiver_name = "",
|
receiver_name = "",
|
||||||
receiver_id = "",
|
receiver_id = "",
|
||||||
reply_to = "",
|
reply_to = "",
|
||||||
reply_to_msg_id = ""
|
reply_to_msg_id = "",
|
||||||
|
is_publish = true # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace("Sent message with $(length(env.payloads)) payloads")
|
log_trace("Sent message with $(length(env.payloads)) payloads")
|
||||||
@@ -114,7 +115,7 @@ function test_dict_send()
|
|||||||
for (i, payload) in enumerate(env.payloads)
|
for (i, payload) in enumerate(env.payloads)
|
||||||
log_trace("Payload $i ('$payload.dataname'):")
|
log_trace("Payload $i ('$payload.dataname'):")
|
||||||
log_trace(" Transport: $(payload.transport)")
|
log_trace(" Transport: $(payload.transport)")
|
||||||
log_trace(" Type: $(payload.type)")
|
log_trace(" Type: $(payload.payload_type)")
|
||||||
log_trace(" Size: $(payload.size) bytes")
|
log_trace(" Size: $(payload.size) bytes")
|
||||||
log_trace(" Encoding: $(payload.encoding)")
|
log_trace(" Encoding: $(payload.encoding)")
|
||||||
|
|
||||||
@@ -79,12 +79,12 @@ function test_large_binary_send()
|
|||||||
# Use smartsend with binary type - will automatically use link transport
|
# Use smartsend with binary type - will automatically use link transport
|
||||||
# if file size exceeds the threshold (1MB by default)
|
# if file size exceeds the threshold (1MB by default)
|
||||||
# API: smartsend(subject, [(dataname, data, type), ...]; keywords...)
|
# API: smartsend(subject, [(dataname, data, type), ...]; keywords...)
|
||||||
env = NATSBridge.smartsend(
|
env, env_json_str = NATSBridge.smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2]; # List of (dataname, data, type) tuples
|
[data1, data2]; # List of (dataname, data, type) tuples
|
||||||
nats_url = NATS_URL;
|
broker_url = NATS_URL;
|
||||||
fileserver_url = FILESERVER_URL,
|
fileserver_url = FILESERVER_URL,
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
fileserver_upload_handler = plik_upload_handler,
|
||||||
size_threshold = 1_000_000,
|
size_threshold = 1_000_000,
|
||||||
correlation_id = correlation_id,
|
correlation_id = correlation_id,
|
||||||
msg_purpose = "chat",
|
msg_purpose = "chat",
|
||||||
@@ -92,11 +92,12 @@ function test_large_binary_send()
|
|||||||
receiver_name = "",
|
receiver_name = "",
|
||||||
receiver_id = "",
|
receiver_id = "",
|
||||||
reply_to = "",
|
reply_to = "",
|
||||||
reply_to_msg_id = ""
|
reply_to_msg_id = "",
|
||||||
|
is_publish = true # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace("Sent message with transport: $(env.payloads[1].transport)")
|
log_trace("Sent message with transport: $(env.payloads[1].transport)")
|
||||||
log_trace("Envelope type: $(env.payloads[1].type)")
|
log_trace("Envelope type: $(env.payloads[1].payload_type)")
|
||||||
|
|
||||||
# Check if link transport was used
|
# Check if link transport was used
|
||||||
if env.payloads[1].transport == "link"
|
if env.payloads[1].transport == "link"
|
||||||
@@ -186,12 +186,12 @@ function test_mix_send()
|
|||||||
]
|
]
|
||||||
|
|
||||||
# Use smartsend with mixed content
|
# Use smartsend with mixed content
|
||||||
env = NATSBridge.smartsend(
|
env, env_json_str = NATSBridge.smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
payloads; # List of (dataname, data, type) tuples
|
payloads; # List of (dataname, data, type) tuples
|
||||||
nats_url = NATS_URL,
|
broker_url = NATS_URL,
|
||||||
fileserver_url = FILESERVER_URL,
|
fileserver_url = FILESERVER_URL,
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
fileserver_upload_handler = plik_upload_handler,
|
||||||
size_threshold = 1_000_000, # 1MB threshold
|
size_threshold = 1_000_000, # 1MB threshold
|
||||||
correlation_id = correlation_id,
|
correlation_id = correlation_id,
|
||||||
msg_purpose = "chat",
|
msg_purpose = "chat",
|
||||||
@@ -199,7 +199,8 @@ function test_mix_send()
|
|||||||
receiver_name = "",
|
receiver_name = "",
|
||||||
receiver_id = "",
|
receiver_id = "",
|
||||||
reply_to = "",
|
reply_to = "",
|
||||||
reply_to_msg_id = ""
|
reply_to_msg_id = "",
|
||||||
|
is_publish = true # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace("Sent message with $(length(env.payloads)) payloads")
|
log_trace("Sent message with $(length(env.payloads)) payloads")
|
||||||
@@ -208,7 +209,7 @@ function test_mix_send()
|
|||||||
for (i, payload) in enumerate(env.payloads)
|
for (i, payload) in enumerate(env.payloads)
|
||||||
log_trace("Payload $i ('$payload.dataname'):")
|
log_trace("Payload $i ('$payload.dataname'):")
|
||||||
log_trace(" Transport: $(payload.transport)")
|
log_trace(" Transport: $(payload.transport)")
|
||||||
log_trace(" Type: $(payload.type)")
|
log_trace(" Type: $(payload.payload_type)")
|
||||||
log_trace(" Size: $(payload.size) bytes")
|
log_trace(" Size: $(payload.size) bytes")
|
||||||
log_trace(" Encoding: $(payload.encoding)")
|
log_trace(" Encoding: $(payload.encoding)")
|
||||||
|
|
||||||
@@ -90,12 +90,12 @@ function test_table_send()
|
|||||||
# Use smartsend with table type
|
# Use smartsend with table type
|
||||||
# For small DataFrame: will use direct transport (Base64 encoded Arrow IPC)
|
# For small DataFrame: will use direct transport (Base64 encoded Arrow IPC)
|
||||||
# For large DataFrame: will use link transport (uploaded to fileserver)
|
# For large DataFrame: will use link transport (uploaded to fileserver)
|
||||||
env = NATSBridge.smartsend(
|
env, env_json_str = NATSBridge.smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2]; # List of (dataname, data, type) tuples
|
[data1, data2]; # List of (dataname, data, type) tuples
|
||||||
nats_url = NATS_URL,
|
broker_url = NATS_URL,
|
||||||
fileserver_url = FILESERVER_URL,
|
fileserver_url = FILESERVER_URL,
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
fileserver_upload_handler = plik_upload_handler,
|
||||||
size_threshold = 1_000_000, # 1MB threshold
|
size_threshold = 1_000_000, # 1MB threshold
|
||||||
correlation_id = correlation_id,
|
correlation_id = correlation_id,
|
||||||
msg_purpose = "chat",
|
msg_purpose = "chat",
|
||||||
@@ -103,7 +103,8 @@ function test_table_send()
|
|||||||
receiver_name = "",
|
receiver_name = "",
|
||||||
receiver_id = "",
|
receiver_id = "",
|
||||||
reply_to = "",
|
reply_to = "",
|
||||||
reply_to_msg_id = ""
|
reply_to_msg_id = "",
|
||||||
|
is_publish = true # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace("Sent message with $(length(env.payloads)) payloads")
|
log_trace("Sent message with $(length(env.payloads)) payloads")
|
||||||
@@ -112,7 +113,7 @@ function test_table_send()
|
|||||||
for (i, payload) in enumerate(env.payloads)
|
for (i, payload) in enumerate(env.payloads)
|
||||||
log_trace("Payload $i ('$payload.dataname'):")
|
log_trace("Payload $i ('$payload.dataname'):")
|
||||||
log_trace(" Transport: $(payload.transport)")
|
log_trace(" Transport: $(payload.transport)")
|
||||||
log_trace(" Type: $(payload.type)")
|
log_trace(" Type: $(payload.payload_type)")
|
||||||
log_trace(" Size: $(payload.size) bytes")
|
log_trace(" Size: $(payload.size) bytes")
|
||||||
log_trace(" Encoding: $(payload.encoding)")
|
log_trace(" Encoding: $(payload.encoding)")
|
||||||
|
|
||||||
@@ -75,12 +75,12 @@ function test_text_send()
|
|||||||
# Use smartsend with text type
|
# Use smartsend with text type
|
||||||
# For small text: will use direct transport (Base64 encoded UTF-8)
|
# For small text: will use direct transport (Base64 encoded UTF-8)
|
||||||
# For large text: will use link transport (uploaded to fileserver)
|
# For large text: will use link transport (uploaded to fileserver)
|
||||||
env = NATSBridge.smartsend(
|
env, env_json_str = NATSBridge.smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2]; # List of (dataname, data, type) tuples
|
[data1, data2]; # List of (dataname, data, type) tuples
|
||||||
nats_url = NATS_URL,
|
broker_url = NATS_URL,
|
||||||
fileserver_url = FILESERVER_URL,
|
fileserver_url = FILESERVER_URL,
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
fileserver_upload_handler = plik_upload_handler,
|
||||||
size_threshold = 1_000_000, # 1MB threshold
|
size_threshold = 1_000_000, # 1MB threshold
|
||||||
correlation_id = correlation_id,
|
correlation_id = correlation_id,
|
||||||
msg_purpose = "chat",
|
msg_purpose = "chat",
|
||||||
@@ -88,7 +88,8 @@ function test_text_send()
|
|||||||
receiver_name = "",
|
receiver_name = "",
|
||||||
receiver_id = "",
|
receiver_id = "",
|
||||||
reply_to = "",
|
reply_to = "",
|
||||||
reply_to_msg_id = ""
|
reply_to_msg_id = "",
|
||||||
|
is_publish = true # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace("Sent message with $(length(env.payloads)) payloads")
|
log_trace("Sent message with $(length(env.payloads)) payloads")
|
||||||
@@ -97,7 +98,7 @@ function test_text_send()
|
|||||||
for (i, payload) in enumerate(env.payloads)
|
for (i, payload) in enumerate(env.payloads)
|
||||||
log_trace("Payload $i ('$payload.dataname'):")
|
log_trace("Payload $i ('$payload.dataname'):")
|
||||||
log_trace(" Transport: $(payload.transport)")
|
log_trace(" Transport: $(payload.transport)")
|
||||||
log_trace(" Type: $(payload.type)")
|
log_trace(" Type: $(payload.payload_type)")
|
||||||
log_trace(" Size: $(payload.size) bytes")
|
log_trace(" Size: $(payload.size) bytes")
|
||||||
log_trace(" Encoding: $(payload.encoding)")
|
log_trace(" Encoding: $(payload.encoding)")
|
||||||
|
|
||||||
@@ -64,7 +64,7 @@ def main():
|
|||||||
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
||||||
|
|
||||||
# Use smartsend with dictionary type
|
# Use smartsend with dictionary type
|
||||||
env = smartsend(
|
env, env_json_str = smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2], # List of (dataname, data, type) tuples
|
[data1, data2], # List of (dataname, data, type) tuples
|
||||||
nats_url=NATS_URL,
|
nats_url=NATS_URL,
|
||||||
@@ -76,7 +76,8 @@ def main():
|
|||||||
receiver_name="",
|
receiver_name="",
|
||||||
receiver_id="",
|
receiver_id="",
|
||||||
reply_to="",
|
reply_to="",
|
||||||
reply_to_msg_id=""
|
reply_to_msg_id="",
|
||||||
|
is_publish=True # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
||||||
|
|||||||
@@ -44,7 +44,7 @@ def main():
|
|||||||
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
||||||
|
|
||||||
# Use smartsend with binary type
|
# Use smartsend with binary type
|
||||||
env = smartsend(
|
env, env_json_str = smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2], # List of (dataname, data, type) tuples
|
[data1, data2], # List of (dataname, data, type) tuples
|
||||||
nats_url=NATS_URL,
|
nats_url=NATS_URL,
|
||||||
@@ -56,7 +56,8 @@ def main():
|
|||||||
receiver_name="",
|
receiver_name="",
|
||||||
receiver_id="",
|
receiver_id="",
|
||||||
reply_to="",
|
reply_to="",
|
||||||
reply_to_msg_id=""
|
reply_to_msg_id="",
|
||||||
|
is_publish=True # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
||||||
|
|||||||
@@ -58,7 +58,7 @@ def main():
|
|||||||
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
||||||
|
|
||||||
# Use smartsend with mixed types
|
# Use smartsend with mixed types
|
||||||
env = smartsend(
|
env, env_json_str = smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
data, # List of (dataname, data, type) tuples
|
data, # List of (dataname, data, type) tuples
|
||||||
nats_url=NATS_URL,
|
nats_url=NATS_URL,
|
||||||
@@ -70,7 +70,8 @@ def main():
|
|||||||
receiver_name="",
|
receiver_name="",
|
||||||
receiver_id="",
|
receiver_id="",
|
||||||
reply_to="",
|
reply_to="",
|
||||||
reply_to_msg_id=""
|
reply_to_msg_id="",
|
||||||
|
is_publish=True # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
||||||
|
|||||||
@@ -46,7 +46,7 @@ def main():
|
|||||||
# Use smartsend with text type
|
# Use smartsend with text type
|
||||||
# For small text: will use direct transport (Base64 encoded UTF-8)
|
# For small text: will use direct transport (Base64 encoded UTF-8)
|
||||||
# For large text: will use link transport (uploaded to fileserver)
|
# For large text: will use link transport (uploaded to fileserver)
|
||||||
env = smartsend(
|
env, env_json_str = smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2], # List of (dataname, data, type) tuples
|
[data1, data2], # List of (dataname, data, type) tuples
|
||||||
nats_url=NATS_URL,
|
nats_url=NATS_URL,
|
||||||
@@ -58,7 +58,8 @@ def main():
|
|||||||
receiver_name="",
|
receiver_name="",
|
||||||
receiver_id="",
|
receiver_id="",
|
||||||
reply_to="",
|
reply_to="",
|
||||||
reply_to_msg_id=""
|
reply_to_msg_id="",
|
||||||
|
is_publish=True # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
||||||
|
|||||||
Reference in New Issue
Block a user