22 Commits

Author SHA1 Message Date
f8a92a45a0 update README.md 2026-02-23 09:39:24 +07:00
cec70e6036 update 2026-02-23 08:11:03 +07:00
f9e08ba628 add Plik fileserver 2026-02-23 07:58:18 +07:00
c12a078149 update README.md 2026-02-23 07:55:10 +07:00
dedd803dc3 fix README.md 2026-02-23 07:24:54 +07:00
e8e927a491 move README.md 2026-02-23 07:17:31 +07:00
ton
d950bbac23 Merge pull request 'smartreceive_return_envelope' (#7) from smartreceive_return_envelope into main
Reviewed-on: #7
2026-02-23 00:11:09 +00:00
fc8da2ebf5 update 2026-02-23 07:08:17 +07:00
f6e50c405f update 2026-02-23 07:06:53 +07:00
ton
c06f508e8f Merge pull request 'smartreceive_return_envelope' (#6) from smartreceive_return_envelope into main
Reviewed-on: #6
2026-02-22 23:59:13 +00:00
97bf1e47f4 update 2026-02-23 06:58:16 +07:00
ef47fddd56 update 2026-02-23 06:28:41 +07:00
896dd84d2a update 2026-02-22 22:19:47 +07:00
def75d8f86 update 2026-02-22 21:55:18 +07:00
69f2173f75 update 2026-02-22 20:52:13 +07:00
075d355c58 update 2026-02-22 20:43:28 +07:00
ton
0de9725ba8 Merge pull request 'add Base64 in project.toml' (#5) from fix_precompile_issue into main
Reviewed-on: #5
2026-02-22 07:16:15 +00:00
6dcccc903f add Base64 in project.toml 2026-02-22 14:15:24 +07:00
ton
507b4951b4 Merge pull request 'add julia project file' (#4) from add_julia_project_file into main
Reviewed-on: #4
2026-02-22 07:02:07 +00:00
a064be0e5c update 2026-02-22 13:54:36 +07:00
ton
8a35f1d4dc Merge pull request 'add micropython support' (#3) from add_micropython into main
Reviewed-on: #3
2026-02-22 06:28:25 +00:00
9e5ee61785 update 2026-02-22 13:26:44 +07:00
43 changed files with 5005 additions and 1855 deletions

View File

@@ -2,7 +2,7 @@
julia_version = "1.12.5"
manifest_format = "2.0"
project_hash = "8a7a8b88d777403234a6816e699fb0ab1e991aac"
project_hash = "b632f853bcf5355f5c53ad3efa7a19f70444dc6c"
[[deps.AliasTables]]
deps = ["PtrArrays", "Random"]
@@ -436,6 +436,12 @@ git-tree-sha1 = "d9d9a189fb9155a460e6b5e8966bf6a66737abf8"
uuid = "55e73f9c-eeeb-467f-b4cc-a633fde63d2a"
version = "0.1.0"
[[deps.NATSBridge]]
deps = ["Arrow", "DataFrames", "Dates", "GeneralUtils", "HTTP", "JSON", "NATS", "PrettyPrinting", "Revise", "UUIDs"]
path = "."
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
version = "0.4.1"
[[deps.NanoDates]]
deps = ["Dates", "Parsers"]
git-tree-sha1 = "850a0557ae5934f6e67ac0dc5ca13d0328422d1f"

View File

@@ -1,5 +1,11 @@
name = "NATSBridge"
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
version = "0.4.2"
authors = ["narawat <narawat@gmail.com>"]
[deps]
Arrow = "69666777-d1a9-59fb-9406-91d4454c9d45"
Base64 = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
GeneralUtils = "c6c72f09-b708-4ac8-ac7c-2084d70108fe"
@@ -9,3 +15,7 @@ NATS = "55e73f9c-eeeb-467f-b4cc-a633fde63d2a"
PrettyPrinting = "54e16d92-306c-5ea0-a30b-337be88ac337"
Revise = "295af30f-e4ad-537b-8983-00126c2a3abe"
UUIDs = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"
[compat]
Base64 = "1.11.0"
JSON = "1.4.0"

957
README.md Normal file
View File

@@ -0,0 +1,957 @@
# NATSBridge
A high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![NATS](https://img.shields.io/badge/NATS-Enabled-green.svg)](https://nats.io)
---
## Table of Contents
- [Overview](#overview)
- [Features](#features)
- [Architecture](#architecture)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [API Reference](#api-reference)
- [Payload Types](#payload-types)
- [Transport Strategies](#transport-strategies)
- [Examples](#examples)
- [Testing](#testing)
- [License](#license)
---
## Overview
NATSBridge enables seamless communication across Julia, JavaScript, and Python/Micropython applications through NATS, with intelligent transport selection based on payload size:
| Transport | Payload Size | Method |
|-----------|--------------|--------|
| **Direct** | < 1MB | Sent directly via NATS (Base64 encoded) |
| **Link** | >= 1MB | Uploaded to HTTP file server, URL sent via NATS |
### Use Cases
- **Chat Applications**: Text, images, audio, video in a single message
- **File Transfer**: Efficient transfer of large files using claim-check pattern
- **Streaming Data**: Sensor data, telemetry, and analytics pipelines
- **Cross-Platform Communication**: Julia ↔ JavaScript ↔ Python/Micropython
- **IoT Devices**: Micropython devices sending data to cloud services
---
## Features
-**Bi-directional messaging** between Julia, JavaScript, and Python/Micropython
-**Multi-payload support** - send multiple payloads with different types in one message
-**Automatic transport selection** - direct vs link based on payload size
-**Claim-Check pattern** for payloads > 1MB
-**Apache Arrow IPC** support for tabular data (zero-copy reading)
-**Exponential backoff** for reliable file server downloads
-**Correlation ID tracking** for message tracing
-**Reply-to support** for request-response patterns
-**JetStream support** for message replay and durability
-**Lightweight Micropython implementation** for microcontrollers
---
## Architecture
### System Components
```
┌─────────────────────────────────────────────────────────────────────┐
│ NATSBridge Architecture │
├─────────────────────────────────────────────────────────────────────┤
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Julia │ │ JavaScript │ │ Python/ │ │
│ │ (NATS.jl) │◄──►│ (nats.js) │◄──►│ Micropython │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ NATS │ │
│ │ (Message Broker) │ │
│ └─────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ File Server │ │
│ │ (HTTP Upload/Get) │ │
│ └──────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
```
### Message Flow
1. **Sender** creates a message envelope with payloads
2. **NATSBridge** serializes and encodes payloads based on type
3. **Transport Decision**: Small payloads go directly to NATS, large payloads are uploaded to file server
4. **NATS** routes messages to subscribers
5. **Receiver** fetches payloads (from NATS or file server)
6. **NATSBridge** deserializes and decodes payloads
---
## Installation
### Prerequisites
- **NATS Server** (v2.10+ recommended)
- **HTTP File Server** (optional, for payloads > 1MB)
### Julia
```julia
using Pkg
Pkg.add("NATS")
Pkg.add("https://git.yiem.cc/ton/NATSBridge")
```
### JavaScript
```bash
npm install nats.js apache-arrow uuid base64-url
```
For Node.js:
```javascript
const { smartsend, smartreceive } = require('./src/NATSBridge');
```
For browser:
```html
<script src="./src/NATSBridge.js"></script>
<script>
// NATSBridge is available as window.NATSBridge
</script>
```
### Python/Micropython
1. Copy [`src/nats_bridge.py`](src/nats_bridge.py) to your device
2. Install dependencies:
**For Python (desktop):**
```bash
pip install nats-py
```
**For Micropython:**
- `urequests` for HTTP requests (built-in for ESP32)
- `base64` for base64 encoding (built-in)
- `json` for JSON handling (built-in)
---
## Quick Start
### Step 1: Start NATS Server
```bash
docker run -p 4222:4222 nats:latest
```
### Step 2: Start HTTP File Server (Optional)
```bash
# Create a directory for file uploads
mkdir -p /tmp/fileserver
# Use Python's built-in server
python3 -m http.server 8080 --directory /tmp/fileserver
```
### Step 3: Send Your First Message
#### Python/Micropython
```python
from nats_bridge import smartsend
# Send a text message
data = [("message", "Hello World", "text")]
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
print("Message sent!")
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
// Send a text message
await smartsend("/chat/room1", [
{ dataname: "message", data: "Hello World", type: "text" }
], { natsUrl: "nats://localhost:4222" });
console.log("Message sent!");
```
#### Julia
```julia
using NATSBridge
# Send a text message
data = [("message", "Hello World", "text")]
env = NATSBridge.smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
println("Message sent!")
```
### Step 4: Receive Messages
#### Python/Micropython
```python
import nats
import asyncio
from nats_bridge import smartreceive
# Configuration
SUBJECT = "/chat/room1"
NATS_URL = "nats://localhost:4222"
async def main():
# Connect to NATS
nc = await nats.connect(NATS_URL)
# Subscribe to the subject - msg comes from the callback
async def message_handler(msg):
# Receive and process message
envelope = smartreceive(msg.data)
for dataname, data, type in envelope["payloads"]:
print(f"Received {dataname}: {data}")
sid = await nc.subscribe(SUBJECT, cb=message_handler)
await asyncio.sleep(120) # Keep listening
await nc.close()
asyncio.run(main())
```
#### JavaScript
```javascript
const { smartreceive } = require('./src/NATSBridge');
const { connect } = require('nats');
// Configuration
const SUBJECT = "/chat/room1";
const NATS_URL = "nats://localhost:4222";
async function main() {
// Connect to NATS
const nc = await connect({ servers: [NATS_URL] });
// Subscribe to the subject - msg comes from the async iteration
const sub = nc.subscribe(SUBJECT);
for await (const msg of sub) {
// Receive and process message
const envelope = await smartreceive(msg);
for (const payload of envelope.payloads) {
console.log(`Received ${payload.dataname}: ${payload.data}`);
}
}
}
main();
```
#### Julia
```julia
using NATS, NATSBridge
# Configuration
const SUBJECT = "/chat/room1"
const NATS_URL = "nats://localhost:4222"
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] $message")
end
# Receiver: Listen for messages - msg comes from the callback
function test_receive()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
log_trace("Received message on $(msg.subject)")
# Receive and process message
envelope = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
for (dataname, data, type) in envelope["payloads"]
println("Received $dataname: $data")
end
end
# Keep listening for 120 seconds
sleep(120)
NATS.drain(conn)
end
test_receive()
```
---
## API Reference
### smartsend
Sends data either directly via NATS or via a fileserver URL, depending on payload size.
#### Python/Micropython
```python
from nats_bridge import smartsend
env = smartsend(
subject, # NATS subject to publish to
data, # List of (dataname, data, type) tuples
nats_url="nats://localhost:4222", # NATS server URL
fileserver_url="http://localhost:8080", # File server URL
fileserver_upload_handler=plik_oneshot_upload, # Upload handler function
size_threshold=1_000_000, # Threshold in bytes (default: 1MB)
correlation_id=None, # Optional correlation ID for tracing
msg_purpose="chat", # Message purpose
sender_name="NATSBridge", # Sender name
receiver_name="", # Receiver name (empty = broadcast)
receiver_id="", # Receiver UUID (empty = broadcast)
reply_to="", # Reply topic
reply_to_msg_id="" # Reply message ID
)
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
const env = await smartsend(
subject, // NATS subject
data, // Array of {dataname, data, type}
{
natsUrl: "nats://localhost:4222",
fileserverUrl: "http://localhost:8080",
fileserverUploadHandler: customUploadHandler,
sizeThreshold: 1_000_000,
correlationId: "custom-id",
msgPurpose: "chat",
senderName: "NATSBridge",
receiverName: "",
receiverId: "",
replyTo: "",
replyToMsgId: ""
}
);
```
#### Julia
```julia
using NATSBridge
env = NATSBridge.smartsend(
subject; # NATS subject
data::AbstractArray{Tuple{String, Any, String}}; # List of (dataname, data, type)
nats_url::String = "nats://localhost:4222",
fileserver_url = "http://localhost:8080",
fileserverUploadHandler::Function = plik_oneshot_upload,
size_threshold::Int = 1_000_000,
correlation_id::Union{String, Nothing} = nothing,
msg_purpose::String = "chat",
sender_name::String = "NATSBridge",
receiver_name::String = "",
receiver_id::String = "",
reply_to::String = "",
reply_to_msg_id::String = ""
)
```
### smartreceive
Receives and processes messages from NATS, handling both direct and link transport.
#### Python/Micropython
```python
from nats_bridge import smartreceive
# Note: For nats-py, use msg.data to pass the raw message data
envelope = smartreceive(
msg.data, # NATS message data (msg.data for nats-py)
fileserver_download_handler=_fetch_with_backoff, # Download handler
max_retries=5, # Max retry attempts
base_delay=100, # Initial delay in ms
max_delay=5000 # Max delay in ms
)
# Returns: Dict with envelope metadata and 'payloads' field
```
#### JavaScript
```javascript
const { smartreceive } = require('./src/NATSBridge');
// Note: msg is the NATS message object from subscription
const envelope = await smartreceive(
msg, // NATS message (raw object from subscription)
{
fileserverDownloadHandler: customDownloadHandler,
maxRetries: 5,
baseDelay: 100,
maxDelay: 5000
}
);
// Returns: Object with envelope metadata and payloads array
```
#### Julia
```julia
using NATSBridge
# Note: msg is a NATS.Msg object passed from the subscription callback
envelope = NATSBridge.smartreceive(
msg::NATS.Msg;
fileserverDownloadHandler::Function = _fetch_with_backoff,
max_retries::Int = 5,
base_delay::Int = 100,
max_delay::Int = 5000
)
# Returns: Dict with envelope metadata and payloads array
```
---
## Payload Types
| Type | Description | Serialization |
|------|-------------|---------------|
| `text` | Plain text strings | UTF-8 bytes |
| `dictionary` | JSON-serializable dictionaries | JSON |
| `table` | Tabular data (DataFrames, arrays) | Apache Arrow IPC |
| `image` | Image data (PNG, JPG) | Raw bytes |
| `audio` | Audio data (WAV, MP3) | Raw bytes |
| `video` | Video data (MP4, AVI) | Raw bytes |
| `binary` | Generic binary data | Raw bytes |
---
## Transport Strategies
### Direct Transport (Payloads < 1MB)
Small payloads are sent directly via NATS with Base64 encoding.
#### Python/Micropython
```python
data = [("message", "Hello", "text")]
smartsend("/topic", data)
```
#### JavaScript
```javascript
await smartsend("/topic", [
{ dataname: "message", data: "Hello", type: "text" }
]);
```
#### Julia
```julia
data = [("message", "Hello", "text")]
smartsend("/topic", data)
```
### Link Transport (Payloads >= 1MB)
Large payloads are uploaded to an HTTP file server.
#### Python/Micropython
```python
data = [("file", large_data, "binary")]
smartsend("/topic", data, fileserver_url="http://localhost:8080")
```
#### JavaScript
```javascript
await smartsend("/topic", [
{ dataname: "file", data: largeData, type: "binary" }
], { fileserverUrl: "http://localhost:8080" });
```
#### Julia
```julia
data = [("file", large_data, "binary")]
smartsend("/topic", data, fileserver_url="http://localhost:8080")
```
---
## Examples
All examples include code for **Julia**, **JavaScript**, and **Python/Micropython** unless otherwise specified.
### Example 1: Chat with Mixed Content
Send text, small image, and large file in one message.
#### Python/Micropython
```python
from nats_bridge import smartsend
data = [
("message_text", "Hello!", "text"),
("user_avatar", image_data, "image"),
("large_document", large_file_data, "binary")
]
smartsend("/chat/room1", data, fileserver_url="http://localhost:8080")
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
await smartsend("/chat/room1", [
{ dataname: "message_text", data: "Hello!", type: "text" },
{ dataname: "user_avatar", data: image_data, type: "image" },
{ dataname: "large_document", data: large_file_data, type: "binary" }
], {
fileserverUrl: "http://localhost:8080"
});
```
#### Julia
```julia
using NATSBridge
data = [
("message_text", "Hello!", "text"),
("user_avatar", image_data, "image"),
("large_document", large_file_data, "binary")
]
NATSBridge.smartsend("/chat/room1", data, fileserver_url="http://localhost:8080")
```
### Example 2: Dictionary Exchange
Send configuration data between platforms.
#### Python/Micropython
```python
from nats_bridge import smartsend
config = {
"wifi_ssid": "MyNetwork",
"wifi_password": "password123",
"update_interval": 60
}
data = [("config", config, "dictionary")]
smartsend("/device/config", data)
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
const config = {
wifi_ssid: "MyNetwork",
wifi_password: "password123",
update_interval: 60
};
await smartsend("/device/config", [
{ dataname: "config", data: config, type: "dictionary" }
]);
```
#### Julia
```julia
using NATSBridge
config = Dict(
"wifi_ssid" => "MyNetwork",
"wifi_password" => "password123",
"update_interval" => 60
)
data = [("config", config, "dictionary")]
NATSBridge.smartsend("/device/config", data)
```
### Example 3: Table Data (Arrow IPC)
Send tabular data using Apache Arrow IPC format.
#### Python/Micropython
```python
import pandas as pd
from nats_bridge import smartsend
df = pd.DataFrame({
"id": [1, 2, 3],
"name": ["Alice", "Bob", "Charlie"],
"score": [95, 88, 92]
})
data = [("students", df, "table")]
smartsend("/data/analysis", data)
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
const tableData = [
{ id: 1, name: "Alice", score: 95 },
{ id: 2, name: "Bob", score: 88 },
{ id: 3, name: "Charlie", score: 92 }
];
await smartsend("/data/analysis", [
{ dataname: "students", data: tableData, type: "table" }
]);
```
#### Julia
```julia
using NATSBridge
using DataFrames
df = DataFrame(
id = [1, 2, 3],
name = ["Alice", "Bob", "Charlie"],
score = [95, 88, 92]
)
data = [("students", df, "table")]
NATSBridge.smartsend("/data/analysis", data)
```
### Example 4: Request-Response Pattern
Bi-directional communication with reply-to support.
#### Python/Micropython (Requester)
```python
from nats_bridge import smartsend
env = smartsend(
"/device/command",
[("command", {"action": "read_sensor"}, "dictionary")],
reply_to="/device/response"
)
```
#### Python/Micropython (Responder)
```python
import nats
import asyncio
from nats_bridge import smartreceive, smartsend
# Configuration
SUBJECT = "/device/command"
REPLY_SUBJECT = "/device/response"
NATS_URL = "nats://localhost:4222"
async def main():
nc = await nats.connect(NATS_URL)
async def message_handler(msg):
envelope = smartreceive(msg.data)
for dataname, data, type in envelope["payloads"]:
if data.get("action") == "read_sensor":
response = {"sensor_id": "sensor-001", "value": 42.5}
smartsend(REPLY_SUBJECT, [("data", response, "dictionary")])
sid = await nc.subscribe(SUBJECT, cb=message_handler)
await asyncio.sleep(120)
await nc.close()
asyncio.run(main())
```
#### JavaScript (Requester)
```javascript
const { smartsend } = require('./src/NATSBridge');
await smartsend("/device/command", [
{ dataname: "command", data: { action: "read_sensor" }, type: "dictionary" }
], {
replyTo: "/device/response"
});
```
#### JavaScript (Responder)
```javascript
const { smartreceive, smartsend } = require('./src/NATSBridge');
const { connect } = require('nats');
// Configuration
const SUBJECT = "/device/command";
const REPLY_SUBJECT = "/device/response";
const NATS_URL = "nats://localhost:4222";
async function main() {
const nc = await connect({ servers: [NATS_URL] });
const sub = nc.subscribe(SUBJECT);
for await (const msg of sub) {
const envelope = await smartreceive(msg);
for (const payload of envelope.payloads) {
if (payload.dataname === "command" && payload.data.action === "read_sensor") {
const response = { sensor_id: "sensor-001", value: 42.5 };
await smartsend(REPLY_SUBJECT, [
{ dataname: "data", data: response, type: "dictionary" }
]);
}
}
}
}
main();
```
#### Julia (Requester)
```julia
using NATSBridge
env = NATSBridge.smartsend(
"/device/command",
[("command", Dict("action" => "read_sensor"), "dictionary")],
reply_to="/device/response"
)
```
#### Julia (Responder)
```julia
using NATS, NATSBridge
# Configuration
const SUBJECT = "/device/command"
const REPLY_SUBJECT = "/device/response"
const NATS_URL = "nats://localhost:4222"
function test_responder()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
envelope = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
for (dataname, data, type) in envelope["payloads"]
if dataname == "command" && data["action"] == "read_sensor"
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
smartsend(REPLY_SUBJECT, [("data", response, "dictionary")])
end
end
end
sleep(120)
NATS.drain(conn)
end
test_responder()
```
### Example 5: Micropython IoT Device
Lightweight Micropython device sending sensor data.
#### Micropython
```python
import nats
import asyncio
from nats_bridge import smartsend, smartreceive
# Configuration
SUBJECT = "/device/sensors"
NATS_URL = "nats://localhost:4222"
async def main():
nc = await nats.connect(NATS_URL)
# Send sensor data
data = [("temperature", "25.5", "text"), ("humidity", 65, "dictionary")]
smartsend("/device/sensors", data, nats_url="nats://localhost:4222")
# Receive commands - msg comes from the callback
async def message_handler(msg):
envelope = smartreceive(msg.data)
for dataname, data, type in envelope["payloads"]:
if type == "dictionary" and data.get("action") == "reboot":
# Execute reboot
pass
sid = await nc.subscribe(SUBJECT, cb=message_handler)
await asyncio.sleep(120)
await nc.close()
asyncio.run(main())
```
#### JavaScript (Receiver)
```javascript
const { smartreceive } = require('./src/NATSBridge');
const { connect } = require('nats');
// Configuration
const SUBJECT = "/device/sensors";
const NATS_URL = "nats://localhost:4222";
async function main() {
const nc = await connect({ servers: [NATS_URL] });
const sub = nc.subscribe(SUBJECT);
for await (const msg of sub) {
const envelope = await smartreceive(msg);
for (const payload of envelope.payloads) {
if (payload.dataname === "temperature") {
console.log(`Temperature: ${payload.data}`);
} else if (payload.dataname === "humidity") {
console.log(`Humidity: ${payload.data}`);
}
}
}
}
main();
```
#### Julia (Receiver)
```julia
using NATS, NATSBridge
# Configuration
const SUBJECT = "/device/sensors"
const NATS_URL = "nats://localhost:4222"
function test_receiver()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
envelope = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
for (dataname, data, type) in envelope["payloads"]
if dataname == "temperature"
println("Temperature: $data")
elseif dataname == "humidity"
println("Humidity: $data")
end
end
end
sleep(120)
NATS.drain(conn)
end
test_receiver()
```
---
## Testing
Run the test scripts to verify functionality:
### Python/Micropython
```bash
# Basic functionality test
python test/test_micropython_basic.py
# Text message exchange
python test/test_micropython_text_sender.py
python test/test_micropython_text_receiver.py
# Dictionary exchange
python test/test_micropython_dict_sender.py
python test/test_micropython_dict_receiver.py
# File transfer
python test/test_micropython_file_sender.py
python test/test_micropython_file_receiver.py
# Mixed payload types
python test/test_micropython_mixed_sender.py
python test/test_micropython_mixed_receiver.py
```
### JavaScript
```bash
# Text message exchange
node test/test_js_text_sender.js
node test/test_js_text_receiver.js
# Dictionary exchange
node test/test_js_dict_sender.js
node test/test_js_dict_receiver.js
# File transfer
node test/test_js_file_sender.js
node test/test_js_file_receiver.js
# Mixed payload types
node test/test_js_mix_payload_sender.js
node test/test_js_mix_payloads_receiver.js
# Table exchange
node test/test_js_table_sender.js
node test/test_js_table_receiver.js
```
### Julia
```julia
# Text message exchange
julia test/test_julia_text_sender.jl
julia test/test_julia_text_receiver.jl
# Dictionary exchange
julia test/test_julia_dict_sender.jl
julia test/test_julia_dict_receiver.jl
# File transfer
julia test/test_julia_file_sender.jl
julia test/test_julia_file_receiver.jl
# Mixed payload types
julia test/test_julia_mix_payloads_sender.jl
julia test/test_julia_mix_payloads_receiver.jl
# Table exchange
julia test/test_julia_table_sender.jl
julia test/test_julia_table_receiver.jl
```
---
## License
MIT License
Copyright (c) 2026 NATSBridge Contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View File

@@ -1,8 +1,13 @@
# Architecture Documentation: Bi-Directional Data Bridge (Julia ↔ JavaScript)
# Architecture Documentation: Bi-Directional Data Bridge
## Overview
This document describes the architecture for a high-performance, bi-directional data bridge between a Julia service and a JavaScript (Node.js) service using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
This document describes the architecture for a high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
The system enables seamless communication across all three platforms:
- **Julia ↔ JavaScript** bi-directional messaging
- **JavaScript ↔ Python/Micropython** bi-directional messaging
- **Julia ↔ Python/Micropython** bi-directional messaging (via JSON serialization)
### File Server Handler Architecture
@@ -35,8 +40,24 @@ The system uses a **standardized list-of-tuples format** for all payload operati
# Input format for smartsend (always a list of tuples with type info)
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
# Output format for smartreceive (always returns a list of tuples)
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
# Output format for smartreceive (returns envelope dictionary with payloads field)
# Returns: Dict with envelope metadata and payloads field containing list of tuples
# {
# "correlationId": "...",
# "msgId": "...",
# "timestamp": "...",
# "sendTo": "...",
# "msgPurpose": "...",
# "senderName": "...",
# "senderId": "...",
# "receiverName": "...",
# "receiverId": "...",
# "replyTo": "...",
# "replyToMsgId": "...",
# "brokerURL": "...",
# "metadata": {...},
# "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
# }
```
**Supported Types:**
@@ -81,9 +102,10 @@ smartsend(
nats_url="nats://localhost:4222"
)
# Receive always returns a list
payloads = smartreceive(msg, fileserverDownloadHandler, max_retries, base_delay, max_delay)
# payloads = [("dataname1", data1, type1), ("dataname2", data2, type2), ...]
# Receive returns a dictionary envelope with all metadata and deserialized payloads
envelope = smartreceive(msg, fileserverDownloadHandler, max_retries, base_delay, max_delay)
# envelope["payloads"] = [("dataname1", data1, type1), ("dataname2", data2, type2), ...]
# envelope["correlationId"], envelope["msgId"], etc.
```
## Architecture Diagram
@@ -118,7 +140,7 @@ flowchart TD
### 1. msgEnvelope_v1 - Message Envelope
The `msgEnvelope_v1` structure provides a comprehensive message format for bidirectional communication between Julia and JavaScript services.
The `msgEnvelope_v1` structure provides a comprehensive message format for bidirectional communication between Julia, JavaScript, and Python/Micropython applications.
**Julia Structure:**
```julia
@@ -194,7 +216,7 @@ end
### 2. msgPayload_v1 - Payload Structure
The `msgPayload_v1` structure provides flexible payload handling for various data types.
The `msgPayload_v1` structure provides flexible payload handling for various data types across all supported platforms.
**Julia Structure:**
```julia
@@ -222,7 +244,7 @@ end
┌─────────────────────────────────────────────────────────────┐
│ smartsend Function │
│ Accepts: [(dataname1, data1, type1), ...] │
│ (No standalone type parameter - type per payload)
│ (Type is per payload, not standalone)
└─────────────────────────────────────────────────────────────┘
@@ -249,19 +271,77 @@ end
└─────────────────┘ └─────────────────┘
```
### 4. Julia Module Architecture
### 4. Cross-Platform Architecture
```mermaid
flowchart TD
subgraph PythonMicropython
Py[Python/Micropython]
PySmartSend[smartsend]
PySmartReceive[smartreceive]
end
subgraph JavaScript
JS[JavaScript]
JSSmartSend[smartsend]
JSSmartReceive[smartreceive]
end
subgraph Julia
Julia[Julia]
JuliaSmartSend[smartsend]
JuliaSmartReceive[smartreceive]
end
subgraph NATS
NATSServer[NATS Server]
end
PySmartSend --> NATSServer
JSSmartSend --> NATSServer
JuliaSmartSend --> NATSServer
NATSServer --> PySmartReceive
NATSServer --> JSSmartReceive
NATSServer --> JuliaSmartReceive
style PythonMicropython fill:#e1f5fe
style JavaScript fill:#f3e5f5
style Julia fill:#e8f5e9
```
### 5. Python/Micropython Module Architecture
```mermaid
graph TD
subgraph JuliaModule
smartsendJulia[smartsend Julia]
subgraph PyModule
PySmartSend[smartsend]
SizeCheck[Size Check]
DirectPath[Direct Path]
LinkPath[Link Path]
HTTPClient[HTTP Client]
end
smartsendJulia --> SizeCheck
PySmartSend --> SizeCheck
SizeCheck -->|< 1MB| DirectPath
SizeCheck -->|>= 1MB| LinkPath
LinkPath --> HTTPClient
style PyModule fill:#b3e5fc
```
### 6. Julia Module Architecture
```mermaid
graph TD
subgraph JuliaModule
JuliaSmartSend[smartsend]
SizeCheck[Size Check]
DirectPath[Direct Path]
LinkPath[Link Path]
HTTPClient[HTTP Client]
end
JuliaSmartSend --> SizeCheck
SizeCheck -->|< 1MB| DirectPath
SizeCheck -->|>= 1MB| LinkPath
LinkPath --> HTTPClient
@@ -269,19 +349,19 @@ graph TD
style JuliaModule fill:#c5e1a5
```
### 5. JavaScript Module Architecture
### 7. JavaScript Module Architecture
```mermaid
graph TD
subgraph JSModule
smartsendJS[smartsend JS]
smartreceiveJS[smartreceive JS]
JSSmartSend[smartsend]
JSSmartReceive[smartreceive]
JetStreamConsumer[JetStream Pull Consumer]
ApacheArrow[Apache Arrow]
end
smartsendJS --> NATS
smartreceiveJS --> JetStreamConsumer
JSSmartSend --> NATS
JSSmartReceive --> JetStreamConsumer
JetStreamConsumer --> ApacheArrow
style JSModule fill:#f3e5f5
@@ -338,23 +418,25 @@ function smartreceive(
# If direct: decode Base64 payload
# If link: fetch from URL with exponential backoff using fileserverDownloadHandler
# Deserialize payload based on type
# Return list of (dataname, data, type) tuples
# Return envelope dictionary with all metadata and deserialized payloads
end
```
**Output Format:**
- Always returns a list of tuples: `[(dataname1, data1, type1), (dataname2, data2, type2), ...]`
- Even for single payloads: `[(dataname1, data1, type1)]`
- Returns a dictionary (key-value map) containing all envelope fields:
- `correlationId`, `msgId`, `timestamp`, `sendTo`, `msgPurpose`, `senderName`, `senderId`, `receiverName`, `receiverId`, `replyTo`, `replyToMsgId`, `brokerURL`
- `metadata` - Message-level metadata dictionary
- `payloads` - List of dictionaries, each containing deserialized payload data
**Process Flow:**
1. Parse the JSON envelope to extract the `payloads` array
1. Parse the JSON envelope to extract all fields
2. Iterate through each payload in `payloads`
3. For each payload:
- Determine transport type (`direct` or `link`)
- If `direct`: decode Base64 data from the message
- If `link`: fetch data from URL using exponential backoff (via `fileserverDownloadHandler`)
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
4. Return list of `(dataname, data, type)` tuples
4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples
**Note:** The `fileserverDownloadHandler` receives `(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)` and returns `Vector{UInt8}`.
@@ -401,21 +483,27 @@ async function smartreceive(msg, options = {})
// - correlationId: optional correlation ID for tracing
```
**Output Format:**
- Returns a dictionary (key-value map) containing all envelope fields:
- `correlationId`, `msgId`, `timestamp`, `sendTo`, `msgPurpose`, `senderName`, `senderId`, `receiverName`, `receiverId`, `replyTo`, `replyToMsgId`, `brokerURL`
- `metadata` - Message-level metadata dictionary
- `payloads` - List of dictionaries, each containing deserialized payload data
**Process Flow:**
1. Parse the JSON envelope to extract the `payloads` array
1. Parse the JSON envelope to extract all fields
2. Iterate through each payload in `payloads`
3. For each payload:
- Determine transport type (`direct` or `link`)
- If `direct`: decode Base64 data from the message
- If `link`: fetch data from URL using exponential backoff
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
4. Return list of `(dataname, data, type)` tuples
4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples
## Scenario Implementations
### Scenario 1: Command & Control (Small Dictionary)
**Julia (Receiver):**
**Julia (Sender/Receiver):**
```julia
# Subscribe to control subject
# Parse JSON envelope
@@ -423,15 +511,21 @@ async function smartreceive(msg, options = {})
# Send acknowledgment
```
**JavaScript (Sender):**
**JavaScript (Sender/Receiver):**
```javascript
// Create small dictionary config
// Send via smartsend with type="dictionary"
```
**Python/Micropython (Sender/Receiver):**
```python
# Create small dictionary config
# Send via smartsend with type="dictionary"
```
### Scenario 2: Deep Dive Analysis (Large Arrow Table)
**Julia (Sender):**
**Julia (Sender/Receiver):**
```julia
# Create large DataFrame
# Convert to Arrow IPC stream
@@ -440,7 +534,7 @@ async function smartreceive(msg, options = {})
# Publish NATS with URL
```
**JavaScript (Receiver):**
**JavaScript (Sender/Receiver):**
```javascript
// Receive NATS message with URL
// Fetch data from HTTP server
@@ -448,42 +542,64 @@ async function smartreceive(msg, options = {})
// Load into Perspective.js or D3
```
**Python/Micropython (Sender/Receiver):**
```python
# Create large DataFrame
# Convert to Arrow IPC stream
# Check size (> 1MB)
# Upload to HTTP server
# Publish NATS with URL
```
### Scenario 3: Live Audio Processing
**JavaScript (Sender):**
**JavaScript (Sender/Receiver):**
```javascript
// Capture audio chunk
// Send as binary with metadata headers
// Use smartsend with type="audio"
```
**Julia (Receiver):**
**Julia (Sender/Receiver):**
```julia
// Receive audio data
// Perform FFT or AI transcription
// Send results back (JSON + Arrow table)
# Receive audio data
# Perform FFT or AI transcription
# Send results back (JSON + Arrow table)
```
**Python/Micropython (Sender/Receiver):**
```python
# Capture audio chunk
# Send as binary with metadata headers
# Use smartsend with type="audio"
```
### Scenario 4: Catch-Up (JetStream)
**Julia (Producer):**
**Julia (Producer/Consumer):**
```julia
# Publish to JetStream
# Include metadata for temporal tracking
```
**JavaScript (Consumer):**
**JavaScript (Producer/Consumer):**
```javascript
// Connect to JetStream
// Request replay from last 10 minutes
// Process historical and real-time messages
```
**Python/Micropython (Producer/Consumer):**
```python
# Publish to JetStream
# Include metadata for temporal tracking
```
### Scenario 5: Selection (Low Bandwidth)
**Focus:** Small Arrow tables, Julia to JavaScript. The Action: Julia wants to send a small DataFrame to show on a JavaScript dashboard for the user to choose.
**Focus:** Small Arrow tables, cross-platform communication. The Action: Any platform wants to send a small DataFrame to show on any receiving application for the user to choose.
**Julia (Sender):**
**Julia (Sender/Receiver):**
```julia
# Create small DataFrame (e.g., 50KB - 500KB)
# Convert to Arrow IPC stream
@@ -492,7 +608,7 @@ async function smartreceive(msg, options = {})
# Include metadata for dashboard selection context
```
**JavaScript (Receiver):**
**JavaScript (Sender/Receiver):**
```javascript
// Receive NATS message with direct transport
// Decode Base64 payload
@@ -502,11 +618,20 @@ async function smartreceive(msg, options = {})
// Send selection back to Julia
```
**Use Case:** Julia server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to JavaScript dashboard for user selection. The selection is then sent back to Julia for processing.
**Python/Micropython (Sender/Receiver):**
```python
# Create small DataFrame (e.g., 50KB - 500KB)
# Convert to Arrow IPC stream
# Check payload size (< 1MB threshold)
# Publish directly to NATS with Base64-encoded payload
# Include metadata for dashboard selection context
```
**Use Case:** Any server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to any receiving application for user selection. The selection is then sent back to the sender for processing.
### Scenario 6: Chat System
**Focus:** Every conversational message is composed of any number and any combination of components, spanning the full spectrum from small to large. This includes text, images, audio, video, tables, and files—specifically accommodating everything from brief snippets to high-resolution images, large audio files, extensive tables, and massive documents. Support for claim-check delivery and full bi-directional messaging.
**Focus:** Every conversational message is composed of any number and any combination of components, spanning the full spectrum from small to large. This includes text, images, audio, video, tables, and files—specifically accommodating everything from brief snippets to high-resolution images, large audio files, extensive tables, and massive documents. Support for claim-check delivery and full bi-directional messaging across all platforms.
**Multi-Payload Support:** The system supports mixed-payload messages where a single message can contain multiple payloads with different transport strategies. The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type.
@@ -545,7 +670,25 @@ async function smartreceive(msg, options = {})
// Support bidirectional reply with claim-check delivery confirmation
```
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components.
**Python/Micropython (Sender/Receiver):**
```python
# Build chat message with mixed payloads:
# - Text: direct transport (Base64)
# - Small images: direct transport (Base64)
# - Large images: link transport (HTTP URL)
# - Audio/video: link transport (HTTP URL)
# - Tables: direct or link depending on size
# - Files: link transport (HTTP URL)
#
# Each payload uses appropriate transport strategy:
# - Size < 1MB → direct (NATS + Base64)
# - Size >= 1MB → link (HTTP upload + NATS URL)
#
# Include claim-check metadata for delivery tracking
# Support bidirectional messaging with replyTo fields
```
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components across all platforms.
**Implementation Note:** The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. See the standard API format in Section 1: `msgEnvelope_v1` supports `AbstractArray{msgPayload_v1}` for multiple payloads.

View File

@@ -2,7 +2,22 @@
## Overview
This document describes the implementation of the high-performance, bi-directional data bridge between Julia and JavaScript services using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
This document describes the implementation of the high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
The system enables seamless communication across all three platforms:
- **Julia ↔ JavaScript** bi-directional messaging
- **JavaScript ↔ Python/Micropython** bi-directional messaging
- **Julia ↔ Python/Micropython** bi-directional messaging (via JSON serialization)
### Implementation Files
NATSBridge is implemented in three languages, each providing the same API:
| Language | Implementation File | Description |
|----------|---------------------|-------------|
| **Julia** | [`src/NATSBridge.jl`](../src/NATSBridge.jl) | Full Julia implementation with Arrow IPC support |
| **JavaScript** | [`src/NATSBridge.js`](../src/NATSBridge.js) | JavaScript implementation for Node.js and browsers |
| **Python/Micropython** | [`src/nats_bridge.py`](../src/nats_bridge.py) | Python implementation for desktop and microcontrollers |
### Multi-Payload Support
@@ -13,8 +28,24 @@ The implementation uses a **standardized list-of-tuples format** for all payload
# Input format for smartsend (always a list of tuples with type info)
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
# Output format for smartreceive (always returns a list of tuples with type info)
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
# Output format for smartreceive (returns envelope dictionary with payloads field)
# Returns: Dict with envelope metadata and payloads field containing list of tuples
# {
# "correlationId": "...",
# "msgId": "...",
# "timestamp": "...",
# "sendTo": "...",
# "msgPurpose": "...",
# "senderName": "...",
# "senderId": "...",
# "receiverName": "...",
# "receiverId": "...",
# "replyTo": "...",
# "replyToMsgId": "...",
# "brokerURL": "...",
# "metadata": {...},
# "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
# }
```
Where `type` can be: `"text"`, `"dictionary"`, `"table"`, `"image"`, `"audio"`, `"video"`, `"binary"`
@@ -27,14 +58,68 @@ smartsend("/test", [(dataname1, data1, "text")], ...)
# Multiple payloads in one message (each payload has its own type)
smartsend("/test", [(dataname1, data1, "dictionary"), (dataname2, data2, "table")], ...)
# Receive always returns a list with type info
payloads = smartreceive(msg, ...)
# payloads = [(dataname1, data1, "text"), (dataname2, data2, "table"), ...]
# Receive returns a dictionary envelope with all metadata and deserialized payloads
envelope = smartreceive(msg, ...)
# envelope["payloads"] = [(dataname1, data1, "text"), (dataname2, data2, "table"), ...]
# envelope["correlationId"], envelope["msgId"], etc.
```
## Cross-Platform Interoperability
NATSBridge is designed for seamless communication between Julia, JavaScript, and Python/Micropython applications. All three implementations share the same interface and data format, ensuring compatibility across platforms.
### Platform-Specific Features
| Feature | Julia | JavaScript | Python/Micropython |
|---------|-------|------------|-------------------|
| Direct NATS transport | ✅ | ✅ | ✅ |
| HTTP file server (Claim-Check) | ✅ | ✅ | ✅ |
| Arrow IPC tables | ✅ | ✅ | ✅ |
| Base64 encoding | ✅ | ✅ | ✅ |
| Exponential backoff | ✅ | ✅ | ✅ |
| Correlation ID tracking | ✅ | ✅ | ✅ |
| Reply-to support | ✅ | ✅ | ✅ |
### Data Type Mapping
| Type | Julia | JavaScript | Python/Micropython |
|------|-------|------------|-------------------|
| `text` | `String` | `String` | `str` |
| `dictionary` | `Dict` | `Object` | `dict` |
| `table` | `DataFrame` | `Array<Object>` | `DataFrame` / `list` |
| `image` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
| `audio` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
| `video` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
| `binary` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
### Example: Julia ↔ Python ↔ JavaScript
```julia
# Julia sender
using NATSBridge
data = [("message", "Hello from Julia!", "text")]
smartsend("/cross_platform", data, nats_url="nats://localhost:4222")
```
```javascript
// JavaScript receiver
const { smartreceive } = require('./src/NATSBridge');
const envelope = await smartreceive(msg);
// envelope.payloads[0].data === "Hello from Julia!"
```
```python
# Python sender
from nats_bridge import smartsend
data = [("response", "Hello from Python!", "text")]
smartsend("/cross_platform", data, nats_url="nats://localhost:4222")
```
All three platforms can communicate seamlessly using the same NATS subjects and data format.
## Architecture
The implementation follows the Claim-Check pattern:
All three implementations (Julia, JavaScript, Python/Micropython) follow the same Claim-Check pattern:
```
┌─────────────────────────────────────────────────────────────────────────┐
@@ -53,7 +138,7 @@ The implementation follows the Claim-Check pattern:
│ (< 1MB) │ │ (> 1MB) │
│ │ │ │
│ • Serialize to │ │ • Serialize to │
IOBuffer │ │ IOBuffer │
│ Buffer │ │ Buffer
│ • Base64 encode │ │ • Upload to │
│ • Publish to │ │ HTTP Server │
│ NATS │ │ • Publish to │
@@ -63,13 +148,13 @@ The implementation follows the Claim-Check pattern:
## Files
### Julia Module: [`src/julia_bridge.jl`](../src/julia_bridge.jl)
### Julia Module: [`src/NATSBridge.jl`](../src/NATSBridge.jl)
The Julia implementation provides:
- **[`MessageEnvelope`](../src/julia_bridge.jl)**: Struct for the unified JSON envelope
- **[`SmartSend()`](../src/julia_bridge.jl)**: Handles transport selection based on payload size
- **[`SmartReceive()`](../src/julia_bridge.jl)**: Handles both direct and link transport
- **[`MessageEnvelope`](src/NATSBridge.jl)**: Struct for the unified JSON envelope
- **[`SmartSend()`](src/NATSBridge.jl)**: Handles transport selection based on payload size
- **[`SmartReceive()`](src/NATSBridge.jl)**: Handles both direct and link transport
### JavaScript Module: [`src/NATSBridge.js`](../src/NATSBridge.js)
@@ -77,8 +162,17 @@ The JavaScript implementation provides:
- **`MessageEnvelope` class**: For the unified JSON envelope
- **`MessagePayload` class**: For individual payload representation
- **[`smartsend()`](../src/NATSBridge.js)**: Handles transport selection based on payload size
- **[`smartreceive()`](../src/NATSBridge.js)**: Handles both direct and link transport
- **[`smartsend()`](src/NATSBridge.js)**: Handles transport selection based on payload size
- **[`smartreceive()`](src/NATSBridge.js)**: Handles both direct and link transport
### Python/Micropython Module: [`src/nats_bridge.py`](../src/nats_bridge.py)
The Python/Micropython implementation provides:
- **`MessageEnvelope` class**: For the unified JSON envelope
- **`MessagePayload` class**: For individual payload representation
- **[`smartsend()`](src/nats_bridge.py)**: Handles transport selection based on payload size
- **[`smartreceive()`](src/nats_bridge.py)**: Handles both direct and link transport
## Installation
@@ -100,6 +194,23 @@ Pkg.add("Dates")
npm install nats.js apache-arrow uuid base64-url
```
### Python/Micropython Dependencies
1. Copy [`src/nats_bridge.py`](../src/nats_bridge.py) to your device
2. Ensure you have the following dependencies:
**For Python (desktop):**
```bash
pip install nats-py
```
**For Micropython:**
- `urequests` for HTTP requests
- `base64` for base64 encoding (built-in)
- `json` for JSON handling (built-in)
- `socket` for networking (built-in)
- `uuid` for UUID generation (built-in)
## Usage Tutorial
### Step 1: Start NATS Server
@@ -138,34 +249,31 @@ node test/scenario3_julia_to_julia.js
### Scenario 0: Basic Multi-Payload Example
#### Julia (Sender)
```julia
using NATSBridge
#### Python/Micropython (Sender)
```python
from nats_bridge import smartsend
# Send multiple payloads in one message (type is required per payload)
smartsend(
"/test",
[("dataname1", data1, "dictionary"), ("dataname2", data2, "table")],
nats_url="nats://localhost:4222",
fileserver_url="http://localhost:8080",
metadata=Dict("custom_key" => "custom_value")
fileserver_url="http://localhost:8080"
)
# Even single payload must be wrapped in a list with type
smartsend("/test", [("single_data", mydata, "dictionary")])
```
#### Julia (Receiver)
```julia
using NATSBridge
#### Python/Micropython (Receiver)
```python
from nats_bridge import smartreceive
# Receive returns a list of payloads with type info
payloads = smartreceive(msg, "http://localhost:8080")
# Receive returns a list of (dataname, data, type) tuples
payloads = smartreceive(msg)
# payloads = [(dataname1, data1, "dictionary"), (dataname2, data2, "table"), ...]
```
### Scenario 1: Command & Control (Small JSON)
#### JavaScript (Sender)
```javascript
const { smartsend } = require('./src/NATSBridge');
@@ -198,27 +306,7 @@ const configs = [
await smartsend("control", configs);
```
#### Julia (Receiver)
```julia
using NATS
using JSON3
# Subscribe to control subject
subscribe(nats, "control") do msg
env = MessageEnvelope(String(msg.data))
config = JSON3.read(env.payload)
# Execute simulation with parameters
step_size = config.step_size
iterations = config.iterations
# Send acknowledgment
response = Dict("status" => "Running", "correlation_id" => env.correlation_id)
publish(nats, "control_response", JSON3.stringify(response))
end
```
### JavaScript (Receiver)
#### JavaScript (Receiver)
```javascript
const { smartreceive } = require('./src/NATSBridge');
@@ -227,13 +315,18 @@ const nc = await connect({ servers: ['nats://localhost:4222'] });
const sub = nc.subscribe("control");
for await (const msg of sub) {
const result = await smartreceive(msg);
const envelope = await smartreceive(msg);
// Process the result
for (const { dataname, data, type } of result) {
// Process the payloads from the envelope
for (const payload of envelope.payloads) {
const { dataname, data, type } = payload;
console.log(`Received ${dataname} of type ${type}`);
console.log(`Data: ${JSON.stringify(data)}`);
}
// Also access envelope metadata
console.log(`Correlation ID: ${envelope.correlationId}`);
console.log(`Message ID: ${envelope.msgId}`);
}
```
@@ -259,15 +352,35 @@ await SmartSend("analysis_results", [("table_data", df, "table")]);
```javascript
const { smartreceive } = require('./src/NATSBridge');
const result = await smartreceive(msg);
const envelope = await smartreceive(msg);
// Use table data for visualization with Perspective.js or D3
// Use table data from the payloads field
// Note: Tables are sent as arrays of objects in JavaScript
const table = result;
const table = envelope.payloads;
```
### Scenario 3: Live Binary Processing
#### Python/Micropython (Sender)
```python
from nats_bridge import smartsend
# Binary data wrapped in a list
binary_data = [
("audio_chunk", binary_buffer, "binary")
]
smartsend(
"binary_input",
binary_data,
nats_url="nats://localhost:4222",
metadata={
"sample_rate": 44100,
"channels": 1
}
)
```
#### JavaScript (Sender)
```javascript
const { smartsend } = require('./src/NATSBridge');
@@ -287,35 +400,35 @@ await smartsend("binary_input", binaryData, {
});
```
#### Julia (Receiver)
```julia
using WAV
using DSP
#### Python/Micropython (Receiver)
```python
from nats_bridge import smartreceive
# Receive binary data
function process_binary(data)
# Perform FFT or AI transcription
spectrum = fft(data)
def process_binary(msg):
envelope = smartreceive(msg)
# Send results back (JSON + Arrow table)
results = Dict("transcription" => "sample text", "spectrum" => spectrum)
await SmartSend("binary_output", results, "json")
end
# Process the binary data from envelope.payloads
for dataname, data, type in envelope["payloads"]:
if type == "binary":
# data is bytes
print(f"Received binary data: {dataname}, size: {len(data)}")
# Perform FFT or AI transcription here
```
### JavaScript (Receiver)
#### JavaScript (Receiver)
```javascript
const { smartreceive } = require('./src/NATSBridge');
// Receive binary data
function process_binary(msg) {
const result = await smartreceive(msg);
const envelope = await smartreceive(msg);
// Process the binary data
for (const { dataname, data, type } of result) {
if (type === "binary") {
// Process the binary data from envelope.payloads
for (const payload of envelope.payloads) {
if (payload.type === "binary") {
// data is an ArrayBuffer or Uint8Array
console.log(`Received binary data: ${dataname}, size: ${data.length}`);
console.log(`Received binary data: ${payload.dataname}, size: ${payload.data.length}`);
// Perform FFT or AI transcription here
}
}
@@ -353,13 +466,71 @@ const consumer = await js.pullSubscribe("health", {
// Process historical and real-time messages
for await (const msg of consumer) {
const result = await smartreceive(msg);
// result contains the list of payloads
const envelope = await smartreceive(msg);
// envelope.payloads contains the list of payloads
// Each payload has: dataname, data, type
msg.ack();
}
```
### Scenario 4: Micropython Device Control
**Focus:** Sending configuration to a Micropython device over NATS. This demonstrates the lightweight nature of the Python implementation suitable for microcontrollers.
**Python/Micropython (Receiver/Device):**
```python
from nats_bridge import smartsend, smartreceive
import json
# Device configuration handler
def handle_device_config(msg):
envelope = smartreceive(msg)
# Process configuration from payloads
for dataname, data, type in envelope["payloads"]:
if type == "dictionary":
print(f"Received configuration: {data}")
# Apply configuration to device
if "wifi_ssid" in data:
wifi_ssid = data["wifi_ssid"]
wifi_password = data["wifi_password"]
update_wifi_config(wifi_ssid, wifi_password)
# Send confirmation back
config = {
"status": "configured",
"wifi_ssid": "MyNetwork",
"ip": get_device_ip()
}
smartsend(
"device/response",
[("config", config, "dictionary")],
nats_url="nats://localhost:4222",
reply_to=envelope.get("replyTo")
)
```
**JavaScript (Sender/Controller):**
```javascript
const { smartsend } = require('./src/NATSBridge');
// Send configuration to Micropython device
await smartsend("device/config", [
{
dataname: "config",
data: {
wifi_ssid: "MyNetwork",
wifi_password: "password123",
update_interval: 60,
temperature_threshold: 30.0
},
type: "dictionary"
}
]);
```
**Use Case:** A controller sends WiFi and operational configuration to a Micropython device (e.g., ESP32). The device receives the configuration, applies it, and sends back a confirmation with its current status.
### Scenario 5: Selection (Low Bandwidth)
**Focus:** Small Arrow tables, Julia to JavaScript. The Action: Julia wants to send a small DataFrame to show on a JavaScript dashboard for the user to choose.
@@ -395,11 +566,11 @@ smartsend(
const { smartreceive, smartsend } = require('./src/NATSBridge');
// Receive NATS message with direct transport
const result = await smartreceive(msg);
const envelope = await smartreceive(msg);
// Decode Base64 payload (for direct transport)
// For tables, data is an array of objects
const table = result; // Array of objects
// For tables, data is in envelope.payloads
const table = envelope.payloads; // Array of objects
// User makes selection
const selection = uiComponent.getSelectedOption();
@@ -558,7 +729,7 @@ await smartsend("chat.room123", message);
### Exponential Backoff
- Maximum retry count: 5
- Base delay: 100ms, max delay: 5000ms
- Implemented in both Julia and JavaScript implementations
- Implemented in all three implementations (Julia, JavaScript, Python/Micropython)
### Correlation ID Logging
- Log correlation_id at every stage
@@ -567,14 +738,73 @@ await smartsend("chat.room123", message);
## Testing
Run the test scripts:
Run the test scripts for each platform:
### Python/Micropython Tests
```bash
# Scenario 1: Command & Control (JavaScript sender)
node test/scenario1_command_control.js
# Basic functionality test
python test/test_micropython_basic.py
```
# Scenario 2: Large Arrow Table (JavaScript sender)
node test/scenario2_large_table.js
### JavaScript Tests
```bash
# Text message exchange
node test/test_js_to_js_text_sender.js
node test/test_js_to_js_text_receiver.js
# Dictionary exchange
node test/test_js_to_js_dict_sender.js
node test/test_js_to_js_dict_receiver.js
# File transfer (direct transport)
node test/test_js_to_js_file_sender.js
node test/test_js_to_js_file_receiver.js
# Mixed payload types
node test/test_js_to_js_mix_payloads_sender.js
node test/test_js_to_js_mix_payloads_receiver.js
# Table (Arrow IPC) exchange
node test/test_js_to_js_table_sender.js
node test/test_js_to_js_table_receiver.js
```
### Julia Tests
```bash
# Text message exchange
julia test/test_julia_to_julia_text_sender.jl
julia test/test_julia_to_julia_text_receiver.jl
# Dictionary exchange
julia test/test_julia_to_julia_dict_sender.jl
julia test/test_julia_to_julia_dict_receiver.jl
# File transfer
julia test/test_julia_to_julia_file_sender.jl
julia test/test_julia_to_julia_file_receiver.jl
# Mixed payload types
julia test/test_julia_to_julia_mix_payloads_sender.jl
julia test/test_julia_to_julia_mix_payloads_receiver.jl
# Table exchange
julia test/test_julia_to_julia_table_sender.jl
julia test/test_julia_to_julia_table_receiver.jl
```
### Cross-Platform Tests
```bash
# Julia ↔ JavaScript communication
julia test/test_julia_to_julia_text_sender.jl
node test/test_js_to_js_text_receiver.js
# Python ↔ JavaScript communication
python test/test_micropython_basic.py
node test/test_js_to_js_text_receiver.js
```
## Troubleshooting
@@ -582,18 +812,24 @@ node test/scenario2_large_table.js
### Common Issues
1. **NATS Connection Failed**
- Ensure NATS server is running
- Check NATS_URL configuration
- **Julia/JavaScript/Python**: Ensure NATS server is running
- **Python/Micropython**: Check `nats_url` parameter and network connectivity
2. **HTTP Upload Failed**
- Ensure file server is running
- Check FILESERVER_URL configuration
- Check `fileserver_url` configuration
- Verify upload permissions
- **Micropython**: Ensure `urequests` is available and network is connected
3. **Arrow IPC Deserialization Error**
- Ensure data is properly serialized to Arrow format
- Check Arrow version compatibility
4. **Python/Micropython Specific Issues**
- **Import Error**: Ensure `nats_bridge.py` is in the correct path
- **Memory Error (Micropython)**: Reduce payload size or use link transport for large payloads
- **Unicode Error**: Ensure proper encoding when sending text data
## License
MIT

21
etc.jl
View File

@@ -1,21 +0,0 @@
Check architecture.jl, NATSBridge.jl and its test files:
- test_julia_to_julia_table_receiver.jl
- test_julia_to_julia_table_sender.jl.
Now I want to test sending a mix-content message from Julia serviceA to Julia serviceB, for example, a chat system.
The test message must show that any combination and any number and any data size of text | json | table | image | audio | video | binary can be send and receive.
Can you write me the following test files:
- test_julia_to_julia_mix_receiver.jl
- test_julia_to_julia_mix_sender.jl
1. create a tutorial file "tutorial_julia.md" for NATSBridge.jl
2. create a walkthrough file "walkthrough_julia.md" for NATSBridge.jl
You may consult architecture.md for more info.

604
examples/tutorial.md Normal file
View File

@@ -0,0 +1,604 @@
# NATSBridge Tutorial
A step-by-step guide to get started with NATSBridge - a high-performance, bi-directional data bridge for **Julia**, **JavaScript**, and **Python/Micropython**.
## Table of Contents
1. [Overview](#overview)
2. [Prerequisites](#prerequisites)
3. [Installation](#installation)
4. [Quick Start](#quick-start)
5. [Basic Examples](#basic-examples)
6. [Advanced Usage](#advanced-usage)
7. [Cross-Platform Communication](#cross-platform-communication)
---
## Overview
NATSBridge enables seamless communication between Julia, JavaScript, and Python/Micropython applications through NATS, with automatic transport selection based on payload size:
- **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded)
- **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL
### Supported Payload Types
| Type | Description |
|------|-------------|
| `text` | Plain text strings |
| `dictionary` | JSON-serializable dictionaries |
| `table` | Tabular data (Arrow IPC format) |
| `image` | Image data (PNG, JPG bytes) |
| `audio` | Audio data (WAV, MP3 bytes) |
| `video` | Video data (MP4, AVI bytes) |
| `binary` | Generic binary data |
---
## Prerequisites
Before you begin, ensure you have:
1. **NATS Server** running (or accessible)
2. **HTTP File Server** (optional, for large payloads > 1MB)
3. **One of the supported platforms**: Julia, JavaScript (Node.js), or Python/Micropython
---
## Installation
### Julia
```julia
using Pkg
Pkg.add("NATS")
Pkg.add("Arrow")
Pkg.add("JSON3")
Pkg.add("HTTP")
Pkg.add("UUIDs")
Pkg.add("Dates")
```
### JavaScript
```bash
npm install nats.js apache-arrow uuid base64-url
```
### Python/Micropython
1. Copy `src/nats_bridge.py` to your device
2. Install dependencies:
**For Python (desktop):**
```bash
pip install nats-py
```
**For Micropython:**
- `urequests` for HTTP requests
- `base64` for base64 encoding (built-in)
- `json` for JSON handling (built-in)
---
## Quick Start
### Step 1: Start NATS Server
```bash
docker run -p 4222:4222 nats:latest
```
### Step 2: Start HTTP File Server (Optional)
```bash
# Create a directory for file uploads
mkdir -p /tmp/fileserver
# Use Python's built-in server
python3 -m http.server 8080 --directory /tmp/fileserver
```
### Step 3: Send Your First Message
#### Python/Micropython
```python
from nats_bridge import smartsend
# Send a text message
data = [("message", "Hello World", "text")]
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
print("Message sent!")
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
// Send a text message
await smartsend("/chat/room1", [
{ dataname: "message", data: "Hello World", type: "text" }
], { natsUrl: "nats://localhost:4222" });
console.log("Message sent!");
```
#### Julia
```julia
using NATSBridge
# Send a text message
data = [("message", "Hello World", "text")]
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
println("Message sent!")
```
### Step 4: Receive Messages
#### Python/Micropython
```python
from nats_bridge import smartreceive
# Receive and process message
envelope = smartreceive(msg)
for dataname, data, type in envelope["payloads"]:
print(f"Received {dataname}: {data}")
```
#### JavaScript
```javascript
const { smartreceive } = require('./src/NATSBridge');
// Receive and process message
const envelope = await smartreceive(msg);
for (const payload of envelope.payloads) {
console.log(`Received ${payload.dataname}: ${payload.data}`);
}
```
#### Julia
```julia
using NATSBridge
# Receive and process message
envelope = smartreceive(msg, fileserverDownloadHandler)
for (dataname, data, type) in envelope["payloads"]
println("Received $dataname: $data")
end
```
---
## Basic Examples
### Example 1: Sending a Dictionary
#### Python/Micropython
```python
from nats_bridge import smartsend
# Create configuration dictionary
config = {
"wifi_ssid": "MyNetwork",
"wifi_password": "password123",
"update_interval": 60
}
# Send as dictionary type
data = [("config", config, "dictionary")]
env = smartsend("/device/config", data, nats_url="nats://localhost:4222")
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
const config = {
wifi_ssid: "MyNetwork",
wifi_password: "password123",
update_interval: 60
};
await smartsend("/device/config", [
{ dataname: "config", data: config, type: "dictionary" }
]);
```
#### Julia
```julia
using NATSBridge
config = Dict(
"wifi_ssid" => "MyNetwork",
"wifi_password" => "password123",
"update_interval" => 60
)
data = [("config", config, "dictionary")]
smartsend("/device/config", data)
```
### Example 2: Sending Binary Data (Image)
#### Python/Micropython
```python
from nats_bridge import smartsend
# Read image file
with open("image.png", "rb") as f:
image_data = f.read()
# Send as binary type
data = [("user_image", image_data, "binary")]
env = smartsend("/chat/image", data, nats_url="nats://localhost:4222")
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
// Read image file (Node.js)
const fs = require('fs');
const image_data = fs.readFileSync('image.png');
await smartsend("/chat/image", [
{ dataname: "user_image", data: image_data, type: "binary" }
]);
```
#### Julia
```julia
using NATSBridge
# Read image file
image_data = read("image.png")
data = [("user_image", image_data, "binary")]
smartsend("/chat/image", data)
```
### Example 3: Request-Response Pattern
#### Python/Micropython (Requester)
```python
from nats_bridge import smartsend
# Send command with reply-to
data = [("command", {"action": "read_sensor"}, "dictionary")]
env = smartsend(
"/device/command",
data,
nats_url="nats://localhost:4222",
reply_to="/device/response",
reply_to_msg_id="cmd-001"
)
```
#### JavaScript (Responder)
```javascript
const { smartreceive, smartsend } = require('./src/NATSBridge');
// Subscribe to command topic
const sub = nc.subscribe("/device/command");
for await (const msg of sub) {
const envelope = await smartreceive(msg);
// Process command
for (const payload of envelope.payloads) {
if (payload.dataname === "command") {
const command = payload.data;
if (command.action === "read_sensor") {
// Read sensor and send response
const response = {
sensor_id: "sensor-001",
value: 42.5,
timestamp: new Date().toISOString()
};
await smartsend("/device/response", [
{ dataname: "sensor_data", data: response, type: "dictionary" }
], {
reply_to: envelope.replyTo,
reply_to_msg_id: envelope.msgId
});
}
}
}
}
```
---
## Advanced Usage
### Example 4: Large Payloads (File Server)
For payloads larger than 1MB, NATSBridge automatically uses the file server:
#### Python/Micropython
```python
from nats_bridge import smartsend
import os
# Create large data (> 1MB)
large_data = os.urandom(2_000_000) # 2MB of random data
# Send with file server URL
env = smartsend(
"/data/large",
[("large_file", large_data, "binary")],
nats_url="nats://localhost:4222",
fileserver_url="http://localhost:8080",
size_threshold=1_000_000
)
# The envelope will contain the download URL
print(f"File uploaded to: {env.payloads[0].data}")
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
// Create large data (> 1MB)
const largeData = new ArrayBuffer(2_000_000);
const view = new Uint8Array(largeData);
view.fill(42); // Fill with some data
await smartsend("/data/large", [
{ dataname: "large_file", data: largeData, type: "binary" }
], {
fileserverUrl: "http://localhost:8080",
sizeThreshold: 1_000_000
});
```
#### Julia
```julia
using NATSBridge
# Create large data (> 1MB)
large_data = rand(UInt8, 2_000_000)
env = smartsend(
"/data/large",
[("large_file", large_data, "binary")],
fileserver_url="http://localhost:8080"
)
# The envelope will contain the download URL
println("File uploaded to: $(env.payloads[1].data)")
```
### Example 5: Mixed Content (Chat with Text + Image)
NATSBridge supports sending multiple payloads with different types in a single message:
#### Python/Micropython
```python
from nats_bridge import smartsend
# Read image file
with open("avatar.png", "rb") as f:
image_data = f.read()
# Send mixed content
data = [
("message_text", "Hello with image!", "text"),
("user_avatar", image_data, "image")
]
env = smartsend("/chat/mixed", data, nats_url="nats://localhost:4222")
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
const fs = require('fs');
await smartsend("/chat/mixed", [
{
dataname: "message_text",
data: "Hello with image!",
type: "text"
},
{
dataname: "user_avatar",
data: fs.readFileSync("avatar.png"),
type: "image"
}
]);
```
#### Julia
```julia
using NATSBridge
image_data = read("avatar.png")
data = [
("message_text", "Hello with image!", "text"),
("user_avatar", image_data, "image")
]
smartsend("/chat/mixed", data)
```
### Example 6: Table Data (Arrow IPC)
For tabular data, NATSBridge uses Apache Arrow IPC format:
#### Python/Micropython
```python
from nats_bridge import smartsend
import pandas as pd
# Create DataFrame
df = pd.DataFrame({
"id": [1, 2, 3],
"name": ["Alice", "Bob", "Charlie"],
"score": [95, 88, 92]
})
# Send as table type
data = [("students", df, "table")]
env = smartsend("/data/students", data, nats_url="nats://localhost:4222")
```
#### Julia
```julia
using NATSBridge
using DataFrames
# Create DataFrame
df = DataFrame(
id = [1, 2, 3],
name = ["Alice", "Bob", "Charlie"],
score = [95, 88, 92]
)
data = [("students", df, "table")]
smartsend("/data/students", data)
```
---
## Cross-Platform Communication
NATSBridge enables seamless communication between different platforms:
### Julia ↔ JavaScript
#### Julia Sender
```julia
using NATSBridge
# Send dictionary from Julia to JavaScript
config = Dict("step_size" => 0.01, "iterations" => 1000)
data = [("config", config, "dictionary")]
smartsend("/analysis/config", data, nats_url="nats://localhost:4222")
```
#### JavaScript Receiver
```javascript
const { smartreceive } = require('./src/NATSBridge');
// Receive dictionary from Julia
const envelope = await smartreceive(msg);
for (const payload of envelope.payloads) {
if (payload.type === "dictionary") {
console.log("Received config:", payload.data);
// payload.data = { step_size: 0.01, iterations: 1000 }
}
}
```
### JavaScript ↔ Python
#### JavaScript Sender
```javascript
const { smartsend } = require('./src/NATSBridge');
await smartsend("/data/transfer", [
{ dataname: "message", data: "Hello from JS!", type: "text" }
]);
```
#### Python Receiver
```python
from nats_bridge import smartreceive
envelope = smartreceive(msg)
for dataname, data, type in envelope["payloads"]:
if type == "text":
print(f"Received from JS: {data}")
```
### Python ↔ Julia
#### Python Sender
```python
from nats_bridge import smartsend
data = [("message", "Hello from Python!", "text")]
smartsend("/chat/python", data)
```
#### Julia Receiver
```julia
using NATSBridge
envelope = smartreceive(msg, fileserverDownloadHandler)
for (dataname, data, type) in envelope["payloads"]
if type == "text"
println("Received from Python: $data")
end
end
```
---
## Next Steps
1. **Explore the test directory** for more examples
2. **Check the documentation** for advanced configuration options
3. **Join the community** to share your use cases
---
## Troubleshooting
### Connection Issues
- Ensure NATS server is running: `docker ps | grep nats`
- Check firewall settings
- Verify NATS URL configuration
### File Server Issues
- Ensure file server is running and accessible
- Check upload permissions
- Verify file server URL configuration
### Serialization Errors
- Verify data type matches the specified type
- Check that binary data is in the correct format (bytes/Vector{UInt8})
---
## License
MIT

1045
examples/walkthrough.md Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,14 @@
services:
plik:
image: rootgg/plik:latest
container_name: plik-server
restart: unless-stopped
ports:
- "8080:8080"
volumes:
# # Mount the config file (created below)
# - ./plikd.cfg:/home/plik/server/plikd.cfg
# Mount local folder for uploads and database
- ./plik-data:/data
# Set user to match your host UID to avoid permission issues
user: "1000:1000"

View File

@@ -10,7 +10,7 @@
#
# Handler Function Signatures:
#
# ```julia
# ```jldoctest
# # Upload handler - uploads data to file server and returns URL
# fileserverUploadHandler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
#
@@ -23,7 +23,7 @@
# Even when sending a single payload, the user must wrap it in a list.
#
# API Standard:
# ```julia
# ```jldoctest
# # Input format for smartsend (always a list of tuples with type info)
# [(dataname1, data1, type1), (dataname2, data2, type2), ...]
#
@@ -45,6 +45,59 @@ const DEFAULT_NATS_URL = "nats://localhost:4222" # Default NATS server URL
const DEFAULT_FILESERVER_URL = "http://localhost:8080" # Default HTTP file server URL for link transport
""" msgPayload_v1 - Internal message payload structure
This structure represents a single payload within a NATS message envelope.
It supports both direct transport (base64-encoded data) and link transport (URL-based).
# Arguments:
- `id::String` - Unique identifier for this payload (e.g., "uuid4")
- `dataname::String` - Name of the payload (e.g., "login_image")
- `type::String` - Payload type: "text", "dictionary", "table", "image", "audio", "video", "binary"
- `transport::String` - Transport method: "direct" or "link"
- `encoding::String` - Encoding method: "none", "json", "base64", "arrow-ipc"
- `size::Integer` - Size of the payload in bytes (e.g., 15433)
- `data::Any` - Payload data (bytes for direct, URL for link)
- `metadata::Dict{String, Any}` - Optional metadata dictionary
# Keyword Arguments:
- `id::String = ""` - Payload ID, auto-generated if empty
- `dataname::String = string(uuid4())` - Payload name, auto-generated UUID if empty
- `transport::String = "direct"` - Transport method
- `encoding::String = "none"` - Encoding method
- `size::Integer = 0` - Payload size
- `metadata::Dict{String, T} = Dict{String, Any}()` - Metadata dictionary
# Return:
- A msgPayload_v1 struct instance
# Example
```jldoctest
using UUIDs
# Create a direct transport payload
payload = msgPayload_v1(
"Hello World",
"text";
id = string(uuid4()),
dataname = "message",
transport = "direct",
encoding = "base64",
size = 11,
metadata = Dict{String, Any}()
)
# Create a link transport payload
payload = msgPayload_v1(
"http://example.com/file.zip",
"binary";
id = string(uuid4()),
dataname = "file",
transport = "link",
encoding = "none",
size = 1000000
)
```
"""
struct msgPayload_v1
id::String # id of this payload e.g. "uuid4"
dataname::String # name of this payload e.g. "login_image"
@@ -80,6 +133,51 @@ function msgPayload_v1(
end
""" msgEnvelope_v1 - Internal message envelope structure
This structure represents a complete NATS message envelope containing multiple payloads
with metadata for routing, tracing, and message context.
# Arguments:
- `sendTo::String` - NATS subject/topic to publish the message to (e.g., "/agent/wine/api/v1/prompt")
- `payloads::AbstractArray{msgPayload_v1}` - List of payloads to include in the message
# Keyword Arguments:
- `correlationId::String = ""` - Unique identifier to track messages across systems; auto-generated if empty
- `msgId::String = ""` - Unique message identifier; auto-generated if empty
- `timestamp::String = string(Dates.now())` - Message publication timestamp
- `msgPurpose::String = ""` - Purpose of the message: "ACK", "NACK", "updateStatus", "shutdown", "chat", etc.
- `senderName::String = ""` - Name of the sender (e.g., "agent-wine-web-frontend")
- `senderId::String = ""` - UUID of the sender; auto-generated if empty
- `receiverName::String = ""` - Name of the receiver (empty string means broadcast)
- `receiverId::String = ""` - UUID of the receiver (empty string means broadcast)
- `replyTo::String = ""` - Topic where receiver should reply (empty string if no reply expected)
- `replyToMsgId::String = ""` - Message ID this message is replying to
- `brokerURL::String = DEFAULT_NATS_URL` - NATS broker URL
- `metadata::Dict{String, Any} = Dict{String, Any}()` - Optional message-level metadata
# Return:
- A msgEnvelope_v1 struct instance
# Example
```jldoctest
using UUIDs, NATSBridge
# Create payloads for the message
payload1 = msgPayload_v1("Hello", "text"; dataname="message", transport="direct", encoding="base64")
payload2 = msgPayload_v1("http://example.com/file.zip", "binary"; dataname="file", transport="link")
# Create message envelope
env = msgEnvelope_v1(
"my.subject",
[payload1, payload2];
correlationId = string(uuid4()),
msgPurpose = "chat",
senderName = "my-app",
receiverName = "receiver-app",
replyTo = "reply.subject"
)
```
"""
struct msgEnvelope_v1
correlationId::String # Unique identifier to track messages across systems. Many senders can talk about the same topic.
msgId::String # this message id
@@ -137,8 +235,34 @@ end
""" Convert msgEnvelope_v1 to JSON string
This function converts the msgEnvelope_v1 struct to a JSON string representation.
""" envelope_to_json - Convert msgEnvelope_v1 to JSON string
This function converts the msgEnvelope_v1 struct to a JSON string representation,
preserving all metadata and payload information for NATS message publishing.
# Function Workflow:
1. Creates a dictionary with envelope metadata (correlationId, msgId, timestamp, etc.)
2. Conditionally includes metadata dictionary if not empty
3. Iterates through payloads and converts each to JSON-compatible dictionary
4. Handles direct transport payloads (Base64 encoding) and link transport payloads (URL)
5. Returns final JSON string representation
# Arguments:
- `env::msgEnvelope_v1` - The msgEnvelope_v1 struct to convert to JSON
# Return:
- `String` - JSON string representation of the envelope
# Example
```jldoctest
using UUIDs
# Create an envelope with payloads
payload = msgPayload_v1("Hello", "text"; dataname="msg", transport="direct", encoding="base64")
env = msgEnvelope_v1("my.subject", [payload])
# Convert to JSON for publishing
json_msg = envelope_to_json(env)
```
"""
function envelope_to_json(env::msgEnvelope_v1)
obj = Dict{String, Any}(
@@ -197,9 +321,24 @@ function envelope_to_json(env::msgEnvelope_v1)
end
""" Log a trace message with correlation ID and timestamp
""" log_trace - Log a trace message with correlation ID and timestamp
This function logs information messages with a correlation ID for tracing purposes,
making it easier to track message flow across distributed systems.
# Arguments:
- `correlation_id::String` - Correlation ID to identify the message flow
- `message::String` - The message content to log
# Return:
- `nothing` - This function performs logging but returns nothing
# Example
```jldoctest
using Dates
log_trace("abc123", "Starting message processing")
# Logs: [2026-02-21T05:39:00] [Correlation: abc123] Starting message processing
```
"""
function log_trace(correlation_id::String, message::String)
timestamp = Dates.now() # Get current timestamp
@@ -216,7 +355,7 @@ Otherwise, it uploads the data to a fileserver (by default using `plik_oneshot_u
The function accepts a list of (dataname, data, type) tuples as input and processes each payload individually.
Each payload can have a different type, enabling mixed-content messages (e.g., chat with text, images, audio).
The function workflow:
# Function Workflow:
1. Iterates through the list of (dataname, data, type) tuples
2. For each payload: extracts the type from the tuple and serializes accordingly
3. Compares the serialized size against `size_threshold`
@@ -247,7 +386,7 @@ The function workflow:
- A `msgEnvelope_v1` object containing metadata and transport information
# Example
```julia
```jldoctest
using UUIDs
# Send a single payload (still wrapped in a list)
@@ -376,26 +515,20 @@ end
""" _serialize_data - Serialize data according to specified format
This function serializes arbitrary Julia data into a binary representation based on the specified format.
It supports multiple serialization formats:
- `"text"`: Treats data as text and converts to UTF-8 bytes
- `"dictionary"`: Serializes data as JSON and returns the UTF-8 byte representation
- `"table"`: Serializes data as an Arrow IPC stream (table format) and returns the byte stream
- `"image"`: Expects binary image data (Vector{UInt8}) and returns it as bytes
- `"audio"`: Expects binary audio data (Vector{UInt8}) and returns it as bytes
- `"video"`: Expects binary video data (Vector{UInt8}) and returns it as bytes
- `"binary"`: Generic binary data (Vector{UInt8} or IOBuffer) and returns bytes
It supports multiple serialization formats for different data types.
The function handles format-specific serialization logic:
1. For `"text"`: Converts string to UTF-8 bytes
2. For `"dictionary"`: Converts Julia data to JSON string, then encodes to bytes
3. For `"table"`: Uses Arrow.jl to write data as an Arrow IPC stream to an in-memory buffer
4. For `"image"`, `"audio"`, `"video"`: Treats data as binary (Vector{UInt8})
5. For `"binary"`: Extracts bytes from `IOBuffer` or returns `Vector{UInt8}` directly
# Function Workflow:
1. Validates the data type against the specified format
2. Converts data to binary representation according to format rules
3. For text: converts string to UTF-8 bytes
4. For dictionary: serializes as JSON then converts to bytes
5. For table: uses Arrow.jl to write as IPC stream
6. For image/audio/video/binary: returns binary data directly
# Arguments:
- `data::Any` - Data to serialize (string for `"text"`, JSON-serializable for `"dictionary"`, table-like for `"table"`, binary for `"image"`, `"audio"`, `"video"`, `"binary"`)
- `type::String` - Target format: "text", "dictionary", "table", "image", "audio", "video", "binary"
# Return:
- `Vector{UInt8}` - Binary representation of the serialized data
@@ -405,7 +538,7 @@ The function handles format-specific serialization logic:
- `Error` if `type` is `"image"`, `"audio"`, or `"video"` but `data` is not `Vector{UInt8}`
# Example
```julia
```jldoctest
using JSON, Arrow, DataFrames
# Text serialization
@@ -505,15 +638,29 @@ function _serialize_data(data::Any, type::String)
end
""" Publish message to NATS
""" publish_message - Publish message to NATS
This internal function publishes a message to a NATS subject with proper
connection management and logging.
Arguments:
- `nats_url::String` - NATS server URL
- `subject::String` - NATS subject to publish to
# Arguments:
- `nats_url::String` - NATS server URL (e.g., "nats://localhost:4222")
- `subject::String` - NATS subject to publish to (e.g., "/agent/wine/api/v1/prompt")
- `message::String` - JSON message to publish
- `correlation_id::String` - Correlation ID for logging
- `correlation_id::String` - Correlation ID for tracing and logging
# Return:
- `nothing` - This function performs publishing but returns nothing
# Example
```jldoctest
using NATS
# Prepare JSON message
message = "{\"correlationId\":\"abc123\",\"payload\":\"test\"}"
# Publish to NATS
publish_message("nats://localhost:4222", "my.subject", message, "abc123")
```
"""
function publish_message(nats_url::String, subject::String, message::String, correlation_id::String)
conn = NATS.connect(nats_url) # Create NATS connection
@@ -532,20 +679,27 @@ This function processes incoming NATS messages, handling both direct transport
It deserializes the data based on the transport type and returns the result.
A HTTP file server is required along with its download function.
Arguments:
# Function Workflow:
1. Parses the JSON envelope from the NATS message
2. Iterates through each payload in the envelope
3. For each payload: determines the transport type (direct or link)
4. For direct transport: decodes Base64 payload and deserializes based on type
5. For link transport: fetches data from URL with exponential backoff, then deserializes
# Arguments:
- `msg::NATS.Msg` - NATS message to process
- `fileserverDownloadHandler::Function` - Function to handle downloading data from file server URLs
Keyword Arguments:
- `max_retries::Int` - Maximum retry attempts for fetching URL (default: 5)
- `base_delay::Int` - Initial delay for exponential backoff in ms (default: 100)
- `max_delay::Int` - Maximum delay for exponential backoff in ms (default: 5000)
# Keyword Arguments:
- `fileserverDownloadHandler::Function = _fetch_with_backoff` - Function to handle downloading data from file server URLs
- `max_retries::Int = 5` - Maximum retry attempts for fetching URL
- `base_delay::Int = 100` - Initial delay for exponential backoff in ms
- `max_delay::Int = 5000` - Maximum delay for exponential backoff in ms
Return:
# Return:
- `AbstractArray{Tuple{String, Any, String}}` - List of (dataname, data, type) tuples
# Example
```julia
```jldoctest
# Receive and process message
msg = nats_message # NATS message
payloads = smartreceive(msg; fileserverDownloadHandler=_fetch_with_backoff, max_retries=5, base_delay=100, max_delay=5000)
@@ -605,24 +759,40 @@ function smartreceive(
error("Unknown transport type for payload '$dataname': $(transport)") # Throw error for unknown transport
end
end
return payloads_list # Return list of (dataname, data, data_type) tuples
json_data["payloads"] = payloads_list
return json_data # Return envelope with list of (dataname, data, data_type) tuples in payloads field
end
""" Fetch data from URL with exponential backoff
""" _fetch_with_backoff - Fetch data from URL with exponential backoff
This internal function retrieves data from a URL with retry logic using
exponential backoff to handle transient failures.
Arguments:
# Function Workflow:
1. Initializes delay with base_delay value
2. Attempts to fetch data from URL in a retry loop
3. On success: logs success and returns response body as bytes
4. On failure: sleeps using exponential backoff and retries
5. After max_retries: throws error indicating failure
# Arguments:
- `url::String` - URL to fetch from
- `max_retries::Int` - Maximum number of retry attempts
- `base_delay::Int` - Initial delay in milliseconds
- `max_delay::Int` - Maximum delay in milliseconds
- `correlation_id::String` - Correlation ID for logging
Return:
- Vector{UInt8} - Fetched data as bytes
# Return:
- `Vector{UInt8}` - Fetched data as bytes
# Throws:
- `Error` if all retry attempts fail
# Example
```jldoctest
# Fetch data with exponential backoff
data = _fetch_with_backoff("http://example.com/file.zip", 5, 100, 5000, "correlation123")
```
"""
function _fetch_with_backoff(
url::String,
@@ -655,18 +825,44 @@ function _fetch_with_backoff(
end
""" Deserialize bytes to data based on type
""" _deserialize_data - Deserialize bytes to data based on type
This internal function converts serialized bytes back to Julia data based on type.
It handles "text" (string), "dictionary" (JSON deserialization), "table" (Arrow IPC deserialization),
"image" (binary data), "audio" (binary data), "video" (binary data), and "binary" (binary data).
Arguments:
# Function Workflow:
1. Validates the data type against supported formats
2. Converts bytes to appropriate Julia data type based on format
3. For text: converts bytes to string
4. For dictionary: converts bytes to JSON string then parses to Julia object
5. For table: reads Arrow IPC format and returns DataFrame
6. For image/audio/video/binary: returns bytes directly
# Arguments:
- `data::Vector{UInt8}` - Serialized data as bytes
- `type::String` - Data type ("text", "dictionary", "table", "image", "audio", "video", "binary")
- `correlation_id::String` - Correlation ID for logging
Return:
# Return:
- Deserialized data (String for "text", DataFrame for "table", JSON data for "dictionary", bytes for "image", "audio", "video", "binary")
# Throws:
- `Error` if `type` is not one of the supported types
# Example
```jldoctest
# Text data
text_bytes = UInt8["Hello World"]
text_data = _deserialize_data(text_bytes, "text", "correlation123")
# JSON data
json_bytes = UInt8[123, 34, 110, 97, 109, 101, 34, 58, 34, 65, 108, 105, 99, 101, 125] # {"name":"Alice"}
json_data = _deserialize_data(json_bytes, "dictionary", "correlation123")
# Arrow IPC data (table)
arrow_bytes = UInt8[1, 2, 3] # Arrow IPC bytes
table_data = _deserialize_data(arrow_bytes, "table", "correlation123")
```
"""
function _deserialize_data(
data::Vector{UInt8},
@@ -697,15 +893,15 @@ end
""" plik_oneshot_upload - Upload a single file to a plik server using one-shot mode
This function uploads a raw byte array to a plik server in one-shot mode (no upload session).
It first creates a one-shot upload session by sending a POST request with `{"OneShot": true}`,
retrieves an upload ID and token, then uploads the file data as multipart form data using the token.
The function workflow:
1. Obtains an upload ID and token from the server
2. Uploads the provided binary data as a file using the `X-UploadToken` header
3. Returns identifiers and download URL for the uploaded file
# Function Workflow:
1. Creates a one-shot upload session by sending POST request with `{"OneShot": true}`
2. Retrieves upload ID and token from server response
3. Uploads binary data as multipart form data using the token
4. Returns identifiers and download URL for the uploaded file
# Arguments:
- `fileServerURL::String` - Base URL of the plik server (e.g., `"http://localhost:8080"`)
@@ -713,14 +909,14 @@ The function workflow:
- `data::Vector{UInt8}` - Raw byte data of the file content
# Return:
- A Dict with keys:
- `Dict{String, Any}` - Dictionary with keys:
- `"status"` - HTTP server response status
- `"uploadid"` - ID of the one-shot upload session
- `"fileid"` - ID of the uploaded file within the session
- `"url"` - Full URL to download the uploaded file
# Example
```julia
```jldoctest
using HTTP, JSON
fileServerURL = "http://localhost:8080"
@@ -776,31 +972,29 @@ end
""" plik_oneshot_upload(fileServerURL::String, filepath::String)
Upload a single file to a plik server using one-shot mode.
This function uploads a file from disk to a plik server in one-shot mode (no upload session).
It first creates a one-shot upload session by sending a POST request with `{"OneShot": true}`,
retrieves an upload ID and token, then uploads the file data as multipart form data using the token.
The function workflow:
1. Obtains an upload ID and token from the server
2. Uploads the file at `filepath` using multipart form data and the `X-UploadToken` header
3. Returns identifiers and download URL for the uploaded file
# Function Workflow:
1. Creates a one-shot upload session by sending POST request with `{"OneShot": true}`
2. Retrieves upload ID and token from server response
3. Uploads the file at `filepath` using multipart form data and the `X-UploadToken` header
4. Returns identifiers and download URL for the uploaded file
# Arguments:
- `fileServerURL::String` - Base URL of the plik server (e.g., `"http://localhost:8080"`)
- `filepath::String` - Full path to the local file to upload
# Return:
- A Dict with keys:
- `Dict{String, Any}` - Dictionary with keys:
- `"status"` - HTTP server response status
- `"uploadid"` - ID of the one-shot upload session
- `"fileid"` - ID of the uploaded file within the session
- `"url"` - Full URL to download the uploaded file
# Example
```julia
```jldoctest
using HTTP, JSON
fileServerURL = "http://localhost:8080"

View File

@@ -603,7 +603,7 @@ async function smartreceive(msg, options = {}) {
* @param {number} options.baseDelay - Initial delay for exponential backoff in ms (default: 100)
* @param {number} options.maxDelay - Maximum delay for exponential backoff in ms (default: 5000)
*
* @returns {Promise<Array>} - List of {dataname, data, type} objects
* @returns {Promise<Object>} - Envelope dictionary with metadata and payloads field containing list of {dataname, data, type} objects
*/
const {
fileserverDownloadHandler = _fetch_with_backoff,
@@ -664,7 +664,10 @@ async function smartreceive(msg, options = {}) {
}
}
return payloads_list;
// Replace payloads array with the processed list of {dataname, data, type} tuples
json_data.payloads = payloads_list;
return json_data;
}
// Export for Node.js

667
src/nats_bridge.py Normal file
View File

@@ -0,0 +1,667 @@
"""
Micropython NATS Bridge - Bi-Directional Data Bridge for Micropython
This module provides functionality for sending and receiving data over NATS
using the Claim-Check pattern for large payloads.
Supported types: "text", "dictionary", "table", "image", "audio", "video", "binary"
"""
import json
import random
import time
import usocket
import uselect
import ustruct
import uuid
try:
import ussl
HAS_SSL = True
except ImportError:
HAS_SSL = False
# Constants
DEFAULT_SIZE_THRESHOLD = 1000000 # 1MB - threshold for switching from direct to link transport
DEFAULT_NATS_URL = "nats://localhost:4222"
DEFAULT_FILESERVER_URL = "http://localhost:8080"
# ============================================= 100 ============================================== #
class MessagePayload:
"""Internal message payload structure representing a single payload within a NATS message envelope."""
def __init__(self, data, msg_type, id="", dataname="", transport="direct",
encoding="none", size=0, metadata=None):
"""
Initialize a MessagePayload.
Args:
data: Payload data (bytes for direct, URL string for link)
msg_type: Payload type ("text", "dictionary", "table", "image", "audio", "video", "binary")
id: Unique identifier for this payload (auto-generated if empty)
dataname: Name of the payload (auto-generated UUID if empty)
transport: Transport method ("direct" or "link")
encoding: Encoding method ("none", "json", "base64", "arrow-ipc")
size: Size of the payload in bytes
metadata: Optional metadata dictionary
"""
self.id = id if id else self._generate_uuid()
self.dataname = dataname if dataname else self._generate_uuid()
self.type = msg_type
self.transport = transport
self.encoding = encoding
self.size = size
self.data = data
self.metadata = metadata if metadata else {}
def _generate_uuid(self):
"""Generate a UUID string."""
return str(uuid.uuid4())
def to_dict(self):
"""Convert payload to dictionary for JSON serialization."""
payload_dict = {
"id": self.id,
"dataname": self.dataname,
"type": self.type,
"transport": self.transport,
"encoding": self.encoding,
"size": self.size,
}
# Include data based on transport type
if self.transport == "direct" and self.data is not None:
if self.encoding == "base64" or self.encoding == "json":
payload_dict["data"] = self.data
else:
# For other encodings, use base64
payload_dict["data"] = self._to_base64(self.data)
elif self.transport == "link" and self.data is not None:
# For link transport, data is a URL string
payload_dict["data"] = self.data
if self.metadata:
payload_dict["metadata"] = self.metadata
return payload_dict
def _to_base64(self, data):
"""Convert bytes to base64 string."""
if isinstance(data, bytes):
# Simple base64 encoding without library
import ubinascii
return ubinascii.b2a_base64(data).decode('utf-8').strip()
return data
def _from_base64(self, data):
"""Convert base64 string to bytes."""
import ubinascii
return ubinascii.a2b_base64(data)
class MessageEnvelope:
"""Internal message envelope structure containing multiple payloads with metadata."""
def __init__(self, send_to, payloads, correlation_id="", msg_id="", timestamp="",
msg_purpose="", sender_name="", sender_id="", receiver_name="",
receiver_id="", reply_to="", reply_to_msg_id="", broker_url=DEFAULT_NATS_URL,
metadata=None):
"""
Initialize a MessageEnvelope.
Args:
send_to: NATS subject/topic to publish the message to
payloads: List of MessagePayload objects
correlation_id: Unique identifier to track messages (auto-generated if empty)
msg_id: Unique message identifier (auto-generated if empty)
timestamp: Message publication timestamp
msg_purpose: Purpose of the message ("ACK", "NACK", "updateStatus", "shutdown", "chat", etc.)
sender_name: Name of the sender
sender_id: UUID of the sender
receiver_name: Name of the receiver (empty means broadcast)
receiver_id: UUID of the receiver (empty means broadcast)
reply_to: Topic where receiver should reply
reply_to_msg_id: Message ID this message is replying to
broker_url: NATS broker URL
metadata: Optional message-level metadata
"""
self.correlation_id = correlation_id if correlation_id else self._generate_uuid()
self.msg_id = msg_id if msg_id else self._generate_uuid()
self.timestamp = timestamp if timestamp else self._get_timestamp()
self.send_to = send_to
self.msg_purpose = msg_purpose
self.sender_name = sender_name
self.sender_id = sender_id if sender_id else self._generate_uuid()
self.receiver_name = receiver_name
self.receiver_id = receiver_id if receiver_id else self._generate_uuid()
self.reply_to = reply_to
self.reply_to_msg_id = reply_to_msg_id
self.broker_url = broker_url
self.metadata = metadata if metadata else {}
self.payloads = payloads
def _generate_uuid(self):
"""Generate a UUID string."""
return str(uuid.uuid4())
def _get_timestamp(self):
"""Get current timestamp in ISO format."""
# Simplified timestamp - Micropython may not have full datetime
return "2026-02-21T" + time.strftime("%H:%M:%S", time.localtime())
def to_json(self):
"""Convert envelope to JSON string."""
obj = {
"correlationId": self.correlation_id,
"msgId": self.msg_id,
"timestamp": self.timestamp,
"sendTo": self.send_to,
"msgPurpose": self.msg_purpose,
"senderName": self.sender_name,
"senderId": self.sender_id,
"receiverName": self.receiver_name,
"receiverId": self.receiver_id,
"replyTo": self.reply_to,
"replyToMsgId": self.reply_to_msg_id,
"brokerURL": self.broker_url
}
# Include metadata if not empty
if self.metadata:
obj["metadata"] = self.metadata
# Convert payloads to JSON array
if self.payloads:
payloads_json = []
for payload in self.payloads:
payloads_json.append(payload.to_dict())
obj["payloads"] = payloads_json
return json.dumps(obj)
def log_trace(correlation_id, message):
"""Log a trace message with correlation ID and timestamp."""
timestamp = time.strftime("%Y-%m-%dT%H:%M:%S", time.localtime())
print("[{}] [Correlation: {}] {}".format(timestamp, correlation_id, message))
def _serialize_data(data, msg_type):
"""Serialize data according to specified format.
Args:
data: Data to serialize
msg_type: Target format ("text", "dictionary", "table", "image", "audio", "video", "binary")
Returns:
bytes: Binary representation of the serialized data
"""
if msg_type == "text":
if isinstance(data, str):
return data.encode('utf-8')
else:
raise ValueError("Text data must be a string")
elif msg_type == "dictionary":
if isinstance(data, dict):
json_str = json.dumps(data)
return json_str.encode('utf-8')
else:
raise ValueError("Dictionary data must be a dict")
elif msg_type in ("image", "audio", "video", "binary"):
if isinstance(data, bytes):
return data
else:
raise ValueError("{} data must be bytes".format(msg_type.capitalize()))
else:
raise ValueError("Unknown type: {}".format(msg_type))
def _deserialize_data(data_bytes, msg_type, correlation_id):
"""Deserialize bytes to data based on type.
Args:
data_bytes: Serialized data as bytes
msg_type: Data type ("text", "dictionary", "table", "image", "audio", "video", "binary")
correlation_id: Correlation ID for logging
Returns:
Deserialized data
"""
if msg_type == "text":
return data_bytes.decode('utf-8')
elif msg_type == "dictionary":
json_str = data_bytes.decode('utf-8')
return json.loads(json_str)
elif msg_type in ("image", "audio", "video", "binary"):
return data_bytes
else:
raise ValueError("Unknown type: {}".format(msg_type))
class NATSConnection:
"""Simple NATS connection for Micropython."""
def __init__(self, url=DEFAULT_NATS_URL):
"""Initialize NATS connection.
Args:
url: NATS server URL (e.g., "nats://localhost:4222")
"""
self.url = url
self.host = "localhost"
self.port = 4222
self.conn = None
self._parse_url(url)
def _parse_url(self, url):
"""Parse NATS URL to extract host and port."""
if url.startswith("nats://"):
url = url[7:]
elif url.startswith("tls://"):
url = url[6:]
if ":" in url:
self.host, port_str = url.split(":")
self.port = int(port_str)
else:
self.host = url
def connect(self):
"""Connect to NATS server."""
addr = usocket.getaddrinfo(self.host, self.port)[0][-1]
self.conn = usocket.socket()
self.conn.connect(addr)
log_trace("", "Connected to NATS server at {}:{}".format(self.host, self.port))
def publish(self, subject, message):
"""Publish a message to a NATS subject.
Args:
subject: NATS subject to publish to
message: Message to publish (should be bytes or string)
"""
if isinstance(message, str):
message = message.encode('utf-8')
# Simple NATS protocol implementation
msg = "PUB {} {}\r\n".format(subject, len(message))
msg = msg.encode('utf-8') + message + b"\r\n"
self.conn.send(msg)
log_trace("", "Message published to {}".format(subject))
def subscribe(self, subject, callback):
"""Subscribe to a NATS subject.
Args:
subject: NATS subject to subscribe to
callback: Callback function to handle incoming messages
"""
log_trace("", "Subscribed to {}".format(subject))
# Simplified subscription - in a real implementation, you'd handle SUB/PUB messages
# For Micropython, we'll use a simple polling approach
self.subscribed_subject = subject
self.subscription_callback = callback
def wait_message(self, timeout=1000):
"""Wait for incoming message.
Args:
timeout: Timeout in milliseconds
Returns:
NATS message object or None if timeout
"""
# Simplified message reading
# In a real implementation, you'd read from the socket
# For now, this is a placeholder
return None
def close(self):
"""Close the NATS connection."""
if self.conn:
self.conn.close()
self.conn = None
log_trace("", "NATS connection closed")
def _fetch_with_backoff(url, max_retries=5, base_delay=100, max_delay=5000, correlation_id=""):
"""Fetch data from URL with exponential backoff.
Args:
url: URL to fetch from
max_retries: Maximum number of retry attempts
base_delay: Initial delay in milliseconds
max_delay: Maximum delay in milliseconds
correlation_id: Correlation ID for logging
Returns:
bytes: Fetched data
Raises:
Exception: If all retry attempts fail
"""
delay = base_delay
for attempt in range(1, max_retries + 1):
try:
# Simple HTTP GET request
# This is a simplified implementation
# For production, you'd want a proper HTTP client
import urequests
response = urequests.get(url)
if response.status_code == 200:
log_trace(correlation_id, "Successfully fetched data from {} on attempt {}".format(url, attempt))
return response.content
else:
raise Exception("Failed to fetch: {}".format(response.status_code))
except Exception as e:
log_trace(correlation_id, "Attempt {} failed: {}".format(attempt, str(e)))
if attempt < max_retries:
time.sleep(delay / 1000.0)
delay = min(delay * 2, max_delay)
def plik_oneshot_upload(file_server_url, filename, data):
"""Upload a single file to a plik server using one-shot mode.
Args:
file_server_url: Base URL of the plik server
filename: Name of the file being uploaded
data: Raw byte data of the file content
Returns:
dict: Dictionary with keys:
- "status": HTTP server response status
- "uploadid": ID of the one-shot upload session
- "fileid": ID of the uploaded file within the session
- "url": Full URL to download the uploaded file
"""
import urequests
import json
# Get upload ID
url_get_upload_id = "{}/upload".format(file_server_url)
headers = {"Content-Type": "application/json"}
body = json.dumps({"OneShot": True})
response = urequests.post(url_get_upload_id, headers=headers, data=body)
response_json = json.loads(response.content)
uploadid = response_json.get("id")
uploadtoken = response_json.get("uploadToken")
# Upload file
url_upload = "{}/file/{}".format(file_server_url, uploadid)
headers = {"X-UploadToken": uploadtoken}
# For Micropython, we need to construct the multipart form data manually
# This is a simplified approach
boundary = "----WebKitFormBoundary{}".format(uuid.uuid4().hex[:16])
# Create multipart body
part1 = "--{}\r\n".format(boundary)
part1 += "Content-Disposition: form-data; name=\"file\"; filename=\"{}\"\r\n".format(filename)
part1 += "Content-Type: application/octet-stream\r\n\r\n"
part1_bytes = part1.encode('utf-8')
part2 = "\r\n--{}--".format(boundary)
part2_bytes = part2.encode('utf-8')
# Combine all parts
full_body = part1_bytes + data + part2_bytes
# Set content type with boundary
content_type = "multipart/form-data; boundary={}".format(boundary)
response = urequests.post(url_upload, headers={"Content-Type": content_type}, data=full_body)
response_json = json.loads(response.content)
fileid = response_json.get("id")
url = "{}/file/{}/{}".format(file_server_url, uploadid, filename)
return {
"status": response.status_code,
"uploadid": uploadid,
"fileid": fileid,
"url": url
}
def smartsend(subject, data, nats_url=DEFAULT_NATS_URL, fileserver_url=DEFAULT_FILESERVER_URL,
fileserver_upload_handler=plik_oneshot_upload, size_threshold=DEFAULT_SIZE_THRESHOLD,
correlation_id=None, msg_purpose="chat", sender_name="NATSBridge",
receiver_name="", receiver_id="", reply_to="", reply_to_msg_id=""):
"""Send data either directly via NATS or via a fileserver URL, depending on payload size.
This function intelligently routes data delivery based on payload size relative to a threshold.
If the serialized payload is smaller than `size_threshold`, it encodes the data as Base64 and
publishes directly over NATS. Otherwise, it uploads the data to a fileserver and publishes
only the download URL over NATS.
Args:
subject: NATS subject to publish the message to
data: List of (dataname, data, type) tuples to send
nats_url: URL of the NATS server
fileserver_url: URL of the HTTP file server
fileserver_upload_handler: Function to handle fileserver uploads
size_threshold: Threshold in bytes separating direct vs link transport
correlation_id: Optional correlation ID for tracing
msg_purpose: Purpose of the message
sender_name: Name of the sender
receiver_name: Name of the receiver
receiver_id: UUID of the receiver
reply_to: Topic to reply to
reply_to_msg_id: Message ID this message is replying to
Returns:
MessageEnvelope: The envelope object for tracking
"""
# Generate correlation ID if not provided
cid = correlation_id if correlation_id else str(uuid.uuid4())
log_trace(cid, "Starting smartsend for subject: {}".format(subject))
# Generate message metadata
msg_id = str(uuid.uuid4())
# Process each payload in the list
payloads = []
for dataname, payload_data, payload_type in data:
# Serialize data based on type
payload_bytes = _serialize_data(payload_data, payload_type)
payload_size = len(payload_bytes)
log_trace(cid, "Serialized payload '{}' (type: {}) size: {} bytes".format(
dataname, payload_type, payload_size))
# Decision: Direct vs Link
if payload_size < size_threshold:
# Direct path - Base64 encode and send via NATS
payload_b64 = _serialize_data(payload_bytes, "binary") # Already bytes
# Convert to base64 string for JSON
import ubinascii
payload_b64_str = ubinascii.b2a_base64(payload_bytes).decode('utf-8').strip()
log_trace(cid, "Using direct transport for {} bytes".format(payload_size))
# Create MessagePayload for direct transport
payload = MessagePayload(
payload_b64_str,
payload_type,
id=str(uuid.uuid4()),
dataname=dataname,
transport="direct",
encoding="base64",
size=payload_size,
metadata={"payload_bytes": payload_size}
)
payloads.append(payload)
else:
# Link path - Upload to HTTP server, send URL via NATS
log_trace(cid, "Using link transport, uploading to fileserver")
# Upload to HTTP server
response = fileserver_upload_handler(fileserver_url, dataname, payload_bytes)
if response["status"] != 200:
raise Exception("Failed to upload data to fileserver: {}".format(response["status"]))
url = response["url"]
log_trace(cid, "Uploaded to URL: {}".format(url))
# Create MessagePayload for link transport
payload = MessagePayload(
url,
payload_type,
id=str(uuid.uuid4()),
dataname=dataname,
transport="link",
encoding="none",
size=payload_size,
metadata={}
)
payloads.append(payload)
# Create MessageEnvelope with all payloads
env = MessageEnvelope(
subject,
payloads,
correlation_id=cid,
msg_id=msg_id,
msg_purpose=msg_purpose,
sender_name=sender_name,
sender_id=str(uuid.uuid4()),
receiver_name=receiver_name,
receiver_id=receiver_id,
reply_to=reply_to,
reply_to_msg_id=reply_to_msg_id,
broker_url=nats_url,
metadata={}
)
msg_json = env.to_json()
# Publish to NATS
nats_conn = NATSConnection(nats_url)
nats_conn.connect()
nats_conn.publish(subject, msg_json)
nats_conn.close()
return env
def smartreceive(msg, fileserver_download_handler=_fetch_with_backoff, max_retries=5,
base_delay=100, max_delay=5000):
"""Receive and process messages from NATS.
This function processes incoming NATS messages, handling both direct transport
(base64 decoded payloads) and link transport (URL-based payloads).
Args:
msg: NATS message to process (dict with payload data)
fileserver_download_handler: Function to handle downloading data from file server URLs
max_retries: Maximum retry attempts for fetching URL
base_delay: Initial delay for exponential backoff in ms
max_delay: Maximum delay for exponential backoff in ms
Returns:
dict: Envelope dictionary with metadata and 'payloads' field containing list of (dataname, data, type) tuples
"""
# Parse the JSON envelope
json_data = msg if isinstance(msg, dict) else json.loads(msg)
log_trace(json_data.get("correlationId", ""), "Processing received message")
# Process all payloads in the envelope
payloads_list = []
# Get number of payloads
num_payloads = len(json_data.get("payloads", []))
for i in range(num_payloads):
payload = json_data["payloads"][i]
transport = payload.get("transport", "")
dataname = payload.get("dataname", "")
if transport == "direct":
log_trace(json_data.get("correlationId", ""),
"Direct transport - decoding payload '{}'".format(dataname))
# Extract base64 payload from the payload
payload_b64 = payload.get("data", "")
# Decode Base64 payload
import ubinascii
payload_bytes = ubinascii.a2b_base64(payload_b64.encode('utf-8'))
# Deserialize based on type
data_type = payload.get("type", "")
data = _deserialize_data(payload_bytes, data_type, json_data.get("correlationId", ""))
payloads_list.append((dataname, data, data_type))
elif transport == "link":
# Extract download URL from the payload
url = payload.get("data", "")
log_trace(json_data.get("correlationId", ""),
"Link transport - fetching '{}' from URL: {}".format(dataname, url))
# Fetch with exponential backoff
downloaded_data = fileserver_download_handler(
url, max_retries, base_delay, max_delay, json_data.get("correlationId", "")
)
# Deserialize based on type
data_type = payload.get("type", "")
data = _deserialize_data(downloaded_data, data_type, json_data.get("correlationId", ""))
payloads_list.append((dataname, data, data_type))
else:
raise ValueError("Unknown transport type for payload '{}': {}".format(dataname, transport))
# Replace payloads field with the processed list of (dataname, data, type) tuples
json_data["payloads"] = payloads_list
return json_data
# Utility functions
def generate_uuid():
"""Generate a UUID string."""
return str(uuid.uuid4())
def get_timestamp():
"""Get current timestamp in ISO format."""
return time.strftime("%Y-%m-%dT%H:%M:%S", time.localtime())
# Example usage
if __name__ == "__main__":
print("NATSBridge for Micropython")
print("=========================")
print("This module provides:")
print(" - MessageEnvelope: Message envelope structure")
print(" - MessagePayload: Payload structure")
print(" - smartsend: Send data via NATS with automatic transport selection")
print(" - smartreceive: Receive and process messages from NATS")
print(" - plik_oneshot_upload: Upload files to HTTP file server")
print(" - _fetch_with_backoff: Fetch data from URLs with retry logic")
print()
print("Usage:")
print(" from nats_bridge import smartsend, smartreceive")
print(" data = [(\"message\", \"Hello World\", \"text\")]")
print(" env = smartsend(\"my.subject\", data)")
print()
print(" # On receiver:")
print(" payloads = smartreceive(msg)")
print(" for dataname, data, type in payloads:")
print(" print(f\"Received {dataname} of type {type}: {data}\")")

View File

@@ -37,8 +37,9 @@ async function test_dict_receive() {
}
);
// Result is a list of {dataname, data, type} objects
for (const { dataname, data, type } of result) {
// Result is an envelope dictionary with payloads field
// Access payloads with result.payloads
for (const { dataname, data, type } of result.payloads) {
if (typeof data === 'object' && data !== null && !Array.isArray(data)) {
log_trace(`Received Dictionary '${dataname}' of type ${type}`);

View File

@@ -36,8 +36,9 @@ async function test_large_binary_receive() {
}
);
// Result is a list of {dataname, data, type} objects
for (const { dataname, data, type } of result) {
// Result is an envelope dictionary with payloads field
// Access payloads with result.payloads
for (const { dataname, data, type } of result.payloads) {
if (data instanceof Uint8Array || Array.isArray(data)) {
const file_size = data.length;
log_trace(`Received ${file_size} bytes of binary data for '${dataname}' of type ${type}`);

View File

@@ -40,10 +40,11 @@ async function test_mix_receive() {
}
);
log_trace(`Received ${result.length} payloads`);
log_trace(`Received ${result.payloads.length} payloads`);
// Result is a list of {dataname, data, type} objects
for (const { dataname, data, type } of result) {
// Result is an envelope dictionary with payloads field
// Access payloads with result.payloads
for (const { dataname, data, type } of result.payloads) {
log_trace(`\n=== Payload: ${dataname} (type: ${type}) ===`);
// Handle different data types
@@ -122,13 +123,13 @@ async function test_mix_receive() {
// Summary
console.log("\n=== Verification Summary ===");
const text_count = result.filter(x => x.type === "text").length;
const dict_count = result.filter(x => x.type === "dictionary").length;
const table_count = result.filter(x => x.type === "table").length;
const image_count = result.filter(x => x.type === "image").length;
const audio_count = result.filter(x => x.type === "audio").length;
const video_count = result.filter(x => x.type === "video").length;
const binary_count = result.filter(x => x.type === "binary").length;
const text_count = result.payloads.filter(x => x.type === "text").length;
const dict_count = result.payloads.filter(x => x.type === "dictionary").length;
const table_count = result.payloads.filter(x => x.type === "table").length;
const image_count = result.payloads.filter(x => x.type === "image").length;
const audio_count = result.payloads.filter(x => x.type === "audio").length;
const video_count = result.payloads.filter(x => x.type === "video").length;
const binary_count = result.payloads.filter(x => x.type === "binary").length;
log_trace(`Text payloads: ${text_count}`);
log_trace(`Dictionary payloads: ${dict_count}`);
@@ -140,7 +141,7 @@ async function test_mix_receive() {
// Print transport type info for each payload if available
console.log("\n=== Payload Details ===");
for (const { dataname, data, type } of result) {
for (const { dataname, data, type } of result.payloads) {
if (["image", "audio", "video", "binary"].includes(type)) {
log_trace(`${dataname}: ${data.length} bytes (binary)`);
} else if (type === "table") {

View File

@@ -40,8 +40,9 @@ async function test_table_receive() {
}
);
// Result is a list of {dataname, data, type} objects
for (const { dataname, data, type } of result) {
// Result is an envelope dictionary with payloads field
// Access payloads with result.payloads
for (const { dataname, data, type } of result.payloads) {
if (Array.isArray(data)) {
log_trace(`Received Table '${dataname}' of type ${type}`);

View File

@@ -37,8 +37,9 @@ async function test_text_receive() {
}
);
// Result is a list of {dataname, data, type} objects
for (const { dataname, data, type } of result) {
// Result is an envelope dictionary with payloads field
// Access payloads with result.payloads
for (const { dataname, data, type } of result.payloads) {
if (typeof data === 'string') {
log_trace(`Received text '${dataname}' of type ${type}`);
log_trace(` Length: ${data.length} characters`);

View File

@@ -42,8 +42,8 @@ function test_dict_receive()
max_delay = 5000
)
# Result is a list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result["payloads"]
if isa(data, JSON.Object{String, Any})
log_trace("Received Dictionary '$dataname' of type $data_type")

View File

@@ -94,7 +94,7 @@ function test_dict_send()
# For large Dictionary: will use link transport (uploaded to fileserver)
env = NATSBridge.smartsend(
SUBJECT,
[data1, data2], # List of (dataname, data, type) tuples
[data1, data2]; # List of (dataname, data, type) tuples
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,

View File

@@ -44,8 +44,8 @@ function test_large_binary_receive()
max_delay = 5000
)
# Result is a list of (dataname, data) tuples
for (dataname, data, data_type) in result
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result["payloads"]
# Check transport type from the envelope
# For link transport, data is the URL string
# For direct transport, data is the actual payload bytes

View File

@@ -81,7 +81,7 @@ function test_large_binary_send()
# API: smartsend(subject, [(dataname, data, type), ...]; keywords...)
env = NATSBridge.smartsend(
SUBJECT,
[data1, data2], # List of (dataname, data, type) tuples
[data1, data2]; # List of (dataname, data, type) tuples
nats_url = NATS_URL;
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,

View File

@@ -45,10 +45,10 @@ function test_mix_receive()
max_delay = 5000
)
log_trace("Received $(length(result)) payloads")
log_trace("Received $(length(result["payloads"])) payloads")
# Result is a list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result["payloads"]
log_trace("\n=== Payload: $dataname (type: $data_type) ===")
# Handle different data types
@@ -178,13 +178,13 @@ function test_mix_receive()
# Summary
println("\n=== Verification Summary ===")
text_count = count(x -> x[3] == "text", result)
dict_count = count(x -> x[3] == "dictionary", result)
table_count = count(x -> x[3] == "table", result)
image_count = count(x -> x[3] == "image", result)
audio_count = count(x -> x[3] == "audio", result)
video_count = count(x -> x[3] == "video", result)
binary_count = count(x -> x[3] == "binary", result)
text_count = count(x -> x[3] == "text", result["payloads"])
dict_count = count(x -> x[3] == "dictionary", result["payloads"])
table_count = count(x -> x[3] == "table", result["payloads"])
image_count = count(x -> x[3] == "image", result["payloads"])
audio_count = count(x -> x[3] == "audio", result["payloads"])
video_count = count(x -> x[3] == "video", result["payloads"])
binary_count = count(x -> x[3] == "binary", result["payloads"])
log_trace("Text payloads: $text_count")
log_trace("Dictionary payloads: $dict_count")
@@ -196,7 +196,7 @@ function test_mix_receive()
# Print transport type info for each payload if available
println("\n=== Payload Details ===")
for (dataname, data, data_type) in result
for (dataname, data, data_type) in result["payloads"]
if data_type in ["image", "audio", "video", "binary"]
log_trace("$dataname: $(length(data)) bytes (binary)")
elseif data_type == "table"

View File

@@ -188,7 +188,7 @@ function test_mix_send()
# Use smartsend with mixed content
env = NATSBridge.smartsend(
SUBJECT,
payloads, # List of (dataname, data, type) tuples
payloads; # List of (dataname, data, type) tuples
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,

View File

@@ -42,8 +42,8 @@ function test_table_receive()
max_delay = 5000
)
# Result is a list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result["payloads"]
data = DataFrame(data)
if isa(data, DataFrame)
log_trace("Received DataFrame '$dataname' of type $data_type")

View File

@@ -92,7 +92,7 @@ function test_table_send()
# For large DataFrame: will use link transport (uploaded to fileserver)
env = NATSBridge.smartsend(
SUBJECT,
[data1, data2], # List of (dataname, data, type) tuples
[data1, data2]; # List of (dataname, data, type) tuples
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,

View File

@@ -42,8 +42,8 @@ function test_text_receive()
max_delay = 5000
)
# Result is a list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result["payloads"]
if isa(data, String)
log_trace("Received text '$dataname' of type $data_type")
log_trace(" Length: $(length(data)) characters")

View File

@@ -77,7 +77,7 @@ function test_text_send()
# For large text: will use link transport (uploaded to fileserver)
env = NATSBridge.smartsend(
SUBJECT,
[data1, data2], # List of (dataname, data, type) tuples
[data1, data2]; # List of (dataname, data, type) tuples
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,

View File

@@ -0,0 +1,207 @@
#!/usr/bin/env python3
"""
Basic functionality test for nats_bridge.py
Tests the core classes and functions without NATS connection
"""
import sys
import os
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import (
MessagePayload,
MessageEnvelope,
smartsend,
smartreceive,
log_trace,
generate_uuid,
get_timestamp,
_serialize_data,
_deserialize_data
)
import json
def test_message_payload():
"""Test MessagePayload class"""
print("\n=== Testing MessagePayload ===")
# Test direct transport with text
payload1 = MessagePayload(
data="Hello World",
msg_type="text",
id="test-id-1",
dataname="message",
transport="direct",
encoding="base64",
size=11
)
assert payload1.id == "test-id-1"
assert payload1.dataname == "message"
assert payload1.type == "text"
assert payload1.transport == "direct"
assert payload1.encoding == "base64"
assert payload1.size == 11
print(" [PASS] MessagePayload with text data")
# Test link transport with URL
payload2 = MessagePayload(
data="http://example.com/file.txt",
msg_type="binary",
id="test-id-2",
dataname="file",
transport="link",
encoding="none",
size=1000
)
assert payload2.transport == "link"
assert payload2.data == "http://example.com/file.txt"
print(" [PASS] MessagePayload with link transport")
# Test to_dict method
payload_dict = payload1.to_dict()
assert "id" in payload_dict
assert "dataname" in payload_dict
assert "type" in payload_dict
assert "transport" in payload_dict
assert "data" in payload_dict
print(" [PASS] MessagePayload.to_dict() method")
def test_message_envelope():
"""Test MessageEnvelope class"""
print("\n=== Testing MessageEnvelope ===")
# Create payloads
payload1 = MessagePayload("Hello", "text", id="p1", dataname="msg1")
payload2 = MessagePayload("http://example.com/file", "binary", id="p2", dataname="file", transport="link")
# Create envelope
env = MessageEnvelope(
send_to="/test/subject",
payloads=[payload1, payload2],
correlation_id="test-correlation-id",
msg_id="test-msg-id",
msg_purpose="chat",
sender_name="test_sender",
receiver_name="test_receiver",
reply_to="/test/reply"
)
assert env.send_to == "/test/subject"
assert env.correlation_id == "test-correlation-id"
assert env.msg_id == "test-msg-id"
assert env.msg_purpose == "chat"
assert len(env.payloads) == 2
print(" [PASS] MessageEnvelope creation")
# Test to_json method
json_str = env.to_json()
json_data = json.loads(json_str)
assert json_data["sendTo"] == "/test/subject"
assert json_data["correlationId"] == "test-correlation-id"
assert json_data["msgPurpose"] == "chat"
assert len(json_data["payloads"]) == 2
print(" [PASS] MessageEnvelope.to_json() method")
def test_serialize_data():
"""Test _serialize_data function"""
print("\n=== Testing _serialize_data ===")
# Test text serialization
text_bytes = _serialize_data("Hello", "text")
assert isinstance(text_bytes, bytes)
assert text_bytes == b"Hello"
print(" [PASS] Text serialization")
# Test dictionary serialization
dict_data = {"key": "value", "number": 42}
dict_bytes = _serialize_data(dict_data, "dictionary")
assert isinstance(dict_bytes, bytes)
parsed = json.loads(dict_bytes.decode('utf-8'))
assert parsed["key"] == "value"
print(" [PASS] Dictionary serialization")
# Test binary serialization
binary_data = b"\x00\x01\x02"
binary_bytes = _serialize_data(binary_data, "binary")
assert binary_bytes == b"\x00\x01\x02"
print(" [PASS] Binary serialization")
# Test image serialization
image_data = bytes([1, 2, 3, 4, 5])
image_bytes = _serialize_data(image_data, "image")
assert image_bytes == image_data
print(" [PASS] Image serialization")
def test_deserialize_data():
"""Test _deserialize_data function"""
print("\n=== Testing _deserialize_data ===")
# Test text deserialization
text_bytes = b"Hello"
text_data = _deserialize_data(text_bytes, "text", "test-correlation-id")
assert text_data == "Hello"
print(" [PASS] Text deserialization")
# Test dictionary deserialization
dict_bytes = b'{"key": "value"}'
dict_data = _deserialize_data(dict_bytes, "dictionary", "test-correlation-id")
assert dict_data == {"key": "value"}
print(" [PASS] Dictionary deserialization")
# Test binary deserialization
binary_data = b"\x00\x01\x02"
binary_result = _deserialize_data(binary_data, "binary", "test-correlation-id")
assert binary_result == b"\x00\x01\x02"
print(" [PASS] Binary deserialization")
def test_utilities():
"""Test utility functions"""
print("\n=== Testing Utility Functions ===")
# Test generate_uuid
uuid1 = generate_uuid()
uuid2 = generate_uuid()
assert uuid1 != uuid2
print(f" [PASS] generate_uuid() - generated: {uuid1}")
# Test get_timestamp
timestamp = get_timestamp()
assert "T" in timestamp
print(f" [PASS] get_timestamp() - generated: {timestamp}")
def main():
"""Run all tests"""
print("=" * 60)
print("NATSBridge Python/Micropython - Basic Functionality Tests")
print("=" * 60)
try:
test_message_payload()
test_message_envelope()
test_serialize_data()
test_deserialize_data()
test_utilities()
print("\n" + "=" * 60)
print("ALL TESTS PASSED!")
print("=" * 60)
except Exception as e:
print(f"\n[FAIL] Test failed with error: {e}")
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,70 @@
#!/usr/bin/env python3
"""
Test script for dictionary transport testing - Receiver
Tests receiving dictionary messages via NATS using nats_bridge.py smartreceive
"""
import sys
import os
import json
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartreceive, log_trace
import nats
import asyncio
# Configuration
SUBJECT = "/NATSBridge_dict_test"
NATS_URL = "nats://nats.yiem.cc:4222"
async def main():
log_trace("", f"Starting dictionary transport receiver test...")
log_trace("", f"Note: This receiver will wait for messages from the sender.")
log_trace("", f"Run test_micropython_dict_sender.py first to send test data.")
# Connect to NATS
nc = await nats.connect(NATS_URL)
log_trace("", f"Connected to NATS at {NATS_URL}")
# Subscribe to the subject
async def message_handler(msg):
log_trace("", f"Received message on {msg.subject}")
# Use smartreceive to handle the data
result = smartreceive(msg.data)
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for dataname, data, data_type in result["payloads"]:
if isinstance(data, dict):
log_trace(result.get("correlationId", ""), f"Received dictionary '{dataname}' of type {data_type}")
log_trace(result.get("correlationId", ""), f" Keys: {list(data.keys())}")
# Display first few items for small dicts
if isinstance(data, dict) and len(data) <= 10:
log_trace(result.get("correlationId", ""), f" Content: {json.dumps(data, indent=2)}")
else:
# For large dicts, show summary
log_trace(result.get("correlationId", ""), f" Summary: {json.dumps(data, default=str)[:200]}...")
# Save to file
output_path = f"./received_{dataname}.json"
with open(output_path, 'w') as f:
json.dump(data, f, indent=2)
log_trace(result.get("correlationId", ""), f"Saved dictionary to {output_path}")
else:
log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}")
sid = await nc.subscribe(SUBJECT, cb=message_handler)
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
# Keep listening for 120 seconds
await asyncio.sleep(120)
await nc.close()
log_trace("", "Test completed.")
if __name__ == "__main__":
asyncio.run(main())

View File

@@ -0,0 +1,99 @@
#!/usr/bin/env python3
"""
Test script for dictionary transport testing - Micropython
Tests sending dictionary messages via NATS using nats_bridge.py smartsend
"""
import sys
import os
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartsend, log_trace
import uuid
# Configuration
SUBJECT = "/NATSBridge_dict_test"
NATS_URL = "nats://nats.yiem.cc:4222"
FILESERVER_URL = "http://192.168.88.104:8080"
SIZE_THRESHOLD = 1_000_000 # 1MB
# Create correlation ID for tracing
correlation_id = str(uuid.uuid4())
def main():
# Create a small dictionary (will use direct transport)
small_dict = {
"name": "test",
"value": 42,
"enabled": True,
"metadata": {
"version": "1.0.0",
"timestamp": "2026-02-22T12:00:00Z"
}
}
# Create a large dictionary (will use link transport if > 1MB)
# Generate a larger dictionary (~2MB to ensure link transport)
large_dict = {
"id": str(uuid.uuid4()),
"items": [
{
"index": i,
"name": f"item_{i}",
"value": i * 1.5,
"data": "x" * 10000 # Large string per item
}
for i in range(200)
],
"metadata": {
"count": 200,
"created": "2026-02-22T12:00:00Z"
}
}
# Test data 1: small dictionary
data1 = ("small_dict", small_dict, "dictionary")
# Test data 2: large dictionary
data2 = ("large_dict", large_dict, "dictionary")
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
# Use smartsend with dictionary type
env = smartsend(
SUBJECT,
[data1, data2], # List of (dataname, data, type) tuples
nats_url=NATS_URL,
fileserver_url=FILESERVER_URL,
size_threshold=SIZE_THRESHOLD,
correlation_id=correlation_id,
msg_purpose="chat",
sender_name="dict_sender",
receiver_name="",
receiver_id="",
reply_to="",
reply_to_msg_id=""
)
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
# Log transport type for each payload
for i, payload in enumerate(env.payloads):
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
log_trace(correlation_id, f" Transport: {payload.transport}")
log_trace(correlation_id, f" Type: {payload.type}")
log_trace(correlation_id, f" Size: {payload.size} bytes")
log_trace(correlation_id, f" Encoding: {payload.encoding}")
if payload.transport == "link":
log_trace(correlation_id, f" URL: {payload.data}")
print(f"Test completed. Correlation ID: {correlation_id}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,65 @@
#!/usr/bin/env python3
"""
Test script for file transport testing - Receiver
Tests receiving binary files via NATS using nats_bridge.py smartreceive
"""
import sys
import os
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartreceive, log_trace
import nats
import asyncio
# Configuration
SUBJECT = "/NATSBridge_file_test"
NATS_URL = "nats://nats.yiem.cc:4222"
async def main():
log_trace("", f"Starting file transport receiver test...")
log_trace("", f"Note: This receiver will wait for messages from the sender.")
log_trace("", f"Run test_micropython_file_sender.py first to send test data.")
# Connect to NATS
nc = await nats.connect(NATS_URL)
log_trace("", f"Connected to NATS at {NATS_URL}")
# Subscribe to the subject
async def message_handler(msg):
log_trace("", f"Received message on {msg.subject}")
# Use smartreceive to handle the data
result = smartreceive(msg.data)
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for dataname, data, data_type in result["payloads"]:
if isinstance(data, bytes):
log_trace(result.get("correlationId", ""), f"Received binary '{dataname}' of type {data_type}")
log_trace(result.get("correlationId", ""), f" Size: {len(data)} bytes")
# Display first 100 bytes as hex
log_trace(result.get("correlationId", ""), f" First 100 bytes (hex): {data[:100].hex()}")
# Save to file
output_path = f"./received_{dataname}.bin"
with open(output_path, 'wb') as f:
f.write(data)
log_trace(result.get("correlationId", ""), f"Saved binary to {output_path}")
else:
log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}")
sid = await nc.subscribe(SUBJECT, cb=message_handler)
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
# Keep listening for 120 seconds
await asyncio.sleep(120)
await nc.close()
log_trace("", "Test completed.")
if __name__ == "__main__":
asyncio.run(main())

View File

@@ -0,0 +1,79 @@
#!/usr/bin/env python3
"""
Test script for file transport testing - Micropython
Tests sending binary files via NATS using nats_bridge.py smartsend
"""
import sys
import os
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartsend, log_trace
import uuid
# Configuration
SUBJECT = "/NATSBridge_file_test"
NATS_URL = "nats://nats.yiem.cc:4222"
FILESERVER_URL = "http://192.168.88.104:8080"
SIZE_THRESHOLD = 1_000_000 # 1MB
# Create correlation ID for tracing
correlation_id = str(uuid.uuid4())
def main():
# Create small binary data (will use direct transport)
small_binary = b"This is small binary data for testing direct transport."
small_binary += b"\x00" * 100 # Add some null bytes
# Create large binary data (will use link transport if > 1MB)
# Generate a larger binary (~2MB to ensure link transport)
large_binary = bytes([
(i * 7) % 256 for i in range(2_000_000)
])
# Test data 1: small binary (direct transport)
data1 = ("small_binary", small_binary, "binary")
# Test data 2: large binary (link transport)
data2 = ("large_binary", large_binary, "binary")
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
# Use smartsend with binary type
env = smartsend(
SUBJECT,
[data1, data2], # List of (dataname, data, type) tuples
nats_url=NATS_URL,
fileserver_url=FILESERVER_URL,
size_threshold=SIZE_THRESHOLD,
correlation_id=correlation_id,
msg_purpose="chat",
sender_name="file_sender",
receiver_name="",
receiver_id="",
reply_to="",
reply_to_msg_id=""
)
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
# Log transport type for each payload
for i, payload in enumerate(env.payloads):
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
log_trace(correlation_id, f" Transport: {payload.transport}")
log_trace(correlation_id, f" Type: {payload.type}")
log_trace(correlation_id, f" Size: {payload.size} bytes")
log_trace(correlation_id, f" Encoding: {payload.encoding}")
if payload.transport == "link":
log_trace(correlation_id, f" URL: {payload.data}")
print(f"Test completed. Correlation ID: {correlation_id}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,97 @@
#!/usr/bin/env python3
"""
Test script for mixed payload testing - Receiver
Tests receiving mixed payload types via NATS using nats_bridge.py smartreceive
"""
import sys
import os
import json
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartreceive, log_trace
import nats
import asyncio
# Configuration
SUBJECT = "/NATSBridge_mixed_test"
NATS_URL = "nats://nats.yiem.cc:4222"
async def main():
log_trace("", f"Starting mixed payload receiver test...")
log_trace("", f"Note: This receiver will wait for messages from the sender.")
log_trace("", f"Run test_micropython_mixed_sender.py first to send test data.")
# Connect to NATS
nc = await nats.connect(NATS_URL)
log_trace("", f"Connected to NATS at {NATS_URL}")
# Subscribe to the subject
async def message_handler(msg):
log_trace("", f"Received message on {msg.subject}")
# Use smartreceive to handle the data
result = smartreceive(msg.data)
log_trace(result.get("correlationId", ""), f"Received envelope with {len(result['payloads'])} payloads")
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for dataname, data, data_type in result["payloads"]:
log_trace(result.get("correlationId", ""), f"\n--- Payload: {dataname} (type: {data_type}) ---")
if isinstance(data, str):
log_trace(result.get("correlationId", ""), f" Type: text/string")
log_trace(result.get("correlationId", ""), f" Length: {len(data)} characters")
if len(data) <= 100:
log_trace(result.get("correlationId", ""), f" Content: {data}")
else:
log_trace(result.get("correlationId", ""), f" First 100 chars: {data[:100]}...")
# Save to file
output_path = f"./received_{dataname}.txt"
with open(output_path, 'w') as f:
f.write(data)
log_trace(result.get("correlationId", ""), f" Saved to: {output_path}")
elif isinstance(data, dict):
log_trace(result.get("correlationId", ""), f" Type: dictionary")
log_trace(result.get("correlationId", ""), f" Keys: {list(data.keys())}")
log_trace(result.get("correlationId", ""), f" Content: {json.dumps(data, indent=2)}")
# Save to file
output_path = f"./received_{dataname}.json"
with open(output_path, 'w') as f:
json.dump(data, f, indent=2)
log_trace(result.get("correlationId", ""), f" Saved to: {output_path}")
elif isinstance(data, bytes):
log_trace(result.get("correlationId", ""), f" Type: binary")
log_trace(result.get("correlationId", ""), f" Size: {len(data)} bytes")
log_trace(result.get("correlationId", ""), f" First 100 bytes (hex): {data[:100].hex()}")
# Save to file
output_path = f"./received_{dataname}.bin"
with open(output_path, 'wb') as f:
f.write(data)
log_trace(result.get("correlationId", ""), f" Saved to: {output_path}")
else:
log_trace(result.get("correlationId", ""), f" Received unexpected data type: {type(data)}")
# Log envelope metadata
log_trace(result.get("correlationId", ""), f"\n--- Envelope Metadata ---")
log_trace(result.get("correlationId", ""), f" Correlation ID: {result.get('correlationId', 'N/A')}")
log_trace(result.get("correlationId", ""), f" Message ID: {result.get('msgId', 'N/A')}")
log_trace(result.get("correlationId", ""), f" Sender: {result.get('senderName', 'N/A')}")
log_trace(result.get("correlationId", ""), f" Purpose: {result.get('msgPurpose', 'N/A')}")
sid = await nc.subscribe(SUBJECT, cb=message_handler)
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
# Keep listening for 120 seconds
await asyncio.sleep(120)
await nc.close()
log_trace("", "Test completed.")
if __name__ == "__main__":
asyncio.run(main())

View File

@@ -0,0 +1,93 @@
#!/usr/bin/env python3
"""
Test script for mixed payload testing - Micropython
Tests sending mixed payload types via NATS using nats_bridge.py smartsend
"""
import sys
import os
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartsend, log_trace
import uuid
# Configuration
SUBJECT = "/NATSBridge_mixed_test"
NATS_URL = "nats://nats.yiem.cc:4222"
FILESERVER_URL = "http://192.168.88.104:8080"
SIZE_THRESHOLD = 1_000_000 # 1MB
# Create correlation ID for tracing
correlation_id = str(uuid.uuid4())
def main():
# Create payloads for mixed content test
# 1. Small text (direct transport)
text_data = "Hello, this is a text message for testing mixed payloads!"
# 2. Small dictionary (direct transport)
dict_data = {
"status": "ok",
"code": 200,
"message": "Test successful",
"items": [1, 2, 3]
}
# 3. Small binary (direct transport)
binary_data = b"\x00\x01\x02\x03\x04\x05" + b"\xff" * 100
# 4. Large text (link transport - will use fileserver)
large_text = "\n".join([
f"Line {i}: This is a large text payload for link transport testing. " * 50
for i in range(100)
])
# Test data list - mixed payload types
data = [
("message_text", text_data, "text"),
("config_dict", dict_data, "dictionary"),
("small_binary", binary_data, "binary"),
("large_text", large_text, "text"),
]
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
# Use smartsend with mixed types
env = smartsend(
SUBJECT,
data, # List of (dataname, data, type) tuples
nats_url=NATS_URL,
fileserver_url=FILESERVER_URL,
size_threshold=SIZE_THRESHOLD,
correlation_id=correlation_id,
msg_purpose="chat",
sender_name="mixed_sender",
receiver_name="",
receiver_id="",
reply_to="",
reply_to_msg_id=""
)
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
# Log transport type for each payload
for i, payload in enumerate(env.payloads):
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
log_trace(correlation_id, f" Transport: {payload.transport}")
log_trace(correlation_id, f" Type: {payload.type}")
log_trace(correlation_id, f" Size: {payload.size} bytes")
log_trace(correlation_id, f" Encoding: {payload.encoding}")
if payload.transport == "link":
log_trace(correlation_id, f" URL: {payload.data}")
print(f"Test completed. Correlation ID: {correlation_id}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,69 @@
#!/usr/bin/env python3
"""
Test script for text transport testing - Receiver
Tests receiving text messages via NATS using nats_bridge.py smartreceive
"""
import sys
import os
import json
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartreceive, log_trace
import nats
import asyncio
# Configuration
SUBJECT = "/NATSBridge_text_test"
NATS_URL = "nats://nats.yiem.cc:4222"
async def main():
log_trace("", f"Starting text transport receiver test...")
log_trace("", f"Note: This receiver will wait for messages from the sender.")
log_trace("", f"Run test_micropython_text_sender.py first to send test data.")
# Connect to NATS
nc = await nats.connect(NATS_URL)
log_trace("", f"Connected to NATS at {NATS_URL}")
# Subscribe to the subject
async def message_handler(msg):
log_trace("", f"Received message on {msg.subject}")
# Use smartreceive to handle the data
result = smartreceive(msg.data)
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for dataname, data, data_type in result["payloads"]:
if isinstance(data, str):
log_trace(result.get("correlationId", ""), f"Received text '{dataname}' of type {data_type}")
log_trace(result.get("correlationId", ""), f" Length: {len(data)} characters")
# Display first 100 characters
if len(data) > 100:
log_trace(result.get("correlationId", ""), f" First 100 characters: {data[:100]}...")
else:
log_trace(result.get("correlationId", ""), f" Content: {data}")
# Save to file
output_path = f"./received_{dataname}.txt"
with open(output_path, 'w') as f:
f.write(data)
log_trace(result.get("correlationId", ""), f"Saved text to {output_path}")
else:
log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}")
sid = await nc.subscribe(SUBJECT, cb=message_handler)
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
# Keep listening for 120 seconds
await asyncio.sleep(120)
await nc.close()
log_trace("", "Test completed.")
if __name__ == "__main__":
asyncio.run(main())

View File

@@ -0,0 +1,81 @@
#!/usr/bin/env python3
"""
Test script for text transport testing - Micropython
Tests sending text messages via NATS using nats_bridge.py smartsend
"""
import sys
import os
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartsend, log_trace
import uuid
# Configuration
SUBJECT = "/NATSBridge_text_test"
NATS_URL = "nats://nats.yiem.cc:4222"
FILESERVER_URL = "http://192.168.88.104:8080"
SIZE_THRESHOLD = 1_000_000 # 1MB
# Create correlation ID for tracing
correlation_id = str(uuid.uuid4())
def main():
# Create a small text (will use direct transport)
small_text = "Hello, this is a small text message. Testing direct transport via NATS."
# Create a large text (will use link transport if > 1MB)
# Generate a larger text (~2MB to ensure link transport)
large_text = "\n".join([
f"Line {i}: This is a sample text line with some content to pad the size. " * 100
for i in range(500)
])
# Test data 1: small text
data1 = ("small_text", small_text, "text")
# Test data 2: large text
data2 = ("large_text", large_text, "text")
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
# Use smartsend with text type
# For small text: will use direct transport (Base64 encoded UTF-8)
# For large text: will use link transport (uploaded to fileserver)
env = smartsend(
SUBJECT,
[data1, data2], # List of (dataname, data, type) tuples
nats_url=NATS_URL,
fileserver_url=FILESERVER_URL,
size_threshold=SIZE_THRESHOLD,
correlation_id=correlation_id,
msg_purpose="chat",
sender_name="text_sender",
receiver_name="",
receiver_id="",
reply_to="",
reply_to_msg_id=""
)
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
# Log transport type for each payload
for i, payload in enumerate(env.payloads):
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
log_trace(correlation_id, f" Transport: {payload.transport}")
log_trace(correlation_id, f" Type: {payload.type}")
log_trace(correlation_id, f" Size: {payload.size} bytes")
log_trace(correlation_id, f" Encoding: {payload.encoding}")
if payload.transport == "link":
log_trace(correlation_id, f" URL: {payload.data}")
print(f"Test completed. Correlation ID: {correlation_id}")
if __name__ == "__main__":
main()

View File

@@ -1,634 +0,0 @@
# NATSBridge.jl Tutorial
A comprehensive tutorial for learning how to use NATSBridge.jl for bi-directional communication between Julia and JavaScript services using NATS.
## Table of Contents
1. [What is NATSBridge.jl?](#what-is-natsbridgejl)
2. [Key Concepts](#key-concepts)
3. [Installation](#installation)
4. [Basic Usage](#basic-usage)
5. [Payload Types](#payload-types)
6. [Transport Strategies](#transport-strategies)
7. [Advanced Features](#advanced-features)
8. [Complete Examples](#complete-examples)
---
## What is NATSBridge.jl?
NATSBridge.jl is a Julia module that provides a high-level API for sending and receiving data across network boundaries using NATS as the message bus. It implements the **Claim-Check pattern** for handling large payloads efficiently.
### Core Features
- **Bi-directional communication**: Julia ↔ JavaScript
- **Smart transport selection**: Automatic direct vs link transport based on payload size
- **Multi-payload support**: Send multiple payloads of different types in a single message
- **Claim-check pattern**: Upload large files to HTTP server, send only URLs via NATS
- **Type-aware serialization**: Different serialization strategies for different data types
---
## Key Concepts
### 1. msgEnvelope_v1 (Message Envelope)
The `msgEnvelope_v1` structure provides a comprehensive message format for bidirectional communication:
```julia
struct msgEnvelope_v1
correlationId::String # Unique identifier to track messages
msgId::String # This message id
timestamp::String # Message published timestamp
sendTo::String # Topic/subject the sender sends to
msgPurpose::String # Purpose (ACK | NACK | updateStatus | shutdown | chat)
senderName::String # Sender name (e.g., "agent-wine-web-frontend")
senderId::String # Sender id (uuid4)
receiverName::String # Message receiver name (e.g., "agent-backend")
receiverId::String # Message receiver id (uuid4 or nothing for broadcast)
replyTo::String # Topic to reply to
replyToMsgId::String # Message id this message is replying to
brokerURL::String # NATS server address
metadata::Dict{String, Any}
payloads::AbstractArray{msgPayload_v1} # Multiple payloads stored here
end
```
### 2. msgPayload_v1 (Payload Structure)
The `msgPayload_v1` structure provides flexible payload handling:
```julia
struct msgPayload_v1
id::String # Id of this payload (e.g., "uuid4")
dataname::String # Name of this payload (e.g., "login_image")
type::String # "text | dictionary | table | image | audio | video | binary"
transport::String # "direct | link"
encoding::String # "none | json | base64 | arrow-ipc"
size::Integer # Data size in bytes
data::Any # Payload data in case of direct transport or a URL in case of link
metadata::Dict{String, Any} # Dict("checksum" => "sha256_hash", ...)
end
```
### 3. Standard API Format
The system uses a **standardized list-of-tuples format** for all payload operations:
```julia
# Input format for smartsend (always a list of tuples with type info)
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
# Output format for smartreceive (always returns a list of tuples)
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
```
**Important**: Even when sending a single payload, you must wrap it in a list.
---
## Installation
```julia
using Pkg
Pkg.add("NATS")
Pkg.add("JSON")
Pkg.add("Arrow")
Pkg.add("HTTP")
Pkg.add("UUIDs")
Pkg.add("Dates")
Pkg.add("Base64")
Pkg.add("PrettyPrinting")
Pkg.add("DataFrames")
```
Then include the NATSBridge module:
```julia
include("NATSBridge.jl")
using .NATSBridge
```
---
## Basic Usage
### Sending Data (smartsend)
```julia
using NATSBridge
# Send a simple dictionary
data = Dict("key" => "value")
env = NATSBridge.smartsend("my.subject", [("dataname1", data, "dictionary")])
```
### Receiving Data (smartreceive)
```julia
using NATSBridge
# Subscribe to a NATS subject
NATS.subscribe("my.subject") do msg
# Process the message
result = NATSBridge.smartreceive(
msg,
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
# result is a list of (dataname, data, type) tuples
for (dataname, data, type) in result
println("Received $dataname of type $type")
println("Data: $data")
end
end
```
---
## Payload Types
NATSBridge.jl supports the following payload types:
| Type | Description | Serialization |
|------|-------------|---------------|
| `text` | Plain text | UTF-8 encoding |
| `dictionary` | JSON-serializable data (Dict, NamedTuple) | JSON |
| `table` | Tabular data (DataFrame, array of structs) | Apache Arrow IPC |
| `image` | Image data (Bitmap, PNG/JPG bytes) | Binary |
| `audio` | Audio data (WAV, MP3 bytes) | Binary |
| `video` | Video data (MP4, AVI bytes) | Binary |
| `binary` | Generic binary data | Binary |
---
## Transport Strategies
NATSBridge.jl automatically selects the appropriate transport strategy based on payload size:
### Direct Transport (< 1MB)
Small payloads are encoded as Base64 and sent directly over NATS.
```julia
# Small data (< 1MB) - uses direct transport
small_data = rand(1000) # ~8KB
env = NATSBridge.smartsend("small", [("data", small_data, "table")])
```
### Link Transport (≥ 1MB)
Large payloads are uploaded to an HTTP file server, and only the URL is sent via NATS.
```julia
# Large data (≥ 1MB) - uses link transport
large_data = rand(10_000_000) # ~80MB
env = NATSBridge.smartsend("large", [("data", large_data, "table")])
```
---
## Complete Examples
### Example 1: Text Message
**Sender:**
```julia
using NATSBridge
using UUIDs
const SUBJECT = "/NATSBridge_text_test"
const NATS_URL = "nats://localhost:4222"
const FILESERVER_URL = "http://localhost:8080"
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
function test_text_send()
small_text = "Hello, this is a small text message."
large_text = join(["Line $i: " for i in 1:50000], "")
data1 = ("small_text", small_text, "text")
data2 = ("large_text", large_text, "text")
env = NATSBridge.smartsend(
SUBJECT,
[data1, data2],
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,
size_threshold = 1_000_000,
correlation_id = string(uuid4()),
msg_purpose = "chat",
sender_name = "text_sender"
)
end
```
**Receiver:**
```julia
using NATSBridge
const SUBJECT = "/NATSBridge_text_test"
const NATS_URL = "nats://localhost:4222"
function test_text_receive()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
result = NATSBridge.smartreceive(
msg,
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
for (dataname, data, data_type) in result
if data_type == "text"
println("Received text: $data")
write("./received_$dataname.txt", data)
end
end
end
sleep(120)
NATS.drain(conn)
end
```
### Example 2: Dictionary (JSON) Message
**Sender:**
```julia
using NATSBridge
using UUIDs
const SUBJECT = "/NATSBridge_dict_test"
const NATS_URL = "nats://localhost:4222"
const FILESERVER_URL = "http://localhost:8080"
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
function test_dict_send()
small_dict = Dict("name" => "Alice", "age" => 30)
large_dict = Dict("ids" => collect(1:50000), "names" => ["User_$i" for i in 1:50000])
data1 = ("small_dict", small_dict, "dictionary")
data2 = ("large_dict", large_dict, "dictionary")
env = NATSBridge.smartsend(
SUBJECT,
[data1, data2],
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,
size_threshold = 1_000_000,
correlation_id = string(uuid4()),
msg_purpose = "chat"
)
end
```
**Receiver:**
```julia
using NATSBridge
const SUBJECT = "/NATSBridge_dict_test"
const NATS_URL = "nats://localhost:4222"
function test_dict_receive()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
result = NATSBridge.smartreceive(
msg,
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
for (dataname, data, data_type) in result
if data_type == "dictionary"
println("Received dictionary: $data")
write("./received_$dataname.json", JSON.json(data, 2))
end
end
end
sleep(120)
NATS.drain(conn)
end
```
### Example 3: DataFrame (Table) Message
**Sender:**
```julia
using NATSBridge
using DataFrames
using UUIDs
const SUBJECT = "/NATSBridge_table_test"
const NATS_URL = "nats://localhost:4222"
const FILESERVER_URL = "http://localhost:8080"
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
function test_table_send()
small_df = DataFrame(id = 1:10, name = ["Alice", "Bob", "Charlie"], score = [95, 88, 92])
large_df = DataFrame(id = 1:50000, name = ["User_$i" for i in 1:50000], score = rand(1:100, 50000))
data1 = ("small_table", small_df, "table")
data2 = ("large_table", large_df, "table")
env = NATSBridge.smartsend(
SUBJECT,
[data1, data2],
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,
size_threshold = 1_000_000,
correlation_id = string(uuid4()),
msg_purpose = "chat"
)
end
```
**Receiver:**
```julia
using NATSBridge
using DataFrames
const SUBJECT = "/NATSBridge_table_test"
const NATS_URL = "nats://localhost:4222"
function test_table_receive()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
result = NATSBridge.smartreceive(
msg,
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
for (dataname, data, data_type) in result
if data_type == "table"
data = DataFrame(data)
println("Received DataFrame with $(size(data, 1)) rows")
display(data[1:min(5, size(data, 1)), :])
end
end
end
sleep(120)
NATS.drain(conn)
end
```
### Example 4: Mixed Content (Chat with Text, Image, Audio)
**Sender:**
```julia
using NATSBridge
using DataFrames
using UUIDs
const SUBJECT = "/NATSBridge_mix_test"
const NATS_URL = "nats://localhost:4222"
const FILESERVER_URL = "http://localhost:8080"
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
function test_mix_send()
# Text data
text_data = "Hello! This is a test chat message. 🎉"
# Dictionary data
dict_data = Dict("type" => "chat", "sender" => "serviceA")
# Small table data
table_data_small = DataFrame(id = 1:10, name = ["msg_$i" for i in 1:10])
# Large table data (link transport)
table_data_large = DataFrame(id = 1:150_000, name = ["msg_$i" for i in 1:150_000])
# Small image data (direct transport)
image_data = UInt8[rand(1:255) for _ in 1:100]
# Large image data (link transport)
large_image_data = UInt8[rand(1:255) for _ in 1:1_500_000]
# Small audio data (direct transport)
audio_data = UInt8[rand(1:255) for _ in 1:100]
# Large audio data (link transport)
large_audio_data = UInt8[rand(1:255) for _ in 1:1_500_000]
# Small video data (direct transport)
video_data = UInt8[rand(1:255) for _ in 1:150]
# Large video data (link transport)
large_video_data = UInt8[rand(1:255) for _ in 1:1_500_000]
# Small binary data (direct transport)
binary_data = UInt8[rand(1:255) for _ in 1:200]
# Large binary data (link transport)
large_binary_data = UInt8[rand(1:255) for _ in 1:1_500_000]
# Create payloads list - mixed content
payloads = [
# Small data (direct transport)
("chat_text", text_data, "text"),
("chat_json", dict_data, "dictionary"),
("chat_table_small", table_data_small, "table"),
# Large data (link transport)
("chat_table_large", table_data_large, "table"),
("user_image_large", large_image_data, "image"),
("audio_clip_large", large_audio_data, "audio"),
("video_clip_large", large_video_data, "video"),
("binary_file_large", large_binary_data, "binary")
]
env = NATSBridge.smartsend(
SUBJECT,
payloads,
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,
size_threshold = 1_000_000,
correlation_id = string(uuid4()),
msg_purpose = "chat",
sender_name = "mix_sender"
)
end
```
**Receiver:**
```julia
using NATSBridge
using DataFrames
const SUBJECT = "/NATSBridge_mix_test"
const NATS_URL = "nats://localhost:4222"
function test_mix_receive()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
result = NATSBridge.smartreceive(
msg,
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
println("Received $(length(result)) payloads")
for (dataname, data, data_type) in result
println("\n=== Payload: $dataname (type: $data_type) ===")
if data_type == "text"
println(" Type: String")
println(" Length: $(length(data)) characters")
elseif data_type == "dictionary"
println(" Type: JSON Object")
println(" Keys: $(keys(data))")
elseif data_type == "table"
data = DataFrame(data)
println(" Type: DataFrame")
println(" Dimensions: $(size(data, 1)) rows x $(size(data, 2)) columns")
elseif data_type == "image"
println(" Type: Vector{UInt8}")
println(" Size: $(length(data)) bytes")
write("./received_$dataname.bin", data)
elseif data_type == "audio"
println(" Type: Vector{UInt8}")
println(" Size: $(length(data)) bytes")
write("./received_$dataname.bin", data)
elseif data_type == "video"
println(" Type: Vector{UInt8}")
println(" Size: $(length(data)) bytes")
write("./received_$dataname.bin", data)
elseif data_type == "binary"
println(" Type: Vector{UInt8}")
println(" Size: $(length(data)) bytes")
write("./received_$dataname.bin", data)
end
end
end
sleep(120)
NATS.drain(conn)
end
```
---
## Best Practices
1. **Always wrap payloads in a list** - Even for single payloads: `[("dataname", data, "type")]`
2. **Use appropriate transport** - Let NATSBridge handle size-based routing (default 1MB threshold)
3. **Customize size threshold** - Use `size_threshold` parameter to adjust the direct/link split
4. **Provide fileserver handler** - Implement `fileserverUploadHandler` for link transport
5. **Include correlation IDs** - Track messages across distributed systems
6. **Handle errors** - Implement proper error handling for network failures
7. **Close connections** - Ensure NATS connections are properly closed using `NATS.drain()`
---
## Conclusion
NATSBridge.jl provides a powerful abstraction for bi-directional communication between Julia and JavaScript services. By understanding the key concepts and following the best practices, you can build robust, scalable applications that leverage the full power of NATS messaging.
For more information, see:
- [`docs/architecture.md`](./architecture.md) - Detailed architecture documentation
- [`docs/implementation.md`](./implementation.md) - Implementation details

View File

@@ -1,939 +0,0 @@
# NATSBridge.jl Walkthrough: Building a Chat System
A step-by-step guided walkthrough for building a real-time chat system using NATSBridge.jl with mixed content support (text, images, audio, video, and files).
## Prerequisites
- Julia 1.7+
- NATS server running
- HTTP file server (Plik) running
## Step 1: Understanding the Chat System Architecture
### System Components
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ Chat System │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ NATS ┌──────────────┐ │
│ │ Julia │◄───────┬───────► │ JavaScript │ │
│ │ Service │ │ │ Client │ │
│ │ │ │ │ │ │
│ │ - Text │ │ │ - Text │ │
│ │ - Images │ │ │ - Images │ │
│ │ - Audio │ ▼ │ - Audio │ │
│ │ - Video │ NATSBridge.jl │ - Files │ │
│ │ - Files │ │ │ - Tables │ │
│ └──────────────┘ │ └──────────────┘ │
│ │ │
│ ┌───────┴───────┐ │
│ │ NATS │ │
│ │ Server │ │
│ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
For large payloads (> 1MB):
┌─────────────────────────────────────────────────────────────────────────────┐
│ File Server (Plik) │
│ │
│ Julia Service ──► Upload ──► File Server ──► Download ◄── JavaScript Client│
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
### Message Format
Each chat message is an envelope containing multiple payloads:
```json
{
"correlationId": "uuid4",
"msgId": "uuid4",
"timestamp": "2024-01-15T10:30:00Z",
"sendTo": "/chat/room1",
"msgPurpose": "chat",
"senderName": "user-1",
"senderId": "uuid4",
"receiverName": "user-2",
"receiverId": "uuid4",
"brokerURL": "nats://localhost:4222",
"payloads": [
{
"id": "uuid4",
"dataname": "message_text",
"type": "text",
"transport": "direct",
"encoding": "base64",
"size": 256,
"data": "SGVsbG8gV29ybGQh",
"metadata": {}
},
{
"id": "uuid4",
"dataname": "user_image",
"type": "image",
"transport": "link",
"encoding": "none",
"size": 15433,
"data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/image.jpg",
"metadata": {}
}
]
}
```
## Step 2: Setting Up the Environment
### 1. Start NATS Server
```bash
# Using Docker
docker run -d -p 4222:4222 -p 8222:8222 --name nats-server nats:latest
# Or download from https://github.com/nats-io/nats-server/releases
./nats-server
```
### 2. Start HTTP File Server (Plik)
```bash
# Using Docker
docker run -d -p 8080:8080 --name plik plik/plik:latest
# Or download from https://github.com/arnaud-lb/plik/releases
./plikd -d
```
### 3. Install Julia Dependencies
```julia
using Pkg
Pkg.add("NATS")
Pkg.add("JSON")
Pkg.add("Arrow")
Pkg.add("HTTP")
Pkg.add("UUIDs")
Pkg.add("Dates")
Pkg.add("Base64")
Pkg.add("PrettyPrinting")
Pkg.add("DataFrames")
```
## Step 3: Basic Text-Only Chat
### Sender (User 1)
```julia
using NATS
using JSON
using UUIDs
using Dates
using PrettyPrinting
using DataFrames
using Arrow
using HTTP
using Base64
# Include the bridge module
include("NATSBridge.jl")
using .NATSBridge
# Configuration
const NATS_URL = "nats://localhost:4222"
const FILESERVER_URL = "http://localhost:8080"
const SUBJECT = "/chat/room1"
# File upload handler for plik server
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
# Send a simple text message
function send_text_message()
message_text = "Hello, how are you today?"
env = NATSBridge.smartsend(
SUBJECT,
[("message", message_text, "text")],
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,
size_threshold = 1_000_000,
correlation_id = string(uuid4()),
msg_purpose = "chat",
sender_name = "user-1"
)
println("Sent text message with correlation ID: $(env.correlationId)")
end
send_text_message()
```
### Receiver (User 2)
```julia
using NATS
using JSON
using UUIDs
using Dates
using PrettyPrinting
using DataFrames
using Arrow
using HTTP
using Base64
# Include the bridge module
include("NATSBridge.jl")
using .NATSBridge
# Configuration
const NATS_URL = "nats://localhost:4222"
const SUBJECT = "/chat/room1"
# Message handler
function message_handler(msg::NATS.Msg)
payloads = NATSBridge.smartreceive(
msg,
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
# Extract the text message
for (dataname, data, data_type) in payloads
if data_type == "text"
println("Received message: $data")
# Save to file
write("./received_$dataname.txt", data)
end
end
end
# Subscribe to the chat room
NATS.subscribe(SUBJECT) do msg
message_handler(msg)
end
# Keep the program running
while true
sleep(1)
end
```
## Step 4: Adding Image Support
### Sending an Image
```julia
using NATS
using JSON
using UUIDs
using Dates
using PrettyPrinting
using DataFrames
using Arrow
using HTTP
using Base64
include("NATSBridge.jl")
using .NATSBridge
const NATS_URL = "nats://localhost:4222"
const FILESERVER_URL = "http://localhost:8080"
const SUBJECT = "/chat/room1"
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
function send_image()
# Read image file
image_data = read("screenshot.png", Vector{UInt8})
# Send with text message
env = NATSBridge.smartsend(
SUBJECT,
[
("text", "Check out this screenshot!", "text"),
("screenshot", image_data, "image")
],
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,
size_threshold = 1_000_000,
correlation_id = string(uuid4()),
msg_purpose = "chat",
sender_name = "user-1"
)
println("Sent image with correlation ID: $(env.correlationId)")
end
send_image()
```
### Receiving an Image
```julia
using NATS
using JSON
using UUIDs
using Dates
using PrettyPrinting
using DataFrames
using Arrow
using HTTP
using Base64
include("NATSBridge.jl")
using .NATSBridge
const NATS_URL = "nats://localhost:4222"
const SUBJECT = "/chat/room1"
function message_handler(msg::NATS.Msg)
payloads = NATSBridge.smartreceive(
msg,
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
for (dataname, data, data_type) in payloads
if data_type == "text"
println("Text: $data")
elseif data_type == "image"
# Save image to file
filename = "received_$dataname.bin"
write(filename, data)
println("Saved image: $filename")
end
end
end
NATS.subscribe(SUBJECT) do msg
message_handler(msg)
end
```
## Step 5: Handling Large Files with Link Transport
### Automatic Transport Selection
```julia
using NATS
using JSON
using UUIDs
using Dates
using PrettyPrinting
using DataFrames
using Arrow
using HTTP
using Base64
include("NATSBridge.jl")
using .NATSBridge
const NATS_URL = "nats://localhost:4222"
const FILESERVER_URL = "http://localhost:8080"
const SUBJECT = "/chat/room1"
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
function send_large_file()
# Create a large file (> 1MB triggers link transport)
large_data = rand(10_000_000) # ~80MB
env = NATSBridge.smartsend(
SUBJECT,
[("large_file", large_data, "binary")],
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,
size_threshold = 1_000_000,
correlation_id = string(uuid4()),
msg_purpose = "chat",
sender_name = "user-1"
)
println("Uploaded large file to: $(env.payloads[1].data)")
println("Correlation ID: $(env.correlationId)")
end
send_large_file()
```
## Step 6: Audio and Video Support
### Sending Audio
```julia
using NATS
using JSON
using UUIDs
using Dates
using PrettyPrinting
using DataFrames
using Arrow
using HTTP
using Base64
include("NATSBridge.jl")
using .NATSBridge
const NATS_URL = "nats://localhost:4222"
const FILESERVER_URL = "http://localhost:8080"
const SUBJECT = "/chat/room1"
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
function send_audio()
# Read audio file (WAV, MP3, etc.)
audio_data = read("voice_message.mp3", Vector{UInt8})
env = NATSBridge.smartsend(
SUBJECT,
[("voice_message", audio_data, "audio")],
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,
size_threshold = 1_000_000,
correlation_id = string(uuid4()),
msg_purpose = "chat",
sender_name = "user-1"
)
println("Sent audio message: $(env.correlationId)")
end
send_audio()
```
### Sending Video
```julia
using NATS
using JSON
using UUIDs
using Dates
using PrettyPrinting
using DataFrames
using Arrow
using HTTP
using Base64
include("NATSBridge.jl")
using .NATSBridge
const NATS_URL = "nats://localhost:4222"
const FILESERVER_URL = "http://localhost:8080"
const SUBJECT = "/chat/room1"
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
function send_video()
# Read video file (MP4, AVI, etc.)
video_data = read("video_message.mp4", Vector{UInt8})
env = NATSBridge.smartsend(
SUBJECT,
[("video_message", video_data, "video")],
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,
size_threshold = 1_000_000,
correlation_id = string(uuid4()),
msg_purpose = "chat",
sender_name = "user-1"
)
println("Sent video message: $(env.correlationId)")
end
send_video()
```
## Step 7: Table/Data Exchange
### Sending Tabular Data
```julia
using NATS
using JSON
using UUIDs
using Dates
using PrettyPrinting
using DataFrames
using Arrow
using HTTP
using Base64
include("NATSBridge.jl")
using .NATSBridge
const NATS_URL = "nats://localhost:4222"
const FILESERVER_URL = "http://localhost:8080"
const SUBJECT = "/chat/room1"
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
function send_table()
# Create a DataFrame
df = DataFrame(
id = 1:5,
name = ["Alice", "Bob", "Charlie", "Diana", "Eve"],
score = [95, 88, 92, 98, 85],
grade = ['A', 'B', 'A', 'B', 'B']
)
env = NATSBridge.smartsend(
SUBJECT,
[("student_scores", df, "table")],
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,
size_threshold = 1_000_000,
correlation_id = string(uuid4()),
msg_purpose = "chat",
sender_name = "user-1"
)
println("Sent table with $(nrow(df)) rows")
end
send_table()
```
### Receiving and Using Tables
```julia
using NATS
using JSON
using UUIDs
using Dates
using PrettyPrinting
using DataFrames
using Arrow
using HTTP
using Base64
include("NATSBridge.jl")
using .NATSBridge
const NATS_URL = "nats://localhost:4222"
const SUBJECT = "/chat/room1"
function message_handler(msg::NATS.Msg)
payloads = NATSBridge.smartreceive(
msg,
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
for (dataname, data, data_type) in payloads
if data_type == "table"
data = DataFrame(data)
println("Received table:")
show(data)
println("\nAverage score: $(mean(data.score))")
end
end
end
NATS.subscribe(SUBJECT) do msg
message_handler(msg)
end
```
## Step 8: Bidirectional Communication
### Request-Response Pattern
```julia
using NATS
using JSON
using UUIDs
using Dates
using PrettyPrinting
using DataFrames
using Arrow
using HTTP
using Base64
include("NATSBridge.jl")
using .NATSBridge
const NATS_URL = "nats://localhost:4222"
const SUBJECT = "/api/query"
const REPLY_SUBJECT = "/api/response"
# Request
function send_request()
query_data = Dict("query" => "SELECT * FROM users")
env = NATSBridge.smartsend(
SUBJECT,
[("sql_query", query_data, "dictionary")],
nats_url = NATS_URL,
fileserver_url = "http://localhost:8080",
fileserverUploadHandler = plik_upload_handler,
size_threshold = 1_000_000,
correlation_id = string(uuid4()),
msg_purpose = "request",
sender_name = "frontend",
receiver_name = "backend",
reply_to = REPLY_SUBJECT,
reply_to_msg_id = string(uuid4())
)
println("Request sent: $(env.correlationId)")
end
# Response handler
function response_handler(msg::NATS.Msg)
payloads = NATSBridge.smartreceive(
msg,
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
for (dataname, data, data_type) in payloads
if data_type == "table"
data = DataFrame(data)
println("Query results:")
show(data)
end
end
end
NATS.subscribe(REPLY_SUBJECT) do msg
response_handler(msg)
end
```
## Step 9: Complete Chat Application
### Full Chat System
```julia
module ChatApp
using NATS
using JSON
using UUIDs
using Dates
using PrettyPrinting
using DataFrames
using Arrow
using HTTP
using Base64
# Include the bridge module
include("../src/NATSBridge.jl")
using .NATSBridge
# Configuration
const NATS_URL = "nats://localhost:4222"
const FILESERVER_URL = "http://localhost:8080"
const SUBJECT = "/chat/room1"
# File upload handler for plik server
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
function send_chat_message(
text::String,
image_path::Union{String, Nothing}=nothing,
audio_path::Union{String, Nothing}=nothing
)
# Build payloads list
payloads = [("message_text", text, "text")]
if image_path !== nothing
image_data = read(image_path, Vector{UInt8})
push!(payloads, ("user_image", image_data, "image"))
end
if audio_path !== nothing
audio_data = read(audio_path, Vector{UInt8})
push!(payloads, ("user_audio", audio_data, "audio"))
end
env = NATSBridge.smartsend(
SUBJECT,
payloads,
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserverUploadHandler = plik_upload_handler,
size_threshold = 1_000_000,
correlation_id = string(uuid4()),
msg_purpose = "chat",
sender_name = "user-1"
)
println("Message sent with correlation ID: $(env.correlationId)")
end
function receive_chat_messages()
function message_handler(msg::NATS.Msg)
payloads = NATSBridge.smartreceive(
msg,
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
println("\n--- New Message ---")
for (dataname, data, data_type) in payloads
if data_type == "text"
println("Text: $data")
elseif data_type == "image"
filename = "received_$dataname.bin"
write(filename, data)
println("Image saved: $filename")
elseif data_type == "audio"
filename = "received_$dataname.bin"
write(filename, data)
println("Audio saved: $filename")
elseif data_type == "table"
println("Table received:")
data = DataFrame(data)
show(data)
end
end
end
NATS.subscribe(SUBJECT) do msg
message_handler(msg)
end
println("Subscribed to: $SUBJECT")
end
function run_interactive_chat()
println("\n=== Interactive Chat ===")
println("1. Send a message")
println("2. Join a chat room")
println("3. Exit")
while true
print("\nSelect option (1-3): ")
choice = readline()
if choice == "1"
print("Enter message text: ")
text = readline()
send_chat_message(text)
elseif choice == "2"
receive_chat_messages()
elseif choice == "3"
break
end
end
end
end # module
# Run the chat app
using .ChatApp
ChatApp.run_interactive_chat()
```
## Step 10: Testing the Chat System
### Test Scenario 1: Text-Only Chat
```bash
# Terminal 1: Start the chat receiver
julia test_julia_to_julia_text_receiver.jl
# Terminal 2: Send a message
julia test_julia_to_julia_text_sender.jl
```
### Test Scenario 2: Image Chat
```bash
# Terminal 1: Receive messages
julia test_julia_to_julia_mix_payloads_receiver.jl
# Terminal 2: Send image
julia test_julia_to_julia_mix_payload_sender.jl
```
### Test Scenario 3: Large File Transfer
```bash
# Terminal 2: Send large file
julia test_julia_to_julia_mix_payload_sender.jl
```
## Conclusion
This walkthrough demonstrated how to build a chat system using NATSBridge.jl with support for:
- Text messages
- Images (direct transport for small, link transport for large)
- Audio files
- Video files
- Tabular data (DataFrames)
- Bidirectional communication
- Mixed-content messages
The key takeaways are:
1. **Always wrap payloads in a list** - Even for single payloads: `[("dataname", data, "type")]`
2. **Use appropriate transport** - NATSBridge automatically handles size-based routing
3. **Support mixed content** - Multiple payloads of different types in one message
4. **Handle errors** - Implement proper error handling for network failures
5. **Use correlation IDs** - Track messages across distributed systems
For more information, see:
- [`docs/architecture.md`](./docs/architecture.md) - Detailed architecture documentation
- [`docs/implementation.md`](./docs/implementation.md) - Implementation details