ton/NATSBridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

update

Browse Source
This commit is contained in:
narawat
2026-02-25 20:27:51 +07:00
parent bee9f783d9
commit f8d93991f5
24 changed files with 579 additions and 4981 deletions
Download Patch File Download Diff File Expand all files Collapse all files

550
README.md
View File

@@ -1,6 +1,6 @@
# NATSBridge
A high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
A high-performance, bi-directional data bridge for **Julia** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![NATS](https://img.shields.io/badge/NATS-Enabled-green.svg)](https://nats.io)
@@ -25,7 +25,7 @@ A high-performance, bi-directional data bridge between **Julia**, **JavaScript**
## Overview
NATSBridge enables seamless communication across Julia, JavaScript, and Python/Micropython applications through NATS, with intelligent transport selection based on payload size:
NATSBridge enables seamless communication for Julia applications through NATS, with intelligent transport selection based on payload size:
| Transport | Payload Size | Method |
|-----------|--------------|--------|
@@ -37,14 +37,13 @@ NATSBridge enables seamless communication across Julia, JavaScript, and Python/M
- **Chat Applications**: Text, images, audio, video in a single message
- **File Transfer**: Efficient transfer of large files using claim-check pattern
- **Streaming Data**: Sensor data, telemetry, and analytics pipelines
- **Cross-Platform Communication**: Julia ↔ JavaScript ↔ Python/Micropython
- **IoT Devices**: Micropython devices sending data to cloud services
---
## Features
- ✅ **Bi-directional messaging** between Julia, JavaScript, and Python/Micropython
- ✅ **Bi-directional messaging** for Julia applications
- ✅ **Multi-payload support** - send multiple payloads with different types in one message
- ✅ **Automatic transport selection** - direct vs link based on payload size
- ✅ **Claim-Check pattern** for payloads > 1MB
@@ -53,7 +52,7 @@ NATSBridge enables seamless communication across Julia, JavaScript, and Python/M
- ✅ **Correlation ID tracking** for message tracing
- ✅ **Reply-to support** for request-response patterns
- ✅ **JetStream support** for message replay and durability
- ✅ **Lightweight Micropython implementation** for microcontrollers
---
@@ -65,16 +64,12 @@ NATSBridge enables seamless communication across Julia, JavaScript, and Python/M
┌─────────────────────────────────────────────────────────────────────┐
│ NATSBridge Architecture │
├─────────────────────────────────────────────────────────────────────┤
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Julia │ │ JavaScript │ │ Python/ │ │
│ │ (NATS.jl) │◄──►│ (nats.js) │◄──►│ Micropython │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ NATS │ │
│ ┌──────────────┐ │ │
│ │ Julia │ ▼ │
│ │ (NATS.jl) │ ┌─────────────────────────┐ │
│ └──────────────┘ │ NATS │ │
│ │ (Message Broker) │ │
│ └─────────────────────────────────────────────────────┘ │
│ └─────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────┐ │
@@ -110,40 +105,6 @@ Pkg.add("NATS")
Pkg.add("https://git.yiem.cc/ton/NATSBridge")
```
### JavaScript
```bash
npm install nats.js apache-arrow uuid base64-url
```
For Node.js:
```javascript
const { smartsend, smartreceive } = require('./src/NATSBridge');
```
For browser:
```html
<script src="./src/NATSBridge.js"></script>
<script>
// NATSBridge is available as window.NATSBridge
</script>
```
### Python/Micropython
1. Copy [`src/nats_bridge.py`](src/nats_bridge.py) to your device
2. Install dependencies:
**For Python (desktop):**
```bash
pip install nats-py
```
**For Micropython:**
- `urequests` for HTTP requests (built-in for ESP32)
- `base64` for base64 encoding (built-in)
- `json` for JSON handling (built-in)
---
## Quick Start
@@ -160,36 +121,12 @@ docker run -p 4222:4222 nats:latest
# Create a directory for file uploads
mkdir -p /tmp/fileserver
# Use Python's built-in server
# Start HTTP file server
python3 -m http.server 8080 --directory /tmp/fileserver
```
### Step 3: Send Your First Message
#### Python/Micropython
```python
from nats_bridge import smartsend
# Send a text message
data = [("message", "Hello World", "text")]
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222")
print("Message sent!")
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
// Send a text message
const { env, env_json_str } = await smartsend("/chat/room1", [
{ dataname: "message", data: "Hello World", type: "text" }
], { broker_url: "nats://localhost:4222" });
console.log("Message sent!");
```
#### Julia
```julia
@@ -203,64 +140,6 @@ println("Message sent!")
### Step 4: Receive Messages
#### Python/Micropython
```python
import nats
import asyncio
from nats_bridge import smartreceive
# Configuration
SUBJECT = "/chat/room1"
NATS_URL = "nats://localhost:4222"
async def main():
# Connect to NATS
nc = await nats.connect(NATS_URL)
# Subscribe to the subject - msg comes from the callback
async def message_handler(msg):
# Receive and process message
env = smartreceive(msg.data)
for dataname, data, type in env["payloads"]:
print(f"Received {dataname}: {data}")
sid = await nc.subscribe(SUBJECT, cb=message_handler)
await asyncio.sleep(120) # Keep listening
await nc.close()
asyncio.run(main())
```
#### JavaScript
```javascript
const { smartreceive } = require('./src/NATSBridge');
const { connect } = require('nats');
// Configuration
const SUBJECT = "/chat/room1";
const NATS_URL = "nats://localhost:4222";
async function main() {
// Connect to NATS
const nc = await connect({ servers: [NATS_URL] });
// Subscribe to the subject - msg comes from the async iteration
const sub = nc.subscribe(SUBJECT);
for await (const msg of sub) {
// Receive and process message
const env = await smartreceive(msg);
for (const payload of env.payloads) {
console.log(`Received ${payload.dataname}: ${payload.data}`);
}
}
}
main();
```
#### Julia
```julia
@@ -305,54 +184,6 @@ test_receive()
Sends data either directly via NATS or via a fileserver URL, depending on payload size.
#### Python/Micropython
```python
from nats_bridge import smartsend
env, env_json_str = smartsend(
subject, # NATS subject to publish to
data, # List of (dataname, data, type) tuples
broker_url="nats://localhost:4222", # NATS server URL
fileserver_url="http://localhost:8080", # File server URL
fileserver_upload_handler=plik_oneshot_upload, # Upload handler function
size_threshold=1_000_000, # Threshold in bytes (default: 1MB)
correlation_id=None, # Optional correlation ID for tracing
msg_purpose="chat", # Message purpose
sender_name="NATSBridge", # Sender name
receiver_name="", # Receiver name (empty = broadcast)
receiver_id="", # Receiver UUID (empty = broadcast)
reply_to="", # Reply topic
reply_to_msg_id="", # Reply message ID
is_publish=True # Whether to automatically publish to NATS
)
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
const { env, env_json_str } = await smartsend(
subject, // NATS subject
data, // Array of {dataname, data, type}
{
broker_url: "nats://localhost:4222",
fileserver_url: "http://localhost:8080",
fileserver_upload_handler: customUploadHandler,
size_threshold: 1_000_000,
correlation_id: "custom-id",
msg_purpose: "chat",
sender_name: "NATSBridge",
receiver_name: "",
receiver_id: "",
reply_to: "",
reply_to_msg_id: "",
is_publish: true // Whether to automatically publish to NATS
}
);
```
#### Julia
```julia
@@ -384,40 +215,6 @@ env, env_json_str = NATSBridge.smartsend(
Receives and processes messages from NATS, handling both direct and link transport.
#### Python/Micropython
```python
from nats_bridge import smartreceive
# Note: For nats-py, use msg.data to pass the raw message data
env = smartreceive(
msg.data, # NATS message data (msg.data for nats-py)
fileserver_download_handler=_fetch_with_backoff, # Download handler
max_retries=5, # Max retry attempts
base_delay=100, # Initial delay in ms
max_delay=5000 # Max delay in ms
)
# Returns: Dict with envelope metadata and 'payloads' field
```
#### JavaScript
```javascript
const { smartreceive } = require('./src/NATSBridge');
// Note: msg is the NATS message object from subscription
const env = await smartreceive(
msg, // NATS message (raw object from subscription)
{
fileserverDownloadHandler: customDownloadHandler,
maxRetries: 5,
baseDelay: 100,
maxDelay: 5000
}
);
// Returns: Object with envelope metadata and payloads array
```
#### Julia
```julia
@@ -485,19 +282,6 @@ NATSBridge.publish_message(conn, "/chat/room1", "{\"correlation_id\":\"abc123\"}
Small payloads are sent directly via NATS with Base64 encoding.
#### Python/Micropython
```python
data = [("message", "Hello", "text")]
smartsend("/topic", data)
```
#### JavaScript
```javascript
await smartsend("/topic", [
{ dataname: "message", data: "Hello", type: "text" }
]);
```
#### Julia
```julia
data = [("message", "Hello", "text")]
@@ -508,19 +292,6 @@ smartsend("/topic", data)
Large payloads are uploaded to an HTTP file server.
#### Python/Micropython
```python
data = [("file", large_data, "binary")]
smartsend("/topic", data, fileserver_url="http://localhost:8080")
```
#### JavaScript
```javascript
await smartsend("/topic", [
{ dataname: "file", data: largeData, type: "binary" }
], { fileserver_url: "http://localhost:8080" });
```
#### Julia
```julia
data = [("file", large_data, "binary")]
@@ -531,38 +302,10 @@ smartsend("/topic", data; fileserver_url="http://localhost:8080")
## Examples
All examples include code for **Julia**, **JavaScript**, and **Python/Micropython** unless otherwise specified.
### Example 1: Chat with Mixed Content
Send text, small image, and large file in one message.
#### Python/Micropython
```python
from nats_bridge import smartsend
data = [
("message_text", "Hello!", "text"),
("user_avatar", image_data, "image"),
("large_document", large_file_data, "binary")
]
env, env_json_str = smartsend("/chat/room1", data, fileserver_url="http://localhost:8080")
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
const { env, env_json_str } = await smartsend("/chat/room1", [
{ dataname: "message_text", data: "Hello!", type: "text" },
{ dataname: "user_avatar", data: image_data, type: "image" },
{ dataname: "large_document", data: large_file_data, type: "binary" }
], {
fileserver_url: "http://localhost:8080"
});
```
#### Julia
```julia
using NATSBridge
@@ -580,35 +323,6 @@ env, env_json_str = NATSBridge.smartsend("/chat/room1", data; fileserver_url="ht
Send configuration data between platforms.
#### Python/Micropython
```python
from nats_bridge import smartsend
config = {
"wifi_ssid": "MyNetwork",
"wifi_password": "password123",
"update_interval": 60
}
data = [("config", config, "dictionary")]
env, env_json_str = smartsend("/device/config", data)
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
const config = {
wifi_ssid: "MyNetwork",
wifi_password: "password123",
update_interval: 60
};
const { env, env_json_str } = await smartsend("/device/config", [
{ dataname: "config", data: config, type: "dictionary" }
]);
```
#### Julia
```julia
using NATSBridge
@@ -627,36 +341,6 @@ env, env_json_str = NATSBridge.smartsend("/device/config", data)
Send tabular data using Apache Arrow IPC format.
#### Python/Micropython
```python
import pandas as pd
from nats_bridge import smartsend
df = pd.DataFrame({
"id": [1, 2, 3],
"name": ["Alice", "Bob", "Charlie"],
"score": [95, 88, 92]
})
data = [("students", df, "table")]
env, env_json_str = smartsend("/data/analysis", data)
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
const tableData = [
{ id: 1, name: "Alice", score: 95 },
{ id: 2, name: "Bob", score: 88 },
{ id: 3, name: "Charlie", score: 92 }
];
const { env, env_json_str } = await smartsend("/data/analysis", [
{ dataname: "students", data: tableData, type: "table" }
]);
```
#### Julia
```julia
using NATSBridge
@@ -676,106 +360,6 @@ env, env_json_str = NATSBridge.smartsend("/data/analysis", data)
Bi-directional communication with reply-to support. The `smartsend` function now returns both the envelope object and a JSON string that can be published directly.
#### Python/Micropython (Requester)
```python
from nats_bridge import smartsend
env, env_json_str = smartsend(
"/device/command",
[("command", {"action": "read_sensor"}, "dictionary")],
broker_url="nats://localhost:4222",
reply_to="/device/response"
)
# env: MessageEnvelope object
# env_json_str: JSON string for publishing to NATS
# The env_json_str can also be published directly using NATS request-reply pattern
# nc.request("/device/command", env_json_str, reply_to="/device/response")
```
#### Python/Micropython (Responder)
```python
import nats
import asyncio
from nats_bridge import smartreceive, smartsend
# Configuration
SUBJECT = "/device/command"
NATS_URL = "nats://localhost:4222"
async def main():
nc = await nats.connect(NATS_URL)
async def message_handler(msg):
# Receive and parse the incoming message envelope
env = smartreceive(msg.data)
# Extract reply_to from the envelope metadata
reply_to = env["reply_to"]
for dataname, data, type in env["payloads"]:
if data.get("action") == "read_sensor":
response = {"sensor_id": "sensor-001", "value": 42.5}
# Send response to the reply_to subject from the request
if reply_to:
smartsend(reply_to, [("data", response, "dictionary")])
sid = await nc.subscribe(SUBJECT, cb=message_handler)
await asyncio.sleep(120)
await nc.close()
asyncio.run(main())
```
#### JavaScript (Requester)
```javascript
const { smartsend } = require('./src/NATSBridge');
const { env, env_json_str } = await smartsend("/device/command", [
{ dataname: "command", data: { action: "read_sensor" }, type: "dictionary" }
], {
broker_url: "nats://localhost:4222",
reply_to: "/device/response"
});
```
#### JavaScript (Responder)
```javascript
const { smartreceive, smartsend } = require('./src/NATSBridge');
const { connect } = require('nats');
// Configuration
const SUBJECT = "/device/command";
const NATS_URL = "nats://localhost:4222";
async function main() {
const nc = await connect({ servers: [NATS_URL] });
const sub = nc.subscribe(SUBJECT);
for await (const msg of sub) {
const env = await smartreceive(msg);
// Extract reply_to from the envelope metadata
const replyTo = env["reply_to"];
for (const payload of env.payloads) {
if (payload.dataname === "command" && payload.data.action === "read_sensor") {
const response = { sensor_id: "sensor-001", value: 42.5 };
// Send response to the reply_to subject from the request
if (replyTo) {
await smartsend(replyTo, [
{ dataname: "data", data: response, type: "dictionary" }
]);
}
}
}
}
}
main();
```
#### Julia (Requester)
```julia
using NATSBridge
@@ -822,70 +406,9 @@ end
test_responder()
```
### Example 5: Micropython IoT Device
### Example 5: IoT Device Sensor Data
Lightweight Micropython device sending sensor data.
#### Micropython
```python
import nats
import asyncio
from nats_bridge import smartsend, smartreceive
# Configuration
SUBJECT = "/device/sensors"
NATS_URL = "nats://localhost:4222"
async def main():
nc = await nats.connect(NATS_URL)
# Send sensor data
data = [("temperature", "25.5", "text"), ("humidity", 65, "dictionary")]
smartsend("/device/sensors", data, broker_url="nats://localhost:4222")
# Receive commands - msg comes from the callback
async def message_handler(msg):
env = smartreceive(msg.data)
for dataname, data, type in env["payloads"]:
if type == "dictionary" and data.get("action") == "reboot":
# Execute reboot
pass
sid = await nc.subscribe(SUBJECT, cb=message_handler)
await asyncio.sleep(120)
await nc.close()
asyncio.run(main())
```
#### JavaScript (Receiver)
```javascript
const { smartreceive } = require('./src/NATSBridge');
const { connect } = require('nats');
// Configuration
const SUBJECT = "/device/sensors";
const NATS_URL = "nats://localhost:4222";
async function main() {
const nc = await connect({ servers: [NATS_URL] });
const sub = nc.subscribe(SUBJECT);
for await (const msg of sub) {
const env = await smartreceive(msg);
for (const payload of env.payloads) {
if (payload.dataname === "temperature") {
console.log(`Temperature: ${payload.data}`);
} else if (payload.dataname === "humidity") {
console.log(`Humidity: ${payload.data}`);
}
}
}
}
main();
```
IoT device sending sensor data.
#### Julia (Receiver)
```julia
@@ -921,53 +444,6 @@ test_receiver()
Run the test scripts to verify functionality:
### Python/Micropython
```bash
# Basic functionality test
python test/test_micropython_basic.py
# Text message exchange
python test/test_micropython_text_sender.py
python test/test_micropython_text_receiver.py
# Dictionary exchange
python test/test_micropython_dict_sender.py
python test/test_micropython_dict_receiver.py
# File transfer
python test/test_micropython_file_sender.py
python test/test_micropython_file_receiver.py
# Mixed payload types
python test/test_micropython_mixed_sender.py
python test/test_micropython_mixed_receiver.py
```
### JavaScript
```bash
# Text message exchange
node test/test_js_text_sender.js
node test/test_js_text_receiver.js
# Dictionary exchange
node test/test_js_dict_sender.js
node test/test_js_dict_receiver.js
# File transfer
node test/test_js_file_sender.js
node test/test_js_file_receiver.js
# Mixed payload types
node test/test_js_mix_payload_sender.js
node test/test_js_mix_payloads_receiver.js
# Table exchange
node test/test_js_table_sender.js
node test/test_js_table_receiver.js
```
### Julia
```julia

436
docs/architecture.md
View File

@@ -2,12 +2,10 @@
## Overview
This document describes the architecture for a high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
This document describes the architecture for a high-performance, bi-directional data bridge for **Julia** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
The system enables seamless communication across all three platforms:
- **Julia ↔ JavaScript** bi-directional messaging
- **JavaScript ↔ Python/Micropython** bi-directional messaging
- **Julia ↔ Python/Micropython** bi-directional messaging (via JSON serialization)
The system enables seamless communication for Julia applications:
- **Julia** messaging with NATS
### File Server Handler Architecture
@@ -113,8 +111,7 @@ env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff, max_ret
```mermaid
flowchart TD
subgraph Client
JS[JavaScript Client]
JSApp[Application Logic]
App[Julia Application]
end
subgraph Server
@@ -123,14 +120,12 @@ flowchart TD
FileServer[HTTP File Server]
end
JS -->|Control/Small Data| JSApp
JSApp -->|NATS| NATS
App -->|NATS| NATS
NATS -->|NATS| Julia
Julia -->|NATS| NATS
Julia -->|HTTP POST| FileServer
JS -->|HTTP GET| FileServer
style JS fill:#e1f5fe
style App fill:#e8f5e9
style Julia fill:#e8f5e9
style NATS fill:#fff3e0
style FileServer fill:#f3e5f5
@@ -140,7 +135,7 @@ flowchart TD
### 1. msg_envelope_v1 - Message Envelope
The `msg_envelope_v1` structure provides a comprehensive message format for bidirectional communication between Julia, JavaScript, and Python/Micropython applications.
The `msg_envelope_v1` structure provides a comprehensive message format for bidirectional communication in Julia applications.
**Julia Structure:**
```julia
@@ -216,7 +211,7 @@ end
### 2. msg_payload_v1 - Payload Structure
The `msg_payload_v1` structure provides flexible payload handling for various data types across all supported platforms.
The `msg_payload_v1` structure provides flexible payload handling for various data types.
**Julia Structure:**
```julia
@@ -271,65 +266,7 @@ end
└─────────────────┘ └─────────────────┘
```
### 4. Cross-Platform Architecture
```mermaid
flowchart TD
subgraph PythonMicropython
Py[Python/Micropython]
PySmartSend[smartsend]
PySmartReceive[smartreceive]
end
subgraph JavaScript
JS[JavaScript]
JSSmartSend[smartsend]
JSSmartReceive[smartreceive]
end
subgraph Julia
Julia[Julia]
JuliaSmartSend[smartsend]
JuliaSmartReceive[smartreceive]
end
subgraph NATS
NATSServer[NATS Server]
end
PySmartSend --> NATSServer
JSSmartSend --> NATSServer
JuliaSmartSend --> NATSServer
NATSServer --> PySmartReceive
NATSServer --> JSSmartReceive
NATSServer --> JuliaSmartReceive
style PythonMicropython fill:#e1f5fe
style JavaScript fill:#f3e5f5
style Julia fill:#e8f5e9
```
### 5. Python/Micropython Module Architecture
```mermaid
graph TD
subgraph PyModule
PySmartSend[smartsend]
SizeCheck[Size Check]
DirectPath[Direct Path]
LinkPath[Link Path]
HTTPClient[HTTP Client]
end
PySmartSend --> SizeCheck
SizeCheck -->|< 1MB| DirectPath
SizeCheck -->|>= 1MB| LinkPath
LinkPath --> HTTPClient
style PyModule fill:#b3e5fc
```
### 6. Julia Module Architecture
### 4. Julia Module Architecture
```mermaid
graph TD
@@ -349,51 +286,8 @@ graph TD
style JuliaModule fill:#c5e1a5
```
### 7. JavaScript Module Architecture
```mermaid
graph TD
subgraph JSModule
JSSmartSend[smartsend]
JSSmartReceive[smartreceive]
JetStreamConsumer[JetStream Pull Consumer]
ApacheArrow[Apache Arrow]
end
JSSmartSend --> NATS
JSSmartReceive --> JetStreamConsumer
JetStreamConsumer --> ApacheArrow
style JSModule fill:#f3e5f5
```
## Implementation Details
### API Consistency Across Languages
**High-Level API (Consistent Across All Languages):**
- `smartsend(subject, data, ...)` - Main publishing function
- `smartreceive(msg, ...)` - Main receiving function
- Message envelope structure (`msg_envelope_v1` / `MessageEnvelope`)
- Payload structure (`msg_payload_v1` / `MessagePayload`)
- Transport strategy (direct vs link based on size threshold)
- Supported payload types: text, dictionary, table, image, audio, video, binary
**Low-Level Native Functions (Language-Specific Conventions):**
- Julia: `NATS.connect()`, `publish_message()`, function overloading
- JavaScript: `nats.js` client, native async/await patterns
- Python: `nats-python` client, native async/await patterns
**Connection Reuse Pattern:**
- **Julia:** Uses `NATS_connection` keyword parameter with function overloading
- **JavaScript/Python:** Achieved by creating NATS client outside the function and reusing it in custom handlers
**Note on `is_publish`:**
- `is_publish` is simply a switch to control automatic publishing
- When `true` (default): Message is published to NATS automatically
- When `false`: Returns `(env, env_json_str)` without publishing, allowing manual publishing
- Connection reuse is achieved separately by creating NATS client outside the function
### Julia Implementation
#### Dependencies
@@ -546,200 +440,6 @@ env, env_json_str = smartsend(
# Uses: publish_message(broker_url, subject, env_json_str, cid)
```
**API Consistency Note:**
- **High-level API (smartsend, smartreceive):** Uses consistent naming across all three languages (Julia, JavaScript, Python/Micropython)
- **Low-level native functions (NATS.connect(), publish_message()):** Follow the conventions of the specific language ecosystem and do not require cross-language consistency
### JavaScript Implementation
#### Dependencies
- `nats.js` - Core NATS functionality
- `apache-arrow` - Arrow IPC serialization
- `uuid` - Correlation ID and message ID generation
- `base64-arraybuffer` - Base64 encoding/decoding
- `node-fetch` or `fetch` - HTTP client for file server
#### smartsend Function
```javascript
async function smartsend(
subject,
data, // List of (dataname, data, type) tuples: [(dataname1, data1, type1), ...]
options = {}
)
```
**Options:**
- `broker_url` (String) - NATS server URL (default: `"nats://localhost:4222"`)
- `fileserver_url` (String) - Base URL of the file server (default: `"http://localhost:8080"`)
- `size_threshold` (Number) - Threshold in bytes for transport selection (default: `1048576` = 1MB)
- `correlation_id` (String) - Optional correlation ID for tracing
- `msg_purpose` (String) - Purpose of the message (default: `"chat"`)
- `sender_name` (String) - Sender name (default: `"NATSBridge"`)
- `receiver_name` (String) - Message receiver name (default: `""`)
- `receiver_id` (String) - Message receiver ID (default: `""`)
- `reply_to` (String) - Topic to reply to (default: `""`)
- `reply_to_msg_id` (String) - Message ID this message is replying to (default: `""`)
- `is_publish` (Boolean) - Whether to automatically publish the message to NATS (default: `true`)
- `fileserver_upload_handler` (Function) - Custom upload handler function
**Note:** JavaScript uses `is_publish` option (instead of `NATS_connection` keyword) to control automatic publishing behavior. Connection reuse can be achieved by creating a NATS client outside the function and reusing it in a custom `fileserver_upload_handler` or custom publish implementation.
**Return Value:**
- Returns a Promise that resolves to an object containing:
- `env` - The envelope object containing all metadata and payloads
- `env_json_str` - JSON string representation of the envelope for publishing
**Input Format:**
- `data` - **Must be a list of (dataname, data, type) tuples**: `[(dataname1, data1, "type1"), (dataname2, data2, "type2"), ...]`
- Even for single payloads: `[(dataname1, data1, "type1")]`
- Each payload can have a different type, enabling mixed-content messages
- Supported types: `"text"`, `"dictionary"`, `"table"`, `"image"`, `"audio"`, `"video"`, `"binary"`
**Flow:**
1. Generate correlation ID and message ID if not provided
2. Iterate through the list of `(dataname, data, type)` tuples
3. For each payload:
- Serialize based on payload type
- Check payload size
- If < threshold: Base64 encode and include in envelope
- If >= threshold: Upload to HTTP server, store URL in envelope
4. Publish the JSON envelope to NATS
5. Return envelope object and JSON string
#### smartreceive Handler
```javascript
async function smartreceive(msg, options = {})
```
**Options:**
- `fileserver_download_handler` (Function) - Custom download handler function
- `max_retries` (Number) - Maximum retry attempts for fetching URL (default: `5`)
- `base_delay` (Number) - Initial delay for exponential backoff in ms (default: `100`)
- `max_delay` (Number) - Maximum delay for exponential backoff in ms (default: `5000`)
- `correlation_id` (String) - Optional correlation ID for tracing
**Output Format:**
- Returns a Promise that resolves to an object containing all envelope fields:
- `correlation_id`, `msg_id`, `timestamp`, `send_to`, `msg_purpose`, `sender_name`, `sender_id`, `receiver_name`, `receiver_id`, `reply_to`, `reply_to_msg_id`, `broker_url`
- `metadata` - Message-level metadata dictionary
- `payloads` - List of tuples, each containing `(dataname, data, type)` with deserialized payload data
**Process Flow:**
1. Parse the JSON envelope to extract all fields
2. Iterate through each payload in `payloads` array
3. For each payload:
- Determine transport type (`direct` or `link`)
- If `direct`: Base64 decode the data from the message
- If `link`: Fetch data from URL using exponential backoff (via `fileserver_download_handler`)
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
4. Return envelope object with `payloads` field containing list of `(dataname, data, type)` tuples
**Note:** The `fileserver_download_handler` receives `(url, max_retries, base_delay, max_delay, correlation_id)` and returns `ArrayBuffer` or `Uint8Array`.
### Python/Micropython Implementation
#### Dependencies
- `nats-python` - Core NATS functionality
- `pyarrow` - Arrow IPC serialization
- `uuid` - Correlation ID and message ID generation
- `base64` - Base64 encoding/decoding
- `requests` or `aiohttp` - HTTP client for file server
#### smartsend Function
```python
def smartsend(
subject: str,
data: List[Tuple[str, Any, str]], # List of (dataname, data, type) tuples
broker_url: str = DEFAULT_BROKER_URL,
fileserver_url: str = DEFAULT_FILESERVER_URL,
fileserver_upload_handler: Callable = plik_oneshot_upload,
size_threshold: int = DEFAULT_SIZE_THRESHOLD,
correlation_id: Union[str, None] = None,
msg_purpose: str = "chat",
sender_name: str = "NATSBridge",
receiver_name: str = "",
receiver_id: str = "",
reply_to: str = "",
reply_to_msg_id: str = "",
is_publish: bool = True
) -> Tuple[MessageEnvelope, str]
```
**Options:**
- `broker_url` (str) - NATS server URL (default: `"nats://localhost:4222"`)
- `fileserver_url` (str) - Base URL of the file server (default: `"http://localhost:8080"`)
- `size_threshold` (int) - Threshold in bytes for transport selection (default: `1048576` = 1MB)
- `correlation_id` (str) - Optional correlation ID for tracing (auto-generated if None)
- `msg_purpose` (str) - Purpose of the message (default: `"chat"`)
- `sender_name` (str) - Sender name (default: `"NATSBridge"`)
- `receiver_name` (str) - Message receiver name (default: `""`)
- `receiver_id` (str) - Message receiver ID (default: `""`)
- `reply_to` (str) - Topic to reply to (default: `""`)
- `reply_to_msg_id` (str) - Message ID this message is replying to (default: `""`)
- `is_publish` (bool) - Whether to automatically publish the message to NATS (default: `True`)
- `fileserver_upload_handler` (Callable) - Custom upload handler function
**Note:** Python uses `is_publish` parameter (instead of `NATS_connection` keyword) to control automatic publishing behavior. Connection reuse can be achieved by creating a NATS client outside the function and reusing it in a custom `fileserver_upload_handler` or custom publish implementation.
**Return Value:**
- Returns a tuple `(env, env_json_str)` where:
- `env` - The envelope dictionary containing all metadata and payloads
- `env_json_str` - JSON string representation of the envelope for publishing
**Input Format:**
- `data` - **Must be a list of (dataname, data, type) tuples**: `[(dataname1, data1, "type1"), (dataname2, data2, "type2"), ...]`
- Even for single payloads: `[(dataname1, data1, "type1")]`
- Each payload can have a different type, enabling mixed-content messages
- Supported types: `"text"`, `"dictionary"`, `"table"`, `"image"`, `"audio"`, `"video"`, `"binary"`
**Flow:**
1. Generate correlation ID and message ID if not provided
2. Iterate through the list of `(dataname, data, type)` tuples
3. For each payload:
- Serialize based on payload type
- Check payload size
- If < threshold: Base64 encode and include in envelope
- If >= threshold: Upload to HTTP server, store URL in envelope
4. Publish the JSON envelope to NATS
5. Return envelope dictionary and JSON string
#### smartreceive Handler
```python
async def smartreceive(
msg: NATS.Message,
options: Dict = {}
)
```
**Options:**
- `fileserver_download_handler` (Callable) - Custom download handler function
- `max_retries` (int) - Maximum retry attempts for fetching URL (default: `5`)
- `base_delay` (int) - Initial delay for exponential backoff in ms (default: `100`)
- `max_delay` (int) - Maximum delay for exponential backoff in ms (default: `5000`)
- `correlation_id` (str) - Optional correlation ID for tracing
**Output Format:**
- Returns a JSON object (dictionary) containing all envelope fields:
- `correlation_id`, `msg_id`, `timestamp`, `send_to`, `msg_purpose`, `sender_name`, `sender_id`, `receiver_name`, `receiver_id`, `reply_to`, `reply_to_msg_id`, `broker_url`
- `metadata` - Message-level metadata dictionary
- `payloads` - List of tuples, each containing `(dataname, data, payload_type)` with deserialized payload data
**Process Flow:**
1. Parse the JSON envelope to extract all fields
2. Iterate through each payload in `payloads` list
3. For each payload:
- Determine transport type (`direct` or `link`)
- If `direct`: Base64 decode the data from the message
- If `link`: Fetch data from URL using exponential backoff (via `fileserver_download_handler`)
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples
**Note:** The `fileserver_download_handler` receives `(url: str, max_retries: int, base_delay: int, max_delay: int, correlation_id: str)` and returns `bytes`.
## Scenario Implementations
### Scenario 1: Command & Control (Small Dictionary)
@@ -752,18 +452,6 @@ async def smartreceive(
# Send acknowledgment
```
**JavaScript (Sender/Receiver):**
```javascript
// Create small dictionary config
// Send via smartsend with type="dictionary"
```
**Python/Micropython (Sender/Receiver):**
```python
# Create small dictionary config
# Send via smartsend with type="dictionary"
```
### Scenario 2: Deep Dive Analysis (Large Arrow Table)
**Julia (Sender/Receiver):**
@@ -775,32 +463,8 @@ async def smartreceive(
# Publish NATS with URL
```
**JavaScript (Sender/Receiver):**
```javascript
// Receive NATS message with URL
// Fetch data from HTTP server
// Parse Arrow IPC with zero-copy
// Load into Perspective.js or D3
```
**Python/Micropython (Sender/Receiver):**
```python
# Create large DataFrame
# Convert to Arrow IPC stream
# Check size (> 1MB)
# Upload to HTTP server
# Publish NATS with URL
```
### Scenario 3: Live Audio Processing
**JavaScript (Sender/Receiver):**
```javascript
// Capture audio chunk
// Send as binary with metadata headers
// Use smartsend with type="audio"
```
**Julia (Sender/Receiver):**
```julia
# Receive audio data
@@ -808,13 +472,6 @@ async def smartreceive(
# Send results back (JSON + Arrow table)
```
**Python/Micropython (Sender/Receiver):**
```python
# Capture audio chunk
# Send as binary with metadata headers
# Use smartsend with type="audio"
```
### Scenario 4: Catch-Up (JetStream)
**Julia (Producer/Consumer):**
@@ -823,22 +480,9 @@ async def smartreceive(
# Include metadata for temporal tracking
```
**JavaScript (Producer/Consumer):**
```javascript
// Connect to JetStream
// Request replay from last 10 minutes
// Process historical and real-time messages
```
**Python/Micropython (Producer/Consumer):**
```python
# Publish to JetStream
# Include metadata for temporal tracking
```
### Scenario 5: Selection (Low Bandwidth)
**Focus:** Small Arrow tables, cross-platform communication. The Action: Any platform wants to send a small DataFrame to show on any receiving application for the user to choose.
**Focus:** Small Arrow tables. The Action: Julia wants to send a small DataFrame to show on a receiving application for the user to choose.
**Julia (Sender/Receiver):**
```julia
@@ -849,30 +493,9 @@ async def smartreceive(
# Include metadata for dashboard selection context
```
**JavaScript (Sender/Receiver):**
```javascript
// Receive NATS message with direct transport
// Decode Base64 payload
// Parse Arrow IPC with zero-copy
// Load into selection UI component (e.g., dropdown, table)
// User makes selection
// Send selection back to Julia
```
**Python/Micropython (Sender/Receiver):**
```python
# Create small DataFrame (e.g., 50KB - 500KB)
# Convert to Arrow IPC stream
# Check payload size (< 1MB threshold)
# Publish directly to NATS with Base64-encoded payload
# Include metadata for dashboard selection context
```
**Use Case:** Any server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to any receiving application for user selection. The selection is then sent back to the sender for processing.
### Scenario 6: Chat System
**Focus:** Every conversational message is composed of any number and any combination of components, spanning the full spectrum from small to large. This includes text, images, audio, video, tables, and files—specifically accommodating everything from brief snippets to high-resolution images, large audio files, extensive tables, and massive documents. Support for claim-check delivery and full bi-directional messaging across all platforms.
**Focus:** Every conversational message is composed of any number and any combination of components, spanning the full spectrum from small to large. This includes text, images, audio, video, tables, and files—specifically accommodating everything from brief snippets to high-resolution images, large audio files, extensive tables, and massive documents. Support for claim-check delivery and full bi-directional messaging.
**Multi-Payload Support:** The system supports mixed-payload messages where a single message can contain multiple payloads with different transport strategies. The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type.
@@ -894,42 +517,7 @@ async def smartreceive(
# Support bidirectional messaging with replyTo fields
```
**JavaScript (Sender/Receiver):**
```javascript
// Build chat message with mixed content:
// - User input text: direct transport
// - Selected image: check size, use appropriate transport
// - Audio recording: link transport for large files
// - File attachment: link transport
//
// Parse received message:
// - Direct payloads: decode Base64
// - Link payloads: fetch from HTTP with exponential backoff
// - Deserialize all payloads appropriately
//
// Render mixed content in chat interface
// Support bidirectional reply with claim-check delivery confirmation
```
**Python/Micropython (Sender/Receiver):**
```python
# Build chat message with mixed payloads:
# - Text: direct transport (Base64)
# - Small images: direct transport (Base64)
# - Large images: link transport (HTTP URL)
# - Audio/video: link transport (HTTP URL)
# - Tables: direct or link depending on size
# - Files: link transport (HTTP URL)
#
# Each payload uses appropriate transport strategy:
# - Size < 1MB → direct (NATS + Base64)
# - Size >= 1MB → link (HTTP upload + NATS URL)
#
# Include claim-check metadata for delivery tracking
# Support bidirectional messaging with replyTo fields
```
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components across all platforms.
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components.
**Implementation Note:** The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. See the standard API format in Section 1: `msg_envelope_v1` supports `Vector{msg_payload_v1}` for multiple payloads.

580
docs/implementation.md
View File

@@ -2,22 +2,17 @@
## Overview
This document describes the implementation of the high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
This document describes the implementation of the high-performance, bi-directional data bridge for **Julia** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
The system enables seamless communication across all three platforms:
- **Julia ↔ JavaScript** bi-directional messaging
- **JavaScript ↔ Python/Micropython** bi-directional messaging
- **Julia ↔ Python/Micropython** bi-directional messaging (via JSON serialization)
The system enables seamless communication for Julia applications.
### Implementation Files
NATSBridge is implemented in three languages, each providing the same API:
NATSBridge is implemented in Julia:
| Language | Implementation File | Description |
|----------|---------------------|-------------|
| **Julia** | [`src/NATSBridge.jl`](../src/NATSBridge.jl) | Full Julia implementation with Arrow IPC support |
| **JavaScript** | [`src/NATSBridge.js`](../src/NATSBridge.js) | JavaScript implementation for Node.js and browsers |
| **Python/Micropython** | [`src/nats_bridge.py`](../src/nats_bridge.py) | Python implementation for desktop and microcontrollers |
### File Server Handler Architecture
@@ -117,64 +112,9 @@ env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff, max_ret
# env is a dictionary containing envelope metadata and payloads field
```
## Cross-Platform Interoperability
NATSBridge is designed for seamless communication between Julia, JavaScript, and Python/Micropython applications. All three implementations share the same interface and data format, ensuring compatibility across platforms.
### Platform-Specific Features
| Feature | Julia | JavaScript | Python/Micropython |
|---------|-------|------------|-------------------|
| Direct NATS transport | ✅ | ✅ | ✅ |
| HTTP file server (Claim-Check) | ✅ | ✅ | ✅ |
| Arrow IPC tables | ✅ | ✅ | ✅ |
| Base64 encoding | ✅ | ✅ | ✅ |
| Exponential backoff | ✅ | ✅ | ✅ |
| Correlation ID tracking | ✅ | ✅ | ✅ |
| Reply-to support | ✅ | ✅ | ✅ |
### Data Type Mapping
| Type | Julia | JavaScript | Python/Micropython |
|------|-------|------------|-------------------|
| `text` | `String` | `String` | `str` |
| `dictionary` | `Dict` | `Object` | `dict` |
| `table` | `DataFrame` | `Array<Object>` | `DataFrame` / `list` |
| `image` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
| `audio` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
| `video` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
| `binary` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
### Example: Julia ↔ Python ↔ JavaScript
```julia
# Julia sender - smartsend returns (env, env_json_str)
using NATSBridge
data = [("message", "Hello from Julia!", "text")]
env, env_json_str = smartsend("/cross_platform", data, broker_url="nats://localhost:4222")
# env: msg_envelope_v1 with all metadata and payloads
# env_json_str: JSON string for publishing
```
```javascript
// JavaScript receiver
const { smartreceive } = require('./src/NATSBridge');
const env = await smartreceive(msg);
// env.payloads[0].data === "Hello from Julia!"
```
```python
# Python sender
from nats_bridge import smartsend
data = [("response", "Hello from Python!", "text")]
smartsend("/cross_platform", data, broker_url="nats://localhost:4222")
```
All three platforms can communicate seamlessly using the same NATS subjects and data format.
## Architecture
All three implementations (Julia, JavaScript, Python/Micropython) follow the same Claim-Check pattern:
The Julia implementation follows the Claim-Check pattern:
```
┌─────────────────────────────────────────────────────────────────────────┐
@@ -227,24 +167,6 @@ The Julia implementation provides:
- **[`smartsend()`](src/NATSBridge.jl)**: Handles transport selection based on payload size
- **[`smartreceive()`](src/NATSBridge.jl)**: Handles both direct and link transport
### JavaScript Module: [`src/NATSBridge.js`](../src/NATSBridge.js)
The JavaScript implementation provides:
- **`MessageEnvelope` class**: For the unified JSON envelope
- **`MessagePayload` class**: For individual payload representation
- **[`smartsend()`](src/NATSBridge.js)**: Handles transport selection based on payload size
- **[`smartreceive()`](src/NATSBridge.js)**: Handles both direct and link transport
### Python/Micropython Module: [`src/nats_bridge.py`](../src/nats_bridge.py)
The Python/Micropython implementation provides:
- **`MessageEnvelope` class**: For the unified JSON envelope
- **`MessagePayload` class**: For individual payload representation
- **[`smartsend()`](src/nats_bridge.py)**: Handles transport selection based on payload size
- **[`smartreceive()`](src/nats_bridge.py)**: Handles both direct and link transport
## Installation
### Julia Dependencies
@@ -259,29 +181,6 @@ Pkg.add("UUIDs")
Pkg.add("Dates")
```
### JavaScript Dependencies
```bash
npm install nats.js apache-arrow uuid base64-url
```
### Python/Micropython Dependencies
1. Copy [`src/nats_bridge.py`](../src/nats_bridge.py) to your device
2. Ensure you have the following dependencies:
**For Python (desktop):**
```bash
pip install nats-py
```
**For Micropython:**
- `urequests` for HTTP requests
- `base64` for base64 encoding (built-in)
- `json` for JSON handling (built-in)
- `socket` for networking (built-in)
- `uuid` for UUID generation (built-in)
## Usage Tutorial
### Step 1: Start NATS Server
@@ -304,48 +203,21 @@ python3 -m http.server 8080 --directory /tmp/fileserver
### Step 3: Run Test Scenarios
```bash
# Scenario 1: Command & Control (JavaScript sender)
node test/scenario1_command_control.js
# Scenario 1: Command & Control
julia test/scenario1_command_control.jl
# Scenario 2: Large Arrow Table (JavaScript sender)
node test/scenario2_large_table.js
# Scenario 2: Large Arrow Table
julia test/scenario2_large_table.jl
# Scenario 3: Julia-to-Julia communication
# Run both Julia and JavaScript versions
julia test/scenario3_julia_to_julia.jl
node test/scenario3_julia_to_julia.js
```
## API Consistency Across Languages
**High-Level API (Consistent Across All Languages):**
- `smartsend(subject, data, ...)` - Main publishing function
- `smartreceive(msg, ...)` - Main receiving function
- Message envelope structure (`msg_envelope_v1` / `MessageEnvelope`)
- Payload structure (`msg_payload_v1` / `MessagePayload`)
- Transport strategy (direct vs link based on size threshold)
- Supported payload types: text, dictionary, table, image, audio, video, binary
**Low-Level Native Functions (Language-Specific Conventions):**
- Julia: `NATS.connect()`, `publish_message()`, function overloading
- JavaScript: `nats.js` client, native async/await patterns
- Python: `nats-python` client, native async/await patterns
**Connection Reuse Pattern - Key Differences:**
- **Julia:** Uses `NATS_connection` keyword parameter with function overloading for automatic connection management
- **JavaScript/Python:** Achieved by creating NATS client outside the function and reusing it in custom handlers or custom publish implementations
**Why the Difference?**
- Julia supports function overloading and keyword arguments, allowing `NATS_connection` to be passed as an optional parameter
- JavaScript/Python use a simpler `is_publish` option to control automatic publishing
- `is_publish` is simply a switch: when `true`, publish automatically; when `false`, return `(env, env_json_str)` without publishing
- For connection reuse in JavaScript/Python, create a NATS client once and reuse it in your custom `fileserver_upload_handler` or custom publish logic
## Usage
### Scenario 1: Command & Control (Small Dictionary)
**Focus:** Sending small dictionary configurations across platforms. This is the simplest use case for command and control scenarios.
**Focus:** Sending small dictionary configurations. This is the simplest use case for command and control scenarios.
**Julia (Sender/Receiver):**
```julia
@@ -386,98 +258,11 @@ NATS.close(conn)
**Use Case:** High-frequency publishing scenarios where connection reuse provides performance benefits by avoiding the overhead of establishing a new NATS connection for each message.
**JavaScript (Sender/Receiver):**
```javascript
const { smartsend } = require('./src/NATSBridge');
// Create small dictionary config
// Send via smartsend with type="dictionary"
const config = {
step_size: 0.01,
iterations: 1000,
threshold: 0.5
};
// Use is_publish option to control automatic publishing
await smartsend("control", [
{ dataname: "config", data: config, type: "dictionary" }
], {
is_publish: true // Automatically publish to NATS
});
```
**Connection Reuse in JavaScript:**
To achieve connection reuse in JavaScript, create a NATS client outside the function and use it in a custom `fileserver_upload_handler` or custom publish implementation:
```javascript
const { connect } = require('nats');
const { smartsend } = require('./src/NATSBridge');
// Create connection once
const nc = await connect({ servers: ['nats://localhost:4222'] });
// Send multiple messages using the same connection
for (let i = 0; i < 100; i++) {
const config = { iteration: i, data: Math.random() };
// Option 1: Use is_publish=false and publish manually with your connection
const { env, env_json_str } = await smartsend("control", [
{ dataname: "config", data: config, type: "dictionary" }
], { is_publish: false });
// Publish with your existing connection
await nc.publish("control", env_json_str);
}
// Close connection when done
await nc.close();
```
**Python/Micropython (Sender/Receiver):**
```python
from nats_bridge import smartsend
# Create small dictionary config
# Send via smartsend with type="dictionary"
config = {
"step_size": 0.01,
"iterations": 1000,
"threshold": 0.5
}
# Use is_publish parameter to control automatic publishing
smartsend("control", [("config", config, "dictionary")], is_publish=True)
```
**Connection Reuse in Python:**
To achieve connection reuse in Python, create a NATS client outside the function and use it in a custom `fileserver_upload_handler` or custom publish implementation:
```python
from nats_bridge import smartsend
import nats
# Create connection once
nc = await nats.connect("nats://localhost:4222")
# Send multiple messages using the same connection
for i in range(100):
config = {"iteration": i, "data": random.random()}
# Option 1: Use is_publish=False and publish manually with your connection
env, env_json_str = smartsend("control", [("config", config, "dictionary")], is_publish=False)
# Publish with your existing connection
await nc.publish("control", env_json_str)
# Close connection when done
await nc.close()
```
### Basic Multi-Payload Example
#### Python/Micropython (Sender)
```python
from nats_bridge import smartsend
#### Julia (Sender)
```julia
using NATSBridge
# Send multiple payloads in one message (type is required per payload)
smartsend(
@@ -491,71 +276,15 @@ smartsend(
smartsend("/test", [("single_data", mydata, "dictionary")], broker_url="nats://localhost:4222")
```
#### Python/Micropython (Receiver)
```python
from nats_bridge import smartreceive
#### Julia (Receiver)
```julia
using NATSBridge
# Receive returns a dictionary with envelope metadata and payloads field
env = smartreceive(msg)
# env["payloads"] = [(dataname1, data1, "dictionary"), (dataname2, data2, "table"), ...]
```
#### JavaScript (Sender)
```javascript
const { smartsend } = require('./src/NATSBridge');
// Single payload wrapped in a list
const config = [{
dataname: "config",
data: { step_size: 0.01, iterations: 1000 },
type: "dictionary"
}];
await smartsend("control", config, {
correlationId: "unique-id"
});
// Multiple payloads
const configs = [
{
dataname: "config1",
data: { step_size: 0.01 },
type: "dictionary"
},
{
dataname: "config2",
data: { iterations: 1000 },
type: "dictionary"
}
];
await smartsend("control", configs);
```
#### JavaScript (Receiver)
```javascript
const { smartreceive } = require('./src/NATSBridge');
// Subscribe to messages
const nc = await connect({ servers: ['nats://localhost:4222'] });
const sub = nc.subscribe("control");
for await (const msg of sub) {
const env = await smartreceive(msg);
// Process the payloads from the envelope
for (const payload of env.payloads) {
const { dataname, data, type } = payload;
console.log(`Received ${dataname} of type ${type}`);
console.log(`Data: ${JSON.stringify(data)}`);
}
// Also access envelope metadata
console.log(`Correlation ID: ${env.correlation_id}`);
console.log(`Message ID: ${env.msg_id}`);
}
```
### Scenario 2: Deep Dive Analysis (Large Arrow Table)
#### Julia (Sender)
@@ -689,91 +418,25 @@ env, env_json_str = smartsend(
**API Consistency Note:**
- **Julia:** Uses `NATS_connection` keyword parameter with function overloading for automatic connection management
- **JavaScript/Python:** Use `is_publish` option and achieve connection reuse by creating NATS client outside the function and reusing it in custom handlers or custom publish implementations
#### JavaScript (Receiver)
```javascript
const { smartreceive } = require('./src/NATSBridge');
const env = await smartreceive(msg);
// Use table data from the payloads field
// Note: Tables are sent as arrays of objects in JavaScript
const table = env.payloads;
```
### Scenario 3: Live Binary Processing
#### Python/Micropython (Sender)
```python
from nats_bridge import smartsend
**Julia (Sender/Receiver):**
```julia
using NATSBridge
# Binary data wrapped in list with type
smartsend(
"binary_input",
[("audio_chunk", binary_buffer, "binary")],
broker_url="nats://localhost:4222",
metadata={"sample_rate": 44100, "channels": 1}
metadata=["sample_rate" => 44100, "channels" => 1]
)
```
#### JavaScript (Sender)
```javascript
const { smartsend } = require('./src/NATSBridge');
// Binary data wrapped in a list
const binaryData = [{
dataname: "audio_chunk",
data: binaryBuffer, // ArrayBuffer or Uint8Array
type: "binary"
}];
await smartsend("binary_input", binaryData, {
metadata: {
sample_rate: 44100,
channels: 1
}
});
```
#### Python/Micropython (Receiver)
```python
from nats_bridge import smartreceive
# Receive binary data
def process_binary(msg):
env = smartreceive(msg)
# Process the binary data from env.payloads
for dataname, data, type in env["payloads"]:
if type == "binary":
# data is bytes
print(f"Received binary data: {dataname}, size: {len(data)}")
# Perform FFT or AI transcription here
```
#### JavaScript (Receiver)
```javascript
const { smartreceive } = require('./src/NATSBridge');
// Receive binary data
function process_binary(msg) {
const env = await smartreceive(msg);
// Process the binary data from env.payloads
for (const payload of env.payloads) {
if (payload.type === "binary") {
// data is an ArrayBuffer or Uint8Array
console.log(`Received binary data: ${payload.dataname}, size: ${payload.data.length}`);
// Perform FFT or AI transcription here
}
}
}
```
### Scenario 4: Catch-Up (JetStream)
#### Julia (Producer)
**Julia (Producer/Consumer):**
```julia
using NATSBridge
@@ -789,93 +452,11 @@ function publish_health_status(broker_url)
end
```
#### JavaScript (Consumer)
```javascript
const { connect } = require('nats');
const { smartreceive } = require('./src/NATSBridge');
const nc = await connect({ servers: ['nats://localhost:4222'] });
const js = nc.jetstream();
// Request replay from last 10 minutes
const consumer = await js.pullSubscribe("health", {
durable_name: "catchup",
max_batch: 100,
max_ack_wait: 30000
});
// Process historical and real-time messages
for await (const msg of consumer) {
const env = await smartreceive(msg);
// env.payloads contains the list of payloads
// Each payload has: dataname, data, type
msg.ack();
}
```
### Scenario 4: Micropython Device Control
**Focus:** Sending configuration to a Micropython device over NATS. This demonstrates the lightweight nature of the Python implementation suitable for microcontrollers.
**Python/Micropython (Receiver/Device):**
```python
from nats_bridge import smartsend, smartreceive
import json
# Device configuration handler
def handle_device_config(msg):
env = smartreceive(msg)
# Process configuration from payloads
for dataname, data, payload_type in env["payloads"]:
if payload_type == "dictionary":
print(f"Received configuration: {data}")
# Apply configuration to device
if "wifi_ssid" in data:
wifi_ssid = data["wifi_ssid"]
wifi_password = data["wifi_password"]
update_wifi_config(wifi_ssid, wifi_password)
# Send confirmation back
config = {
"status": "configured",
"wifi_ssid": "MyNetwork",
"ip": get_device_ip()
}
smartsend(
"device/response",
[("config", config, "dictionary")],
broker_url="nats://localhost:4222",
reply_to=env.get("reply_to")
)
```
**JavaScript (Sender/Controller):**
```javascript
const { smartsend } = require('./src/NATSBridge');
// Send configuration to Micropython device
await smartsend("device/config", [
{
dataname: "config",
data: {
wifi_ssid: "MyNetwork",
wifi_password: "password123",
update_interval: 60,
temperature_threshold: 30.0
},
type: "dictionary"
}
]);
```
**Use Case:** A controller sends WiFi and operational configuration to a Micropython device (e.g., ESP32). The device receives the configuration, applies it, and sends back a confirmation with its current status.
### Scenario 5: Selection (Low Bandwidth)
**Focus:** Small Arrow tables, Julia to JavaScript. The Action: Julia wants to send a small DataFrame to show on a JavaScript dashboard for the user to choose.
**Focus:** Small Arrow tables. The Action: Julia wants to send a small DataFrame to show on a receiving application for the user to choose.
**Julia (Sender):**
**Julia (Sender/Receiver):**
```julia
using NATSBridge
using DataFrames
@@ -903,27 +484,7 @@ env, env_json_str = smartsend(
# env_json_str: JSON string for publishing
```
**JavaScript (Receiver):**
```javascript
const { smartreceive, smartsend } = require('./src/NATSBridge');
// Receive NATS message with direct transport
const env = await smartreceive(msg);
// Decode Base64 payload (for direct transport)
// For tables, data is in env.payloads
const table = env.payloads; // Array of objects
// User makes selection
const selection = uiComponent.getSelectedOption();
// Send selection back to Julia
await smartsend("dashboard.response", [
{ dataname: "selected_option", data: selection, type: "dictionary" }
]);
```
**Use Case:** Julia server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to JavaScript dashboard for user selection. The selection is then sent back to Julia for processing.
**Use Case:** Julia server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to a receiving application for user selection. The selection is then sent back to Julia for processing.
### Scenario 6: Chat System
@@ -968,46 +529,6 @@ env, env_json_str = smartsend(
# env_json_str: JSON string for publishing
```
**JavaScript (Sender/Receiver):**
```javascript
const { smartsend, smartreceive } = require('./src/NATSBridge');
// Build chat message with mixed content:
// - User input text: direct transport
// - Selected image: check size, use appropriate transport
// - Audio recording: link transport for large files
// - File attachment: link transport
//
// Parse received message:
// - Direct payloads: decode Base64
// - Link payloads: fetch from HTTP with exponential backoff
// - Deserialize all payloads appropriately
//
// Render mixed content in chat interface
// Support bidirectional reply with claim-check delivery confirmation
// Example: Send chat with mixed content
const message = [
{
dataname: "text",
data: "Hello from JavaScript!",
type: "text"
},
{
dataname: "image",
data: selectedImageBuffer, // Small image (ArrayBuffer or Uint8Array)
type: "image"
},
{
dataname: "audio",
data: audioUrl, // Large audio, link transport
type: "audio"
}
];
await smartsend("chat.room123", message);
```
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components.
**Implementation Note:** The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. See the standard API format in Section 1: `msg_envelope_v1` supports `Vector{msg_payload_v1}` for multiple payloads.
@@ -1072,7 +593,6 @@ await smartsend("chat.room123", message);
### Exponential Backoff
- Maximum retry count: 5
- Base delay: 100ms, max delay: 5000ms
- Implemented in all three implementations (Julia, JavaScript, Python/Micropython)
### Correlation ID Logging
- Log correlation_id at every stage
@@ -1081,38 +601,7 @@ await smartsend("chat.room123", message);
## Testing
Run the test scripts for each platform:
### Python/Micropython Tests
```bash
# Basic functionality test
python test/test_micropython_basic.py
```
### JavaScript Tests
```bash
# Text message exchange
node test/test_js_to_js_text_sender.js
node test/test_js_to_js_text_receiver.js
# Dictionary exchange
node test/test_js_to_js_dict_sender.js
node test/test_js_to_js_dict_receiver.js
# File transfer (direct transport)
node test/test_js_to_js_file_sender.js
node test/test_js_to_js_file_receiver.js
# Mixed payload types
node test/test_js_to_js_mix_payloads_sender.js
node test/test_js_to_js_mix_payloads_receiver.js
# Table (Arrow IPC) exchange
node test/test_js_to_js_table_sender.js
node test/test_js_to_js_table_receiver.js
```
Run the test scripts for Julia:
### Julia Tests
@@ -1138,41 +627,22 @@ julia test/test_julia_to_julia_table_sender.jl
julia test/test_julia_to_julia_table_receiver.jl
```
### Cross-Platform Tests
```bash
# Julia ↔ JavaScript communication
julia test/test_julia_to_julia_text_sender.jl
node test/test_js_to_js_text_receiver.js
# Python ↔ JavaScript communication
python test/test_micropython_basic.py
node test/test_js_to_js_text_receiver.js
```
## Troubleshooting
### Common Issues
1. **NATS Connection Failed**
- **Julia/JavaScript/Python**: Ensure NATS server is running
- **Python/Micropython**: Check `nats_url` parameter and network connectivity
- Ensure NATS server is running
2. **HTTP Upload Failed**
- Ensure file server is running
- Check `fileserver_url` configuration
- Verify upload permissions
- **Micropython**: Ensure `urequests` is available and network is connected
3. **Arrow IPC Deserialization Error**
- Ensure data is properly serialized to Arrow format
- Check Arrow version compatibility
4. **Python/Micropython Specific Issues**
- **Import Error**: Ensure `nats_bridge.py` is in the correct path
- **Memory Error (Micropython)**: Reduce payload size or use link transport for large payloads
- **Unicode Error**: Ensure proper encoding when sending text data
## License
MIT

402
examples/tutorial.md
View File

@@ -1,6 +1,6 @@
# NATSBridge Tutorial
A step-by-step guide to get started with NATSBridge - a high-performance, bi-directional data bridge for **Julia**, **JavaScript**, and **Python/Micropython**.
A step-by-step guide to get started with NATSBridge - a high-performance, bi-directional data bridge for **Julia**.
## Table of Contents
@@ -10,13 +10,12 @@ A step-by-step guide to get started with NATSBridge - a high-performance, bi-dir
4. [Quick Start](#quick-start)
5. [Basic Examples](#basic-examples)
6. [Advanced Usage](#advanced-usage)
7. [Cross-Platform Communication](#cross-platform-communication)
---
## Overview
NATSBridge enables seamless communication between Julia, JavaScript, and Python/Micropython applications through NATS, with automatic transport selection based on payload size:
NATSBridge enables seamless communication for Julia applications through NATS, with automatic transport selection based on payload size:
- **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded)
- **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL
@@ -41,7 +40,7 @@ Before you begin, ensure you have:
1. **NATS Server** running (or accessible)
2. **HTTP File Server** (optional, for large payloads > 1MB)
3. **One of the supported platforms**: Julia, JavaScript (Node.js), or Python/Micropython
3. **Julia** with required packages
---
@@ -59,27 +58,6 @@ Pkg.add("UUIDs")
Pkg.add("Dates")
```
### JavaScript
```bash
npm install nats.js apache-arrow uuid base64-url
```
### Python/Micropython
1. Copy `src/nats_bridge.py` to your device
2. Install dependencies:
**For Python (desktop):**
```bash
pip install nats-py
```
**For Micropython:**
- `urequests` for HTTP requests
- `base64` for base64 encoding (built-in)
- `json` for JSON handling (built-in)
---
## Quick Start
@@ -96,48 +74,12 @@ docker run -p 4222:4222 nats:latest
# Create a directory for file uploads
mkdir -p /tmp/fileserver
# Use Python's built-in server
# Use any HTTP server that supports POST for file uploads
python3 -m http.server 8080 --directory /tmp/fileserver
```
### Step 3: Send Your First Message
#### Python/Micropython
```python
from nats_bridge import smartsend
# Send a text message (is_publish=True by default)
data = [("message", "Hello World", "text")]
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222")
print("Message sent!")
# Or use is_publish=False to get envelope and JSON without publishing
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222", is_publish=False)
# env: MessageEnvelope object
# env_json_str: JSON string for publishing to NATS
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
// Send a text message (isPublish=true by default)
await smartsend("/chat/room1", [
{ dataname: "message", data: "Hello World", type: "text" }
], { brokerUrl: "nats://localhost:4222" });
console.log("Message sent!");
// Or use isPublish=false to get envelope and JSON without publishing
const { env, env_json_str } = await smartsend("/chat/room1", [
{ dataname: "message", data: "Hello World", type: "text" }
], { brokerUrl: "nats://localhost:4222", isPublish: false });
// env: MessageEnvelope object
// env_json_str: JSON string for publishing to NATS
```
#### Julia
```julia
@@ -149,33 +91,15 @@ env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:
# env: msg_envelope_v1 object with all metadata and payloads
# env_json_str: JSON string representation of the envelope for publishing
println("Message sent!")
# Or use is_publish=false to get envelope and JSON without publishing
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222", is_publish=false)
# env: msg_envelope_v1 object
# env_json_str: JSON string for publishing to NATS
```
### Step 4: Receive Messages
#### Python/Micropython
```python
from nats_bridge import smartreceive
# Receive and process message
env = smartreceive(msg)
for dataname, data, type in env["payloads"]:
print(f"Received {dataname}: {data}")
```
#### JavaScript
```javascript
const { smartreceive } = require('./src/NATSBridge');
// Receive and process message
const env = await smartreceive(msg);
for (const payload of env.payloads) {
console.log(`Received ${payload.dataname}: ${payload.data}`);
}
```
#### Julia
```julia
@@ -194,39 +118,6 @@ end
### Example 1: Sending a Dictionary
#### Python/Micropython
```python
from nats_bridge import smartsend
# Create configuration dictionary
config = {
"wifi_ssid": "MyNetwork",
"wifi_password": "password123",
"update_interval": 60
}
# Send as dictionary type
data = [("config", config, "dictionary")]
env, env_json_str = smartsend("/device/config", data, broker_url="nats://localhost:4222")
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
const config = {
wifi_ssid: "MyNetwork",
wifi_password: "password123",
update_interval: 60
};
const { env, env_json_str } = await smartsend("/device/config", [
{ dataname: "config", data: config, type: "dictionary" }
], { brokerUrl: "nats://localhost:4222" });
```
#### Julia
```julia
@@ -244,34 +135,6 @@ env, env_json_str = smartsend("/device/config", data, broker_url="nats://localho
### Example 2: Sending Binary Data (Image)
#### Python/Micropython
```python
from nats_bridge import smartsend
# Read image file
with open("image.png", "rb") as f:
image_data = f.read()
# Send as binary type
data = [("user_image", image_data, "binary")]
env, env_json_str = smartsend("/chat/image", data, broker_url="nats://localhost:4222")
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
// Read image file (Node.js)
const fs = require('fs');
const image_data = fs.readFileSync('image.png');
const { env, env_json_str } = await smartsend("/chat/image", [
{ dataname: "user_image", data: image_data, type: "binary" }
], { brokerUrl: "nats://localhost:4222" });
```
#### Julia
```julia
@@ -286,13 +149,13 @@ env, env_json_str = smartsend("/chat/image", data, broker_url="nats://localhost:
### Example 3: Request-Response Pattern
#### Python/Micropython (Requester)
#### Julia (Requester)
```python
from nats_bridge import smartsend
```julia
using NATSBridge
# Send command with reply-to
data = [("command", {"action": "read_sensor"}, "dictionary")]
data = [("command", Dict("action" => "read_sensor"), "dictionary")]
env, env_json_str = smartsend(
"/device/command",
data,
@@ -300,44 +163,43 @@ env, env_json_str = smartsend(
reply_to="/device/response",
reply_to_msg_id="cmd-001"
)
# env: MessageEnvelope object
# env: msg_envelope_v1 object
# env_json_str: JSON string for publishing to NATS
```
#### JavaScript (Responder)
#### Julia (Responder)
```javascript
const { smartreceive, smartsend } = require('./src/NATSBridge');
```julia
using NATS, NATSBridge
// Subscribe to command topic
const sub = nc.subscribe("/device/command");
# Configuration
const SUBJECT = "/device/command"
const NATS_URL = "nats://localhost:4222"
for await (const msg of sub) {
const env = await smartreceive(msg);
function test_responder()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
env = smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
// Process command
for (const payload of env.payloads) {
if (payload.dataname === "command") {
const command = payload.data;
# Extract reply_to from the envelope metadata
reply_to = env["reply_to"]
if (command.action === "read_sensor") {
// Read sensor and send response
const response = {
sensor_id: "sensor-001",
value: 42.5,
timestamp: new Date().toISOString()
};
for (dataname, data, type) in env["payloads"]
if dataname == "command" && data["action"] == "read_sensor"
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
# Send response to the reply_to subject from the request
if !isempty(reply_to)
smartsend(reply_to, [("data", response, "dictionary")])
end
end
end
end
await smartsend("/device/response", [
{ dataname: "sensor_data", data: response, type: "dictionary" }
], {
reply_to: env.replyTo,
reply_to_msg_id: env.msgId
});
}
}
}
}
sleep(120)
NATS.drain(conn)
end
test_responder()
```
---
@@ -348,47 +210,6 @@ for await (const msg of sub) {
For payloads larger than 1MB, NATSBridge automatically uses the file server:
#### Python/Micropython
```python
from nats_bridge import smartsend
import os
# Create large data (> 1MB)
large_data = os.urandom(2_000_000) # 2MB of random data
# Send with file server URL
env, env_json_str = smartsend(
"/data/large",
[("large_file", large_data, "binary")],
broker_url="nats://localhost:4222",
fileserver_url="http://localhost:8080",
size_threshold=1_000_000
)
# The envelope will contain the download URL
print(f"File uploaded to: {env.payloads[0].data}")
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
// Create large data (> 1MB)
const largeData = new ArrayBuffer(2_000_000);
const view = new Uint8Array(largeData);
view.fill(42); // Fill with some data
const { env, env_json_str } = await smartsend("/data/large", [
{ dataname: "large_file", data: largeData, type: "binary" }
], {
brokerUrl: "nats://localhost:4222",
fileserverUrl: "http://localhost:8080",
sizeThreshold: 1_000_000
});
```
#### Julia
```julia
@@ -412,45 +233,6 @@ println("File uploaded to: $(env.payloads[1].data)")
NATSBridge supports sending multiple payloads with different types in a single message:
#### Python/Micropython
```python
from nats_bridge import smartsend
# Read image file
with open("avatar.png", "rb") as f:
image_data = f.read()
# Send mixed content
data = [
("message_text", "Hello with image!", "text"),
("user_avatar", image_data, "image")
]
env, env_json_str = smartsend("/chat/mixed", data, broker_url="nats://localhost:4222")
```
#### JavaScript
```javascript
const { smartsend } = require('./src/NATSBridge');
const fs = require('fs');
const { env, env_json_str } = await smartsend("/chat/mixed", [
{
dataname: "message_text",
data: "Hello with image!",
type: "text"
},
{
dataname: "user_avatar",
data: fs.readFileSync("avatar.png"),
type: "image"
}
], { brokerUrl: "nats://localhost:4222" });
```
#### Julia
```julia
@@ -470,24 +252,6 @@ env, env_json_str = smartsend("/chat/mixed", data, broker_url="nats://localhost:
For tabular data, NATSBridge uses Apache Arrow IPC format:
#### Python/Micropython
```python
from nats_bridge import smartsend
import pandas as pd
# Create DataFrame
df = pd.DataFrame({
"id": [1, 2, 3],
"name": ["Alice", "Bob", "Charlie"],
"score": [95, 88, 92]
})
# Send as table type
data = [("students", df, "table")]
env, env_json_str = smartsend("/data/students", data, broker_url="nats://localhost:4222")
```
#### Julia
```julia
@@ -507,92 +271,10 @@ env, env_json_str = smartsend("/data/students", data, broker_url="nats://localho
---
## Cross-Platform Communication
NATSBridge enables seamless communication between different platforms:
### Julia ↔ JavaScript
#### Julia Sender
```julia
using NATSBridge
# Send dictionary from Julia to JavaScript
config = Dict("step_size" => 0.01, "iterations" => 1000)
data = [("config", config, "dictionary")]
env, env_json_str = smartsend("/analysis/config", data, broker_url="nats://localhost:4222")
```
#### JavaScript Receiver
```javascript
const { smartreceive } = require('./src/NATSBridge');
// Receive dictionary from Julia
const env = await smartreceive(msg);
for (const payload of env.payloads) {
if (payload.type === "dictionary") {
console.log("Received config:", payload.data);
// payload.data = { step_size: 0.01, iterations: 1000 }
}
}
```
### JavaScript ↔ Python
#### JavaScript Sender
```javascript
const { smartsend } = require('./src/NATSBridge');
const { env, env_json_str } = await smartsend("/data/transfer", [
{ dataname: "message", data: "Hello from JS!", type: "text" }
], { brokerUrl: "nats://localhost:4222" });
```
#### Python Receiver
```python
from nats_bridge import smartreceive
env = smartreceive(msg)
for dataname, data, type in env["payloads"]:
if type == "text":
print(f"Received from JS: {data}")
```
### Python ↔ Julia
#### Python Sender
```python
from nats_bridge import smartsend
data = [("message", "Hello from Python!", "text")]
env, env_json_str = smartsend("/chat/python", data, broker_url="nats://localhost:4222")
```
#### Julia Receiver
```julia
using NATSBridge
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff)
for (dataname, data, type) in env["payloads"]
if type == "text"
println("Received from Python: $data")
end
end
```
---
## Next Steps
1. **Explore the test directory** for more examples
2. **Check the documentation** for advanced configuration options
3. **Join the community** to share your use cases
---
@@ -613,7 +295,7 @@ end
### Serialization Errors
- Verify data type matches the specified type
- Check that binary data is in the correct format (bytes/Vector{UInt8})
- Check that binary data is in the correct format (Vector{UInt8})
---

1208
examples/walkthrough.md
View File

File diff suppressed because it is too large Load Diff

80
test/test_js_dict_receiver.js
View File

@@ -1,80 +0,0 @@
#!/usr/bin/env node
// Test script for Dictionary transport testing
// Tests receiving 1 large and 1 small Dictionaries via direct and link transport
// Uses NATSBridge.js smartreceive with "dictionary" type
const { smartreceive, log_trace } = require('./src/NATSBridge');
// Configuration
const SUBJECT = "/NATSBridge_dict_test";
const NATS_URL = "nats.yiem.cc";
// Helper: Log with correlation ID
function log_trace(message) {
const timestamp = new Date().toISOString();
console.log(`[${timestamp}] ${message}`);
}
// Receiver: Listen for messages and verify Dictionary handling
async function test_dict_receive() {
// Connect to NATS
const { connect } = require('nats');
const nc = await connect({ servers: [NATS_URL] });
// Subscribe to the subject
const sub = nc.subscribe(SUBJECT);
for await (const msg of sub) {
log_trace(`Received message on ${msg.subject}`);
// Use NATSBridge.smartreceive to handle the data
const result = await smartreceive(
msg,
{
maxRetries: 5,
baseDelay: 100,
maxDelay: 5000
}
);
// Result is an envelope dictionary with payloads field
// Access payloads with result.payloads
for (const { dataname, data, type } of result.payloads) {
if (typeof data === 'object' && data !== null && !Array.isArray(data)) {
log_trace(`Received Dictionary '${dataname}' of type ${type}`);
// Display dictionary contents
console.log(" Contents:");
for (const [key, value] of Object.entries(data)) {
console.log(` ${key} => ${value}`);
}
// Save to JSON file
const fs = require('fs');
const output_path = `./received_${dataname}.json`;
const json_str = JSON.stringify(data, null, 2);
fs.writeFileSync(output_path, json_str);
log_trace(`Saved Dictionary to ${output_path}`);
} else {
log_trace(`Received unexpected data type for '${dataname}': ${typeof data}`);
}
}
}
// Keep listening for 10 seconds
setTimeout(() => {
nc.close();
process.exit(0);
}, 120000);
}
// Run the test
console.log("Starting Dictionary transport test...");
console.log("Note: This receiver will wait for messages from the sender.");
console.log("Run test_js_to_js_dict_sender.js first to send test data.");
// Run receiver
console.log("testing smartreceive");
test_dict_receive();
console.log("Test completed.");

165
test/test_js_dict_sender.js
View File

@@ -1,165 +0,0 @@
#!/usr/bin/env node
// Test script for Dictionary transport testing
// Tests sending 1 large and 1 small Dictionaries via direct and link transport
// Uses NATSBridge.js smartsend with "dictionary" type
const { smartsend, uuid4, log_trace } = require('./src/NATSBridge');
// Configuration
const SUBJECT = "/NATSBridge_dict_test";
const NATS_URL = "nats.yiem.cc";
const FILESERVER_URL = "http://192.168.88.104:8080";
// Create correlation ID for tracing
const correlation_id = uuid4();
// Helper: Log with correlation ID
function log_trace(message) {
const timestamp = new Date().toISOString();
console.log(`[${timestamp}] [Correlation: ${correlation_id}] ${message}`);
}
// File upload handler for plik server
async function plik_upload_handler(fileserver_url, dataname, data, correlation_id) {
// Get upload ID
const url_getUploadID = `${fileserver_url}/upload`;
const headers = {
"Content-Type": "application/json"
};
const body = JSON.stringify({ OneShot: true });
let response = await fetch(url_getUploadID, {
method: "POST",
headers: headers,
body: body
});
if (!response.ok) {
throw new Error(`Failed to get upload ID: ${response.status} ${response.statusText}`);
}
const responseJson = await response.json();
const uploadid = responseJson.id;
const uploadtoken = responseJson.uploadToken;
// Upload file
const formData = new FormData();
const blob = new Blob([data], { type: "application/octet-stream" });
formData.append("file", blob, dataname);
response = await fetch(`${fileserver_url}/file/${uploadid}`, {
method: "POST",
headers: {
"X-UploadToken": uploadtoken
},
body: formData
});
if (!response.ok) {
throw new Error(`Failed to upload file: ${response.status} ${response.statusText}`);
}
const fileResponseJson = await response.json();
const fileid = fileResponseJson.id;
const url = `${fileserver_url}/file/${uploadid}/${fileid}/${encodeURIComponent(dataname)}`;
return {
status: response.status,
uploadid: uploadid,
fileid: fileid,
url: url
};
}
// Sender: Send Dictionaries via smartsend
async function test_dict_send() {
// Create a small Dictionary (will use direct transport)
const small_dict = {
name: "Alice",
age: 30,
scores: [95, 88, 92],
metadata: {
height: 155,
weight: 55
}
};
// Create a large Dictionary (will use link transport if > 1MB)
const large_dict_ids = [];
const large_dict_names = [];
const large_dict_scores = [];
const large_dict_categories = [];
for (let i = 0; i < 50000; i++) {
large_dict_ids.push(i + 1);
large_dict_names.push(`User_${i}`);
large_dict_scores.push(Math.floor(Math.random() * 100) + 1);
large_dict_categories.push(`Category_${Math.floor(Math.random() * 10) + 1}`);
}
const large_dict = {
ids: large_dict_ids,
names: large_dict_names,
scores: large_dict_scores,
categories: large_dict_categories,
metadata: {
source: "test_generator",
timestamp: new Date().toISOString()
}
};
// Test data 1: small Dictionary
const data1 = { dataname: "small_dict", data: small_dict, type: "dictionary" };
// Test data 2: large Dictionary
const data2 = { dataname: "large_dict", data: large_dict, type: "dictionary" };
// Use smartsend with dictionary type
// For small Dictionary: will use direct transport (JSON encoded)
// For large Dictionary: will use link transport (uploaded to fileserver)
const { env, env_json_str } = await smartsend(
SUBJECT,
[data1, data2],
{
natsUrl: NATS_URL,
fileserverUrl: FILESERVER_URL,
fileserverUploadHandler: plik_upload_handler,
sizeThreshold: 1_000_000,
correlationId: correlation_id,
msgPurpose: "chat",
senderName: "dict_sender",
receiverName: "",
receiverId: "",
replyTo: "",
replyToMsgId: "",
isPublish: true // Publish the message to NATS
}
);
log_trace(`Sent message with ${env.payloads.length} payloads`);
// Log transport type for each payload
for (let i = 0; i < env.payloads.length; i++) {
const payload = env.payloads[i];
log_trace(`Payload ${i + 1} ('${payload.dataname}'):`);
log_trace(` Transport: ${payload.transport}`);
log_trace(` Type: ${payload.type}`);
log_trace(` Size: ${payload.size} bytes`);
log_trace(` Encoding: ${payload.encoding}`);
if (payload.transport === "link") {
log_trace(` URL: ${payload.data}`);
}
}
}
// Run the test
console.log("Starting Dictionary transport test...");
console.log(`Correlation ID: ${correlation_id}`);
// Run sender
console.log("start smartsend for dictionaries");
test_dict_send();
console.log("Test completed.");

71
test/test_js_file_receiver.js
View File

@@ -1,71 +0,0 @@
#!/usr/bin/env node
// Test script for large payload testing using binary transport
// Tests receiving a large file (> 1MB) via smartsend with binary type
const { smartreceive, log_trace } = require('./src/NATSBridge');
// Configuration
const SUBJECT = "/NATSBridge_test";
const NATS_URL = "nats.yiem.cc";
// Helper: Log with correlation ID
function log_trace(message) {
const timestamp = new Date().toISOString();
console.log(`[${timestamp}] ${message}`);
}
// Receiver: Listen for messages and verify large payload handling
async function test_large_binary_receive() {
// Connect to NATS
const { connect } = require('nats');
const nc = await connect({ servers: [NATS_URL] });
// Subscribe to the subject
const sub = nc.subscribe(SUBJECT);
for await (const msg of sub) {
log_trace(`Received message on ${msg.subject}`);
// Use NATSBridge.smartreceive to handle the data
const result = await smartreceive(
msg,
{
maxRetries: 5,
baseDelay: 100,
maxDelay: 5000
}
);
// Result is an envelope dictionary with payloads field
// Access payloads with result.payloads
for (const { dataname, data, type } of result.payloads) {
if (data instanceof Uint8Array || Array.isArray(data)) {
const file_size = data.length;
log_trace(`Received ${file_size} bytes of binary data for '${dataname}' of type ${type}`);
// Save received data to a test file
const fs = require('fs');
const output_path = `./new_${dataname}`;
fs.writeFileSync(output_path, Buffer.from(data));
log_trace(`Saved received data to ${output_path}`);
} else {
log_trace(`Received unexpected data type for '${dataname}': ${typeof data}`);
}
}
}
// Keep listening for 10 seconds
setTimeout(() => {
nc.close();
process.exit(0);
}, 120000);
}
// Run the test
console.log("Starting large binary payload test...");
// Run receiver
console.log("testing smartreceive");
test_large_binary_receive();
console.log("Test completed.");

144
test/test_js_file_sender.js
View File

@@ -1,144 +0,0 @@
#!/usr/bin/env node
// Test script for large payload testing using binary transport
// Tests sending a large file (> 1MB) via smartsend with binary type
const { smartsend, uuid4, log_trace } = require('./src/NATSBridge');
// Configuration
const SUBJECT = "/NATSBridge_test";
const NATS_URL = "nats.yiem.cc";
const FILESERVER_URL = "http://192.168.88.104:8080";
// Create correlation ID for tracing
const correlation_id = uuid4();
// Helper: Log with correlation ID
function log_trace(message) {
const timestamp = new Date().toISOString();
console.log(`[${timestamp}] [Correlation: ${correlation_id}] ${message}`);
}
// File upload handler for plik server
async function plik_upload_handler(fileserver_url, dataname, data, correlation_id) {
log_trace(correlation_id, `Uploading ${dataname} to fileserver: ${fileserver_url}`);
// Step 1: Get upload ID and token
const url_getUploadID = `${fileserver_url}/upload`;
const headers = {
"Content-Type": "application/json"
};
const body = JSON.stringify({ OneShot: true });
let response = await fetch(url_getUploadID, {
method: "POST",
headers: headers,
body: body
});
if (!response.ok) {
throw new Error(`Failed to get upload ID: ${response.status} ${response.statusText}`);
}
const responseJson = await response.json();
const uploadid = responseJson.id;
const uploadtoken = responseJson.uploadToken;
// Step 2: Upload file data
const url_upload = `${fileserver_url}/file/${uploadid}`;
// Create multipart form data
const formData = new FormData();
const blob = new Blob([data], { type: "application/octet-stream" });
formData.append("file", blob, dataname);
response = await fetch(url_upload, {
method: "POST",
headers: {
"X-UploadToken": uploadtoken
},
body: formData
});
if (!response.ok) {
throw new Error(`Failed to upload file: ${response.status} ${response.statusText}`);
}
const fileResponseJson = await response.json();
const fileid = fileResponseJson.id;
// Build the download URL
const url = `${fileserver_url}/file/${uploadid}/${fileid}/${encodeURIComponent(dataname)}`;
log_trace(correlation_id, `Uploaded to URL: ${url}`);
return {
status: response.status,
uploadid: uploadid,
fileid: fileid,
url: url
};
}
// Sender: Send large binary file via smartsend
async function test_large_binary_send() {
// Read the large file as binary data
const fs = require('fs');
// Test data 1
const file_path1 = './testFile_large.zip';
const file_data1 = fs.readFileSync(file_path1);
const filename1 = 'testFile_large.zip';
const data1 = { dataname: filename1, data: file_data1, type: "binary" };
// Test data 2
const file_path2 = './testFile_small.zip';
const file_data2 = fs.readFileSync(file_path2);
const filename2 = 'testFile_small.zip';
const data2 = { dataname: filename2, data: file_data2, type: "binary" };
// Use smartsend with binary type - will automatically use link transport
// if file size exceeds the threshold (1MB by default)
const { env, env_json_str } = await smartsend(
SUBJECT,
[data1, data2],
{
natsUrl: NATS_URL,
fileserverUrl: FILESERVER_URL,
fileserverUploadHandler: plik_upload_handler,
sizeThreshold: 1_000_000,
correlationId: correlation_id,
msgPurpose: "chat",
senderName: "sender",
receiverName: "",
receiverId: "",
replyTo: "",
replyToMsgId: "",
isPublish: true // Publish the message to NATS
}
);
log_trace(`Sent message with transport: ${env.payloads[0].transport}`);
log_trace(`Envelope type: ${env.payloads[0].type}`);
// Check if link transport was used
if (env.payloads[0].transport === "link") {
log_trace("Using link transport - file uploaded to HTTP server");
log_trace(`URL: ${env.payloads[0].data}`);
} else {
log_trace("Using direct transport - payload sent via NATS");
}
}
// Run the test
console.log("Starting large binary payload test...");
console.log(`Correlation ID: ${correlation_id}`);
// Run sender first
console.log("start smartsend");
test_large_binary_send();
// Run receiver
// console.log("testing smartreceive");
// test_large_binary_receive();
console.log("Test completed.");

277
test/test_js_mix_payload_sender.js
View File

@@ -1,277 +0,0 @@
#!/usr/bin/env node
// Test script for mixed-content message testing
// Tests sending a mix of text, json, table, image, audio, video, and binary data
// from JavaScript serviceA to JavaScript serviceB using NATSBridge.js smartsend
//
// This test demonstrates that any combination and any number of mixed content
// can be sent and received correctly.
const { smartsend, uuid4, log_trace, _serialize_data } = require('./src/NATSBridge');
// Configuration
const SUBJECT = "/NATSBridge_mix_test";
const NATS_URL = "nats.yiem.cc";
const FILESERVER_URL = "http://192.168.88.104:8080";
// Create correlation ID for tracing
const correlation_id = uuid4();
// Helper: Log with correlation ID
function log_trace(message) {
const timestamp = new Date().toISOString();
console.log(`[${timestamp}] [Correlation: ${correlation_id}] ${message}`);
}
// File upload handler for plik server
async function plik_upload_handler(fileserver_url, dataname, data, correlation_id) {
log_trace(correlation_id, `Uploading ${dataname} to fileserver: ${fileserver_url}`);
// Step 1: Get upload ID and token
const url_getUploadID = `${fileserver_url}/upload`;
const headers = {
"Content-Type": "application/json"
};
const body = JSON.stringify({ OneShot: true });
let response = await fetch(url_getUploadID, {
method: "POST",
headers: headers,
body: body
});
if (!response.ok) {
throw new Error(`Failed to get upload ID: ${response.status} ${response.statusText}`);
}
const responseJson = await response.json();
const uploadid = responseJson.id;
const uploadtoken = responseJson.uploadToken;
// Step 2: Upload file data
const url_upload = `${fileserver_url}/file/${uploadid}`;
// Create multipart form data
const formData = new FormData();
const blob = new Blob([data], { type: "application/octet-stream" });
formData.append("file", blob, dataname);
response = await fetch(url_upload, {
method: "POST",
headers: {
"X-UploadToken": uploadtoken
},
body: formData
});
if (!response.ok) {
throw new Error(`Failed to upload file: ${response.status} ${response.statusText}`);
}
const fileResponseJson = await response.json();
const fileid = fileResponseJson.id;
// Build the download URL
const url = `${fileserver_url}/file/${uploadid}/${fileid}/${encodeURIComponent(dataname)}`;
log_trace(correlation_id, `Uploaded to URL: ${url}`);
return {
status: response.status,
uploadid: uploadid,
fileid: fileid,
url: url
};
}
// Helper: Create sample data for each type
function create_sample_data() {
// Text data (small - direct transport)
const text_data = "Hello! This is a test chat message. 🎉\nHow are you doing today? 😊";
// Dictionary/JSON data (medium - could be direct or link)
const dict_data = {
type: "chat",
sender: "serviceA",
receiver: "serviceB",
metadata: {
timestamp: new Date().toISOString(),
priority: "high",
tags: ["urgent", "chat", "test"]
},
content: {
text: "This is a JSON-formatted chat message with nested structure.",
format: "markdown",
mentions: ["user1", "user2"]
}
};
// Table data (small - direct transport) - NOT IMPLEMENTED (requires apache-arrow)
// const table_data_small = {...};
// Table data (large - link transport) - NOT IMPLEMENTED (requires apache-arrow)
// const table_data_large = {...};
// Image data (small binary - direct transport)
// Create a simple 10x10 pixel PNG-like data
const image_width = 10;
const image_height = 10;
let image_data = new Uint8Array(128); // PNG header + pixel data
// PNG header
image_data[0] = 0x89;
image_data[1] = 0x50;
image_data[2] = 0x4E;
image_data[3] = 0x47;
image_data[4] = 0x0D;
image_data[5] = 0x0A;
image_data[6] = 0x1A;
image_data[7] = 0x0A;
// Simple RGB data (10*10*3 = 300 bytes)
for (let i = 0; i < 300; i++) {
image_data[i + 8] = 0xFF; // Red pixel
}
// Image data (large - link transport)
const large_image_width = 500;
const large_image_height = 1000;
const large_image_data = new Uint8Array(large_image_width * large_image_height * 3 + 8);
// PNG header
large_image_data[0] = 0x89;
large_image_data[1] = 0x50;
large_image_data[2] = 0x4E;
large_image_data[3] = 0x47;
large_image_data[4] = 0x0D;
large_image_data[5] = 0x0A;
large_image_data[6] = 0x1A;
large_image_data[7] = 0x0A;
// Random RGB data
for (let i = 0; i < large_image_width * large_image_height * 3; i++) {
large_image_data[i + 8] = Math.floor(Math.random() * 255);
}
// Audio data (small binary - direct transport)
const audio_data = new Uint8Array(100);
for (let i = 0; i < 100; i++) {
audio_data[i] = Math.floor(Math.random() * 255);
}
// Audio data (large - link transport)
const large_audio_data = new Uint8Array(1_500_000);
for (let i = 0; i < 1_500_000; i++) {
large_audio_data[i] = Math.floor(Math.random() * 255);
}
// Video data (small binary - direct transport)
const video_data = new Uint8Array(150);
for (let i = 0; i < 150; i++) {
video_data[i] = Math.floor(Math.random() * 255);
}
// Video data (large - link transport)
const large_video_data = new Uint8Array(1_500_000);
for (let i = 0; i < 1_500_000; i++) {
large_video_data[i] = Math.floor(Math.random() * 255);
}
// Binary data (small - direct transport)
const binary_data = new Uint8Array(200);
for (let i = 0; i < 200; i++) {
binary_data[i] = Math.floor(Math.random() * 255);
}
// Binary data (large - link transport)
const large_binary_data = new Uint8Array(1_500_000);
for (let i = 0; i < 1_500_000; i++) {
large_binary_data[i] = Math.floor(Math.random() * 255);
}
return {
text_data,
dict_data,
// table_data_small,
// table_data_large,
image_data,
large_image_data,
audio_data,
large_audio_data,
video_data,
large_video_data,
binary_data,
large_binary_data
};
}
// Sender: Send mixed content via smartsend
async function test_mix_send() {
// Create sample data
const { text_data, dict_data, image_data, large_image_data, audio_data, large_audio_data, video_data, large_video_data, binary_data, large_binary_data } = create_sample_data();
// Create payloads list - mixed content with both small and large data
// Small data uses direct transport, large data uses link transport
const payloads = [
// Small data (direct transport) - text, dictionary
{ dataname: "chat_text", data: text_data, type: "text" },
{ dataname: "chat_json", data: dict_data, type: "dictionary" },
// { dataname: "chat_table_small", data: table_data_small, type: "table" },
// Large data (link transport) - large image, large audio, large video, large binary
// { dataname: "chat_table_large", data: table_data_large, type: "table" },
{ dataname: "user_image_large", data: large_image_data, type: "image" },
{ dataname: "audio_clip_large", data: large_audio_data, type: "audio" },
{ dataname: "video_clip_large", data: large_video_data, type: "video" },
{ dataname: "binary_file_large", data: large_binary_data, type: "binary" }
];
// Use smartsend with mixed content
const { env, env_json_str } = await smartsend(
SUBJECT,
payloads,
{
natsUrl: NATS_URL,
fileserverUrl: FILESERVER_URL,
fileserverUploadHandler: plik_upload_handler,
sizeThreshold: 1_000_000,
correlationId: correlation_id,
msgPurpose: "chat",
senderName: "mix_sender",
receiverName: "",
receiverId: "",
replyTo: "",
replyToMsgId: "",
isPublish: true // Publish the message to NATS
}
);
log_trace(`Sent message with ${env.payloads.length} payloads`);
// Log transport type for each payload
for (let i = 0; i < env.payloads.length; i++) {
const payload = env.payloads[i];
log_trace(`Payload ${i + 1} ('${payload.dataname}'):`);
log_trace(` Transport: ${payload.transport}`);
log_trace(` Type: ${payload.type}`);
log_trace(` Size: ${payload.size} bytes`);
log_trace(` Encoding: ${payload.encoding}`);
if (payload.transport === "link") {
log_trace(` URL: ${payload.data}`);
}
}
// Summary
console.log("\n--- Transport Summary ---");
const direct_count = env.payloads.filter(p => p.transport === "direct").length;
const link_count = env.payloads.filter(p => p.transport === "link").length;
log_trace(`Direct transport: ${direct_count} payloads`);
log_trace(`Link transport: ${link_count} payloads`);
}
// Run the test
console.log("Starting mixed-content transport test...");
console.log(`Correlation ID: ${correlation_id}`);
// Run sender
console.log("start smartsend for mixed content");
test_mix_send();
console.log("\nTest completed.");
console.log("Note: Run test_js_to_js_mix_receiver.js to receive the messages.");

173
test/test_js_mix_payloads_receiver.js
View File

@@ -1,173 +0,0 @@
#!/usr/bin/env node
// Test script for mixed-content message testing
// Tests receiving a mix of text, json, table, image, audio, video, and binary data
// from JavaScript serviceA to JavaScript serviceB using NATSBridge.js smartreceive
//
// This test demonstrates that any combination and any number of mixed content
// can be sent and received correctly.
const { smartreceive, log_trace } = require('./src/NATSBridge');
// Configuration
const SUBJECT = "/NATSBridge_mix_test";
const NATS_URL = "nats.yiem.cc";
// Helper: Log with correlation ID
function log_trace(message) {
const timestamp = new Date().toISOString();
console.log(`[${timestamp}] ${message}`);
}
// Receiver: Listen for messages and verify mixed content handling
async function test_mix_receive() {
// Connect to NATS
const { connect } = require('nats');
const nc = await connect({ servers: [NATS_URL] });
// Subscribe to the subject
const sub = nc.subscribe(SUBJECT);
for await (const msg of sub) {
log_trace(`Received message on ${msg.subject}`);
// Use NATSBridge.smartreceive to handle the data
const result = await smartreceive(
msg,
{
maxRetries: 5,
baseDelay: 100,
maxDelay: 5000
}
);
log_trace(`Received ${result.payloads.length} payloads`);
// Result is an envelope dictionary with payloads field
// Access payloads with result.payloads
for (const { dataname, data, type } of result.payloads) {
log_trace(`\n=== Payload: ${dataname} (type: ${type}) ===`);
// Handle different data types
if (type === "text") {
// Text data - should be a String
if (typeof data === 'string') {
log_trace(` Type: String`);
log_trace(` Length: ${data.length} characters`);
// Display first 200 characters
if (data.length > 200) {
log_trace(` First 200 chars: ${data.substring(0, 200)}...`);
} else {
log_trace(` Content: ${data}`);
}
// Save to file
const fs = require('fs');
const output_path = `./received_${dataname}.txt`;
fs.writeFileSync(output_path, data);
log_trace(` Saved to: ${output_path}`);
} else {
log_trace(` ERROR: Expected String, got ${typeof data}`);
}
} else if (type === "dictionary") {
// Dictionary data - should be an object
if (typeof data === 'object' && data !== null && !Array.isArray(data)) {
log_trace(` Type: Object`);
log_trace(` Keys: ${Object.keys(data).join(', ')}`);
// Display nested content
for (const [key, value] of Object.entries(data)) {
log_trace(` ${key} => ${value}`);
}
// Save to JSON file
const fs = require('fs');
const output_path = `./received_${dataname}.json`;
const json_str = JSON.stringify(data, null, 2);
fs.writeFileSync(output_path, json_str);
log_trace(` Saved to: ${output_path}`);
} else {
log_trace(` ERROR: Expected Object, got ${typeof data}`);
}
} else if (type === "table") {
// Table data - should be an array of objects (requires apache-arrow)
log_trace(` Type: Array (requires apache-arrow for full deserialization)`);
if (Array.isArray(data)) {
log_trace(` Length: ${data.length} items`);
log_trace(` First item: ${JSON.stringify(data[0])}`);
} else {
log_trace(` ERROR: Expected Array, got ${typeof data}`);
}
} else if (type === "image" || type === "audio" || type === "video" || type === "binary") {
// Binary data - should be Uint8Array
if (data instanceof Uint8Array || Array.isArray(data)) {
log_trace(` Type: Uint8Array (binary)`);
log_trace(` Size: ${data.length} bytes`);
// Save to file
const fs = require('fs');
const output_path = `./received_${dataname}.bin`;
fs.writeFileSync(output_path, Buffer.from(data));
log_trace(` Saved to: ${output_path}`);
} else {
log_trace(` ERROR: Expected Uint8Array, got ${typeof data}`);
}
} else {
log_trace(` ERROR: Unknown data type '${type}'`);
}
}
// Summary
console.log("\n=== Verification Summary ===");
const text_count = result.payloads.filter(x => x.type === "text").length;
const dict_count = result.payloads.filter(x => x.type === "dictionary").length;
const table_count = result.payloads.filter(x => x.type === "table").length;
const image_count = result.payloads.filter(x => x.type === "image").length;
const audio_count = result.payloads.filter(x => x.type === "audio").length;
const video_count = result.payloads.filter(x => x.type === "video").length;
const binary_count = result.payloads.filter(x => x.type === "binary").length;
log_trace(`Text payloads: ${text_count}`);
log_trace(`Dictionary payloads: ${dict_count}`);
log_trace(`Table payloads: ${table_count}`);
log_trace(`Image payloads: ${image_count}`);
log_trace(`Audio payloads: ${audio_count}`);
log_trace(`Video payloads: ${video_count}`);
log_trace(`Binary payloads: ${binary_count}`);
// Print transport type info for each payload if available
console.log("\n=== Payload Details ===");
for (const { dataname, data, type } of result.payloads) {
if (["image", "audio", "video", "binary"].includes(type)) {
log_trace(`${dataname}: ${data.length} bytes (binary)`);
} else if (type === "table") {
log_trace(`${dataname}: ${data.length} items (Array)`);
} else if (type === "dictionary") {
log_trace(`${dataname}: ${JSON.stringify(data).length} bytes (Object)`);
} else if (type === "text") {
log_trace(`${dataname}: ${data.length} characters (String)`);
}
}
}
// Keep listening for 2 minutes
setTimeout(() => {
nc.close();
process.exit(0);
}, 120000);
}
// Run the test
console.log("Starting mixed-content transport test...");
console.log("Note: This receiver will wait for messages from the sender.");
console.log("Run test_js_to_js_mix_sender.js first to send test data.");
// Run receiver
console.log("\ntesting smartreceive for mixed content");
test_mix_receive();
console.log("\nTest completed.");

87
test/test_js_table_receiver.js
View File

@@ -1,87 +0,0 @@
#!/usr/bin/env node
// Test script for Table transport testing
// Tests receiving 1 large and 1 small Tables via direct and link transport
// Uses NATSBridge.js smartreceive with "table" type
//
// Note: This test requires the apache-arrow library to deserialize table data.
// The JavaScript implementation uses apache-arrow for Arrow IPC deserialization.
const { smartreceive, log_trace } = require('./src/NATSBridge');
// Configuration
const SUBJECT = "/NATSBridge_table_test";
const NATS_URL = "nats.yiem.cc";
// Helper: Log with correlation ID
function log_trace(message) {
const timestamp = new Date().toISOString();
console.log(`[${timestamp}] ${message}`);
}
// Receiver: Listen for messages and verify Table handling
async function test_table_receive() {
// Connect to NATS
const { connect } = require('nats');
const nc = await connect({ servers: [NATS_URL] });
// Subscribe to the subject
const sub = nc.subscribe(SUBJECT);
for await (const msg of sub) {
log_trace(`Received message on ${msg.subject}`);
// Use NATSBridge.smartreceive to handle the data
const result = await smartreceive(
msg,
{
maxRetries: 5,
baseDelay: 100,
maxDelay: 5000
}
);
// Result is an envelope dictionary with payloads field
// Access payloads with result.payloads
for (const { dataname, data, type } of result.payloads) {
if (Array.isArray(data)) {
log_trace(`Received Table '${dataname}' of type ${type}`);
// Display table contents
console.log(` Dimensions: ${data.length} rows x ${data.length > 0 ? Object.keys(data[0]).length : 0} columns`);
console.log(` Columns: ${data.length > 0 ? Object.keys(data[0]).join(', ') : ''}`);
// Display first few rows
console.log(` First 5 rows:`);
for (let i = 0; i < Math.min(5, data.length); i++) {
console.log(` Row ${i}: ${JSON.stringify(data[i])}`);
}
// Save to JSON file
const fs = require('fs');
const output_path = `./received_${dataname}.json`;
const json_str = JSON.stringify(data, null, 2);
fs.writeFileSync(output_path, json_str);
log_trace(`Saved Table to ${output_path}`);
} else {
log_trace(`Received unexpected data type for '${dataname}': ${typeof data}`);
}
}
}
// Keep listening for 10 seconds
setTimeout(() => {
nc.close();
process.exit(0);
}, 120000);
}
// Run the test
console.log("Starting Table transport test...");
console.log("Note: This receiver will wait for messages from the sender.");
console.log("Run test_js_to_js_table_sender.js first to send test data.");
// Run receiver
console.log("testing smartreceive");
test_table_receive();
console.log("Test completed.");

165
test/test_js_table_sender.js
View File

@@ -1,165 +0,0 @@
#!/usr/bin/env node
// Test script for Table transport testing
// Tests sending 1 large and 1 small Tables via direct and link transport
// Uses NATSBridge.js smartsend with "table" type
//
// Note: This test requires the apache-arrow library to serialize/deserialize table data.
// The JavaScript implementation uses apache-arrow for Arrow IPC serialization.
const { smartsend, uuid4, log_trace } = require('./src/NATSBridge');
// Configuration
const SUBJECT = "/NATSBridge_table_test";
const NATS_URL = "nats.yiem.cc";
const FILESERVER_URL = "http://192.168.88.104:8080";
// Create correlation ID for tracing
const correlation_id = uuid4();
// Helper: Log with correlation ID
function log_trace(message) {
const timestamp = new Date().toISOString();
console.log(`[${timestamp}] [Correlation: ${correlation_id}] ${message}`);
}
// File upload handler for plik server
async function plik_upload_handler(fileserver_url, dataname, data, correlation_id) {
log_trace(correlation_id, `Uploading ${dataname} to fileserver: ${fileserver_url}`);
// Step 1: Get upload ID and token
const url_getUploadID = `${fileserver_url}/upload`;
const headers = {
"Content-Type": "application/json"
};
const body = JSON.stringify({ OneShot: true });
let response = await fetch(url_getUploadID, {
method: "POST",
headers: headers,
body: body
});
if (!response.ok) {
throw new Error(`Failed to get upload ID: ${response.status} ${response.statusText}`);
}
const responseJson = await response.json();
const uploadid = responseJson.id;
const uploadtoken = responseJson.uploadToken;
// Step 2: Upload file data
const url_upload = `${fileserver_url}/file/${uploadid}`;
// Create multipart form data
const formData = new FormData();
const blob = new Blob([data], { type: "application/octet-stream" });
formData.append("file", blob, dataname);
response = await fetch(url_upload, {
method: "POST",
headers: {
"X-UploadToken": uploadtoken
},
body: formData
});
if (!response.ok) {
throw new Error(`Failed to upload file: ${response.status} ${response.statusText}`);
}
const fileResponseJson = await response.json();
const fileid = fileResponseJson.id;
// Build the download URL
const url = `${fileserver_url}/file/${uploadid}/${fileid}/${encodeURIComponent(dataname)}`;
log_trace(correlation_id, `Uploaded to URL: ${url}`);
return {
status: response.status,
uploadid: uploadid,
fileid: fileid,
url: url
};
}
// Sender: Send Tables via smartsend
async function test_table_send() {
// Note: This test requires apache-arrow library to create Arrow IPC data.
// For now, we'll use a simple array of objects as table data.
// In production, you would use the apache-arrow library to create Arrow IPC data.
// Create a small Table (will use direct transport)
const small_table = [
{ id: 1, name: "Alice", score: 95 },
{ id: 2, name: "Bob", score: 88 },
{ id: 3, name: "Charlie", score: 92 }
];
// Create a large Table (will use link transport if > 1MB)
// Generate a larger dataset (~2MB to ensure link transport)
const large_table = [];
for (let i = 0; i < 50000; i++) {
large_table.push({
id: i,
message: `msg_${i}`,
sender: `sender_${i}`,
timestamp: new Date().toISOString(),
priority: Math.floor(Math.random() * 3) + 1
});
}
// Test data 1: small Table
const data1 = { dataname: "small_table", data: small_table, type: "table" };
// Test data 2: large Table
const data2 = { dataname: "large_table", data: large_table, type: "table" };
// Use smartsend with table type
// For small Table: will use direct transport (Arrow IPC encoded)
// For large Table: will use link transport (uploaded to fileserver)
const { env, env_json_str } = await smartsend(
SUBJECT,
[data1, data2],
{
natsUrl: NATS_URL,
fileserverUrl: FILESERVER_URL,
fileserverUploadHandler: plik_upload_handler,
sizeThreshold: 1_000_000,
correlationId: correlation_id,
msgPurpose: "chat",
senderName: "table_sender",
receiverName: "",
receiverId: "",
replyTo: "",
replyToMsgId: "",
isPublish: true // Publish the message to NATS
}
);
log_trace(`Sent message with ${env.payloads.length} payloads`);
// Log transport type for each payload
for (let i = 0; i < env.payloads.length; i++) {
const payload = env.payloads[i];
log_trace(`Payload ${i + 1} ('${payload.dataname}'):`);
log_trace(` Transport: ${payload.transport}`);
log_trace(` Type: ${payload.type}`);
log_trace(` Size: ${payload.size} bytes`);
log_trace(` Encoding: ${payload.encoding}`);
if (payload.transport === "link") {
log_trace(` URL: ${payload.data}`);
}
}
}
// Run the test
console.log("Starting Table transport test...");
console.log(`Correlation ID: ${correlation_id}`);
// Run sender
console.log("start smartsend for tables");
test_table_send();
console.log("Test completed.");

81
test/test_js_text_receiver.js
View File

@@ -1,81 +0,0 @@
#!/usr/bin/env node
// Test script for text transport testing
// Tests receiving 1 large and 1 small text from JavaScript serviceA to JavaScript serviceB
// Uses NATSBridge.js smartreceive with "text" type
const { smartreceive, log_trace } = require('./src/NATSBridge');
// Configuration
const SUBJECT = "/NATSBridge_text_test";
const NATS_URL = "nats.yiem.cc";
// Helper: Log with correlation ID
function log_trace(message) {
const timestamp = new Date().toISOString();
console.log(`[${timestamp}] ${message}`);
}
// Receiver: Listen for messages and verify text handling
async function test_text_receive() {
// Connect to NATS
const { connect } = require('nats');
const nc = await connect({ servers: [NATS_URL] });
// Subscribe to the subject
const sub = nc.subscribe(SUBJECT);
for await (const msg of sub) {
log_trace(`Received message on ${msg.subject}`);
// Use NATSBridge.smartreceive to handle the data
const result = await smartreceive(
msg,
{
maxRetries: 5,
baseDelay: 100,
maxDelay: 5000
}
);
// Result is an envelope dictionary with payloads field
// Access payloads with result.payloads
for (const { dataname, data, type } of result.payloads) {
if (typeof data === 'string') {
log_trace(`Received text '${dataname}' of type ${type}`);
log_trace(` Length: ${data.length} characters`);
// Display first 100 characters
if (data.length > 100) {
log_trace(` First 100 characters: ${data.substring(0, 100)}...`);
} else {
log_trace(` Content: ${data}`);
}
// Save to file
const fs = require('fs');
const output_path = `./received_${dataname}.txt`;
fs.writeFileSync(output_path, data);
log_trace(`Saved text to ${output_path}`);
} else {
log_trace(`Received unexpected data type for '${dataname}': ${typeof data}`);
}
}
}
// Keep listening for 10 seconds
setTimeout(() => {
nc.close();
process.exit(0);
}, 120000);
}
// Run the test
console.log("Starting text transport test...");
console.log("Note: This receiver will wait for messages from the sender.");
console.log("Run test_js_to_js_text_sender.js first to send test data.");
// Run receiver
console.log("testing smartreceive for text");
test_text_receive();
console.log("Test completed.");

141
test/test_js_text_sender.js
View File

@@ -1,141 +0,0 @@
#!/usr/bin/env node
// Test script for text transport testing
// Tests sending 1 large and 1 small text from JavaScript serviceA to JavaScript serviceB
// Uses NATSBridge.js smartsend with "text" type
const { smartsend, uuid4, log_trace } = require('./src/NATSBridge');
// Configuration
const SUBJECT = "/NATSBridge_text_test";
const NATS_URL = "nats.yiem.cc";
const FILESERVER_URL = "http://192.168.88.104:8080";
// Create correlation ID for tracing
const correlation_id = uuid4();
// Helper: Log with correlation ID
function log_trace(message) {
const timestamp = new Date().toISOString();
console.log(`[${timestamp}] [Correlation: ${correlation_id}] ${message}`);
}
// File upload handler for plik server
async function plik_upload_handler(fileserver_url, dataname, data, correlation_id) {
// Get upload ID
const url_getUploadID = `${fileserver_url}/upload`;
const headers = {
"Content-Type": "application/json"
};
const body = JSON.stringify({ OneShot: true });
let response = await fetch(url_getUploadID, {
method: "POST",
headers: headers,
body: body
});
if (!response.ok) {
throw new Error(`Failed to get upload ID: ${response.status} ${response.statusText}`);
}
const responseJson = await response.json();
const uploadid = responseJson.id;
const uploadtoken = responseJson.uploadToken;
// Upload file
const formData = new FormData();
const blob = new Blob([data], { type: "application/octet-stream" });
formData.append("file", blob, dataname);
response = await fetch(`${fileserver_url}/file/${uploadid}`, {
method: "POST",
headers: {
"X-UploadToken": uploadtoken
},
body: formData
});
if (!response.ok) {
throw new Error(`Failed to upload file: ${response.status} ${response.statusText}`);
}
const fileResponseJson = await response.json();
const fileid = fileResponseJson.id;
const url = `${fileserver_url}/file/${uploadid}/${fileid}/${encodeURIComponent(dataname)}`;
return {
status: response.status,
uploadid: uploadid,
fileid: fileid,
url: url
};
}
// Sender: Send text via smartsend
async function test_text_send() {
// Create a small text (will use direct transport)
const small_text = "Hello, this is a small text message. Testing direct transport via NATS.";
// Create a large text (will use link transport if > 1MB)
// Generate a larger text (~2MB to ensure link transport)
const large_text_lines = [];
for (let i = 0; i < 50000; i++) {
large_text_lines.push(`Line ${i}: This is a sample text line with some content to pad the size. `);
}
const large_text = large_text_lines.join("");
// Test data 1: small text
const data1 = { dataname: "small_text", data: small_text, type: "text" };
// Test data 2: large text
const data2 = { dataname: "large_text", data: large_text, type: "text" };
// Use smartsend with text type
// For small text: will use direct transport (Base64 encoded UTF-8)
// For large text: will use link transport (uploaded to fileserver)
const { env, env_json_str } = await smartsend(
SUBJECT,
[data1, data2],
{
natsUrl: NATS_URL,
fileserverUrl: FILESERVER_URL,
fileserverUploadHandler: plik_upload_handler,
sizeThreshold: 1_000_000,
correlationId: correlation_id,
msgPurpose: "chat",
senderName: "text_sender",
receiverName: "",
receiverId: "",
replyTo: "",
replyToMsgId: "",
isPublish: true // Publish the message to NATS
}
);
log_trace(`Sent message with ${env.payloads.length} payloads`);
// Log transport type for each payload
for (let i = 0; i < env.payloads.length; i++) {
const payload = env.payloads[i];
log_trace(`Payload ${i + 1} ('${payload.dataname}'):`);
log_trace(` Transport: ${payload.transport}`);
log_trace(` Type: ${payload.type}`);
log_trace(` Size: ${payload.size} bytes`);
log_trace(` Encoding: ${payload.encoding}`);
if (payload.transport === "link") {
log_trace(` URL: ${payload.data}`);
}
}
}
// Run the test
console.log("Starting text transport test...");
console.log(`Correlation ID: ${correlation_id}`);
// Run sender
console.log("start smartsend for text");
test_text_send();
console.log("Test completed.");

207
test/test_micropython_basic.py
View File

@@ -1,207 +0,0 @@
#!/usr/bin/env python3
"""
Basic functionality test for nats_bridge.py
Tests the core classes and functions without NATS connection
"""
import sys
import os
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import (
MessagePayload,
MessageEnvelope,
smartsend,
smartreceive,
log_trace,
generate_uuid,
get_timestamp,
_serialize_data,
_deserialize_data
)
import json
def test_message_payload():
"""Test MessagePayload class"""
print("\n=== Testing MessagePayload ===")
# Test direct transport with text
payload1 = MessagePayload(
data="Hello World",
msg_type="text",
id="test-id-1",
dataname="message",
transport="direct",
encoding="base64",
size=11
)
assert payload1.id == "test-id-1"
assert payload1.dataname == "message"
assert payload1.type == "text"
assert payload1.transport == "direct"
assert payload1.encoding == "base64"
assert payload1.size == 11
print(" [PASS] MessagePayload with text data")
# Test link transport with URL
payload2 = MessagePayload(
data="http://example.com/file.txt",
msg_type="binary",
id="test-id-2",
dataname="file",
transport="link",
encoding="none",
size=1000
)
assert payload2.transport == "link"
assert payload2.data == "http://example.com/file.txt"
print(" [PASS] MessagePayload with link transport")
# Test to_dict method
payload_dict = payload1.to_dict()
assert "id" in payload_dict
assert "dataname" in payload_dict
assert "type" in payload_dict
assert "transport" in payload_dict
assert "data" in payload_dict
print(" [PASS] MessagePayload.to_dict() method")
def test_message_envelope():
"""Test MessageEnvelope class"""
print("\n=== Testing MessageEnvelope ===")
# Create payloads
payload1 = MessagePayload("Hello", "text", id="p1", dataname="msg1")
payload2 = MessagePayload("http://example.com/file", "binary", id="p2", dataname="file", transport="link")
# Create envelope
env = MessageEnvelope(
send_to="/test/subject",
payloads=[payload1, payload2],
correlation_id="test-correlation-id",
msg_id="test-msg-id",
msg_purpose="chat",
sender_name="test_sender",
receiver_name="test_receiver",
reply_to="/test/reply"
)
assert env.send_to == "/test/subject"
assert env.correlation_id == "test-correlation-id"
assert env.msg_id == "test-msg-id"
assert env.msg_purpose == "chat"
assert len(env.payloads) == 2
print(" [PASS] MessageEnvelope creation")
# Test to_json method
json_str = env.to_json()
json_data = json.loads(json_str)
assert json_data["sendTo"] == "/test/subject"
assert json_data["correlationId"] == "test-correlation-id"
assert json_data["msgPurpose"] == "chat"
assert len(json_data["payloads"]) == 2
print(" [PASS] MessageEnvelope.to_json() method")
def test_serialize_data():
"""Test _serialize_data function"""
print("\n=== Testing _serialize_data ===")
# Test text serialization
text_bytes = _serialize_data("Hello", "text")
assert isinstance(text_bytes, bytes)
assert text_bytes == b"Hello"
print(" [PASS] Text serialization")
# Test dictionary serialization
dict_data = {"key": "value", "number": 42}
dict_bytes = _serialize_data(dict_data, "dictionary")
assert isinstance(dict_bytes, bytes)
parsed = json.loads(dict_bytes.decode('utf-8'))
assert parsed["key"] == "value"
print(" [PASS] Dictionary serialization")
# Test binary serialization
binary_data = b"\x00\x01\x02"
binary_bytes = _serialize_data(binary_data, "binary")
assert binary_bytes == b"\x00\x01\x02"
print(" [PASS] Binary serialization")
# Test image serialization
image_data = bytes([1, 2, 3, 4, 5])
image_bytes = _serialize_data(image_data, "image")
assert image_bytes == image_data
print(" [PASS] Image serialization")
def test_deserialize_data():
"""Test _deserialize_data function"""
print("\n=== Testing _deserialize_data ===")
# Test text deserialization
text_bytes = b"Hello"
text_data = _deserialize_data(text_bytes, "text", "test-correlation-id")
assert text_data == "Hello"
print(" [PASS] Text deserialization")
# Test dictionary deserialization
dict_bytes = b'{"key": "value"}'
dict_data = _deserialize_data(dict_bytes, "dictionary", "test-correlation-id")
assert dict_data == {"key": "value"}
print(" [PASS] Dictionary deserialization")
# Test binary deserialization
binary_data = b"\x00\x01\x02"
binary_result = _deserialize_data(binary_data, "binary", "test-correlation-id")
assert binary_result == b"\x00\x01\x02"
print(" [PASS] Binary deserialization")
def test_utilities():
"""Test utility functions"""
print("\n=== Testing Utility Functions ===")
# Test generate_uuid
uuid1 = generate_uuid()
uuid2 = generate_uuid()
assert uuid1 != uuid2
print(f" [PASS] generate_uuid() - generated: {uuid1}")
# Test get_timestamp
timestamp = get_timestamp()
assert "T" in timestamp
print(f" [PASS] get_timestamp() - generated: {timestamp}")
def main():
"""Run all tests"""
print("=" * 60)
print("NATSBridge Python/Micropython - Basic Functionality Tests")
print("=" * 60)
try:
test_message_payload()
test_message_envelope()
test_serialize_data()
test_deserialize_data()
test_utilities()
print("\n" + "=" * 60)
print("ALL TESTS PASSED!")
print("=" * 60)
except Exception as e:
print(f"\n[FAIL] Test failed with error: {e}")
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == "__main__":
main()

70
test/test_micropython_dict_receiver.py
View File

@@ -1,70 +0,0 @@
#!/usr/bin/env python3
"""
Test script for dictionary transport testing - Receiver
Tests receiving dictionary messages via NATS using nats_bridge.py smartreceive
"""
import sys
import os
import json
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartreceive, log_trace
import nats
import asyncio
# Configuration
SUBJECT = "/NATSBridge_dict_test"
NATS_URL = "nats://nats.yiem.cc:4222"
async def main():
log_trace("", f"Starting dictionary transport receiver test...")
log_trace("", f"Note: This receiver will wait for messages from the sender.")
log_trace("", f"Run test_micropython_dict_sender.py first to send test data.")
# Connect to NATS
nc = await nats.connect(NATS_URL)
log_trace("", f"Connected to NATS at {NATS_URL}")
# Subscribe to the subject
async def message_handler(msg):
log_trace("", f"Received message on {msg.subject}")
# Use smartreceive to handle the data
result = smartreceive(msg.data)
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for dataname, data, data_type in result["payloads"]:
if isinstance(data, dict):
log_trace(result.get("correlationId", ""), f"Received dictionary '{dataname}' of type {data_type}")
log_trace(result.get("correlationId", ""), f" Keys: {list(data.keys())}")
# Display first few items for small dicts
if isinstance(data, dict) and len(data) <= 10:
log_trace(result.get("correlationId", ""), f" Content: {json.dumps(data, indent=2)}")
else:
# For large dicts, show summary
log_trace(result.get("correlationId", ""), f" Summary: {json.dumps(data, default=str)[:200]}...")
# Save to file
output_path = f"./received_{dataname}.json"
with open(output_path, 'w') as f:
json.dump(data, f, indent=2)
log_trace(result.get("correlationId", ""), f"Saved dictionary to {output_path}")
else:
log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}")
sid = await nc.subscribe(SUBJECT, cb=message_handler)
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
# Keep listening for 120 seconds
await asyncio.sleep(120)
await nc.close()
log_trace("", "Test completed.")
if __name__ == "__main__":
asyncio.run(main())

100
test/test_micropython_dict_sender.py
View File

@@ -1,100 +0,0 @@
#!/usr/bin/env python3
"""
Test script for dictionary transport testing - Micropython
Tests sending dictionary messages via NATS using nats_bridge.py smartsend
"""
import sys
import os
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartsend, log_trace
import uuid
# Configuration
SUBJECT = "/NATSBridge_dict_test"
NATS_URL = "nats://nats.yiem.cc:4222"
FILESERVER_URL = "http://192.168.88.104:8080"
SIZE_THRESHOLD = 1_000_000 # 1MB
# Create correlation ID for tracing
correlation_id = str(uuid.uuid4())
def main():
# Create a small dictionary (will use direct transport)
small_dict = {
"name": "test",
"value": 42,
"enabled": True,
"metadata": {
"version": "1.0.0",
"timestamp": "2026-02-22T12:00:00Z"
}
}
# Create a large dictionary (will use link transport if > 1MB)
# Generate a larger dictionary (~2MB to ensure link transport)
large_dict = {
"id": str(uuid.uuid4()),
"items": [
{
"index": i,
"name": f"item_{i}",
"value": i * 1.5,
"data": "x" * 10000 # Large string per item
}
for i in range(200)
],
"metadata": {
"count": 200,
"created": "2026-02-22T12:00:00Z"
}
}
# Test data 1: small dictionary
data1 = ("small_dict", small_dict, "dictionary")
# Test data 2: large dictionary
data2 = ("large_dict", large_dict, "dictionary")
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
# Use smartsend with dictionary type
env, env_json_str = smartsend(
SUBJECT,
[data1, data2], # List of (dataname, data, type) tuples
nats_url=NATS_URL,
fileserver_url=FILESERVER_URL,
size_threshold=SIZE_THRESHOLD,
correlation_id=correlation_id,
msg_purpose="chat",
sender_name="dict_sender",
receiver_name="",
receiver_id="",
reply_to="",
reply_to_msg_id="",
is_publish=True # Publish the message to NATS
)
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
# Log transport type for each payload
for i, payload in enumerate(env.payloads):
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
log_trace(correlation_id, f" Transport: {payload.transport}")
log_trace(correlation_id, f" Type: {payload.type}")
log_trace(correlation_id, f" Size: {payload.size} bytes")
log_trace(correlation_id, f" Encoding: {payload.encoding}")
if payload.transport == "link":
log_trace(correlation_id, f" URL: {payload.data}")
print(f"Test completed. Correlation ID: {correlation_id}")
if __name__ == "__main__":
main()

65
test/test_micropython_file_receiver.py
View File

@@ -1,65 +0,0 @@
#!/usr/bin/env python3
"""
Test script for file transport testing - Receiver
Tests receiving binary files via NATS using nats_bridge.py smartreceive
"""
import sys
import os
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartreceive, log_trace
import nats
import asyncio
# Configuration
SUBJECT = "/NATSBridge_file_test"
NATS_URL = "nats://nats.yiem.cc:4222"
async def main():
log_trace("", f"Starting file transport receiver test...")
log_trace("", f"Note: This receiver will wait for messages from the sender.")
log_trace("", f"Run test_micropython_file_sender.py first to send test data.")
# Connect to NATS
nc = await nats.connect(NATS_URL)
log_trace("", f"Connected to NATS at {NATS_URL}")
# Subscribe to the subject
async def message_handler(msg):
log_trace("", f"Received message on {msg.subject}")
# Use smartreceive to handle the data
result = smartreceive(msg.data)
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for dataname, data, data_type in result["payloads"]:
if isinstance(data, bytes):
log_trace(result.get("correlationId", ""), f"Received binary '{dataname}' of type {data_type}")
log_trace(result.get("correlationId", ""), f" Size: {len(data)} bytes")
# Display first 100 bytes as hex
log_trace(result.get("correlationId", ""), f" First 100 bytes (hex): {data[:100].hex()}")
# Save to file
output_path = f"./received_{dataname}.bin"
with open(output_path, 'wb') as f:
f.write(data)
log_trace(result.get("correlationId", ""), f"Saved binary to {output_path}")
else:
log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}")
sid = await nc.subscribe(SUBJECT, cb=message_handler)
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
# Keep listening for 120 seconds
await asyncio.sleep(120)
await nc.close()
log_trace("", "Test completed.")
if __name__ == "__main__":
asyncio.run(main())

80
test/test_micropython_file_sender.py
View File

@@ -1,80 +0,0 @@
#!/usr/bin/env python3
"""
Test script for file transport testing - Micropython
Tests sending binary files via NATS using nats_bridge.py smartsend
"""
import sys
import os
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartsend, log_trace
import uuid
# Configuration
SUBJECT = "/NATSBridge_file_test"
NATS_URL = "nats://nats.yiem.cc:4222"
FILESERVER_URL = "http://192.168.88.104:8080"
SIZE_THRESHOLD = 1_000_000 # 1MB
# Create correlation ID for tracing
correlation_id = str(uuid.uuid4())
def main():
# Create small binary data (will use direct transport)
small_binary = b"This is small binary data for testing direct transport."
small_binary += b"\x00" * 100 # Add some null bytes
# Create large binary data (will use link transport if > 1MB)
# Generate a larger binary (~2MB to ensure link transport)
large_binary = bytes([
(i * 7) % 256 for i in range(2_000_000)
])
# Test data 1: small binary (direct transport)
data1 = ("small_binary", small_binary, "binary")
# Test data 2: large binary (link transport)
data2 = ("large_binary", large_binary, "binary")
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
# Use smartsend with binary type
env, env_json_str = smartsend(
SUBJECT,
[data1, data2], # List of (dataname, data, type) tuples
nats_url=NATS_URL,
fileserver_url=FILESERVER_URL,
size_threshold=SIZE_THRESHOLD,
correlation_id=correlation_id,
msg_purpose="chat",
sender_name="file_sender",
receiver_name="",
receiver_id="",
reply_to="",
reply_to_msg_id="",
is_publish=True # Publish the message to NATS
)
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
# Log transport type for each payload
for i, payload in enumerate(env.payloads):
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
log_trace(correlation_id, f" Transport: {payload.transport}")
log_trace(correlation_id, f" Type: {payload.type}")
log_trace(correlation_id, f" Size: {payload.size} bytes")
log_trace(correlation_id, f" Encoding: {payload.encoding}")
if payload.transport == "link":
log_trace(correlation_id, f" URL: {payload.data}")
print(f"Test completed. Correlation ID: {correlation_id}")
if __name__ == "__main__":
main()

97
test/test_micropython_mixed_receiver.py
View File

@@ -1,97 +0,0 @@
#!/usr/bin/env python3
"""
Test script for mixed payload testing - Receiver
Tests receiving mixed payload types via NATS using nats_bridge.py smartreceive
"""
import sys
import os
import json
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartreceive, log_trace
import nats
import asyncio
# Configuration
SUBJECT = "/NATSBridge_mixed_test"
NATS_URL = "nats://nats.yiem.cc:4222"
async def main():
log_trace("", f"Starting mixed payload receiver test...")
log_trace("", f"Note: This receiver will wait for messages from the sender.")
log_trace("", f"Run test_micropython_mixed_sender.py first to send test data.")
# Connect to NATS
nc = await nats.connect(NATS_URL)
log_trace("", f"Connected to NATS at {NATS_URL}")
# Subscribe to the subject
async def message_handler(msg):
log_trace("", f"Received message on {msg.subject}")
# Use smartreceive to handle the data
result = smartreceive(msg.data)
log_trace(result.get("correlationId", ""), f"Received envelope with {len(result['payloads'])} payloads")
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for dataname, data, data_type in result["payloads"]:
log_trace(result.get("correlationId", ""), f"\n--- Payload: {dataname} (type: {data_type}) ---")
if isinstance(data, str):
log_trace(result.get("correlationId", ""), f" Type: text/string")
log_trace(result.get("correlationId", ""), f" Length: {len(data)} characters")
if len(data) <= 100:
log_trace(result.get("correlationId", ""), f" Content: {data}")
else:
log_trace(result.get("correlationId", ""), f" First 100 chars: {data[:100]}...")
# Save to file
output_path = f"./received_{dataname}.txt"
with open(output_path, 'w') as f:
f.write(data)
log_trace(result.get("correlationId", ""), f" Saved to: {output_path}")
elif isinstance(data, dict):
log_trace(result.get("correlationId", ""), f" Type: dictionary")
log_trace(result.get("correlationId", ""), f" Keys: {list(data.keys())}")
log_trace(result.get("correlationId", ""), f" Content: {json.dumps(data, indent=2)}")
# Save to file
output_path = f"./received_{dataname}.json"
with open(output_path, 'w') as f:
json.dump(data, f, indent=2)
log_trace(result.get("correlationId", ""), f" Saved to: {output_path}")
elif isinstance(data, bytes):
log_trace(result.get("correlationId", ""), f" Type: binary")
log_trace(result.get("correlationId", ""), f" Size: {len(data)} bytes")
log_trace(result.get("correlationId", ""), f" First 100 bytes (hex): {data[:100].hex()}")
# Save to file
output_path = f"./received_{dataname}.bin"
with open(output_path, 'wb') as f:
f.write(data)
log_trace(result.get("correlationId", ""), f" Saved to: {output_path}")
else:
log_trace(result.get("correlationId", ""), f" Received unexpected data type: {type(data)}")
# Log envelope metadata
log_trace(result.get("correlationId", ""), f"\n--- Envelope Metadata ---")
log_trace(result.get("correlationId", ""), f" Correlation ID: {result.get('correlationId', 'N/A')}")
log_trace(result.get("correlationId", ""), f" Message ID: {result.get('msgId', 'N/A')}")
log_trace(result.get("correlationId", ""), f" Sender: {result.get('senderName', 'N/A')}")
log_trace(result.get("correlationId", ""), f" Purpose: {result.get('msgPurpose', 'N/A')}")
sid = await nc.subscribe(SUBJECT, cb=message_handler)
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
# Keep listening for 120 seconds
await asyncio.sleep(120)
await nc.close()
log_trace("", "Test completed.")
if __name__ == "__main__":
asyncio.run(main())

94
test/test_micropython_mixed_sender.py
View File

@@ -1,94 +0,0 @@
#!/usr/bin/env python3
"""
Test script for mixed payload testing - Micropython
Tests sending mixed payload types via NATS using nats_bridge.py smartsend
"""
import sys
import os
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartsend, log_trace
import uuid
# Configuration
SUBJECT = "/NATSBridge_mixed_test"
NATS_URL = "nats://nats.yiem.cc:4222"
FILESERVER_URL = "http://192.168.88.104:8080"
SIZE_THRESHOLD = 1_000_000 # 1MB
# Create correlation ID for tracing
correlation_id = str(uuid.uuid4())
def main():
# Create payloads for mixed content test
# 1. Small text (direct transport)
text_data = "Hello, this is a text message for testing mixed payloads!"
# 2. Small dictionary (direct transport)
dict_data = {
"status": "ok",
"code": 200,
"message": "Test successful",
"items": [1, 2, 3]
}
# 3. Small binary (direct transport)
binary_data = b"\x00\x01\x02\x03\x04\x05" + b"\xff" * 100
# 4. Large text (link transport - will use fileserver)
large_text = "\n".join([
f"Line {i}: This is a large text payload for link transport testing. " * 50
for i in range(100)
])
# Test data list - mixed payload types
data = [
("message_text", text_data, "text"),
("config_dict", dict_data, "dictionary"),
("small_binary", binary_data, "binary"),
("large_text", large_text, "text"),
]
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
# Use smartsend with mixed types
env, env_json_str = smartsend(
SUBJECT,
data, # List of (dataname, data, type) tuples
nats_url=NATS_URL,
fileserver_url=FILESERVER_URL,
size_threshold=SIZE_THRESHOLD,
correlation_id=correlation_id,
msg_purpose="chat",
sender_name="mixed_sender",
receiver_name="",
receiver_id="",
reply_to="",
reply_to_msg_id="",
is_publish=True # Publish the message to NATS
)
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
# Log transport type for each payload
for i, payload in enumerate(env.payloads):
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
log_trace(correlation_id, f" Transport: {payload.transport}")
log_trace(correlation_id, f" Type: {payload.type}")
log_trace(correlation_id, f" Size: {payload.size} bytes")
log_trace(correlation_id, f" Encoding: {payload.encoding}")
if payload.transport == "link":
log_trace(correlation_id, f" URL: {payload.data}")
print(f"Test completed. Correlation ID: {correlation_id}")
if __name__ == "__main__":
main()

69
test/test_micropython_text_receiver.py
View File

@@ -1,69 +0,0 @@
#!/usr/bin/env python3
"""
Test script for text transport testing - Receiver
Tests receiving text messages via NATS using nats_bridge.py smartreceive
"""
import sys
import os
import json
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartreceive, log_trace
import nats
import asyncio
# Configuration
SUBJECT = "/NATSBridge_text_test"
NATS_URL = "nats://nats.yiem.cc:4222"
async def main():
log_trace("", f"Starting text transport receiver test...")
log_trace("", f"Note: This receiver will wait for messages from the sender.")
log_trace("", f"Run test_micropython_text_sender.py first to send test data.")
# Connect to NATS
nc = await nats.connect(NATS_URL)
log_trace("", f"Connected to NATS at {NATS_URL}")
# Subscribe to the subject
async def message_handler(msg):
log_trace("", f"Received message on {msg.subject}")
# Use smartreceive to handle the data
result = smartreceive(msg.data)
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for dataname, data, data_type in result["payloads"]:
if isinstance(data, str):
log_trace(result.get("correlationId", ""), f"Received text '{dataname}' of type {data_type}")
log_trace(result.get("correlationId", ""), f" Length: {len(data)} characters")
# Display first 100 characters
if len(data) > 100:
log_trace(result.get("correlationId", ""), f" First 100 characters: {data[:100]}...")
else:
log_trace(result.get("correlationId", ""), f" Content: {data}")
# Save to file
output_path = f"./received_{dataname}.txt"
with open(output_path, 'w') as f:
f.write(data)
log_trace(result.get("correlationId", ""), f"Saved text to {output_path}")
else:
log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}")
sid = await nc.subscribe(SUBJECT, cb=message_handler)
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
# Keep listening for 120 seconds
await asyncio.sleep(120)
await nc.close()
log_trace("", "Test completed.")
if __name__ == "__main__":
asyncio.run(main())

82
test/test_micropython_text_sender.py
View File

@@ -1,82 +0,0 @@
#!/usr/bin/env python3
"""
Test script for text transport testing - Micropython
Tests sending text messages via NATS using nats_bridge.py smartsend
"""
import sys
import os
# Add src to path for import
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
from nats_bridge import smartsend, log_trace
import uuid
# Configuration
SUBJECT = "/NATSBridge_text_test"
NATS_URL = "nats://nats.yiem.cc:4222"
FILESERVER_URL = "http://192.168.88.104:8080"
SIZE_THRESHOLD = 1_000_000 # 1MB
# Create correlation ID for tracing
correlation_id = str(uuid.uuid4())
def main():
# Create a small text (will use direct transport)
small_text = "Hello, this is a small text message. Testing direct transport via NATS."
# Create a large text (will use link transport if > 1MB)
# Generate a larger text (~2MB to ensure link transport)
large_text = "\n".join([
f"Line {i}: This is a sample text line with some content to pad the size. " * 100
for i in range(500)
])
# Test data 1: small text
data1 = ("small_text", small_text, "text")
# Test data 2: large text
data2 = ("large_text", large_text, "text")
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
# Use smartsend with text type
# For small text: will use direct transport (Base64 encoded UTF-8)
# For large text: will use link transport (uploaded to fileserver)
env, env_json_str = smartsend(
SUBJECT,
[data1, data2], # List of (dataname, data, type) tuples
nats_url=NATS_URL,
fileserver_url=FILESERVER_URL,
size_threshold=SIZE_THRESHOLD,
correlation_id=correlation_id,
msg_purpose="chat",
sender_name="text_sender",
receiver_name="",
receiver_id="",
reply_to="",
reply_to_msg_id="",
is_publish=True # Publish the message to NATS
)
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
# Log transport type for each payload
for i, payload in enumerate(env.payloads):
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
log_trace(correlation_id, f" Transport: {payload.transport}")
log_trace(correlation_id, f" Type: {payload.type}")
log_trace(correlation_id, f" Size: {payload.size} bytes")
log_trace(correlation_id, f" Encoding: {payload.encoding}")
if payload.transport == "link":
log_trace(correlation_id, f" URL: {payload.data}")
print(f"Test completed. Correlation ID: {correlation_id}")
if __name__ == "__main__":
main()