ton/NATSBridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

update

Browse Source
This commit is contained in:
narawat
2026-02-25 20:27:51 +07:00
parent bee9f783d9
commit f8d93991f5
24 changed files with 579 additions and 4981 deletions
Download Patch File Download Diff File Expand all files Collapse all files

580
docs/implementation.md
View File

@@ -2,22 +2,17 @@
## Overview
This document describes the implementation of the high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
This document describes the implementation of the high-performance, bi-directional data bridge for **Julia** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
The system enables seamless communication across all three platforms:
- **Julia ↔ JavaScript** bi-directional messaging
- **JavaScript ↔ Python/Micropython** bi-directional messaging
- **Julia ↔ Python/Micropython** bi-directional messaging (via JSON serialization)
The system enables seamless communication for Julia applications.
### Implementation Files
NATSBridge is implemented in three languages, each providing the same API:
NATSBridge is implemented in Julia:
| Language | Implementation File | Description |
|----------|---------------------|-------------|
| **Julia** | [`src/NATSBridge.jl`](../src/NATSBridge.jl) | Full Julia implementation with Arrow IPC support |
| **JavaScript** | [`src/NATSBridge.js`](../src/NATSBridge.js) | JavaScript implementation for Node.js and browsers |
| **Python/Micropython** | [`src/nats_bridge.py`](../src/nats_bridge.py) | Python implementation for desktop and microcontrollers |
### File Server Handler Architecture
@@ -117,64 +112,9 @@ env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff, max_ret
# env is a dictionary containing envelope metadata and payloads field
```
## Cross-Platform Interoperability
NATSBridge is designed for seamless communication between Julia, JavaScript, and Python/Micropython applications. All three implementations share the same interface and data format, ensuring compatibility across platforms.
### Platform-Specific Features
| Feature | Julia | JavaScript | Python/Micropython |
|---------|-------|------------|-------------------|
| Direct NATS transport | ✅ | ✅ | ✅ |
| HTTP file server (Claim-Check) | ✅ | ✅ | ✅ |
| Arrow IPC tables | ✅ | ✅ | ✅ |
| Base64 encoding | ✅ | ✅ | ✅ |
| Exponential backoff | ✅ | ✅ | ✅ |
| Correlation ID tracking | ✅ | ✅ | ✅ |
| Reply-to support | ✅ | ✅ | ✅ |
### Data Type Mapping
| Type | Julia | JavaScript | Python/Micropython |
|------|-------|------------|-------------------|
| `text` | `String` | `String` | `str` |
| `dictionary` | `Dict` | `Object` | `dict` |
| `table` | `DataFrame` | `Array<Object>` | `DataFrame` / `list` |
| `image` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
| `audio` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
| `video` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
| `binary` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
### Example: Julia ↔ Python ↔ JavaScript
```julia
# Julia sender - smartsend returns (env, env_json_str)
using NATSBridge
data = [("message", "Hello from Julia!", "text")]
env, env_json_str = smartsend("/cross_platform", data, broker_url="nats://localhost:4222")
# env: msg_envelope_v1 with all metadata and payloads
# env_json_str: JSON string for publishing
```
```javascript
// JavaScript receiver
const { smartreceive } = require('./src/NATSBridge');
const env = await smartreceive(msg);
// env.payloads[0].data === "Hello from Julia!"
```
```python
# Python sender
from nats_bridge import smartsend
data = [("response", "Hello from Python!", "text")]
smartsend("/cross_platform", data, broker_url="nats://localhost:4222")
```
All three platforms can communicate seamlessly using the same NATS subjects and data format.
## Architecture
All three implementations (Julia, JavaScript, Python/Micropython) follow the same Claim-Check pattern:
The Julia implementation follows the Claim-Check pattern:
```
┌─────────────────────────────────────────────────────────────────────────┐
@@ -227,24 +167,6 @@ The Julia implementation provides:
- **[`smartsend()`](src/NATSBridge.jl)**: Handles transport selection based on payload size
- **[`smartreceive()`](src/NATSBridge.jl)**: Handles both direct and link transport
### JavaScript Module: [`src/NATSBridge.js`](../src/NATSBridge.js)
The JavaScript implementation provides:
- **`MessageEnvelope` class**: For the unified JSON envelope
- **`MessagePayload` class**: For individual payload representation
- **[`smartsend()`](src/NATSBridge.js)**: Handles transport selection based on payload size
- **[`smartreceive()`](src/NATSBridge.js)**: Handles both direct and link transport
### Python/Micropython Module: [`src/nats_bridge.py`](../src/nats_bridge.py)
The Python/Micropython implementation provides:
- **`MessageEnvelope` class**: For the unified JSON envelope
- **`MessagePayload` class**: For individual payload representation
- **[`smartsend()`](src/nats_bridge.py)**: Handles transport selection based on payload size
- **[`smartreceive()`](src/nats_bridge.py)**: Handles both direct and link transport
## Installation
### Julia Dependencies
@@ -259,29 +181,6 @@ Pkg.add("UUIDs")
Pkg.add("Dates")
```
### JavaScript Dependencies
```bash
npm install nats.js apache-arrow uuid base64-url
```
### Python/Micropython Dependencies
1. Copy [`src/nats_bridge.py`](../src/nats_bridge.py) to your device
2. Ensure you have the following dependencies:
**For Python (desktop):**
```bash
pip install nats-py
```
**For Micropython:**
- `urequests` for HTTP requests
- `base64` for base64 encoding (built-in)
- `json` for JSON handling (built-in)
- `socket` for networking (built-in)
- `uuid` for UUID generation (built-in)
## Usage Tutorial
### Step 1: Start NATS Server
@@ -304,48 +203,21 @@ python3 -m http.server 8080 --directory /tmp/fileserver
### Step 3: Run Test Scenarios
```bash
# Scenario 1: Command & Control (JavaScript sender)
node test/scenario1_command_control.js
# Scenario 1: Command & Control
julia test/scenario1_command_control.jl
# Scenario 2: Large Arrow Table (JavaScript sender)
node test/scenario2_large_table.js
# Scenario 2: Large Arrow Table
julia test/scenario2_large_table.jl
# Scenario 3: Julia-to-Julia communication
# Run both Julia and JavaScript versions
julia test/scenario3_julia_to_julia.jl
node test/scenario3_julia_to_julia.js
```
## API Consistency Across Languages
**High-Level API (Consistent Across All Languages):**
- `smartsend(subject, data, ...)` - Main publishing function
- `smartreceive(msg, ...)` - Main receiving function
- Message envelope structure (`msg_envelope_v1` / `MessageEnvelope`)
- Payload structure (`msg_payload_v1` / `MessagePayload`)
- Transport strategy (direct vs link based on size threshold)
- Supported payload types: text, dictionary, table, image, audio, video, binary
**Low-Level Native Functions (Language-Specific Conventions):**
- Julia: `NATS.connect()`, `publish_message()`, function overloading
- JavaScript: `nats.js` client, native async/await patterns
- Python: `nats-python` client, native async/await patterns
**Connection Reuse Pattern - Key Differences:**
- **Julia:** Uses `NATS_connection` keyword parameter with function overloading for automatic connection management
- **JavaScript/Python:** Achieved by creating NATS client outside the function and reusing it in custom handlers or custom publish implementations
**Why the Difference?**
- Julia supports function overloading and keyword arguments, allowing `NATS_connection` to be passed as an optional parameter
- JavaScript/Python use a simpler `is_publish` option to control automatic publishing
- `is_publish` is simply a switch: when `true`, publish automatically; when `false`, return `(env, env_json_str)` without publishing
- For connection reuse in JavaScript/Python, create a NATS client once and reuse it in your custom `fileserver_upload_handler` or custom publish logic
## Usage
### Scenario 1: Command & Control (Small Dictionary)
**Focus:** Sending small dictionary configurations across platforms. This is the simplest use case for command and control scenarios.
**Focus:** Sending small dictionary configurations. This is the simplest use case for command and control scenarios.
**Julia (Sender/Receiver):**
```julia
@@ -386,98 +258,11 @@ NATS.close(conn)
**Use Case:** High-frequency publishing scenarios where connection reuse provides performance benefits by avoiding the overhead of establishing a new NATS connection for each message.
**JavaScript (Sender/Receiver):**
```javascript
const { smartsend } = require('./src/NATSBridge');
// Create small dictionary config
// Send via smartsend with type="dictionary"
const config = {
step_size: 0.01,
iterations: 1000,
threshold: 0.5
};
// Use is_publish option to control automatic publishing
await smartsend("control", [
{ dataname: "config", data: config, type: "dictionary" }
], {
is_publish: true // Automatically publish to NATS
});
```
**Connection Reuse in JavaScript:**
To achieve connection reuse in JavaScript, create a NATS client outside the function and use it in a custom `fileserver_upload_handler` or custom publish implementation:
```javascript
const { connect } = require('nats');
const { smartsend } = require('./src/NATSBridge');
// Create connection once
const nc = await connect({ servers: ['nats://localhost:4222'] });
// Send multiple messages using the same connection
for (let i = 0; i < 100; i++) {
const config = { iteration: i, data: Math.random() };
// Option 1: Use is_publish=false and publish manually with your connection
const { env, env_json_str } = await smartsend("control", [
{ dataname: "config", data: config, type: "dictionary" }
], { is_publish: false });
// Publish with your existing connection
await nc.publish("control", env_json_str);
}
// Close connection when done
await nc.close();
```
**Python/Micropython (Sender/Receiver):**
```python
from nats_bridge import smartsend
# Create small dictionary config
# Send via smartsend with type="dictionary"
config = {
"step_size": 0.01,
"iterations": 1000,
"threshold": 0.5
}
# Use is_publish parameter to control automatic publishing
smartsend("control", [("config", config, "dictionary")], is_publish=True)
```
**Connection Reuse in Python:**
To achieve connection reuse in Python, create a NATS client outside the function and use it in a custom `fileserver_upload_handler` or custom publish implementation:
```python
from nats_bridge import smartsend
import nats
# Create connection once
nc = await nats.connect("nats://localhost:4222")
# Send multiple messages using the same connection
for i in range(100):
config = {"iteration": i, "data": random.random()}
# Option 1: Use is_publish=False and publish manually with your connection
env, env_json_str = smartsend("control", [("config", config, "dictionary")], is_publish=False)
# Publish with your existing connection
await nc.publish("control", env_json_str)
# Close connection when done
await nc.close()
```
### Basic Multi-Payload Example
#### Python/Micropython (Sender)
```python
from nats_bridge import smartsend
#### Julia (Sender)
```julia
using NATSBridge
# Send multiple payloads in one message (type is required per payload)
smartsend(
@@ -491,71 +276,15 @@ smartsend(
smartsend("/test", [("single_data", mydata, "dictionary")], broker_url="nats://localhost:4222")
```
#### Python/Micropython (Receiver)
```python
from nats_bridge import smartreceive
#### Julia (Receiver)
```julia
using NATSBridge
# Receive returns a dictionary with envelope metadata and payloads field
env = smartreceive(msg)
# env["payloads"] = [(dataname1, data1, "dictionary"), (dataname2, data2, "table"), ...]
```
#### JavaScript (Sender)
```javascript
const { smartsend } = require('./src/NATSBridge');
// Single payload wrapped in a list
const config = [{
dataname: "config",
data: { step_size: 0.01, iterations: 1000 },
type: "dictionary"
}];
await smartsend("control", config, {
correlationId: "unique-id"
});
// Multiple payloads
const configs = [
{
dataname: "config1",
data: { step_size: 0.01 },
type: "dictionary"
},
{
dataname: "config2",
data: { iterations: 1000 },
type: "dictionary"
}
];
await smartsend("control", configs);
```
#### JavaScript (Receiver)
```javascript
const { smartreceive } = require('./src/NATSBridge');
// Subscribe to messages
const nc = await connect({ servers: ['nats://localhost:4222'] });
const sub = nc.subscribe("control");
for await (const msg of sub) {
const env = await smartreceive(msg);
// Process the payloads from the envelope
for (const payload of env.payloads) {
const { dataname, data, type } = payload;
console.log(`Received ${dataname} of type ${type}`);
console.log(`Data: ${JSON.stringify(data)}`);
}
// Also access envelope metadata
console.log(`Correlation ID: ${env.correlation_id}`);
console.log(`Message ID: ${env.msg_id}`);
}
```
### Scenario 2: Deep Dive Analysis (Large Arrow Table)
#### Julia (Sender)
@@ -689,91 +418,25 @@ env, env_json_str = smartsend(
**API Consistency Note:**
- **Julia:** Uses `NATS_connection` keyword parameter with function overloading for automatic connection management
- **JavaScript/Python:** Use `is_publish` option and achieve connection reuse by creating NATS client outside the function and reusing it in custom handlers or custom publish implementations
#### JavaScript (Receiver)
```javascript
const { smartreceive } = require('./src/NATSBridge');
const env = await smartreceive(msg);
// Use table data from the payloads field
// Note: Tables are sent as arrays of objects in JavaScript
const table = env.payloads;
```
### Scenario 3: Live Binary Processing
#### Python/Micropython (Sender)
```python
from nats_bridge import smartsend
**Julia (Sender/Receiver):**
```julia
using NATSBridge
# Binary data wrapped in list with type
smartsend(
"binary_input",
[("audio_chunk", binary_buffer, "binary")],
broker_url="nats://localhost:4222",
metadata={"sample_rate": 44100, "channels": 1}
metadata=["sample_rate" => 44100, "channels" => 1]
)
```
#### JavaScript (Sender)
```javascript
const { smartsend } = require('./src/NATSBridge');
// Binary data wrapped in a list
const binaryData = [{
dataname: "audio_chunk",
data: binaryBuffer, // ArrayBuffer or Uint8Array
type: "binary"
}];
await smartsend("binary_input", binaryData, {
metadata: {
sample_rate: 44100,
channels: 1
}
});
```
#### Python/Micropython (Receiver)
```python
from nats_bridge import smartreceive
# Receive binary data
def process_binary(msg):
env = smartreceive(msg)
# Process the binary data from env.payloads
for dataname, data, type in env["payloads"]:
if type == "binary":
# data is bytes
print(f"Received binary data: {dataname}, size: {len(data)}")
# Perform FFT or AI transcription here
```
#### JavaScript (Receiver)
```javascript
const { smartreceive } = require('./src/NATSBridge');
// Receive binary data
function process_binary(msg) {
const env = await smartreceive(msg);
// Process the binary data from env.payloads
for (const payload of env.payloads) {
if (payload.type === "binary") {
// data is an ArrayBuffer or Uint8Array
console.log(`Received binary data: ${payload.dataname}, size: ${payload.data.length}`);
// Perform FFT or AI transcription here
}
}
}
```
### Scenario 4: Catch-Up (JetStream)
#### Julia (Producer)
**Julia (Producer/Consumer):**
```julia
using NATSBridge
@@ -789,93 +452,11 @@ function publish_health_status(broker_url)
end
```
#### JavaScript (Consumer)
```javascript
const { connect } = require('nats');
const { smartreceive } = require('./src/NATSBridge');
const nc = await connect({ servers: ['nats://localhost:4222'] });
const js = nc.jetstream();
// Request replay from last 10 minutes
const consumer = await js.pullSubscribe("health", {
durable_name: "catchup",
max_batch: 100,
max_ack_wait: 30000
});
// Process historical and real-time messages
for await (const msg of consumer) {
const env = await smartreceive(msg);
// env.payloads contains the list of payloads
// Each payload has: dataname, data, type
msg.ack();
}
```
### Scenario 4: Micropython Device Control
**Focus:** Sending configuration to a Micropython device over NATS. This demonstrates the lightweight nature of the Python implementation suitable for microcontrollers.
**Python/Micropython (Receiver/Device):**
```python
from nats_bridge import smartsend, smartreceive
import json
# Device configuration handler
def handle_device_config(msg):
env = smartreceive(msg)
# Process configuration from payloads
for dataname, data, payload_type in env["payloads"]:
if payload_type == "dictionary":
print(f"Received configuration: {data}")
# Apply configuration to device
if "wifi_ssid" in data:
wifi_ssid = data["wifi_ssid"]
wifi_password = data["wifi_password"]
update_wifi_config(wifi_ssid, wifi_password)
# Send confirmation back
config = {
"status": "configured",
"wifi_ssid": "MyNetwork",
"ip": get_device_ip()
}
smartsend(
"device/response",
[("config", config, "dictionary")],
broker_url="nats://localhost:4222",
reply_to=env.get("reply_to")
)
```
**JavaScript (Sender/Controller):**
```javascript
const { smartsend } = require('./src/NATSBridge');
// Send configuration to Micropython device
await smartsend("device/config", [
{
dataname: "config",
data: {
wifi_ssid: "MyNetwork",
wifi_password: "password123",
update_interval: 60,
temperature_threshold: 30.0
},
type: "dictionary"
}
]);
```
**Use Case:** A controller sends WiFi and operational configuration to a Micropython device (e.g., ESP32). The device receives the configuration, applies it, and sends back a confirmation with its current status.
### Scenario 5: Selection (Low Bandwidth)
**Focus:** Small Arrow tables, Julia to JavaScript. The Action: Julia wants to send a small DataFrame to show on a JavaScript dashboard for the user to choose.
**Focus:** Small Arrow tables. The Action: Julia wants to send a small DataFrame to show on a receiving application for the user to choose.
**Julia (Sender):**
**Julia (Sender/Receiver):**
```julia
using NATSBridge
using DataFrames
@@ -903,27 +484,7 @@ env, env_json_str = smartsend(
# env_json_str: JSON string for publishing
```
**JavaScript (Receiver):**
```javascript
const { smartreceive, smartsend } = require('./src/NATSBridge');
// Receive NATS message with direct transport
const env = await smartreceive(msg);
// Decode Base64 payload (for direct transport)
// For tables, data is in env.payloads
const table = env.payloads; // Array of objects
// User makes selection
const selection = uiComponent.getSelectedOption();
// Send selection back to Julia
await smartsend("dashboard.response", [
{ dataname: "selected_option", data: selection, type: "dictionary" }
]);
```
**Use Case:** Julia server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to JavaScript dashboard for user selection. The selection is then sent back to Julia for processing.
**Use Case:** Julia server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to a receiving application for user selection. The selection is then sent back to Julia for processing.
### Scenario 6: Chat System
@@ -968,46 +529,6 @@ env, env_json_str = smartsend(
# env_json_str: JSON string for publishing
```
**JavaScript (Sender/Receiver):**
```javascript
const { smartsend, smartreceive } = require('./src/NATSBridge');
// Build chat message with mixed content:
// - User input text: direct transport
// - Selected image: check size, use appropriate transport
// - Audio recording: link transport for large files
// - File attachment: link transport
//
// Parse received message:
// - Direct payloads: decode Base64
// - Link payloads: fetch from HTTP with exponential backoff
// - Deserialize all payloads appropriately
//
// Render mixed content in chat interface
// Support bidirectional reply with claim-check delivery confirmation
// Example: Send chat with mixed content
const message = [
{
dataname: "text",
data: "Hello from JavaScript!",
type: "text"
},
{
dataname: "image",
data: selectedImageBuffer, // Small image (ArrayBuffer or Uint8Array)
type: "image"
},
{
dataname: "audio",
data: audioUrl, // Large audio, link transport
type: "audio"
}
];
await smartsend("chat.room123", message);
```
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components.
**Implementation Note:** The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. See the standard API format in Section 1: `msg_envelope_v1` supports `Vector{msg_payload_v1}` for multiple payloads.
@@ -1072,7 +593,6 @@ await smartsend("chat.room123", message);
### Exponential Backoff
- Maximum retry count: 5
- Base delay: 100ms, max delay: 5000ms
- Implemented in all three implementations (Julia, JavaScript, Python/Micropython)
### Correlation ID Logging
- Log correlation_id at every stage
@@ -1081,38 +601,7 @@ await smartsend("chat.room123", message);
## Testing
Run the test scripts for each platform:
### Python/Micropython Tests
```bash
# Basic functionality test
python test/test_micropython_basic.py
```
### JavaScript Tests
```bash
# Text message exchange
node test/test_js_to_js_text_sender.js
node test/test_js_to_js_text_receiver.js
# Dictionary exchange
node test/test_js_to_js_dict_sender.js
node test/test_js_to_js_dict_receiver.js
# File transfer (direct transport)
node test/test_js_to_js_file_sender.js
node test/test_js_to_js_file_receiver.js
# Mixed payload types
node test/test_js_to_js_mix_payloads_sender.js
node test/test_js_to_js_mix_payloads_receiver.js
# Table (Arrow IPC) exchange
node test/test_js_to_js_table_sender.js
node test/test_js_to_js_table_receiver.js
```
Run the test scripts for Julia:
### Julia Tests
@@ -1138,41 +627,22 @@ julia test/test_julia_to_julia_table_sender.jl
julia test/test_julia_to_julia_table_receiver.jl
```
### Cross-Platform Tests
```bash
# Julia ↔ JavaScript communication
julia test/test_julia_to_julia_text_sender.jl
node test/test_js_to_js_text_receiver.js
# Python ↔ JavaScript communication
python test/test_micropython_basic.py
node test/test_js_to_js_text_receiver.js
```
## Troubleshooting
### Common Issues
1. **NATS Connection Failed**
- **Julia/JavaScript/Python**: Ensure NATS server is running
- **Python/Micropython**: Check `nats_url` parameter and network connectivity
- Ensure NATS server is running
2. **HTTP Upload Failed**
- Ensure file server is running
- Check `fileserver_url` configuration
- Verify upload permissions
- **Micropython**: Ensure `urequests` is available and network is connected
3. **Arrow IPC Deserialization Error**
- Ensure data is properly serialized to Arrow format
- Check Arrow version compatibility
4. **Python/Micropython Specific Issues**
- **Import Error**: Ensure `nats_bridge.py` is in the correct path
- **Memory Error (Micropython)**: Reduce payload size or use link transport for large payloads
- **Unicode Error**: Ensure proper encoding when sending text data
## License
MIT