This commit is contained in:
2026-02-22 21:55:18 +07:00
parent 69f2173f75
commit def75d8f86
5 changed files with 541 additions and 127 deletions

View File

@@ -1,8 +1,13 @@
# Architecture Documentation: Bi-Directional Data Bridge (Julia ↔ JavaScript)
# Architecture Documentation: Bi-Directional Data Bridge
## Overview
This document describes the architecture for a high-performance, bi-directional data bridge between a Julia service and a JavaScript (Node.js) service using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
This document describes the architecture for a high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
The system enables seamless communication across all three platforms:
- **Julia ↔ JavaScript** bi-directional messaging
- **JavaScript ↔ Python/Micropython** bi-directional messaging
- **Julia ↔ Python/Micropython** bi-directional messaging (via JSON serialization)
### File Server Handler Architecture
@@ -135,7 +140,7 @@ flowchart TD
### 1. msgEnvelope_v1 - Message Envelope
The `msgEnvelope_v1` structure provides a comprehensive message format for bidirectional communication between Julia and JavaScript services.
The `msgEnvelope_v1` structure provides a comprehensive message format for bidirectional communication between Julia, JavaScript, and Python/Micropython applications.
**Julia Structure:**
```julia
@@ -211,7 +216,7 @@ end
### 2. msgPayload_v1 - Payload Structure
The `msgPayload_v1` structure provides flexible payload handling for various data types.
The `msgPayload_v1` structure provides flexible payload handling for various data types across all supported platforms.
**Julia Structure:**
```julia
@@ -239,7 +244,7 @@ end
┌─────────────────────────────────────────────────────────────┐
│ smartsend Function │
│ Accepts: [(dataname1, data1, type1), ...] │
│ (No standalone type parameter - type per payload)
│ (Type is per payload, not standalone)
└─────────────────────────────────────────────────────────────┘
@@ -266,19 +271,77 @@ end
└─────────────────┘ └─────────────────┘
```
### 4. Julia Module Architecture
### 4. Cross-Platform Architecture
```mermaid
flowchart TD
subgraph PythonMicropython
Py[Python/Micropython]
PySmartSend[smartsend]
PySmartReceive[smartreceive]
end
subgraph JavaScript
JS[JavaScript]
JSSmartSend[smartsend]
JSSmartReceive[smartreceive]
end
subgraph Julia
Julia[Julia]
JuliaSmartSend[smartsend]
JuliaSmartReceive[smartreceive]
end
subgraph NATS
NATSServer[NATS Server]
end
PySmartSend --> NATSServer
JSSmartSend --> NATSServer
JuliaSmartSend --> NATSServer
NATSServer --> PySmartReceive
NATSServer --> JSSmartReceive
NATSServer --> JuliaSmartReceive
style PythonMicropython fill:#e1f5fe
style JavaScript fill:#f3e5f5
style Julia fill:#e8f5e9
```
### 5. Python/Micropython Module Architecture
```mermaid
graph TD
subgraph JuliaModule
smartsendJulia[smartsend Julia]
subgraph PyModule
PySmartSend[smartsend]
SizeCheck[Size Check]
DirectPath[Direct Path]
LinkPath[Link Path]
HTTPClient[HTTP Client]
end
smartsendJulia --> SizeCheck
PySmartSend --> SizeCheck
SizeCheck -->|< 1MB| DirectPath
SizeCheck -->|>= 1MB| LinkPath
LinkPath --> HTTPClient
style PyModule fill:#b3e5fc
```
### 6. Julia Module Architecture
```mermaid
graph TD
subgraph JuliaModule
JuliaSmartSend[smartsend]
SizeCheck[Size Check]
DirectPath[Direct Path]
LinkPath[Link Path]
HTTPClient[HTTP Client]
end
JuliaSmartSend --> SizeCheck
SizeCheck -->|< 1MB| DirectPath
SizeCheck -->|>= 1MB| LinkPath
LinkPath --> HTTPClient
@@ -286,19 +349,19 @@ graph TD
style JuliaModule fill:#c5e1a5
```
### 5. JavaScript Module Architecture
### 7. JavaScript Module Architecture
```mermaid
graph TD
subgraph JSModule
smartsendJS[smartsend JS]
smartreceiveJS[smartreceive JS]
JSSmartSend[smartsend]
JSSmartReceive[smartreceive]
JetStreamConsumer[JetStream Pull Consumer]
ApacheArrow[Apache Arrow]
end
smartsendJS --> NATS
smartreceiveJS --> JetStreamConsumer
JSSmartSend --> NATS
JSSmartReceive --> JetStreamConsumer
JetStreamConsumer --> ApacheArrow
style JSModule fill:#f3e5f5
@@ -440,7 +503,7 @@ async function smartreceive(msg, options = {})
### Scenario 1: Command & Control (Small Dictionary)
**Julia (Receiver):**
**Julia (Sender/Receiver):**
```julia
# Subscribe to control subject
# Parse JSON envelope
@@ -448,15 +511,21 @@ async function smartreceive(msg, options = {})
# Send acknowledgment
```
**JavaScript (Sender):**
**JavaScript (Sender/Receiver):**
```javascript
// Create small dictionary config
// Send via smartsend with type="dictionary"
```
**Python/Micropython (Sender/Receiver):**
```python
# Create small dictionary config
# Send via smartsend with type="dictionary"
```
### Scenario 2: Deep Dive Analysis (Large Arrow Table)
**Julia (Sender):**
**Julia (Sender/Receiver):**
```julia
# Create large DataFrame
# Convert to Arrow IPC stream
@@ -465,7 +534,7 @@ async function smartreceive(msg, options = {})
# Publish NATS with URL
```
**JavaScript (Receiver):**
**JavaScript (Sender/Receiver):**
```javascript
// Receive NATS message with URL
// Fetch data from HTTP server
@@ -473,42 +542,64 @@ async function smartreceive(msg, options = {})
// Load into Perspective.js or D3
```
**Python/Micropython (Sender/Receiver):**
```python
# Create large DataFrame
# Convert to Arrow IPC stream
# Check size (> 1MB)
# Upload to HTTP server
# Publish NATS with URL
```
### Scenario 3: Live Audio Processing
**JavaScript (Sender):**
**JavaScript (Sender/Receiver):**
```javascript
// Capture audio chunk
// Send as binary with metadata headers
// Use smartsend with type="audio"
```
**Julia (Receiver):**
**Julia (Sender/Receiver):**
```julia
// Receive audio data
// Perform FFT or AI transcription
// Send results back (JSON + Arrow table)
# Receive audio data
# Perform FFT or AI transcription
# Send results back (JSON + Arrow table)
```
**Python/Micropython (Sender/Receiver):**
```python
# Capture audio chunk
# Send as binary with metadata headers
# Use smartsend with type="audio"
```
### Scenario 4: Catch-Up (JetStream)
**Julia (Producer):**
**Julia (Producer/Consumer):**
```julia
# Publish to JetStream
# Include metadata for temporal tracking
```
**JavaScript (Consumer):**
**JavaScript (Producer/Consumer):**
```javascript
// Connect to JetStream
// Request replay from last 10 minutes
// Process historical and real-time messages
```
**Python/Micropython (Producer/Consumer):**
```python
# Publish to JetStream
# Include metadata for temporal tracking
```
### Scenario 5: Selection (Low Bandwidth)
**Focus:** Small Arrow tables, Julia to JavaScript. The Action: Julia wants to send a small DataFrame to show on a JavaScript dashboard for the user to choose.
**Focus:** Small Arrow tables, cross-platform communication. The Action: Any platform wants to send a small DataFrame to show on any receiving application for the user to choose.
**Julia (Sender):**
**Julia (Sender/Receiver):**
```julia
# Create small DataFrame (e.g., 50KB - 500KB)
# Convert to Arrow IPC stream
@@ -517,7 +608,7 @@ async function smartreceive(msg, options = {})
# Include metadata for dashboard selection context
```
**JavaScript (Receiver):**
**JavaScript (Sender/Receiver):**
```javascript
// Receive NATS message with direct transport
// Decode Base64 payload
@@ -527,11 +618,20 @@ async function smartreceive(msg, options = {})
// Send selection back to Julia
```
**Use Case:** Julia server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to JavaScript dashboard for user selection. The selection is then sent back to Julia for processing.
**Python/Micropython (Sender/Receiver):**
```python
# Create small DataFrame (e.g., 50KB - 500KB)
# Convert to Arrow IPC stream
# Check payload size (< 1MB threshold)
# Publish directly to NATS with Base64-encoded payload
# Include metadata for dashboard selection context
```
**Use Case:** Any server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to any receiving application for user selection. The selection is then sent back to the sender for processing.
### Scenario 6: Chat System
**Focus:** Every conversational message is composed of any number and any combination of components, spanning the full spectrum from small to large. This includes text, images, audio, video, tables, and files—specifically accommodating everything from brief snippets to high-resolution images, large audio files, extensive tables, and massive documents. Support for claim-check delivery and full bi-directional messaging.
**Focus:** Every conversational message is composed of any number and any combination of components, spanning the full spectrum from small to large. This includes text, images, audio, video, tables, and files—specifically accommodating everything from brief snippets to high-resolution images, large audio files, extensive tables, and massive documents. Support for claim-check delivery and full bi-directional messaging across all platforms.
**Multi-Payload Support:** The system supports mixed-payload messages where a single message can contain multiple payloads with different transport strategies. The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type.
@@ -570,7 +670,25 @@ async function smartreceive(msg, options = {})
// Support bidirectional reply with claim-check delivery confirmation
```
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components.
**Python/Micropython (Sender/Receiver):**
```python
# Build chat message with mixed payloads:
# - Text: direct transport (Base64)
# - Small images: direct transport (Base64)
# - Large images: link transport (HTTP URL)
# - Audio/video: link transport (HTTP URL)
# - Tables: direct or link depending on size
# - Files: link transport (HTTP URL)
#
# Each payload uses appropriate transport strategy:
# - Size < 1MB → direct (NATS + Base64)
# - Size >= 1MB → link (HTTP upload + NATS URL)
#
# Include claim-check metadata for delivery tracking
# Support bidirectional messaging with replyTo fields
```
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components across all platforms.
**Implementation Note:** The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. See the standard API format in Section 1: `msgEnvelope_v1` supports `AbstractArray{msgPayload_v1}` for multiple payloads.

View File

@@ -2,7 +2,22 @@
## Overview
This document describes the implementation of the high-performance, bi-directional data bridge between Julia and JavaScript services using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
This document describes the implementation of the high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
The system enables seamless communication across all three platforms:
- **Julia ↔ JavaScript** bi-directional messaging
- **JavaScript ↔ Python/Micropython** bi-directional messaging
- **Julia ↔ Python/Micropython** bi-directional messaging (via JSON serialization)
### Implementation Files
NATSBridge is implemented in three languages, each providing the same API:
| Language | Implementation File | Description |
|----------|---------------------|-------------|
| **Julia** | [`src/NATSBridge.jl`](../src/NATSBridge.jl) | Full Julia implementation with Arrow IPC support |
| **JavaScript** | [`src/NATSBridge.js`](../src/NATSBridge.js) | JavaScript implementation for Node.js and browsers |
| **Python/Micropython** | [`src/nats_bridge.py`](../src/nats_bridge.py) | Python implementation for desktop and microcontrollers |
### Multi-Payload Support
@@ -49,9 +64,62 @@ envelope = smartreceive(msg, ...)
# envelope["correlationId"], envelope["msgId"], etc.
```
## Cross-Platform Interoperability
NATSBridge is designed for seamless communication between Julia, JavaScript, and Python/Micropython applications. All three implementations share the same interface and data format, ensuring compatibility across platforms.
### Platform-Specific Features
| Feature | Julia | JavaScript | Python/Micropython |
|---------|-------|------------|-------------------|
| Direct NATS transport | ✅ | ✅ | ✅ |
| HTTP file server (Claim-Check) | ✅ | ✅ | ✅ |
| Arrow IPC tables | ✅ | ✅ | ✅ |
| Base64 encoding | ✅ | ✅ | ✅ |
| Exponential backoff | ✅ | ✅ | ✅ |
| Correlation ID tracking | ✅ | ✅ | ✅ |
| Reply-to support | ✅ | ✅ | ✅ |
### Data Type Mapping
| Type | Julia | JavaScript | Python/Micropython |
|------|-------|------------|-------------------|
| `text` | `String` | `String` | `str` |
| `dictionary` | `Dict` | `Object` | `dict` |
| `table` | `DataFrame` | `Array<Object>` | `DataFrame` / `list` |
| `image` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
| `audio` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
| `video` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
| `binary` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
### Example: Julia ↔ Python ↔ JavaScript
```julia
# Julia sender
using NATSBridge
data = [("message", "Hello from Julia!", "text")]
smartsend("/cross_platform", data, nats_url="nats://localhost:4222")
```
```javascript
// JavaScript receiver
const { smartreceive } = require('./src/NATSBridge');
const envelope = await smartreceive(msg);
// envelope.payloads[0].data === "Hello from Julia!"
```
```python
# Python sender
from nats_bridge import smartsend
data = [("response", "Hello from Python!", "text")]
smartsend("/cross_platform", data, nats_url="nats://localhost:4222")
```
All three platforms can communicate seamlessly using the same NATS subjects and data format.
## Architecture
The implementation follows the Claim-Check pattern:
All three implementations (Julia, JavaScript, Python/Micropython) follow the same Claim-Check pattern:
```
┌─────────────────────────────────────────────────────────────────────────┐
@@ -70,7 +138,7 @@ The implementation follows the Claim-Check pattern:
│ (< 1MB) │ │ (> 1MB) │
│ │ │ │
│ • Serialize to │ │ • Serialize to │
IOBuffer │ │ IOBuffer │
│ Buffer │ │ Buffer
│ • Base64 encode │ │ • Upload to │
│ • Publish to │ │ HTTP Server │
│ NATS │ │ • Publish to │
@@ -80,13 +148,13 @@ The implementation follows the Claim-Check pattern:
## Files
### Julia Module: [`src/julia_bridge.jl`](../src/julia_bridge.jl)
### Julia Module: [`src/NATSBridge.jl`](../src/NATSBridge.jl)
The Julia implementation provides:
- **[`MessageEnvelope`](../src/julia_bridge.jl)**: Struct for the unified JSON envelope
- **[`SmartSend()`](../src/julia_bridge.jl)**: Handles transport selection based on payload size
- **[`SmartReceive()`](../src/julia_bridge.jl)**: Handles both direct and link transport
- **[`MessageEnvelope`](src/NATSBridge.jl)**: Struct for the unified JSON envelope
- **[`SmartSend()`](src/NATSBridge.jl)**: Handles transport selection based on payload size
- **[`SmartReceive()`](src/NATSBridge.jl)**: Handles both direct and link transport
### JavaScript Module: [`src/NATSBridge.js`](../src/NATSBridge.js)
@@ -94,8 +162,17 @@ The JavaScript implementation provides:
- **`MessageEnvelope` class**: For the unified JSON envelope
- **`MessagePayload` class**: For individual payload representation
- **[`smartsend()`](../src/NATSBridge.js)**: Handles transport selection based on payload size
- **[`smartreceive()`](../src/NATSBridge.js)**: Handles both direct and link transport
- **[`smartsend()`](src/NATSBridge.js)**: Handles transport selection based on payload size
- **[`smartreceive()`](src/NATSBridge.js)**: Handles both direct and link transport
### Python/Micropython Module: [`src/nats_bridge.py`](../src/nats_bridge.py)
The Python/Micropython implementation provides:
- **`MessageEnvelope` class**: For the unified JSON envelope
- **`MessagePayload` class**: For individual payload representation
- **[`smartsend()`](src/nats_bridge.py)**: Handles transport selection based on payload size
- **[`smartreceive()`](src/nats_bridge.py)**: Handles both direct and link transport
## Installation
@@ -117,6 +194,23 @@ Pkg.add("Dates")
npm install nats.js apache-arrow uuid base64-url
```
### Python/Micropython Dependencies
1. Copy [`src/nats_bridge.py`](../src/nats_bridge.py) to your device
2. Ensure you have the following dependencies:
**For Python (desktop):**
```bash
pip install nats-py
```
**For Micropython:**
- `urequests` for HTTP requests
- `base64` for base64 encoding (built-in)
- `json` for JSON handling (built-in)
- `socket` for networking (built-in)
- `uuid` for UUID generation (built-in)
## Usage Tutorial
### Step 1: Start NATS Server
@@ -155,35 +249,31 @@ node test/scenario3_julia_to_julia.js
### Scenario 0: Basic Multi-Payload Example
#### Julia (Sender)
```julia
using NATSBridge
#### Python/Micropython (Sender)
```python
from nats_bridge import smartsend
# Send multiple payloads in one message (type is required per payload)
smartsend(
"/test",
[("dataname1", data1, "dictionary"), ("dataname2", data2, "table")],
nats_url="nats://localhost:4222",
fileserver_url="http://localhost:8080",
metadata=Dict("custom_key" => "custom_value")
fileserver_url="http://localhost:8080"
)
# Even single payload must be wrapped in a list with type
smartsend("/test", [("single_data", mydata, "dictionary")])
```
#### Julia (Receiver)
```julia
using NATSBridge
#### Python/Micropython (Receiver)
```python
from nats_bridge import smartreceive
# Receive returns a dictionary envelope with all metadata and deserialized payloads
envelope = smartreceive(msg, "http://localhost:8080")
# envelope["payloads"] = [(dataname1, data1, "dictionary"), (dataname2, data2, "table"), ...]
# envelope["correlationId"], envelope["msgId"], etc.
# Receive returns a list of (dataname, data, type) tuples
payloads = smartreceive(msg)
# payloads = [(dataname1, data1, "dictionary"), (dataname2, data2, "table"), ...]
```
### Scenario 1: Command & Control (Small JSON)
#### JavaScript (Sender)
```javascript
const { smartsend } = require('./src/NATSBridge');
@@ -216,27 +306,7 @@ const configs = [
await smartsend("control", configs);
```
#### Julia (Receiver)
```julia
using NATS
using JSON3
# Subscribe to control subject
subscribe(nats, "control") do msg
env = MessageEnvelope(String(msg.data))
config = JSON3.read(env.payload)
# Execute simulation with parameters
step_size = config.step_size
iterations = config.iterations
# Send acknowledgment
response = Dict("status" => "Running", "correlation_id" => env.correlation_id)
publish(nats, "control_response", JSON3.stringify(response))
end
```
### JavaScript (Receiver)
#### JavaScript (Receiver)
```javascript
const { smartreceive } = require('./src/NATSBridge');
@@ -291,6 +361,26 @@ const table = envelope.payloads;
### Scenario 3: Live Binary Processing
#### Python/Micropython (Sender)
```python
from nats_bridge import smartsend
# Binary data wrapped in a list
binary_data = [
("audio_chunk", binary_buffer, "binary")
]
smartsend(
"binary_input",
binary_data,
nats_url="nats://localhost:4222",
metadata={
"sample_rate": 44100,
"channels": 1
}
)
```
#### JavaScript (Sender)
```javascript
const { smartsend } = require('./src/NATSBridge');
@@ -310,23 +400,23 @@ await smartsend("binary_input", binaryData, {
});
```
#### Julia (Receiver)
```julia
using WAV
using DSP
#### Python/Micropython (Receiver)
```python
from nats_bridge import smartreceive
# Receive binary data
function process_binary(data)
# Perform FFT or AI transcription
spectrum = fft(data)
def process_binary(msg):
envelope = smartreceive(msg)
# Send results back (JSON + Arrow table)
results = Dict("transcription" => "sample text", "spectrum" => spectrum)
await SmartSend("binary_output", results, "json")
end
# Process the binary data from envelope.payloads
for dataname, data, type in envelope["payloads"]:
if type == "binary":
# data is bytes
print(f"Received binary data: {dataname}, size: {len(data)}")
# Perform FFT or AI transcription here
```
### JavaScript (Receiver)
#### JavaScript (Receiver)
```javascript
const { smartreceive } = require('./src/NATSBridge');
@@ -383,6 +473,64 @@ for await (const msg of consumer) {
}
```
### Scenario 4: Micropython Device Control
**Focus:** Sending configuration to a Micropython device over NATS. This demonstrates the lightweight nature of the Python implementation suitable for microcontrollers.
**Python/Micropython (Receiver/Device):**
```python
from nats_bridge import smartsend, smartreceive
import json
# Device configuration handler
def handle_device_config(msg):
envelope = smartreceive(msg)
# Process configuration from payloads
for dataname, data, type in envelope["payloads"]:
if type == "dictionary":
print(f"Received configuration: {data}")
# Apply configuration to device
if "wifi_ssid" in data:
wifi_ssid = data["wifi_ssid"]
wifi_password = data["wifi_password"]
update_wifi_config(wifi_ssid, wifi_password)
# Send confirmation back
config = {
"status": "configured",
"wifi_ssid": "MyNetwork",
"ip": get_device_ip()
}
smartsend(
"device/response",
[("config", config, "dictionary")],
nats_url="nats://localhost:4222",
reply_to=envelope.get("replyTo")
)
```
**JavaScript (Sender/Controller):**
```javascript
const { smartsend } = require('./src/NATSBridge');
// Send configuration to Micropython device
await smartsend("device/config", [
{
dataname: "config",
data: {
wifi_ssid: "MyNetwork",
wifi_password: "password123",
update_interval: 60,
temperature_threshold: 30.0
},
type: "dictionary"
}
]);
```
**Use Case:** A controller sends WiFi and operational configuration to a Micropython device (e.g., ESP32). The device receives the configuration, applies it, and sends back a confirmation with its current status.
### Scenario 5: Selection (Low Bandwidth)
**Focus:** Small Arrow tables, Julia to JavaScript. The Action: Julia wants to send a small DataFrame to show on a JavaScript dashboard for the user to choose.
@@ -581,7 +729,7 @@ await smartsend("chat.room123", message);
### Exponential Backoff
- Maximum retry count: 5
- Base delay: 100ms, max delay: 5000ms
- Implemented in both Julia and JavaScript implementations
- Implemented in all three implementations (Julia, JavaScript, Python/Micropython)
### Correlation ID Logging
- Log correlation_id at every stage
@@ -590,14 +738,73 @@ await smartsend("chat.room123", message);
## Testing
Run the test scripts:
Run the test scripts for each platform:
### Python/Micropython Tests
```bash
# Scenario 1: Command & Control (JavaScript sender)
node test/scenario1_command_control.js
# Basic functionality test
python test/test_micropython_basic.py
```
# Scenario 2: Large Arrow Table (JavaScript sender)
node test/scenario2_large_table.js
### JavaScript Tests
```bash
# Text message exchange
node test/test_js_to_js_text_sender.js
node test/test_js_to_js_text_receiver.js
# Dictionary exchange
node test/test_js_to_js_dict_sender.js
node test/test_js_to_js_dict_receiver.js
# File transfer (direct transport)
node test/test_js_to_js_file_sender.js
node test/test_js_to_js_file_receiver.js
# Mixed payload types
node test/test_js_to_js_mix_payloads_sender.js
node test/test_js_to_js_mix_payloads_receiver.js
# Table (Arrow IPC) exchange
node test/test_js_to_js_table_sender.js
node test/test_js_to_js_table_receiver.js
```
### Julia Tests
```bash
# Text message exchange
julia test/test_julia_to_julia_text_sender.jl
julia test/test_julia_to_julia_text_receiver.jl
# Dictionary exchange
julia test/test_julia_to_julia_dict_sender.jl
julia test/test_julia_to_julia_dict_receiver.jl
# File transfer
julia test/test_julia_to_julia_file_sender.jl
julia test/test_julia_to_julia_file_receiver.jl
# Mixed payload types
julia test/test_julia_to_julia_mix_payloads_sender.jl
julia test/test_julia_to_julia_mix_payloads_receiver.jl
# Table exchange
julia test/test_julia_to_julia_table_sender.jl
julia test/test_julia_to_julia_table_receiver.jl
```
### Cross-Platform Tests
```bash
# Julia ↔ JavaScript communication
julia test/test_julia_to_julia_text_sender.jl
node test/test_js_to_js_text_receiver.js
# Python ↔ JavaScript communication
python test/test_micropython_basic.py
node test/test_js_to_js_text_receiver.js
```
## Troubleshooting
@@ -605,18 +812,24 @@ node test/scenario2_large_table.js
### Common Issues
1. **NATS Connection Failed**
- Ensure NATS server is running
- Check NATS_URL configuration
- **Julia/JavaScript/Python**: Ensure NATS server is running
- **Python/Micropython**: Check `nats_url` parameter and network connectivity
2. **HTTP Upload Failed**
- Ensure file server is running
- Check FILESERVER_URL configuration
- Check `fileserver_url` configuration
- Verify upload permissions
- **Micropython**: Ensure `urequests` is available and network is connected
3. **Arrow IPC Deserialization Error**
- Ensure data is properly serialized to Arrow format
- Check Arrow version compatibility
4. **Python/Micropython Specific Issues**
- **Import Error**: Ensure `nats_bridge.py` is in the correct path
- **Memory Error (Micropython)**: Reduce payload size or use link transport for large payloads
- **Unicode Error**: Ensure proper encoding when sending text data
## License
MIT

View File

@@ -1,17 +1,17 @@
# NATSBridge for Micropython
# NATSBridge
A high-performance, bi-directional data bridge for Micropython devices using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
A high-performance, bi-directional data bridge for **Julia**, **JavaScript**, and **Python/Micropython** using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
## Overview
This module provides functionality for sending and receiving data over NATS with automatic transport selection based on payload size:
NATSBridge enables seamless communication between Julia, JavaScript, and Python/Micropython applications through NATS, with automatic transport selection based on payload size:
- **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded)
- **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL
## Features
- ✅ Bi-directional NATS communication
- ✅ Bi-directional NATS communication across Julia ↔ JavaScript ↔ Python/Micropython
- ✅ Multi-payload support (mixed content in single message)
- ✅ Automatic transport selection based on payload size
- ✅ File server integration for large payloads
@@ -31,19 +31,56 @@ This module provides functionality for sending and receiving data over NATS with
| `video` | Video data (MP4, AVI bytes) |
| `binary` | Generic binary data |
## Implementation Guides
### [Julia Implementation](../tutorial_julia.md)
See the [Julia tutorial](../tutorial_julia.md) for getting started with Julia.
### [JavaScript Implementation](#javascript-implementation)
See [`NATSBridge.js`](NATSBridge.js) for the JavaScript implementation.
### [Python/Micropython Implementation](#pythonmicropython-implementation)
See [`nats_bridge.py`](nats_bridge.py) for the Python/Micropython implementation.
## Installation
1. Copy `nats_bridge.py` to your Micropython device
### Julia
```julia
using Pkg
Pkg.add("NATS")
Pkg.add("Arrow")
Pkg.add("JSON3")
Pkg.add("HTTP")
Pkg.add("UUIDs")
Pkg.add("Dates")
```
### JavaScript
```bash
npm install nats.js apache-arrow uuid base64-url
```
### Python/Micropython
1. Copy `nats_bridge.py` to your device
2. Ensure you have the following dependencies:
- `urequests` for HTTP requests
- `ubinascii` for base64 encoding
- `ujson` for JSON handling
- `usocket` for networking
- `urequests` for HTTP requests (Micropython)
- `requests` for HTTP requests (Python)
- `base64` for base64 encoding
- `json` for JSON handling
- `socket` for networking (Micropython)
## Usage
### Basic Text Message
#### Python/Micropython
```python
from nats_bridge import smartsend, smartreceive
@@ -57,8 +94,39 @@ for dataname, data, type in payloads:
print("Received {}: {}".format(dataname, data))
```
#### Julia
```julia
using NATSBridge
# Sender
data = [("message", "Hello World", "text")]
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
# Receiver
envelope = smartreceive(msg, fileserverDownloadHandler)
# envelope["payloads"] = [("message", "Hello World", "text"), ...]
```
#### JavaScript
```javascript
const { smartsend, smartreceive } = require('./src/NATSBridge');
// Sender
await smartsend("/chat/room1", [
{ dataname: "message", data: "Hello World", type: "text" }
], { natsUrl: "nats://localhost:4222" });
// Receiver
const envelope = await smartreceive(msg);
// envelope.payloads = [{ dataname: "message", data: "Hello World", type: "text" }, ...]
```
### Sending JSON Configuration
#### Python/Micropython
```python
from nats_bridge import smartsend
@@ -74,6 +142,8 @@ env = smartsend("/device/config", data, nats_url="nats://localhost:4222")
### Mixed Content (Chat with Text + Image)
#### Python/Micropython
```python
from nats_bridge import smartsend
@@ -89,6 +159,8 @@ env = smartsend("/chat/mixed", data, nats_url="nats://localhost:4222")
### Request-Response Pattern
#### Python/Micropython
```python
from nats_bridge import smartsend
@@ -105,6 +177,8 @@ env = smartsend(
### Large Payloads (File Server)
#### Python/Micropython
```python
from nats_bridge import smartsend
@@ -191,21 +265,30 @@ Represents a single payload within a message envelope.
## Examples
See `examples/micropython_example.py` for more detailed examples.
See [`examples/micropython_example.py`](../examples/micropython_example.py) for more detailed examples.
## Testing
Run the test suite:
```bash
# Python/Micropython
python test/test_micropython_basic.py
# JavaScript
node test/test_js_to_js_text_sender.js
node test/test_js_to_js_text_receiver.js
# Julia
julia test/test_julia_to_julia_text_sender.jl
julia test/test_julia_to_julia_text_receiver.jl
```
## Requirements
- Micropython with networking support
- NATS server (nats.io)
- HTTP file server (optional, for large payloads)
- **Julia**: NATS server (nats.io), HTTP file server (optional)
- **JavaScript**: NATS server (nats.io), HTTP file server (optional)
- **Python/Micropython**: NATS server (nats.io), HTTP file server (optional)
## License