update
This commit is contained in:
715
README.md
715
README.md
@@ -1,6 +1,6 @@
|
||||
# NATSBridge
|
||||
# NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||
|
||||
A high-performance, bi-directional data bridge for **Julia** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
A high-performance, bi-directional data bridge for **Julia, JavaScript, Python, and MicroPython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://nats.io)
|
||||
@@ -10,6 +10,7 @@ A high-performance, bi-directional data bridge for **Julia** applications using
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Cross-Platform Support](#cross-platform-support)
|
||||
- [Features](#features)
|
||||
- [Architecture](#architecture)
|
||||
- [Installation](#installation)
|
||||
@@ -17,7 +18,7 @@ A high-performance, bi-directional data bridge for **Julia** applications using
|
||||
- [API Reference](#api-reference)
|
||||
- [Payload Types](#payload-types)
|
||||
- [Transport Strategies](#transport-strategies)
|
||||
- [Examples](#examples)
|
||||
- [Cross-Platform Examples](#cross-platform-examples)
|
||||
- [Testing](#testing)
|
||||
- [License](#license)
|
||||
|
||||
@@ -25,7 +26,7 @@ A high-performance, bi-directional data bridge for **Julia** applications using
|
||||
|
||||
## Overview
|
||||
|
||||
NATSBridge enables seamless communication for Julia applications through NATS, with intelligent transport selection based on payload size:
|
||||
NATSBridge enables seamless communication across multiple platforms through NATS, with intelligent transport selection based on payload size:
|
||||
|
||||
| Transport | Payload Size | Method |
|
||||
|-----------|--------------|--------|
|
||||
@@ -36,14 +37,40 @@ NATSBridge enables seamless communication for Julia applications through NATS, w
|
||||
|
||||
- **Chat Applications**: Text, images, audio, video in a single message
|
||||
- **File Transfer**: Efficient transfer of large files using claim-check pattern
|
||||
- **Streaming Data**: Sensor data, telemetry, and analytics pipelines
|
||||
- **IoT/Embedded**: Sensor data, telemetry, and analytics pipelines (MicroPython)
|
||||
- **Cross-Platform Communication**: Interoperability between Julia, JavaScript, Python, and MicroPython systems
|
||||
|
||||
---
|
||||
|
||||
## Cross-Platform Support
|
||||
|
||||
| Platform | Implementation | Features |
|
||||
|----------|----------------|----------|
|
||||
| **Julia** | [`src/NATSBridge.jl`](src/NATSBridge.jl) | Full feature set, Arrow IPC, multiple dispatch |
|
||||
| **JavaScript** | [`src/natbridge.js`](src/natbridge.js) | Node.js & browser, async/await |
|
||||
| **Python** | [`src/natbridge.py`](src/natbridge.py) | Desktop Python, asyncio, type hints |
|
||||
| **MicroPython** | [`src/natbridge_mpy.py`](src/natbridge_mpy.py) | Memory-constrained, synchronous API |
|
||||
|
||||
### Platform Comparison
|
||||
|
||||
| Feature | Julia | JavaScript | Python | MicroPython |
|
||||
|---------|-------|------------|--------|-------------|
|
||||
| Multiple Dispatch | ✅ Native | ❌ | ❌ | ❌ |
|
||||
| Async/Await | ❌ | ✅ Native | ✅ Native | ⚠️ (uasyncio) |
|
||||
| Type Safety | ✅ Strong | ⚠️ (TypeScript) | ✅ (Type hints) | ❌ |
|
||||
| Memory Management | ✅ GC | ✅ GC | ✅ GC | ⚠️ (Manual) |
|
||||
| Arrow IPC | ✅ Native | ✅ | ✅ | ❌ |
|
||||
| Direct Transport | ✅ | ✅ | ✅ | ✅ |
|
||||
| Link Transport | ✅ | ✅ | ✅ | ⚠️ (Limited) |
|
||||
| Handler Functions | ✅ | ✅ | ✅ | ✅ |
|
||||
| Cross-Platform API | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
- ✅ **Bi-directional messaging** for Julia applications
|
||||
- ✅ **Cross-platform messaging** for Julia, JavaScript, Python, and MicroPython applications
|
||||
- ✅ **Bi-directional messaging** with request-reply patterns
|
||||
- ✅ **Multi-payload support** - send multiple payloads with different types in one message
|
||||
- ✅ **Automatic transport selection** - direct vs link based on payload size
|
||||
- ✅ **Claim-Check pattern** for payloads > 1MB
|
||||
@@ -51,8 +78,7 @@ NATSBridge enables seamless communication for Julia applications through NATS, w
|
||||
- ✅ **Exponential backoff** for reliable file server downloads
|
||||
- ✅ **Correlation ID tracking** for message tracing
|
||||
- ✅ **Reply-to support** for request-response patterns
|
||||
- ✅ **JetStream support** for message replay and durability
|
||||
|
||||
- ✅ **Handler function abstraction** - pluggable file server implementations (Plik, AWS S3, custom)
|
||||
|
||||
---
|
||||
|
||||
@@ -62,13 +88,13 @@ NATSBridge enables seamless communication for Julia applications through NATS, w
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Sender["Julia Application (Sender)"]
|
||||
subgraph Sender["Application (Sender)"]
|
||||
SenderApp[App Code]
|
||||
NATSBridge_Send[NATSBridge]
|
||||
NATS_Client[<b>NATS.jl</b>]
|
||||
end
|
||||
|
||||
subgraph Receiver["Julia Application (Receiver)"]
|
||||
subgraph Receiver["Application (Receiver)"]
|
||||
ReceiverApp[App Code]
|
||||
NATSBridge_Recv[NATSBridge]
|
||||
NATS_Client_Recv[<b>NATS.jl</b>]
|
||||
@@ -96,14 +122,6 @@ flowchart TB
|
||||
style FileServer fill:#f3e5f5
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
| Component | Description |
|
||||
|-----------|-------------|
|
||||
| **Julia Application** | Sender and receiver applications using the NATSBridge module |
|
||||
| **NATS Server** | Message broker for transporting message envelopes |
|
||||
| **HTTP File Server** | Independent HTTP server for large payload storage (e.g., Plik) |
|
||||
|
||||
### Message Flow
|
||||
|
||||
1. **Sender** creates a message envelope with payloads using `smartsend()`
|
||||
@@ -124,11 +142,53 @@ The system uses handler functions to abstract file server operations:
|
||||
|
||||
| Handler | Purpose |
|
||||
|---------|---------|
|
||||
| `plik_oneshot_upload()` | Uploads payload bytes to file server, returns URL |
|
||||
| `_fetch_with_backoff()` | Downloads data from URL with exponential backoff retry |
|
||||
| `plik_oneshot_upload()` / `plikOneshotUpload()` | Uploads payload bytes to file server, returns URL |
|
||||
| `_fetch_with_backoff()` / `fetchWithBackoff()` | Downloads data from URL with exponential backoff retry |
|
||||
|
||||
This abstraction allows support for different file server implementations (Plik, AWS S3, custom HTTP server).
|
||||
|
||||
### Message Envelope Schema
|
||||
|
||||
All platforms use identical JSON schemas for message envelopes:
|
||||
|
||||
```json
|
||||
{
|
||||
"correlation_id": "uuid-v4-string",
|
||||
"msg_id": "uuid-v4-string",
|
||||
"timestamp": "2024-01-15T10:30:00Z",
|
||||
"send_to": "topic/subject",
|
||||
"msg_purpose": "ACK | NACK | updateStatus | shutdown | chat",
|
||||
"sender_name": "agent-wine-web-frontend",
|
||||
"sender_id": "uuid4",
|
||||
"receiver_name": "agent-backend",
|
||||
"receiver_id": "uuid4",
|
||||
"reply_to": "topic",
|
||||
"reply_to_msg_id": "uuid4",
|
||||
"broker_url": "nats://localhost:4222",
|
||||
"metadata": {},
|
||||
"payloads": [
|
||||
{
|
||||
"id": "uuid4",
|
||||
"dataname": "login_image",
|
||||
"payload_type": "image",
|
||||
"transport": "direct",
|
||||
"encoding": "base64",
|
||||
"size": 15433,
|
||||
"data": "base64-encoded-string"
|
||||
},
|
||||
{
|
||||
"id": "uuid4",
|
||||
"dataname": "large_table",
|
||||
"payload_type": "table",
|
||||
"transport": "link",
|
||||
"encoding": "none",
|
||||
"size": 524288,
|
||||
"data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/data.arrow"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
@@ -138,14 +198,53 @@ This abstraction allows support for different file server implementations (Plik,
|
||||
- **NATS Server** (v2.10+ recommended)
|
||||
- **HTTP File Server** (optional, for payloads > 1MB)
|
||||
|
||||
### Julia
|
||||
### Platform-Specific Dependencies
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using Pkg
|
||||
Pkg.add("NATS")
|
||||
Pkg.add("https://git.yiem.cc/ton/NATSBridge")
|
||||
Pkg.add("Arrow")
|
||||
Pkg.add("JSON3")
|
||||
Pkg.add("HTTP")
|
||||
Pkg.add("UUIDs")
|
||||
Pkg.add("Dates")
|
||||
```
|
||||
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```bash
|
||||
npm install nats uuid apache-arrow node-fetch
|
||||
# or
|
||||
yarn add nats uuid apache-arrow node-fetch
|
||||
```
|
||||
|
||||
#### JavaScript (Browser)
|
||||
|
||||
```bash
|
||||
npm install nats uuid apache-arrow
|
||||
# or use CDN:
|
||||
# https://unpkg.com/nats-js/dist/bundle/nats.min.js
|
||||
# https://unpkg.com/apache-arrow/arrow.min.js
|
||||
```
|
||||
|
||||
#### Python (Desktop)
|
||||
|
||||
```bash
|
||||
pip install nats-py aiohttp pyarrow pandas python-dateutil
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
MicroPython uses built-in modules:
|
||||
- `network` - NATS connection (custom implementation)
|
||||
- `time` - Timestamps
|
||||
- `uos` - File operations
|
||||
- `base64` - Base64 encoding
|
||||
- `json` - JSON parsing
|
||||
- `struct` - Binary data handling
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
@@ -166,61 +265,39 @@ mkdir -p /tmp/fileserver
|
||||
python3 -m http.server 8080 --directory /tmp/fileserver
|
||||
```
|
||||
|
||||
### Step 3: Send Your First Message
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Send a text message
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; broker_url="nats://localhost:4222")
|
||||
println("Message sent!")
|
||||
```
|
||||
|
||||
### Step 4: Receive Messages
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATS, NATSBridge
|
||||
|
||||
# Configuration
|
||||
const SUBJECT = "/chat/room1"
|
||||
const NATS_URL = "nats://localhost:4222"
|
||||
|
||||
# Helper: Log with correlation ID
|
||||
function log_trace(message)
|
||||
timestamp = Dates.now()
|
||||
println("[$timestamp] $message")
|
||||
end
|
||||
|
||||
# Receiver: Listen for messages - msg comes from the callback
|
||||
function test_receive()
|
||||
conn = NATS.connect(NATS_URL)
|
||||
NATS.subscribe(conn, SUBJECT) do msg
|
||||
log_trace("Received message on $(msg.subject)")
|
||||
|
||||
# Receive and process message
|
||||
env, env_json_str = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
println("Received $dataname: $data")
|
||||
end
|
||||
end
|
||||
|
||||
# Keep listening for 120 seconds
|
||||
sleep(120)
|
||||
NATS.drain(conn)
|
||||
end
|
||||
|
||||
test_receive()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Unified API Standard
|
||||
|
||||
All platforms use the same input/output format for payloads:
|
||||
|
||||
**Input format for smartsend:**
|
||||
```
|
||||
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||
```
|
||||
|
||||
**Output format for smartreceive:**
|
||||
```
|
||||
{
|
||||
"correlation_id": "...",
|
||||
"msg_id": "...",
|
||||
"timestamp": "...",
|
||||
"send_to": "...",
|
||||
"msg_purpose": "...",
|
||||
"sender_name": "...",
|
||||
"sender_id": "...",
|
||||
"receiver_name": "...",
|
||||
"receiver_id": "...",
|
||||
"reply_to": "...",
|
||||
"reply_to_msg_id": "...",
|
||||
"broker_url": "...",
|
||||
"metadata": {...},
|
||||
"payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||
}
|
||||
```
|
||||
|
||||
### smartsend
|
||||
|
||||
Sends data either directly via NATS or via a fileserver URL, depending on payload size.
|
||||
@@ -231,27 +308,96 @@ Sends data either directly via NATS or via a fileserver URL, depending on payloa
|
||||
using NATSBridge
|
||||
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
subject, # NATS subject
|
||||
subject::String, # NATS subject
|
||||
data::AbstractArray{Tuple{String, Any, String}}; # List of (dataname, data, type)
|
||||
broker_url::String = "nats://localhost:4222",
|
||||
fileserver_url = "http://localhost:8080",
|
||||
fileserver_upload_handler::Function = plik_oneshot_upload,
|
||||
size_threshold::Int = 1_000_000,
|
||||
correlation_id::String = string(uuid4()), # Correlation ID for tracing (auto-generated UUID)
|
||||
correlation_id::String = string(uuid4()),
|
||||
msg_purpose::String = "chat",
|
||||
sender_name::String = "NATSBridge",
|
||||
receiver_name::String = "",
|
||||
receiver_id::String = "",
|
||||
reply_to::String = "",
|
||||
reply_to_msg_id::String = "",
|
||||
is_publish::Bool = true, # Whether to automatically publish to NATS
|
||||
NATS_connection::Union{NATS.Connection, Nothing} = nothing, # Pre-existing NATS connection (optional, saves connection overhead)
|
||||
msg_id::String = string(uuid4()), # Message ID (auto-generated UUID)
|
||||
sender_id::String = string(uuid4()) # Sender ID (auto-generated UUID)
|
||||
is_publish::Bool = true,
|
||||
NATS_connection::Union{NATS.Connection, Nothing} = nothing,
|
||||
msg_id::String = string(uuid4()),
|
||||
sender_id::String = string(uuid4())
|
||||
)
|
||||
# Returns: ::Tuple{msg_envelope_v1, String}
|
||||
# - env: msg_envelope_v1 object with all envelope metadata and payloads
|
||||
# - env_json_str: JSON string representation of the envelope for publishing
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natbridge');
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
subject,
|
||||
data, // Array of [dataname, data, type] tuples
|
||||
{
|
||||
broker_url: 'nats://localhost:4222',
|
||||
fileserver_url: 'http://localhost:8080',
|
||||
fileserver_upload_handler: NATSBridge.plikOneshotUpload,
|
||||
size_threshold: 1_000_000,
|
||||
correlation_id: uuidv4(),
|
||||
msg_purpose: 'chat',
|
||||
sender_name: 'NATSBridge',
|
||||
receiver_name: '',
|
||||
receiver_id: '',
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
is_publish: true,
|
||||
nats_connection: null,
|
||||
msg_id: uuidv4(),
|
||||
sender_id: uuidv4()
|
||||
}
|
||||
);
|
||||
// Returns: Promise<[env, env_json_str]>
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import NATSBridge
|
||||
|
||||
env, env_json_str = await NATSBridge.smartsend(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
broker_url: str = "nats://localhost:4222",
|
||||
fileserver_url: str = "http://localhost:8080",
|
||||
fileserver_upload_handler: Callable = plik_oneshot_upload,
|
||||
size_threshold: int = 1_000_000,
|
||||
correlation_id: str = None,
|
||||
msg_purpose: str = "chat",
|
||||
sender_name: str = "NATSBridge",
|
||||
receiver_name: str = "",
|
||||
receiver_id: str = "",
|
||||
reply_to: str = "",
|
||||
reply_to_msg_id: str = "",
|
||||
is_publish: bool = True,
|
||||
nats_connection: Any = None,
|
||||
msg_id: str = None,
|
||||
sender_id: str = None
|
||||
)
|
||||
# Returns: Tuple[Dict, str]
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
from natbridge import NATSBridge
|
||||
|
||||
# Limited to direct transport (< 100KB threshold)
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
subject,
|
||||
data, # List of (dataname, data, type) tuples
|
||||
broker_url="nats://localhost:4222",
|
||||
size_threshold=100000 # Lower threshold for memory constraints
|
||||
)
|
||||
# Returns: Tuple[Dict, str]
|
||||
```
|
||||
|
||||
### smartreceive
|
||||
@@ -263,7 +409,6 @@ Receives and processes messages from NATS, handling both direct and link transpo
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Note: msg is a NATS.Msg object passed from the subscription callback
|
||||
env = NATSBridge.smartreceive(
|
||||
msg::NATS.Msg;
|
||||
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||
@@ -271,51 +416,63 @@ env = NATSBridge.smartreceive(
|
||||
base_delay::Int = 100,
|
||||
max_delay::Int = 5000
|
||||
)
|
||||
# Returns: ::JSON.Object{String, Any} - key-value structure resemble msg_envelope_v1
|
||||
# Returns: ::JSON.Object{String, Any}
|
||||
```
|
||||
|
||||
### publish_message
|
||||
#### JavaScript
|
||||
|
||||
Publish a message to a NATS subject. This function is available in Julia with two overloads:
|
||||
```javascript
|
||||
const env = await NATSBridge.smartreceive(
|
||||
msg,
|
||||
{
|
||||
fileserver_download_handler: NATSBridge.fetchWithBackoff,
|
||||
max_retries: 5,
|
||||
base_delay: 100,
|
||||
max_delay: 5000
|
||||
}
|
||||
);
|
||||
// Returns: Promise<env_object>
|
||||
```
|
||||
|
||||
#### Julia
|
||||
#### Python
|
||||
|
||||
**Using broker URL (creates new connection):**
|
||||
```julia
|
||||
using NATSBridge, NATS
|
||||
|
||||
# Publish with URL - creates a new connection
|
||||
NATSBridge.publish_message(
|
||||
"nats://localhost:4222", # broker_url
|
||||
"/chat/room1", # subject
|
||||
"{\"correlation_id\":\"abc123\"}", # message
|
||||
"abc123" # correlation_id
|
||||
```python
|
||||
env = await NATSBridge.smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=fetch_with_backoff,
|
||||
max_retries=5,
|
||||
base_delay=100,
|
||||
max_delay=5000
|
||||
)
|
||||
# Returns: Dict with "payloads" key
|
||||
```
|
||||
|
||||
**Using pre-existing connection (saves connection overhead):**
|
||||
```julia
|
||||
using NATSBridge, NATS
|
||||
#### MicroPython
|
||||
|
||||
# Create connection once and reuse
|
||||
conn = NATS.connect("nats://localhost:4222")
|
||||
NATSBridge.publish_message(conn, "/chat/room1", "{\"correlation_id\":\"abc123\"}", "abc123")
|
||||
# Connection is automatically drained after publish
|
||||
```python
|
||||
env = NATSBridge.smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=_sync_fileserver_download,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
# Returns: Dict with "payloads" key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Payload Types
|
||||
|
||||
| Type | Description | Serialization |
|
||||
|------|-------------|---------------|
|
||||
| `text` | Plain text strings | UTF-8 bytes |
|
||||
| `dictionary` | JSON-serializable dictionaries | JSON |
|
||||
| `table` | Tabular data (DataFrames, arrays) | Apache Arrow IPC |
|
||||
| `image` | Image data (PNG, JPG) | Raw bytes |
|
||||
| `audio` | Audio data (WAV, MP3) | Raw bytes |
|
||||
| `video` | Video data (MP4, AVI) | Raw bytes |
|
||||
| `binary` | Generic binary data | Raw bytes |
|
||||
| Type | Julia | JavaScript | Python | MicroPython | Description |
|
||||
|------|-------|------------|--------|-------------|-------------|
|
||||
| `text` | `String` | `string` | `str` | `str` | Plain text strings |
|
||||
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable dictionaries |
|
||||
| `table` | `DataFrame`, `Arrow.Table` | `Array<Object>` | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) |
|
||||
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image data (PNG, JPG) |
|
||||
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio data (WAV, MP3) |
|
||||
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video data (MP4, AVI) |
|
||||
| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `bytearray` | Generic binary data |
|
||||
|
||||
---
|
||||
|
||||
@@ -325,31 +482,60 @@ NATSBridge.publish_message(conn, "/chat/room1", "{\"correlation_id\":\"abc123\"}
|
||||
|
||||
Small payloads are sent directly via NATS with Base64 encoding.
|
||||
|
||||
#### Julia
|
||||
#### Cross-Platform
|
||||
|
||||
```julia
|
||||
# Julia
|
||||
data = [("message", "Hello", "text")]
|
||||
smartsend("/topic", data)
|
||||
```
|
||||
|
||||
```javascript
|
||||
// JavaScript
|
||||
const data = [["message", "Hello", "text"]];
|
||||
smartsend("/topic", data);
|
||||
```
|
||||
|
||||
```python
|
||||
# Python
|
||||
data = [("message", "Hello", "text")]
|
||||
await smartsend("/topic", data)
|
||||
```
|
||||
|
||||
### Link Transport (Payloads >= 1MB)
|
||||
|
||||
Large payloads are uploaded to an HTTP file server.
|
||||
|
||||
#### Julia
|
||||
#### Cross-Platform
|
||||
|
||||
```julia
|
||||
# Julia
|
||||
data = [("file", large_data, "binary")]
|
||||
smartsend("/topic", data; fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
```javascript
|
||||
// JavaScript
|
||||
const data = [["file", largeData, "binary"]];
|
||||
smartsend("/topic", data, { fileserver_url: 'http://localhost:8080' });
|
||||
```
|
||||
|
||||
```python
|
||||
# Python
|
||||
data = [("file", large_data, "binary")]
|
||||
await smartsend("/topic", data, fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
## Cross-Platform Examples
|
||||
|
||||
### Example 1: Chat with Mixed Content
|
||||
|
||||
Send text, small image, and large file in one message.
|
||||
Send text, image, and large file in one message.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
@@ -362,11 +548,48 @@ data = [
|
||||
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natbridge');
|
||||
|
||||
const data = [
|
||||
["message_text", "Hello!", "text"],
|
||||
["user_avatar", imageData, "image"],
|
||||
["large_document", largeFileData, "binary"]
|
||||
];
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ fileserver_url: 'http://localhost:8080' }
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import NATSBridge
|
||||
|
||||
data = [
|
||||
("message_text", "Hello!", "text"),
|
||||
("user_avatar", image_data, "image"),
|
||||
("large_document", large_file_data, "binary")
|
||||
]
|
||||
|
||||
env, env_json_str = await NATSBridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
fileserver_url="http://localhost:8080"
|
||||
)
|
||||
```
|
||||
|
||||
### Example 2: Dictionary Exchange
|
||||
|
||||
Send configuration data between platforms.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
@@ -380,11 +603,44 @@ data = [("config", config, "dictionary")]
|
||||
env, env_json_str = NATSBridge.smartsend("/device/config", data)
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natbridge');
|
||||
|
||||
const config = {
|
||||
wifi_ssid: "MyNetwork",
|
||||
wifi_password: "password123",
|
||||
update_interval: 60
|
||||
};
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/device/config",
|
||||
[["config", config, "dictionary"]]
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import NATSBridge
|
||||
|
||||
config = {
|
||||
"wifi_ssid": "MyNetwork",
|
||||
"wifi_password": "password123",
|
||||
"update_interval": 60
|
||||
}
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env, env_json_str = await NATSBridge.smartsend("/device/config", data)
|
||||
```
|
||||
|
||||
### Example 3: Table Data (Arrow IPC)
|
||||
|
||||
Send tabular data using Apache Arrow IPC format.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
using DataFrames
|
||||
@@ -399,14 +655,49 @@ data = [("students", df, "table")]
|
||||
env, env_json_str = NATSBridge.smartsend("/data/analysis", data)
|
||||
```
|
||||
|
||||
### Example 4: Request-Response Pattern with Envelope JSON
|
||||
#### JavaScript
|
||||
|
||||
Bi-directional communication with reply-to support. The `smartsend` function now returns both the envelope object and a JSON string that can be published directly.
|
||||
```javascript
|
||||
const NATSBridge = require('natbridge');
|
||||
|
||||
const df = [
|
||||
{ id: 1, name: "Alice", score: 95 },
|
||||
{ id: 2, name: "Bob", score: 88 },
|
||||
{ id: 3, name: "Charlie", score: 92 }
|
||||
];
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/data/analysis",
|
||||
[["students", df, "table"]]
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import NATSBridge
|
||||
import pandas as pd
|
||||
|
||||
df = pd.DataFrame({
|
||||
"id": [1, 2, 3],
|
||||
"name": ["Alice", "Bob", "Charlie"],
|
||||
"score": [95, 88, 92]
|
||||
})
|
||||
|
||||
data = [("students", df, "table")]
|
||||
env, env_json_str = await NATSBridge.smartsend("/data/analysis", data)
|
||||
```
|
||||
|
||||
### Example 4: Request-Response Pattern
|
||||
|
||||
Bi-directional communication with reply-to support.
|
||||
|
||||
#### Julia
|
||||
|
||||
#### Julia (Requester)
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Requester
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
"/device/command",
|
||||
[("command", Dict("action" => "read_sensor"), "dictionary")];
|
||||
@@ -415,26 +706,20 @@ env, env_json_str = NATSBridge.smartsend(
|
||||
)
|
||||
```
|
||||
|
||||
#### Julia (Responder)
|
||||
```julia
|
||||
# Responder
|
||||
using NATS, NATSBridge
|
||||
|
||||
# Configuration
|
||||
const SUBJECT = "/device/command"
|
||||
const NATS_URL = "nats://localhost:4222"
|
||||
|
||||
function test_responder()
|
||||
conn = NATS.connect(NATS_URL)
|
||||
NATS.subscribe(conn, SUBJECT) do msg
|
||||
conn = NATS.connect("nats://localhost:4222")
|
||||
NATS.subscribe(conn, "/device/command") do msg
|
||||
env = NATSBridge.smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
|
||||
|
||||
# Extract reply_to from the envelope metadata
|
||||
reply_to = env["reply_to"]
|
||||
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
if dataname == "command" && data["action"] == "read_sensor"
|
||||
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
|
||||
# Send response to the reply_to subject from the request
|
||||
if !isempty(reply_to)
|
||||
smartsend(reply_to, [("data", response, "dictionary")])
|
||||
end
|
||||
@@ -445,51 +730,118 @@ function test_responder()
|
||||
sleep(120)
|
||||
NATS.drain(conn)
|
||||
end
|
||||
|
||||
test_responder()
|
||||
```
|
||||
|
||||
### Example 5: IoT Device Sensor Data
|
||||
#### JavaScript
|
||||
|
||||
IoT device sending sensor data.
|
||||
```javascript
|
||||
const NATSBridge = require('natbridge');
|
||||
|
||||
#### Julia (Receiver)
|
||||
```julia
|
||||
using NATS, NATSBridge
|
||||
// Requester
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/device/command",
|
||||
[["command", { action: "read_sensor" }, "dictionary"]],
|
||||
{ broker_url: 'nats://localhost:4222', reply_to: '/device/response' }
|
||||
);
|
||||
```
|
||||
|
||||
# Configuration
|
||||
const SUBJECT = "/device/sensors"
|
||||
const NATS_URL = "nats://localhost:4222"
|
||||
```javascript
|
||||
// Responder
|
||||
const nats = require('nats');
|
||||
const NATSBridge = require('natbridge');
|
||||
|
||||
function test_receiver()
|
||||
conn = NATS.connect(NATS_URL)
|
||||
NATS.subscribe(conn, SUBJECT) do msg
|
||||
env, env_json_str = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
if dataname == "temperature"
|
||||
println("Temperature: $data")
|
||||
elseif dataname == "humidity"
|
||||
println("Humidity: $data")
|
||||
end
|
||||
end
|
||||
end
|
||||
async function testResponder() {
|
||||
const conn = await nats.connect('nats://localhost:4222');
|
||||
|
||||
sleep(120)
|
||||
NATS.drain(conn)
|
||||
end
|
||||
const subscription = await conn.subscribe('/device/command');
|
||||
|
||||
for await (const msg of subscription) {
|
||||
const env = await NATSBridge.smartreceive(msg, {
|
||||
fileserver_download_handler: NATSBridge.fetchWithBackoff
|
||||
});
|
||||
|
||||
const replyTo = env.reply_to;
|
||||
|
||||
for (const [dataname, data, type] of env.payloads) {
|
||||
if (dataname === 'command' && data.action === 'read_sensor') {
|
||||
const response = { sensor_id: 'sensor-001', value: 42.5 };
|
||||
if (replyTo) {
|
||||
await NATSBridge.smartsend(
|
||||
replyTo,
|
||||
[["data", response, "dictionary"]]
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
setTimeout(() => conn.close(), 120000);
|
||||
}
|
||||
```
|
||||
|
||||
test_receiver()
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import NATSBridge
|
||||
|
||||
# Requester
|
||||
env, env_json_str = await NATSBridge.smartsend(
|
||||
"/device/command",
|
||||
[("command", {"action": "read_sensor"}, "dictionary")],
|
||||
broker_url="nats://localhost:4222",
|
||||
reply_to="/device/response"
|
||||
)
|
||||
```
|
||||
|
||||
```python
|
||||
# Responder
|
||||
from natbridge import NATSBridge
|
||||
import asyncio
|
||||
import nats
|
||||
|
||||
async def test_responder():
|
||||
nc = await nats.connect('nats://localhost:4222')
|
||||
|
||||
async def msg_handler(msg):
|
||||
env = await NATSBridge.smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=fetch_with_backoff
|
||||
)
|
||||
|
||||
reply_to = env["reply_to"]
|
||||
|
||||
for dataname, data, type_ in env["payloads"]:
|
||||
if dataname == "command" and data["action"] == "read_sensor":
|
||||
response = {"sensor_id": "sensor-001", "value": 42.5}
|
||||
if reply_to:
|
||||
await NATSBridge.smartsend(
|
||||
reply_to,
|
||||
[("data", response, "dictionary")]
|
||||
)
|
||||
|
||||
await nc.subscribe('/device/command', cb=msg_handler)
|
||||
|
||||
await asyncio.sleep(120)
|
||||
await nc.drain()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
Run the test scripts to verify functionality:
|
||||
### Test File Organization
|
||||
|
||||
### Julia
|
||||
| Platform | Sender Tests | Receiver Tests |
|
||||
|----------|--------------|----------------|
|
||||
| **Julia** | `test/test_julia_*_sender.jl` | `test/test_julia_*_receiver.jl` |
|
||||
| **JavaScript** | `test/test_js_*_sender.js` | `test/test_js_*_receiver.js` |
|
||||
| **Python** | `test/test_py_*_sender.py` | `test/test_py_*_receiver.py` |
|
||||
|
||||
```julia
|
||||
### Run Tests
|
||||
|
||||
#### Julia
|
||||
|
||||
```bash
|
||||
# Text message exchange
|
||||
julia test/test_julia_text_sender.jl
|
||||
julia test/test_julia_text_receiver.jl
|
||||
@@ -511,6 +863,55 @@ julia test/test_julia_table_sender.jl
|
||||
julia test/test_julia_table_receiver.jl
|
||||
```
|
||||
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```bash
|
||||
# Text message exchange
|
||||
node test/test_js_text_sender.js
|
||||
node test/test_js_text_receiver.js
|
||||
|
||||
# Dictionary exchange
|
||||
node test/test_js_dictionary_sender.js
|
||||
node test/test_js_dictionary_receiver.js
|
||||
|
||||
# Binary transfer
|
||||
node test/test_js_binary_sender.js
|
||||
node test/test_js_binary_receiver.js
|
||||
|
||||
# Table exchange
|
||||
node test/test_js_table_sender.js
|
||||
node test/test_js_table_receiver.js
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```bash
|
||||
# Text message exchange
|
||||
python3 test/test_py_text_sender.py
|
||||
python3 test/test_py_text_receiver.py
|
||||
|
||||
# Dictionary exchange
|
||||
python3 test/test_py_dictionary_sender.py
|
||||
python3 test/test_py_dictionary_receiver.py
|
||||
|
||||
# Binary transfer
|
||||
python3 test/test_py_binary_sender.py
|
||||
python3 test/test_py_binary_receiver.py
|
||||
|
||||
# Table exchange
|
||||
python3 test/test_py_table_sender.py
|
||||
python3 test/test_py_table_receiver.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
For detailed architecture and implementation information, see:
|
||||
|
||||
- [Architecture Documentation](docs/architecture.md) - Cross-platform architecture, API parity, platform-specific patterns
|
||||
- [Implementation Guide](docs/implementation.md) - Detailed implementation for each platform, handler functions, testing
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
Reference in New Issue
Block a user