495 lines
14 KiB
Markdown
495 lines
14 KiB
Markdown
# NATSBridge
|
|
|
|
A high-performance, bi-directional data bridge for **Julia** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
|
|
|
[](https://opensource.org/licenses/MIT)
|
|
[](https://nats.io)
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
|
|
- [Overview](#overview)
|
|
- [Features](#features)
|
|
- [Architecture](#architecture)
|
|
- [Installation](#installation)
|
|
- [Quick Start](#quick-start)
|
|
- [API Reference](#api-reference)
|
|
- [Payload Types](#payload-types)
|
|
- [Transport Strategies](#transport-strategies)
|
|
- [Examples](#examples)
|
|
- [Testing](#testing)
|
|
- [License](#license)
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
NATSBridge enables seamless communication for Julia applications through NATS, with intelligent transport selection based on payload size:
|
|
|
|
| Transport | Payload Size | Method |
|
|
|-----------|--------------|--------|
|
|
| **Direct** | < 1MB | Sent directly via NATS (Base64 encoded) |
|
|
| **Link** | >= 1MB | Uploaded to HTTP file server, URL sent via NATS |
|
|
|
|
### Use Cases
|
|
|
|
- **Chat Applications**: Text, images, audio, video in a single message
|
|
- **File Transfer**: Efficient transfer of large files using claim-check pattern
|
|
- **Streaming Data**: Sensor data, telemetry, and analytics pipelines
|
|
|
|
|
|
---
|
|
|
|
## Features
|
|
|
|
- ✅ **Bi-directional messaging** for Julia applications
|
|
- ✅ **Multi-payload support** - send multiple payloads with different types in one message
|
|
- ✅ **Automatic transport selection** - direct vs link based on payload size
|
|
- ✅ **Claim-Check pattern** for payloads > 1MB
|
|
- ✅ **Apache Arrow IPC** support for tabular data (zero-copy reading)
|
|
- ✅ **Exponential backoff** for reliable file server downloads
|
|
- ✅ **Correlation ID tracking** for message tracing
|
|
- ✅ **Reply-to support** for request-response patterns
|
|
- ✅ **JetStream support** for message replay and durability
|
|
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
### System Components
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ NATSBridge Architecture │
|
|
├─────────────────────────────────────────────────────────────────────┤
|
|
│ ┌──────────────┐ │ │
|
|
│ │ Julia │ ▼ │
|
|
│ │ (NATS.jl) │ ┌─────────────────────────┐ │
|
|
│ └──────────────┘ │ NATS │ │
|
|
│ │ (Message Broker) │ │
|
|
│ └─────────────────────────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌──────────────────────┐ │
|
|
│ │ File Server │ │
|
|
│ │ (HTTP Upload/Get) │ │
|
|
│ └──────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Message Flow
|
|
|
|
1. **Sender** creates a message envelope with payloads
|
|
2. **NATSBridge** serializes and encodes payloads based on type
|
|
3. **Transport Decision**: Small payloads go directly to NATS, large payloads are uploaded to file server
|
|
4. **NATS** routes messages to subscribers
|
|
5. **Receiver** fetches payloads (from NATS or file server)
|
|
6. **NATSBridge** deserializes and decodes payloads
|
|
|
|
---
|
|
|
|
## Installation
|
|
|
|
### Prerequisites
|
|
|
|
- **NATS Server** (v2.10+ recommended)
|
|
- **HTTP File Server** (optional, for payloads > 1MB)
|
|
|
|
### Julia
|
|
|
|
```julia
|
|
using Pkg
|
|
Pkg.add("NATS")
|
|
Pkg.add("https://git.yiem.cc/ton/NATSBridge")
|
|
```
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
### Step 1: Start NATS Server
|
|
|
|
```bash
|
|
docker run -p 4222:4222 nats:latest
|
|
```
|
|
|
|
### Step 2: Start HTTP File Server (Optional)
|
|
|
|
```bash
|
|
# Create a directory for file uploads
|
|
mkdir -p /tmp/fileserver
|
|
|
|
# Start HTTP file server
|
|
python3 -m http.server 8080 --directory /tmp/fileserver
|
|
```
|
|
|
|
### Step 3: Send Your First Message
|
|
|
|
#### Julia
|
|
|
|
```julia
|
|
using NATSBridge
|
|
|
|
# Send a text message
|
|
data = [("message", "Hello World", "text")]
|
|
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; broker_url="nats://localhost:4222")
|
|
println("Message sent!")
|
|
```
|
|
|
|
### Step 4: Receive Messages
|
|
|
|
#### Julia
|
|
|
|
```julia
|
|
using NATS, NATSBridge
|
|
|
|
# Configuration
|
|
const SUBJECT = "/chat/room1"
|
|
const NATS_URL = "nats://localhost:4222"
|
|
|
|
# Helper: Log with correlation ID
|
|
function log_trace(message)
|
|
timestamp = Dates.now()
|
|
println("[$timestamp] $message")
|
|
end
|
|
|
|
# Receiver: Listen for messages - msg comes from the callback
|
|
function test_receive()
|
|
conn = NATS.connect(NATS_URL)
|
|
NATS.subscribe(conn, SUBJECT) do msg
|
|
log_trace("Received message on $(msg.subject)")
|
|
|
|
# Receive and process message
|
|
env, env_json_str = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
|
|
for (dataname, data, type) in env["payloads"]
|
|
println("Received $dataname: $data")
|
|
end
|
|
end
|
|
|
|
# Keep listening for 120 seconds
|
|
sleep(120)
|
|
NATS.drain(conn)
|
|
end
|
|
|
|
test_receive()
|
|
```
|
|
|
|
---
|
|
|
|
## API Reference
|
|
|
|
### smartsend
|
|
|
|
Sends data either directly via NATS or via a fileserver URL, depending on payload size.
|
|
|
|
#### Julia
|
|
|
|
```julia
|
|
using NATSBridge
|
|
|
|
env, env_json_str = NATSBridge.smartsend(
|
|
subject, # NATS subject
|
|
data::AbstractArray{Tuple{String, Any, String}}; # List of (dataname, data, type)
|
|
broker_url::String = "nats://localhost:4222",
|
|
fileserver_url = "http://localhost:8080",
|
|
fileserver_upload_handler::Function = plik_oneshot_upload,
|
|
size_threshold::Int = 1_000_000,
|
|
correlation_id::Union{String, Nothing} = nothing,
|
|
msg_purpose::String = "chat",
|
|
sender_name::String = "NATSBridge",
|
|
receiver_name::String = "",
|
|
receiver_id::String = "",
|
|
reply_to::String = "",
|
|
reply_to_msg_id::String = "",
|
|
is_publish::Bool = true, # Whether to automatically publish to NATS
|
|
NATS_connection::Union{NATS.Connection, Nothing} = nothing # Pre-existing NATS connection (optional, saves connection overhead)
|
|
)
|
|
# Returns: (msgEnvelope_v1, JSON string)
|
|
# - env: msgEnvelope_v1 object with all envelope metadata and payloads
|
|
# - env_json_str: JSON string representation of the envelope for publishing
|
|
```
|
|
|
|
### smartreceive
|
|
|
|
Receives and processes messages from NATS, handling both direct and link transport.
|
|
|
|
#### Julia
|
|
|
|
```julia
|
|
using NATSBridge
|
|
|
|
# Note: msg is a NATS.Msg object passed from the subscription callback
|
|
env = NATSBridge.smartreceive(
|
|
msg::NATS.Msg;
|
|
fileserver_download_handler::Function = _fetch_with_backoff,
|
|
max_retries::Int = 5,
|
|
base_delay::Int = 100,
|
|
max_delay::Int = 5000
|
|
)
|
|
# Returns: Dict with envelope metadata and payloads array
|
|
```
|
|
|
|
### publish_message
|
|
|
|
Publish a message to a NATS subject. This function is available in Julia with two overloads:
|
|
|
|
#### Julia
|
|
|
|
**Using broker URL (creates new connection):**
|
|
```julia
|
|
using NATSBridge, NATS
|
|
|
|
# Publish with URL - creates a new connection
|
|
NATSBridge.publish_message(
|
|
"nats://localhost:4222", # broker_url
|
|
"/chat/room1", # subject
|
|
"{\"correlation_id\":\"abc123\"}", # message
|
|
"abc123" # correlation_id
|
|
)
|
|
```
|
|
|
|
**Using pre-existing connection (saves connection overhead):**
|
|
```julia
|
|
using NATSBridge, NATS
|
|
|
|
# Create connection once and reuse
|
|
conn = NATS.connect("nats://localhost:4222")
|
|
NATSBridge.publish_message(conn, "/chat/room1", "{\"correlation_id\":\"abc123\"}", "abc123")
|
|
# Connection is automatically drained after publish
|
|
```
|
|
|
|
---
|
|
|
|
## Payload Types
|
|
|
|
| Type | Description | Serialization |
|
|
|------|-------------|---------------|
|
|
| `text` | Plain text strings | UTF-8 bytes |
|
|
| `dictionary` | JSON-serializable dictionaries | JSON |
|
|
| `table` | Tabular data (DataFrames, arrays) | Apache Arrow IPC |
|
|
| `image` | Image data (PNG, JPG) | Raw bytes |
|
|
| `audio` | Audio data (WAV, MP3) | Raw bytes |
|
|
| `video` | Video data (MP4, AVI) | Raw bytes |
|
|
| `binary` | Generic binary data | Raw bytes |
|
|
|
|
---
|
|
|
|
## Transport Strategies
|
|
|
|
### Direct Transport (Payloads < 1MB)
|
|
|
|
Small payloads are sent directly via NATS with Base64 encoding.
|
|
|
|
#### Julia
|
|
```julia
|
|
data = [("message", "Hello", "text")]
|
|
smartsend("/topic", data)
|
|
```
|
|
|
|
### Link Transport (Payloads >= 1MB)
|
|
|
|
Large payloads are uploaded to an HTTP file server.
|
|
|
|
#### Julia
|
|
```julia
|
|
data = [("file", large_data, "binary")]
|
|
smartsend("/topic", data; fileserver_url="http://localhost:8080")
|
|
```
|
|
|
|
---
|
|
|
|
## Examples
|
|
|
|
### Example 1: Chat with Mixed Content
|
|
|
|
Send text, small image, and large file in one message.
|
|
|
|
#### Julia
|
|
```julia
|
|
using NATSBridge
|
|
|
|
data = [
|
|
("message_text", "Hello!", "text"),
|
|
("user_avatar", image_data, "image"),
|
|
("large_document", large_file_data, "binary")
|
|
]
|
|
|
|
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; fileserver_url="http://localhost:8080")
|
|
```
|
|
|
|
### Example 2: Dictionary Exchange
|
|
|
|
Send configuration data between platforms.
|
|
|
|
#### Julia
|
|
```julia
|
|
using NATSBridge
|
|
|
|
config = Dict(
|
|
"wifi_ssid" => "MyNetwork",
|
|
"wifi_password" => "password123",
|
|
"update_interval" => 60
|
|
)
|
|
|
|
data = [("config", config, "dictionary")]
|
|
env, env_json_str = NATSBridge.smartsend("/device/config", data)
|
|
```
|
|
|
|
### Example 3: Table Data (Arrow IPC)
|
|
|
|
Send tabular data using Apache Arrow IPC format.
|
|
|
|
#### Julia
|
|
```julia
|
|
using NATSBridge
|
|
using DataFrames
|
|
|
|
df = DataFrame(
|
|
id = [1, 2, 3],
|
|
name = ["Alice", "Bob", "Charlie"],
|
|
score = [95, 88, 92]
|
|
)
|
|
|
|
data = [("students", df, "table")]
|
|
env, env_json_str = NATSBridge.smartsend("/data/analysis", data)
|
|
```
|
|
|
|
### Example 4: Request-Response Pattern with Envelope JSON
|
|
|
|
Bi-directional communication with reply-to support. The `smartsend` function now returns both the envelope object and a JSON string that can be published directly.
|
|
|
|
#### Julia (Requester)
|
|
```julia
|
|
using NATSBridge
|
|
|
|
env, env_json_str = NATSBridge.smartsend(
|
|
"/device/command",
|
|
[("command", Dict("action" => "read_sensor"), "dictionary")];
|
|
broker_url="nats://localhost:4222",
|
|
reply_to="/device/response"
|
|
)
|
|
```
|
|
|
|
#### Julia (Responder)
|
|
```julia
|
|
using NATS, NATSBridge
|
|
|
|
# Configuration
|
|
const SUBJECT = "/device/command"
|
|
const NATS_URL = "nats://localhost:4222"
|
|
|
|
function test_responder()
|
|
conn = NATS.connect(NATS_URL)
|
|
NATS.subscribe(conn, SUBJECT) do msg
|
|
env = NATSBridge.smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
|
|
|
|
# Extract reply_to from the envelope metadata
|
|
reply_to = env["reply_to"]
|
|
|
|
for (dataname, data, type) in env["payloads"]
|
|
if dataname == "command" && data["action"] == "read_sensor"
|
|
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
|
|
# Send response to the reply_to subject from the request
|
|
if !isempty(reply_to)
|
|
smartsend(reply_to, [("data", response, "dictionary")])
|
|
end
|
|
end
|
|
end
|
|
end
|
|
|
|
sleep(120)
|
|
NATS.drain(conn)
|
|
end
|
|
|
|
test_responder()
|
|
```
|
|
|
|
### Example 5: IoT Device Sensor Data
|
|
|
|
IoT device sending sensor data.
|
|
|
|
#### Julia (Receiver)
|
|
```julia
|
|
using NATS, NATSBridge
|
|
|
|
# Configuration
|
|
const SUBJECT = "/device/sensors"
|
|
const NATS_URL = "nats://localhost:4222"
|
|
|
|
function test_receiver()
|
|
conn = NATS.connect(NATS_URL)
|
|
NATS.subscribe(conn, SUBJECT) do msg
|
|
env, env_json_str = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
|
|
for (dataname, data, type) in env["payloads"]
|
|
if dataname == "temperature"
|
|
println("Temperature: $data")
|
|
elseif dataname == "humidity"
|
|
println("Humidity: $data")
|
|
end
|
|
end
|
|
end
|
|
|
|
sleep(120)
|
|
NATS.drain(conn)
|
|
end
|
|
|
|
test_receiver()
|
|
```
|
|
|
|
---
|
|
|
|
## Testing
|
|
|
|
Run the test scripts to verify functionality:
|
|
|
|
### Julia
|
|
|
|
```julia
|
|
# Text message exchange
|
|
julia test/test_julia_text_sender.jl
|
|
julia test/test_julia_text_receiver.jl
|
|
|
|
# Dictionary exchange
|
|
julia test/test_julia_dict_sender.jl
|
|
julia test/test_julia_dict_receiver.jl
|
|
|
|
# File transfer
|
|
julia test/test_julia_file_sender.jl
|
|
julia test/test_julia_file_receiver.jl
|
|
|
|
# Mixed payload types
|
|
julia test/test_julia_mix_payloads_sender.jl
|
|
julia test/test_julia_mix_payloads_receiver.jl
|
|
|
|
# Table exchange
|
|
julia test/test_julia_table_sender.jl
|
|
julia test/test_julia_table_receiver.jl
|
|
```
|
|
|
|
---
|
|
|
|
## License
|
|
|
|
MIT License
|
|
|
|
Copyright (c) 2026 NATSBridge Contributors
|
|
|
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
of this software and associated documentation files (the "Software"), to deal
|
|
in the Software without restriction, including without limitation the rights
|
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
copies of the Software, and to permit persons to whom the Software is
|
|
furnished to do so, subject to the following conditions:
|
|
|
|
The above copyright notice and this permission notice shall be included in all
|
|
copies or substantial portions of the Software.
|
|
|
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
SOFTWARE. |