update
This commit is contained in:
538
README.md
Normal file
538
README.md
Normal file
@@ -0,0 +1,538 @@
|
||||
# NATSBridge
|
||||
|
||||
A high-performance, bi-directional data bridge for **Julia** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://nats.io)
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Features](#features)
|
||||
- [Architecture](#architecture)
|
||||
- [Installation](#installation)
|
||||
- [Quick Start](#quick-start)
|
||||
- [API Reference](#api-reference)
|
||||
- [Payload Types](#payload-types)
|
||||
- [Transport Strategies](#transport-strategies)
|
||||
- [Examples](#examples)
|
||||
- [Testing](#testing)
|
||||
- [License](#license)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
NATSBridge enables seamless communication for Julia applications through NATS, with intelligent transport selection based on payload size:
|
||||
|
||||
| Transport | Payload Size | Method |
|
||||
|-----------|--------------|--------|
|
||||
| **Direct** | < 1MB | Sent directly via NATS (Base64 encoded) |
|
||||
| **Link** | >= 1MB | Uploaded to HTTP file server, URL sent via NATS |
|
||||
|
||||
### Use Cases
|
||||
|
||||
- **Chat Applications**: Text, images, audio, video in a single message
|
||||
- **File Transfer**: Efficient transfer of large files using claim-check pattern
|
||||
- **Streaming Data**: Sensor data, telemetry, and analytics pipelines
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
- ✅ **Bi-directional messaging** for Julia applications
|
||||
- ✅ **Multi-payload support** - send multiple payloads with different types in one message
|
||||
- ✅ **Automatic transport selection** - direct vs link based on payload size
|
||||
- ✅ **Claim-Check pattern** for payloads > 1MB
|
||||
- ✅ **Apache Arrow IPC** support for tabular data (zero-copy reading)
|
||||
- ✅ **Exponential backoff** for reliable file server downloads
|
||||
- ✅ **Correlation ID tracking** for message tracing
|
||||
- ✅ **Reply-to support** for request-response patterns
|
||||
- ✅ **JetStream support** for message replay and durability
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### System Components
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Sender["Julia Application (Sender)"]
|
||||
SenderApp[App Code]
|
||||
NATSBridge_Send[NATSBridge]
|
||||
NATS_Client[<b>NATS.jl</b>]
|
||||
end
|
||||
|
||||
subgraph Receiver["Julia Application (Receiver)"]
|
||||
ReceiverApp[App Code]
|
||||
NATSBridge_Recv[NATSBridge]
|
||||
NATS_Client_Recv[<b>NATS.jl</b>]
|
||||
end
|
||||
|
||||
subgraph Infrastructure["Infrastructure"]
|
||||
NATS[<b>NATS Server</b><br/>Message Broker]
|
||||
FileServer[<b>HTTP File Server</b><br/>Upload/Download]
|
||||
end
|
||||
|
||||
SenderApp --> NATSBridge_Send
|
||||
NATSBridge_Send --> NATS_Client
|
||||
NATS_Client --> NATS
|
||||
|
||||
NATS --> NATS_Client_Recv
|
||||
NATS_Client_Recv --> NATSBridge_Recv
|
||||
NATSBridge_Recv --> ReceiverApp
|
||||
|
||||
NATSBridge_Send -.->|HTTP POST upload| FileServer
|
||||
FileServer -.->|HTTP GET download| NATSBridge_Recv
|
||||
|
||||
style SenderApp fill:#e8f5e9
|
||||
style ReceiverApp fill:#e8f5e9
|
||||
style NATS fill:#fff3e0
|
||||
style FileServer fill:#f3e5f5
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
| Component | Description |
|
||||
|-----------|-------------|
|
||||
| **Julia Application** | Sender and receiver applications using the NATSBridge module |
|
||||
| **NATS Server** | Message broker for transporting message envelopes |
|
||||
| **HTTP File Server** | Independent HTTP server for large payload storage (e.g., Plik) |
|
||||
|
||||
### Message Flow
|
||||
|
||||
1. **Sender** creates a message envelope with payloads using `smartsend()`
|
||||
2. **NATSBridge** serializes and encodes each payload based on type
|
||||
3. **Transport Decision**:
|
||||
- **Direct** (< 1MB): Payload encoded as Base64, published to NATS
|
||||
- **Link** (≥ 1MB): Payload uploaded to HTTP file server, URL published to NATS
|
||||
4. **NATS** routes message envelope to subscribers
|
||||
5. **Receiver** receives message via NATS subscription callback
|
||||
6. **NATSBridge** processes envelope:
|
||||
- Decodes Base64 payloads from NATS message
|
||||
- Fetches URLs from file server with exponential backoff
|
||||
7. **Receiver** deserializes payloads based on their type
|
||||
|
||||
### File Server Handler Abstraction
|
||||
|
||||
The system uses handler functions to abstract file server operations:
|
||||
|
||||
| Handler | Purpose |
|
||||
|---------|---------|
|
||||
| `plik_oneshot_upload()` | Uploads payload bytes to file server, returns URL |
|
||||
| `_fetch_with_backoff()` | Downloads data from URL with exponential backoff retry |
|
||||
|
||||
This abstraction allows support for different file server implementations (Plik, AWS S3, custom HTTP server).
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **NATS Server** (v2.10+ recommended)
|
||||
- **HTTP File Server** (optional, for payloads > 1MB)
|
||||
|
||||
### Julia
|
||||
|
||||
```julia
|
||||
using Pkg
|
||||
Pkg.add("NATS")
|
||||
Pkg.add("https://git.yiem.cc/ton/NATSBridge")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Step 1: Start NATS Server
|
||||
|
||||
```bash
|
||||
docker run -p 4222:4222 nats:latest
|
||||
```
|
||||
|
||||
### Step 2: Start HTTP File Server (Optional)
|
||||
|
||||
```bash
|
||||
# Create a directory for file uploads
|
||||
mkdir -p /tmp/fileserver
|
||||
|
||||
# Start HTTP file server
|
||||
python3 -m http.server 8080 --directory /tmp/fileserver
|
||||
```
|
||||
|
||||
### Step 3: Send Your First Message
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Send a text message
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; broker_url="nats://localhost:4222")
|
||||
println("Message sent!")
|
||||
```
|
||||
|
||||
### Step 4: Receive Messages
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATS, NATSBridge
|
||||
|
||||
# Configuration
|
||||
const SUBJECT = "/chat/room1"
|
||||
const NATS_URL = "nats://localhost:4222"
|
||||
|
||||
# Helper: Log with correlation ID
|
||||
function log_trace(message)
|
||||
timestamp = Dates.now()
|
||||
println("[$timestamp] $message")
|
||||
end
|
||||
|
||||
# Receiver: Listen for messages - msg comes from the callback
|
||||
function test_receive()
|
||||
conn = NATS.connect(NATS_URL)
|
||||
NATS.subscribe(conn, SUBJECT) do msg
|
||||
log_trace("Received message on $(msg.subject)")
|
||||
|
||||
# Receive and process message
|
||||
env, env_json_str = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
println("Received $dataname: $data")
|
||||
end
|
||||
end
|
||||
|
||||
# Keep listening for 120 seconds
|
||||
sleep(120)
|
||||
NATS.drain(conn)
|
||||
end
|
||||
|
||||
test_receive()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### smartsend
|
||||
|
||||
Sends data either directly via NATS or via a fileserver URL, depending on payload size.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
subject, # NATS subject
|
||||
data::AbstractArray{Tuple{String, Any, String}}; # List of (dataname, data, type)
|
||||
broker_url::String = "nats://localhost:4222",
|
||||
fileserver_url = "http://localhost:8080",
|
||||
fileserver_upload_handler::Function = plik_oneshot_upload,
|
||||
size_threshold::Int = 1_000_000,
|
||||
correlation_id::String = string(uuid4()), # Correlation ID for tracing (auto-generated UUID)
|
||||
msg_purpose::String = "chat",
|
||||
sender_name::String = "NATSBridge",
|
||||
receiver_name::String = "",
|
||||
receiver_id::String = "",
|
||||
reply_to::String = "",
|
||||
reply_to_msg_id::String = "",
|
||||
is_publish::Bool = true, # Whether to automatically publish to NATS
|
||||
NATS_connection::Union{NATS.Connection, Nothing} = nothing, # Pre-existing NATS connection (optional, saves connection overhead)
|
||||
msg_id::String = string(uuid4()), # Message ID (auto-generated UUID)
|
||||
sender_id::String = string(uuid4()) # Sender ID (auto-generated UUID)
|
||||
)
|
||||
# Returns: ::Tuple{msg_envelope_v1, String}
|
||||
# - env: msg_envelope_v1 object with all envelope metadata and payloads
|
||||
# - env_json_str: JSON string representation of the envelope for publishing
|
||||
```
|
||||
|
||||
### smartreceive
|
||||
|
||||
Receives and processes messages from NATS, handling both direct and link transport.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Note: msg is a NATS.Msg object passed from the subscription callback
|
||||
env = NATSBridge.smartreceive(
|
||||
msg::NATS.Msg;
|
||||
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||
max_retries::Int = 5,
|
||||
base_delay::Int = 100,
|
||||
max_delay::Int = 5000
|
||||
)
|
||||
# Returns: ::JSON.Object{String, Any} - key-value structure resemble msg_envelope_v1
|
||||
```
|
||||
|
||||
### publish_message
|
||||
|
||||
Publish a message to a NATS subject. This function is available in Julia with two overloads:
|
||||
|
||||
#### Julia
|
||||
|
||||
**Using broker URL (creates new connection):**
|
||||
```julia
|
||||
using NATSBridge, NATS
|
||||
|
||||
# Publish with URL - creates a new connection
|
||||
NATSBridge.publish_message(
|
||||
"nats://localhost:4222", # broker_url
|
||||
"/chat/room1", # subject
|
||||
"{\"correlation_id\":\"abc123\"}", # message
|
||||
"abc123" # correlation_id
|
||||
)
|
||||
```
|
||||
|
||||
**Using pre-existing connection (saves connection overhead):**
|
||||
```julia
|
||||
using NATSBridge, NATS
|
||||
|
||||
# Create connection once and reuse
|
||||
conn = NATS.connect("nats://localhost:4222")
|
||||
NATSBridge.publish_message(conn, "/chat/room1", "{\"correlation_id\":\"abc123\"}", "abc123")
|
||||
# Connection is automatically drained after publish
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Payload Types
|
||||
|
||||
| Type | Description | Serialization |
|
||||
|------|-------------|---------------|
|
||||
| `text` | Plain text strings | UTF-8 bytes |
|
||||
| `dictionary` | JSON-serializable dictionaries | JSON |
|
||||
| `table` | Tabular data (DataFrames, arrays) | Apache Arrow IPC |
|
||||
| `image` | Image data (PNG, JPG) | Raw bytes |
|
||||
| `audio` | Audio data (WAV, MP3) | Raw bytes |
|
||||
| `video` | Video data (MP4, AVI) | Raw bytes |
|
||||
| `binary` | Generic binary data | Raw bytes |
|
||||
|
||||
---
|
||||
|
||||
## Transport Strategies
|
||||
|
||||
### Direct Transport (Payloads < 1MB)
|
||||
|
||||
Small payloads are sent directly via NATS with Base64 encoding.
|
||||
|
||||
#### Julia
|
||||
```julia
|
||||
data = [("message", "Hello", "text")]
|
||||
smartsend("/topic", data)
|
||||
```
|
||||
|
||||
### Link Transport (Payloads >= 1MB)
|
||||
|
||||
Large payloads are uploaded to an HTTP file server.
|
||||
|
||||
#### Julia
|
||||
```julia
|
||||
data = [("file", large_data, "binary")]
|
||||
smartsend("/topic", data; fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Chat with Mixed Content
|
||||
|
||||
Send text, small image, and large file in one message.
|
||||
|
||||
#### Julia
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
data = [
|
||||
("message_text", "Hello!", "text"),
|
||||
("user_avatar", image_data, "image"),
|
||||
("large_document", large_file_data, "binary")
|
||||
]
|
||||
|
||||
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
### Example 2: Dictionary Exchange
|
||||
|
||||
Send configuration data between platforms.
|
||||
|
||||
#### Julia
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
config = Dict(
|
||||
"wifi_ssid" => "MyNetwork",
|
||||
"wifi_password" => "password123",
|
||||
"update_interval" => 60
|
||||
)
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env, env_json_str = NATSBridge.smartsend("/device/config", data)
|
||||
```
|
||||
|
||||
### Example 3: Table Data (Arrow IPC)
|
||||
|
||||
Send tabular data using Apache Arrow IPC format.
|
||||
|
||||
#### Julia
|
||||
```julia
|
||||
using NATSBridge
|
||||
using DataFrames
|
||||
|
||||
df = DataFrame(
|
||||
id = [1, 2, 3],
|
||||
name = ["Alice", "Bob", "Charlie"],
|
||||
score = [95, 88, 92]
|
||||
)
|
||||
|
||||
data = [("students", df, "table")]
|
||||
env, env_json_str = NATSBridge.smartsend("/data/analysis", data)
|
||||
```
|
||||
|
||||
### Example 4: Request-Response Pattern with Envelope JSON
|
||||
|
||||
Bi-directional communication with reply-to support. The `smartsend` function now returns both the envelope object and a JSON string that can be published directly.
|
||||
|
||||
#### Julia (Requester)
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
"/device/command",
|
||||
[("command", Dict("action" => "read_sensor"), "dictionary")];
|
||||
broker_url="nats://localhost:4222",
|
||||
reply_to="/device/response"
|
||||
)
|
||||
```
|
||||
|
||||
#### Julia (Responder)
|
||||
```julia
|
||||
using NATS, NATSBridge
|
||||
|
||||
# Configuration
|
||||
const SUBJECT = "/device/command"
|
||||
const NATS_URL = "nats://localhost:4222"
|
||||
|
||||
function test_responder()
|
||||
conn = NATS.connect(NATS_URL)
|
||||
NATS.subscribe(conn, SUBJECT) do msg
|
||||
env = NATSBridge.smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
|
||||
|
||||
# Extract reply_to from the envelope metadata
|
||||
reply_to = env["reply_to"]
|
||||
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
if dataname == "command" && data["action"] == "read_sensor"
|
||||
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
|
||||
# Send response to the reply_to subject from the request
|
||||
if !isempty(reply_to)
|
||||
smartsend(reply_to, [("data", response, "dictionary")])
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
sleep(120)
|
||||
NATS.drain(conn)
|
||||
end
|
||||
|
||||
test_responder()
|
||||
```
|
||||
|
||||
### Example 5: IoT Device Sensor Data
|
||||
|
||||
IoT device sending sensor data.
|
||||
|
||||
#### Julia (Receiver)
|
||||
```julia
|
||||
using NATS, NATSBridge
|
||||
|
||||
# Configuration
|
||||
const SUBJECT = "/device/sensors"
|
||||
const NATS_URL = "nats://localhost:4222"
|
||||
|
||||
function test_receiver()
|
||||
conn = NATS.connect(NATS_URL)
|
||||
NATS.subscribe(conn, SUBJECT) do msg
|
||||
env, env_json_str = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
if dataname == "temperature"
|
||||
println("Temperature: $data")
|
||||
elseif dataname == "humidity"
|
||||
println("Humidity: $data")
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
sleep(120)
|
||||
NATS.drain(conn)
|
||||
end
|
||||
|
||||
test_receiver()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
Run the test scripts to verify functionality:
|
||||
|
||||
### Julia
|
||||
|
||||
```julia
|
||||
# Text message exchange
|
||||
julia test/test_julia_text_sender.jl
|
||||
julia test/test_julia_text_receiver.jl
|
||||
|
||||
# Dictionary exchange
|
||||
julia test/test_julia_dict_sender.jl
|
||||
julia test/test_julia_dict_receiver.jl
|
||||
|
||||
# File transfer
|
||||
julia test/test_julia_file_sender.jl
|
||||
julia test/test_julia_file_receiver.jl
|
||||
|
||||
# Mixed payload types
|
||||
julia test/test_julia_mix_payloads_sender.jl
|
||||
julia test/test_julia_mix_payloads_receiver.jl
|
||||
|
||||
# Table exchange
|
||||
julia test/test_julia_table_sender.jl
|
||||
julia test/test_julia_table_receiver.jl
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 NATSBridge Contributors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
Reference in New Issue
Block a user