ton/NATSBridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

update readme

Browse Source
This commit is contained in:
narawat
2026-03-09 15:54:09 +07:00
parent 3fcd27f41a
commit db377ead3c
1 changed files with 53 additions and 321 deletions
Download Patch File Download Diff File Expand all files Collapse all files

374
README.md
View File

@@ -12,14 +12,12 @@ A high-performance, bi-directional data bridge for **Julia, JavaScript, Python,
- [Overview](#overview) - [Overview](#overview)
- [Cross-Platform Support](#cross-platform-support) - [Cross-Platform Support](#cross-platform-support)
- [Features](#features) - [Features](#features)
- [Architecture](#architecture)
- [Installation](#installation)
- [Quick Start](#quick-start) - [Quick Start](#quick-start)
- [API Reference](#api-reference) - [API Reference](#api-reference)
- [Payload Types](#payload-types) - [Payload Types](#payload-types)
- [Transport Strategies](#transport-strategies)
- [Cross-Platform Examples](#cross-platform-examples) - [Cross-Platform Examples](#cross-platform-examples)
- [Testing](#testing) - [Testing](#testing)
- [Documentation](#documentation)
- [License](#license) - [License](#license)
--- ---
@@ -58,7 +56,6 @@ NATSBridge enables seamless communication across multiple platforms through NATS
| Multiple Dispatch | ✅ Native | ❌ | ❌ | ❌ | | Multiple Dispatch | ✅ Native | ❌ | ❌ | ❌ |
| Async/Await | ❌ | ✅ Native | ✅ Native | ⚠️ (uasyncio) | | Async/Await | ❌ | ✅ Native | ✅ Native | ⚠️ (uasyncio) |
| Type Safety | ✅ Strong | ⚠️ (TypeScript) | ✅ (Type hints) | ❌ | | Type Safety | ✅ Strong | ⚠️ (TypeScript) | ✅ (Type hints) | ❌ |
| Memory Management | ✅ GC | ✅ GC | ✅ GC | ⚠️ (Manual) |
| Arrow IPC | ✅ Native | ✅ | ✅ | ❌ | | Arrow IPC | ✅ Native | ✅ | ✅ | ❌ |
| Direct Transport | ✅ | ✅ | ✅ | ✅ | | Direct Transport | ✅ | ✅ | ✅ | ✅ |
| Link Transport | ✅ | ✅ | ✅ | ⚠️ (Limited) | | Link Transport | ✅ | ✅ | ✅ | ⚠️ (Limited) |
@@ -82,171 +79,6 @@ NATSBridge enables seamless communication across multiple platforms through NATS
--- ---
## Architecture
### System Components
```mermaid
flowchart TB
subgraph Sender["Application (Sender)"]
SenderApp[App Code]
NATSBridge_Send[NATSBridge]
NATS_Client[<b>NATS.jl</b>]
end
subgraph Receiver["Application (Receiver)"]
ReceiverApp[App Code]
NATSBridge_Recv[NATSBridge]
NATS_Client_Recv[<b>NATS.jl</b>]
end
subgraph Infrastructure["Infrastructure"]
NATS[<b>NATS Server</b><br/>Message Broker]
FileServer[<b>HTTP File Server</b><br/>Upload/Download]
end
SenderApp --> NATSBridge_Send
NATSBridge_Send --> NATS_Client
NATS_Client --> NATS
NATS --> NATS_Client_Recv
NATS_Client_Recv --> NATSBridge_Recv
NATSBridge_Recv --> ReceiverApp
NATSBridge_Send -.->|HTTP POST upload| FileServer
FileServer -.->|HTTP GET download| NATSBridge_Recv
style SenderApp fill:#e8f5e9
style ReceiverApp fill:#e8f5e9
style NATS fill:#fff3e0
style FileServer fill:#f3e5f5
```
### Message Flow
1. **Sender** creates a message envelope with payloads using `smartsend()`
2. **NATSBridge** serializes and encodes each payload based on type
3. **Transport Decision**:
- **Direct** (< 1MB): Payload encoded as Base64, published to NATS
- **Link** (≥ 1MB): Payload uploaded to HTTP file server, URL published to NATS
4. **NATS** routes message envelope to subscribers
5. **Receiver** receives message via NATS subscription callback
6. **NATSBridge** processes envelope:
- Decodes Base64 payloads from NATS message
- Fetches URLs from file server with exponential backoff
7. **Receiver** deserializes payloads based on their type
### File Server Handler Abstraction
The system uses handler functions to abstract file server operations:
| Handler | Purpose |
|---------|---------|
| `plik_oneshot_upload()` / `plikOneshotUpload()` | Uploads payload bytes to file server, returns URL |
| `_fetch_with_backoff()` / `fetchWithBackoff()` | Downloads data from URL with exponential backoff retry |
This abstraction allows support for different file server implementations (Plik, AWS S3, custom HTTP server).
### Message Envelope Schema
All platforms use identical JSON schemas for message envelopes:
```json
{
"correlation_id": "uuid-v4-string",
"msg_id": "uuid-v4-string",
"timestamp": "2024-01-15T10:30:00Z",
"send_to": "topic/subject",
"msg_purpose": "ACK | NACK | updateStatus | shutdown | chat",
"sender_name": "agent-wine-web-frontend",
"sender_id": "uuid4",
"receiver_name": "agent-backend",
"receiver_id": "uuid4",
"reply_to": "topic",
"reply_to_msg_id": "uuid4",
"broker_url": "nats://localhost:4222",
"metadata": {},
"payloads": [
{
"id": "uuid4",
"dataname": "login_image",
"payload_type": "image",
"transport": "direct",
"encoding": "base64",
"size": 15433,
"data": "base64-encoded-string"
},
{
"id": "uuid4",
"dataname": "large_table",
"payload_type": "table",
"transport": "link",
"encoding": "none",
"size": 524288,
"data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/data.arrow"
}
]
}
```
---
## Installation
### Prerequisites
- **NATS Server** (v2.10+ recommended)
- **HTTP File Server** (optional, for payloads > 1MB)
### Platform-Specific Dependencies
#### Julia
```julia
using Pkg
Pkg.add("NATS")
Pkg.add("Arrow")
Pkg.add("JSON3")
Pkg.add("HTTP")
Pkg.add("UUIDs")
Pkg.add("Dates")
```
#### JavaScript (Node.js)
```bash
npm install nats uuid apache-arrow node-fetch
# or
yarn add nats uuid apache-arrow node-fetch
```
#### JavaScript (Browser)
```bash
npm install nats uuid apache-arrow
# or use CDN:
# https://unpkg.com/nats-js/dist/bundle/nats.min.js
# https://unpkg.com/apache-arrow/arrow.min.js
```
#### Python (Desktop)
```bash
pip install nats-py aiohttp pyarrow pandas python-dateutil
```
#### MicroPython
MicroPython uses built-in modules:
- `network` - NATS connection (custom implementation)
- `time` - Timestamps
- `uos` - File operations
- `base64` - Base64 encoding
- `json` - JSON parsing
- `struct` - Binary data handling
---
## Quick Start ## Quick Start
### Step 1: Start NATS Server ### Step 1: Start NATS Server
@@ -265,6 +97,46 @@ mkdir -p /tmp/fileserver
python3 -m http.server 8080 --directory /tmp/fileserver python3 -m http.server 8080 --directory /tmp/fileserver
``` ```
### Step 3: Send Your First Message
#### Julia
```julia
using NATSBridge
data = [("message", "Hello World", "text")]
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222")
println("Message sent!")
```
#### JavaScript
```javascript
const NATSBridge = require('./src/natsbridge.js');
const data = [["message", "Hello World", "text"]];
const [env, env_json_str] = await NATSBridge.smartsend(
"/chat/room1",
data,
{ broker_url: "nats://localhost:4222" }
);
console.log("Message sent!");
```
#### Python
```python
from natsbridge import smartsend
data = [("message", "Hello World", "text")]
env, env_json_str = await smartsend(
"/chat/room1",
data,
broker_url="nats://localhost:4222"
)
print("Message sent!")
```
--- ---
## API Reference ## API Reference
@@ -308,8 +180,8 @@ Sends data either directly via NATS or via a fileserver URL, depending on payloa
using NATSBridge using NATSBridge
env, env_json_str = NATSBridge.smartsend( env, env_json_str = NATSBridge.smartsend(
subject::String, # NATS subject subject::String,
data::AbstractArray{Tuple{String, Any, String}}; # List of (dataname, data, type) data::AbstractArray{Tuple{String, Any, String}};
broker_url::String = "nats://localhost:4222", broker_url::String = "nats://localhost:4222",
fileserver_url = "http://localhost:8080", fileserver_url = "http://localhost:8080",
fileserver_upload_handler::Function = plik_oneshot_upload, fileserver_upload_handler::Function = plik_oneshot_upload,
@@ -468,7 +340,9 @@ env = NATSBridge.smartreceive(
|------|-------|------------|--------|-------------|-------------| |------|-------|------------|--------|-------------|-------------|
| `text` | `String` | `string` | `str` | `str` | Plain text strings | | `text` | `String` | `string` | `str` | `str` | Plain text strings |
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable dictionaries | | `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable dictionaries |
| `table` | `DataFrame`, `Arrow.Table` | `Array<Object>` → `Arrow.Table` | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) | | `arrowtable` | `DataFrame`, `Arrow.Table` | `Array<Object>` | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) |
| `jsontable` | `Vector{NamedTuple}` | `Array<Object>` | `list[dict]` | ❌ | Tabular data (JSON) |
| `table` | ❌ | ❌ | `pandas.DataFrame` | ❌ | Python unified table type |
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image data (PNG, JPG) | | `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image data (PNG, JPG) |
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio data (WAV, MP3) | | `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio data (WAV, MP3) |
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video data (MP4, AVI) | | `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video data (MP4, AVI) |
@@ -476,58 +350,6 @@ env = NATSBridge.smartreceive(
--- ---
## Transport Strategies
### Direct Transport (Payloads < 1MB)
Small payloads are sent directly via NATS with Base64 encoding.
#### Cross-Platform
```julia
# Julia
data = [("message", "Hello", "text")]
smartsend("/topic", data)
```
```javascript
// JavaScript
const data = [["message", "Hello", "text"]];
smartsend("/topic", data);
```
```python
# Python
data = [("message", "Hello", "text")]
await smartsend("/topic", data)
```
### Link Transport (Payloads >= 1MB)
Large payloads are uploaded to an HTTP file server.
#### Cross-Platform
```julia
# Julia
data = [("file", large_data, "binary")]
smartsend("/topic", data; fileserver_url="http://localhost:8080")
```
```javascript
// JavaScript
const data = [["file", largeData, "binary"]];
smartsend("/topic", data, { fileserver_url: 'http://localhost:8080' });
```
```python
# Python
data = [("file", large_data, "binary")]
await smartsend("/topic", data, fileserver_url="http://localhost:8080")
```
---
## Cross-Platform Examples ## Cross-Platform Examples
### Example 1: Chat with Mixed Content ### Example 1: Chat with Mixed Content
@@ -651,7 +473,7 @@ df = DataFrame(
score = [95, 88, 92] score = [95, 88, 92]
) )
data = [("students", df, "table")] data = [("students", df, "arrowtable")]
env, env_json_str = NATSBridge.smartsend("/data/analysis", data) env, env_json_str = NATSBridge.smartsend("/data/analysis", data)
``` ```
@@ -668,7 +490,7 @@ const df = [
const [env, env_json_str] = await NATSBridge.smartsend( const [env, env_json_str] = await NATSBridge.smartsend(
"/data/analysis", "/data/analysis",
[["students", df, "table"]] [["students", df, "arrowtable"]]
); );
``` ```
@@ -706,32 +528,6 @@ env, env_json_str = NATSBridge.smartsend(
) )
``` ```
```julia
# Responder
using NATS, NATSBridge
function test_responder()
conn = NATS.connect("nats://localhost:4222")
NATS.subscribe(conn, "/device/command") do msg
env = NATSBridge.smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
reply_to = env["reply_to"]
for (dataname, data, type) in env["payloads"]
if dataname == "command" && data["action"] == "read_sensor"
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
if !isempty(reply_to)
smartsend(reply_to, [("data", response, "dictionary")])
end
end
end
end
sleep(120)
NATS.drain(conn)
end
```
#### JavaScript #### JavaScript
```javascript ```javascript
@@ -745,40 +541,6 @@ const [env, env_json_str] = await NATSBridge.smartsend(
); );
``` ```
```javascript
// Responder
const nats = require('nats');
const NATSBridge = require('natsbridge');
async function testResponder() {
const conn = await nats.connect('nats://localhost:4222');
const subscription = await conn.subscribe('/device/command');
for await (const msg of subscription) {
const env = await NATSBridge.smartreceive(msg, {
fileserver_download_handler: NATSBridge.fetchWithBackoff
});
const replyTo = env.reply_to;
for (const [dataname, data, type] of env.payloads) {
if (dataname === 'command' && data.action === 'read_sensor') {
const response = { sensor_id: 'sensor-001', value: 42.5 };
if (replyTo) {
await NATSBridge.smartsend(
replyTo,
[["data", response, "dictionary"]]
);
}
}
}
}
setTimeout(() => conn.close(), 120000);
}
```
#### Python #### Python
```python ```python
@@ -793,38 +555,6 @@ env, env_json_str = await NATSBridge.smartsend(
) )
``` ```
```python
# Responder
from natsbridge import NATSBridge
import asyncio
import nats
async def test_responder():
nc = await nats.connect('nats://localhost:4222')
async def msg_handler(msg):
env = await NATSBridge.smartreceive(
msg,
fileserver_download_handler=fetch_with_backoff
)
reply_to = env["reply_to"]
for dataname, data, type_ in env["payloads"]:
if dataname == "command" and data["action"] == "read_sensor":
response = {"sensor_id": "sensor-001", "value": 42.5}
if reply_to:
await NATSBridge.smartsend(
reply_to,
[("data", response, "dictionary")]
)
await nc.subscribe('/device/command', cb=msg_handler)
await asyncio.sleep(120)
await nc.drain()
```
--- ---
## Testing ## Testing
@@ -909,8 +639,10 @@ python3 test/test_py_table_receiver.py
For detailed architecture and implementation information, see: For detailed architecture and implementation information, see:
- [Architecture Documentation](docs/architecture.md) - Cross-platform architecture, API parity, platform-specific patterns - [Architecture Documentation](docs/architecture_updated.md) - Cross-platform architecture, API parity, platform-specific patterns
- [Implementation Guide](docs/implementation.md) - Detailed implementation for each platform, handler functions, testing - [Implementation Guide](docs/implementation_updated.md) - Detailed implementation for each platform, handler functions, testing
- [Tutorial](docs/tutorial_updated.md) - Step-by-step getting started guide
- [Walkthrough](docs/walkthrough_updated.md) - Real-world application building guides
--- ---
@@ -936,4 +668,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE. SOFTWARE.