update readme
This commit is contained in:
372
README.md
372
README.md
@@ -12,14 +12,12 @@ A high-performance, bi-directional data bridge for **Julia, JavaScript, Python,
|
||||
- [Overview](#overview)
|
||||
- [Cross-Platform Support](#cross-platform-support)
|
||||
- [Features](#features)
|
||||
- [Architecture](#architecture)
|
||||
- [Installation](#installation)
|
||||
- [Quick Start](#quick-start)
|
||||
- [API Reference](#api-reference)
|
||||
- [Payload Types](#payload-types)
|
||||
- [Transport Strategies](#transport-strategies)
|
||||
- [Cross-Platform Examples](#cross-platform-examples)
|
||||
- [Testing](#testing)
|
||||
- [Documentation](#documentation)
|
||||
- [License](#license)
|
||||
|
||||
---
|
||||
@@ -58,7 +56,6 @@ NATSBridge enables seamless communication across multiple platforms through NATS
|
||||
| Multiple Dispatch | ✅ Native | ❌ | ❌ | ❌ |
|
||||
| Async/Await | ❌ | ✅ Native | ✅ Native | ⚠️ (uasyncio) |
|
||||
| Type Safety | ✅ Strong | ⚠️ (TypeScript) | ✅ (Type hints) | ❌ |
|
||||
| Memory Management | ✅ GC | ✅ GC | ✅ GC | ⚠️ (Manual) |
|
||||
| Arrow IPC | ✅ Native | ✅ | ✅ | ❌ |
|
||||
| Direct Transport | ✅ | ✅ | ✅ | ✅ |
|
||||
| Link Transport | ✅ | ✅ | ✅ | ⚠️ (Limited) |
|
||||
@@ -82,171 +79,6 @@ NATSBridge enables seamless communication across multiple platforms through NATS
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### System Components
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Sender["Application (Sender)"]
|
||||
SenderApp[App Code]
|
||||
NATSBridge_Send[NATSBridge]
|
||||
NATS_Client[<b>NATS.jl</b>]
|
||||
end
|
||||
|
||||
subgraph Receiver["Application (Receiver)"]
|
||||
ReceiverApp[App Code]
|
||||
NATSBridge_Recv[NATSBridge]
|
||||
NATS_Client_Recv[<b>NATS.jl</b>]
|
||||
end
|
||||
|
||||
subgraph Infrastructure["Infrastructure"]
|
||||
NATS[<b>NATS Server</b><br/>Message Broker]
|
||||
FileServer[<b>HTTP File Server</b><br/>Upload/Download]
|
||||
end
|
||||
|
||||
SenderApp --> NATSBridge_Send
|
||||
NATSBridge_Send --> NATS_Client
|
||||
NATS_Client --> NATS
|
||||
|
||||
NATS --> NATS_Client_Recv
|
||||
NATS_Client_Recv --> NATSBridge_Recv
|
||||
NATSBridge_Recv --> ReceiverApp
|
||||
|
||||
NATSBridge_Send -.->|HTTP POST upload| FileServer
|
||||
FileServer -.->|HTTP GET download| NATSBridge_Recv
|
||||
|
||||
style SenderApp fill:#e8f5e9
|
||||
style ReceiverApp fill:#e8f5e9
|
||||
style NATS fill:#fff3e0
|
||||
style FileServer fill:#f3e5f5
|
||||
```
|
||||
|
||||
### Message Flow
|
||||
|
||||
1. **Sender** creates a message envelope with payloads using `smartsend()`
|
||||
2. **NATSBridge** serializes and encodes each payload based on type
|
||||
3. **Transport Decision**:
|
||||
- **Direct** (< 1MB): Payload encoded as Base64, published to NATS
|
||||
- **Link** (≥ 1MB): Payload uploaded to HTTP file server, URL published to NATS
|
||||
4. **NATS** routes message envelope to subscribers
|
||||
5. **Receiver** receives message via NATS subscription callback
|
||||
6. **NATSBridge** processes envelope:
|
||||
- Decodes Base64 payloads from NATS message
|
||||
- Fetches URLs from file server with exponential backoff
|
||||
7. **Receiver** deserializes payloads based on their type
|
||||
|
||||
### File Server Handler Abstraction
|
||||
|
||||
The system uses handler functions to abstract file server operations:
|
||||
|
||||
| Handler | Purpose |
|
||||
|---------|---------|
|
||||
| `plik_oneshot_upload()` / `plikOneshotUpload()` | Uploads payload bytes to file server, returns URL |
|
||||
| `_fetch_with_backoff()` / `fetchWithBackoff()` | Downloads data from URL with exponential backoff retry |
|
||||
|
||||
This abstraction allows support for different file server implementations (Plik, AWS S3, custom HTTP server).
|
||||
|
||||
### Message Envelope Schema
|
||||
|
||||
All platforms use identical JSON schemas for message envelopes:
|
||||
|
||||
```json
|
||||
{
|
||||
"correlation_id": "uuid-v4-string",
|
||||
"msg_id": "uuid-v4-string",
|
||||
"timestamp": "2024-01-15T10:30:00Z",
|
||||
"send_to": "topic/subject",
|
||||
"msg_purpose": "ACK | NACK | updateStatus | shutdown | chat",
|
||||
"sender_name": "agent-wine-web-frontend",
|
||||
"sender_id": "uuid4",
|
||||
"receiver_name": "agent-backend",
|
||||
"receiver_id": "uuid4",
|
||||
"reply_to": "topic",
|
||||
"reply_to_msg_id": "uuid4",
|
||||
"broker_url": "nats://localhost:4222",
|
||||
"metadata": {},
|
||||
"payloads": [
|
||||
{
|
||||
"id": "uuid4",
|
||||
"dataname": "login_image",
|
||||
"payload_type": "image",
|
||||
"transport": "direct",
|
||||
"encoding": "base64",
|
||||
"size": 15433,
|
||||
"data": "base64-encoded-string"
|
||||
},
|
||||
{
|
||||
"id": "uuid4",
|
||||
"dataname": "large_table",
|
||||
"payload_type": "table",
|
||||
"transport": "link",
|
||||
"encoding": "none",
|
||||
"size": 524288,
|
||||
"data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/data.arrow"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **NATS Server** (v2.10+ recommended)
|
||||
- **HTTP File Server** (optional, for payloads > 1MB)
|
||||
|
||||
### Platform-Specific Dependencies
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using Pkg
|
||||
Pkg.add("NATS")
|
||||
Pkg.add("Arrow")
|
||||
Pkg.add("JSON3")
|
||||
Pkg.add("HTTP")
|
||||
Pkg.add("UUIDs")
|
||||
Pkg.add("Dates")
|
||||
```
|
||||
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```bash
|
||||
npm install nats uuid apache-arrow node-fetch
|
||||
# or
|
||||
yarn add nats uuid apache-arrow node-fetch
|
||||
```
|
||||
|
||||
#### JavaScript (Browser)
|
||||
|
||||
```bash
|
||||
npm install nats uuid apache-arrow
|
||||
# or use CDN:
|
||||
# https://unpkg.com/nats-js/dist/bundle/nats.min.js
|
||||
# https://unpkg.com/apache-arrow/arrow.min.js
|
||||
```
|
||||
|
||||
#### Python (Desktop)
|
||||
|
||||
```bash
|
||||
pip install nats-py aiohttp pyarrow pandas python-dateutil
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
MicroPython uses built-in modules:
|
||||
- `network` - NATS connection (custom implementation)
|
||||
- `time` - Timestamps
|
||||
- `uos` - File operations
|
||||
- `base64` - Base64 encoding
|
||||
- `json` - JSON parsing
|
||||
- `struct` - Binary data handling
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Step 1: Start NATS Server
|
||||
@@ -265,6 +97,46 @@ mkdir -p /tmp/fileserver
|
||||
python3 -m http.server 8080 --directory /tmp/fileserver
|
||||
```
|
||||
|
||||
### Step 3: Send Your First Message
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222")
|
||||
println("Message sent!")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natsbridge.js');
|
||||
|
||||
const data = [["message", "Hello World", "text"]];
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222" }
|
||||
);
|
||||
console.log("Message sent!");
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import smartsend
|
||||
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = await smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
broker_url="nats://localhost:4222"
|
||||
)
|
||||
print("Message sent!")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
@@ -308,8 +180,8 @@ Sends data either directly via NATS or via a fileserver URL, depending on payloa
|
||||
using NATSBridge
|
||||
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
subject::String, # NATS subject
|
||||
data::AbstractArray{Tuple{String, Any, String}}; # List of (dataname, data, type)
|
||||
subject::String,
|
||||
data::AbstractArray{Tuple{String, Any, String}};
|
||||
broker_url::String = "nats://localhost:4222",
|
||||
fileserver_url = "http://localhost:8080",
|
||||
fileserver_upload_handler::Function = plik_oneshot_upload,
|
||||
@@ -468,7 +340,9 @@ env = NATSBridge.smartreceive(
|
||||
|------|-------|------------|--------|-------------|-------------|
|
||||
| `text` | `String` | `string` | `str` | `str` | Plain text strings |
|
||||
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable dictionaries |
|
||||
| `table` | `DataFrame`, `Arrow.Table` | `Array<Object>` → `Arrow.Table` | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) |
|
||||
| `arrowtable` | `DataFrame`, `Arrow.Table` | `Array<Object>` | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) |
|
||||
| `jsontable` | `Vector{NamedTuple}` | `Array<Object>` | `list[dict]` | ❌ | Tabular data (JSON) |
|
||||
| `table` | ❌ | ❌ | `pandas.DataFrame` | ❌ | Python unified table type |
|
||||
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image data (PNG, JPG) |
|
||||
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio data (WAV, MP3) |
|
||||
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video data (MP4, AVI) |
|
||||
@@ -476,58 +350,6 @@ env = NATSBridge.smartreceive(
|
||||
|
||||
---
|
||||
|
||||
## Transport Strategies
|
||||
|
||||
### Direct Transport (Payloads < 1MB)
|
||||
|
||||
Small payloads are sent directly via NATS with Base64 encoding.
|
||||
|
||||
#### Cross-Platform
|
||||
|
||||
```julia
|
||||
# Julia
|
||||
data = [("message", "Hello", "text")]
|
||||
smartsend("/topic", data)
|
||||
```
|
||||
|
||||
```javascript
|
||||
// JavaScript
|
||||
const data = [["message", "Hello", "text"]];
|
||||
smartsend("/topic", data);
|
||||
```
|
||||
|
||||
```python
|
||||
# Python
|
||||
data = [("message", "Hello", "text")]
|
||||
await smartsend("/topic", data)
|
||||
```
|
||||
|
||||
### Link Transport (Payloads >= 1MB)
|
||||
|
||||
Large payloads are uploaded to an HTTP file server.
|
||||
|
||||
#### Cross-Platform
|
||||
|
||||
```julia
|
||||
# Julia
|
||||
data = [("file", large_data, "binary")]
|
||||
smartsend("/topic", data; fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
```javascript
|
||||
// JavaScript
|
||||
const data = [["file", largeData, "binary"]];
|
||||
smartsend("/topic", data, { fileserver_url: 'http://localhost:8080' });
|
||||
```
|
||||
|
||||
```python
|
||||
# Python
|
||||
data = [("file", large_data, "binary")]
|
||||
await smartsend("/topic", data, fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cross-Platform Examples
|
||||
|
||||
### Example 1: Chat with Mixed Content
|
||||
@@ -651,7 +473,7 @@ df = DataFrame(
|
||||
score = [95, 88, 92]
|
||||
)
|
||||
|
||||
data = [("students", df, "table")]
|
||||
data = [("students", df, "arrowtable")]
|
||||
env, env_json_str = NATSBridge.smartsend("/data/analysis", data)
|
||||
```
|
||||
|
||||
@@ -668,7 +490,7 @@ const df = [
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/data/analysis",
|
||||
[["students", df, "table"]]
|
||||
[["students", df, "arrowtable"]]
|
||||
);
|
||||
```
|
||||
|
||||
@@ -706,32 +528,6 @@ env, env_json_str = NATSBridge.smartsend(
|
||||
)
|
||||
```
|
||||
|
||||
```julia
|
||||
# Responder
|
||||
using NATS, NATSBridge
|
||||
|
||||
function test_responder()
|
||||
conn = NATS.connect("nats://localhost:4222")
|
||||
NATS.subscribe(conn, "/device/command") do msg
|
||||
env = NATSBridge.smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
|
||||
|
||||
reply_to = env["reply_to"]
|
||||
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
if dataname == "command" && data["action"] == "read_sensor"
|
||||
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
|
||||
if !isempty(reply_to)
|
||||
smartsend(reply_to, [("data", response, "dictionary")])
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
sleep(120)
|
||||
NATS.drain(conn)
|
||||
end
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
@@ -745,40 +541,6 @@ const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
);
|
||||
```
|
||||
|
||||
```javascript
|
||||
// Responder
|
||||
const nats = require('nats');
|
||||
const NATSBridge = require('natsbridge');
|
||||
|
||||
async function testResponder() {
|
||||
const conn = await nats.connect('nats://localhost:4222');
|
||||
|
||||
const subscription = await conn.subscribe('/device/command');
|
||||
|
||||
for await (const msg of subscription) {
|
||||
const env = await NATSBridge.smartreceive(msg, {
|
||||
fileserver_download_handler: NATSBridge.fetchWithBackoff
|
||||
});
|
||||
|
||||
const replyTo = env.reply_to;
|
||||
|
||||
for (const [dataname, data, type] of env.payloads) {
|
||||
if (dataname === 'command' && data.action === 'read_sensor') {
|
||||
const response = { sensor_id: 'sensor-001', value: 42.5 };
|
||||
if (replyTo) {
|
||||
await NATSBridge.smartsend(
|
||||
replyTo,
|
||||
[["data", response, "dictionary"]]
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
setTimeout(() => conn.close(), 120000);
|
||||
}
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
@@ -793,38 +555,6 @@ env, env_json_str = await NATSBridge.smartsend(
|
||||
)
|
||||
```
|
||||
|
||||
```python
|
||||
# Responder
|
||||
from natsbridge import NATSBridge
|
||||
import asyncio
|
||||
import nats
|
||||
|
||||
async def test_responder():
|
||||
nc = await nats.connect('nats://localhost:4222')
|
||||
|
||||
async def msg_handler(msg):
|
||||
env = await NATSBridge.smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=fetch_with_backoff
|
||||
)
|
||||
|
||||
reply_to = env["reply_to"]
|
||||
|
||||
for dataname, data, type_ in env["payloads"]:
|
||||
if dataname == "command" and data["action"] == "read_sensor":
|
||||
response = {"sensor_id": "sensor-001", "value": 42.5}
|
||||
if reply_to:
|
||||
await NATSBridge.smartsend(
|
||||
reply_to,
|
||||
[("data", response, "dictionary")]
|
||||
)
|
||||
|
||||
await nc.subscribe('/device/command', cb=msg_handler)
|
||||
|
||||
await asyncio.sleep(120)
|
||||
await nc.drain()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
@@ -909,8 +639,10 @@ python3 test/test_py_table_receiver.py
|
||||
|
||||
For detailed architecture and implementation information, see:
|
||||
|
||||
- [Architecture Documentation](docs/architecture.md) - Cross-platform architecture, API parity, platform-specific patterns
|
||||
- [Implementation Guide](docs/implementation.md) - Detailed implementation for each platform, handler functions, testing
|
||||
- [Architecture Documentation](docs/architecture_updated.md) - Cross-platform architecture, API parity, platform-specific patterns
|
||||
- [Implementation Guide](docs/implementation_updated.md) - Detailed implementation for each platform, handler functions, testing
|
||||
- [Tutorial](docs/tutorial_updated.md) - Step-by-step getting started guide
|
||||
- [Walkthrough](docs/walkthrough_updated.md) - Real-world application building guides
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user