update readme
This commit is contained in:
228
README.md
228
README.md
@@ -1,6 +1,6 @@
|
||||
# NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||
|
||||
A high-performance, bi-directional data bridge for **Julia, JavaScript, Python, and MicroPython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
A high-performance, bi-directional data bridge for **Julia**, **JavaScript**, **Python**, and **MicroPython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://nats.io)
|
||||
@@ -28,8 +28,8 @@ NATSBridge enables seamless communication across multiple platforms through NATS
|
||||
|
||||
| Transport | Payload Size | Method |
|
||||
|-----------|--------------|--------|
|
||||
| **Direct** | < 1MB | Sent directly via NATS (Base64 encoded) |
|
||||
| **Link** | >= 1MB | Uploaded to HTTP file server, URL sent via NATS |
|
||||
| **Direct** | < 500KB | Sent directly via NATS (Base64 encoded) |
|
||||
| **Link** | ≥ 500KB | Uploaded to HTTP file server, URL sent via NATS |
|
||||
|
||||
### Use Cases
|
||||
|
||||
@@ -45,7 +45,7 @@ NATSBridge enables seamless communication across multiple platforms through NATS
|
||||
| Platform | Implementation | Features |
|
||||
|----------|----------------|----------|
|
||||
| **Julia** | [`src/NATSBridge.jl`](src/NATSBridge.jl) | Full feature set, Arrow IPC, multiple dispatch |
|
||||
| **JavaScript** | [`src/natsbridge.js`](src/natsbridge.js) | Node.js, async/await |
|
||||
| **JavaScript (Node.js)** | [`src/natsbridge.js`](src/natsbridge_ssr.js) | Node.js, async/await |
|
||||
| **JavaScript (Browser)** | [`src/natsbridge_csr.js`](src/natsbridge_csr.js) | Browser, WebSocket NATS, async/await |
|
||||
| **Python** | [`src/natsbridge.py`](src/natsbridge.py) | Desktop Python, asyncio, type hints |
|
||||
| **MicroPython** | [`src/natsbridge_mpy.py`](src/natsbridge_mpy.py) | Memory-constrained, synchronous API |
|
||||
@@ -72,7 +72,7 @@ NATSBridge enables seamless communication across multiple platforms through NATS
|
||||
- ✅ **Bi-directional messaging** with request-reply patterns
|
||||
- ✅ **Multi-payload support** - send multiple payloads with different types in one message
|
||||
- ✅ **Automatic transport selection** - direct vs link based on payload size
|
||||
- ✅ **Claim-Check pattern** for payloads > 1MB
|
||||
- ✅ **Claim-Check pattern** for payloads ≥ 500KB
|
||||
- ✅ **Apache Arrow IPC** support for tabular data (zero-copy reading)
|
||||
- ✅ **Exponential backoff** for reliable file server downloads
|
||||
- ✅ **Correlation ID tracking** for message tracing
|
||||
@@ -83,23 +83,24 @@ NATSBridge enables seamless communication across multiple platforms through NATS
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Step 1: Start NATS Server
|
||||
### Prerequisites
|
||||
|
||||
```bash
|
||||
docker run -p 4222:4222 nats:latest
|
||||
```
|
||||
1. **NATS Server** - Install and run a NATS server:
|
||||
```bash
|
||||
docker run -p 4222:4222 nats:latest
|
||||
```
|
||||
|
||||
### Step 2: Start HTTP File Server (Optional)
|
||||
2. **HTTP File Server** (optional, for large payloads) - Install and run a file server:
|
||||
```bash
|
||||
# Using Plik
|
||||
docker run -p 8080:8080 -v /tmp/fileserver:/var/lib/plik -e PLIK_ADMIN_PASSWORD=admin plik/plik
|
||||
|
||||
# OR using simple Python HTTP server
|
||||
mkdir -p /tmp/fileserver
|
||||
python3 -m http.server 8080 --directory /tmp/fileserver
|
||||
```
|
||||
|
||||
```bash
|
||||
# Create a directory for file uploads
|
||||
mkdir -p /tmp/fileserver
|
||||
|
||||
# Start HTTP file server
|
||||
python3 -m http.server 8080 --directory /tmp/fileserver
|
||||
```
|
||||
|
||||
### Step 3: Send Your First Message
|
||||
### Send Your First Message
|
||||
|
||||
#### Julia
|
||||
|
||||
@@ -107,14 +108,14 @@ python3 -m http.server 8080 --directory /tmp/fileserver
|
||||
using NATSBridge
|
||||
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222")
|
||||
env, env_json_str = smartsend("/chat/room1", data; broker_url="nats://localhost:4222")
|
||||
println("Message sent!")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natsbridge.js');
|
||||
import NATSBridge from './src/natsbridge_ssr.js';
|
||||
|
||||
const data = [["message", "Hello World", "text"]];
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
@@ -125,6 +126,20 @@ const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
console.log("Message sent!");
|
||||
```
|
||||
|
||||
#### JavaScript (Browser)
|
||||
|
||||
```javascript
|
||||
import NATSBridge from './src/natsbridge_csr.js';
|
||||
|
||||
const data = [["message", "Hello World", "text"]];
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ broker_url: "ws://localhost:4222" }
|
||||
);
|
||||
console.log("Message sent!");
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
@@ -139,6 +154,21 @@ env, env_json_str = await smartsend(
|
||||
print("Message sent!")
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
from natsbridge import smartsend
|
||||
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
broker_url="nats://localhost:4222",
|
||||
size_threshold=100000 # 100KB for MicroPython
|
||||
)
|
||||
print("Message sent!")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
@@ -147,13 +177,13 @@ print("Message sent!")
|
||||
|
||||
All platforms use the same input/output format for payloads:
|
||||
|
||||
**Input format for smartsend:**
|
||||
**Input format for `smartsend`:**
|
||||
```
|
||||
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||
```
|
||||
|
||||
**Output format for smartreceive:**
|
||||
```
|
||||
**Output format for `smartreceive`:**
|
||||
```json
|
||||
{
|
||||
"correlation_id": "...",
|
||||
"msg_id": "...",
|
||||
@@ -187,7 +217,7 @@ env, env_json_str = NATSBridge.smartsend(
|
||||
broker_url::String = "nats://localhost:4222",
|
||||
fileserver_url = "http://localhost:8080",
|
||||
fileserver_upload_handler::Function = plik_oneshot_upload,
|
||||
size_threshold::Int = 1_000_000,
|
||||
size_threshold::Int = 500_000,
|
||||
correlation_id::String = string(uuid4()),
|
||||
msg_purpose::String = "chat",
|
||||
sender_name::String = "NATSBridge",
|
||||
@@ -203,10 +233,10 @@ env, env_json_str = NATSBridge.smartsend(
|
||||
# Returns: ::Tuple{msg_envelope_v1, String}
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natsbridge');
|
||||
import NATSBridge from './src/natsbridge_ssr.js';
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
subject,
|
||||
@@ -215,7 +245,36 @@ const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
broker_url: 'nats://localhost:4222',
|
||||
fileserver_url: 'http://localhost:8080',
|
||||
fileserver_upload_handler: NATSBridge.plikOneshotUpload,
|
||||
size_threshold: 1_000_000,
|
||||
size_threshold: 500_000,
|
||||
correlation_id: uuidv4(),
|
||||
msg_purpose: 'chat',
|
||||
sender_name: 'NATSBridge',
|
||||
receiver_name: '',
|
||||
receiver_id: '',
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
is_publish: true,
|
||||
nats_connection: null,
|
||||
msg_id: uuidv4(),
|
||||
sender_id: uuidv4()
|
||||
}
|
||||
);
|
||||
// Returns: Promise<[env, env_json_str]>
|
||||
```
|
||||
|
||||
#### JavaScript (Browser)
|
||||
|
||||
```javascript
|
||||
import NATSBridge from './src/natsbridge_csr.js';
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
subject,
|
||||
data,
|
||||
{
|
||||
broker_url: 'ws://localhost:4222',
|
||||
fileserver_url: 'http://localhost:8080',
|
||||
fileserver_upload_handler: NATSBridge.plikOneshotUpload,
|
||||
size_threshold: 500_000,
|
||||
correlation_id: uuidv4(),
|
||||
msg_purpose: 'chat',
|
||||
sender_name: 'NATSBridge',
|
||||
@@ -243,7 +302,7 @@ env, env_json_str = await NATSBridge.smartsend(
|
||||
broker_url: str = "nats://localhost:4222",
|
||||
fileserver_url: str = "http://localhost:8080",
|
||||
fileserver_upload_handler: Callable = plik_oneshot_upload,
|
||||
size_threshold: int = 1_000_000,
|
||||
size_threshold: int = 500_000,
|
||||
correlation_id: str = None,
|
||||
msg_purpose: str = "chat",
|
||||
sender_name: str = "NATSBridge",
|
||||
@@ -293,9 +352,28 @@ env = NATSBridge.smartreceive(
|
||||
# Returns: ::JSON.Object{String, Any}
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```javascript
|
||||
import NATSBridge from './src/natsbridge_ssr.js';
|
||||
|
||||
const env = await NATSBridge.smartreceive(
|
||||
msg,
|
||||
{
|
||||
fileserver_download_handler: NATSBridge.fetchWithBackoff,
|
||||
max_retries: 5,
|
||||
base_delay: 100,
|
||||
max_delay: 5000
|
||||
}
|
||||
);
|
||||
// Returns: Promise<env_object>
|
||||
```
|
||||
|
||||
#### JavaScript (Browser)
|
||||
|
||||
```javascript
|
||||
import NATSBridge from './src/natsbridge_csr.js';
|
||||
|
||||
const env = await NATSBridge.smartreceive(
|
||||
msg,
|
||||
{
|
||||
@@ -311,6 +389,8 @@ const env = await NATSBridge.smartreceive(
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import NATSBridge
|
||||
|
||||
env = await NATSBridge.smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=fetch_with_backoff,
|
||||
@@ -324,6 +404,8 @@ env = await NATSBridge.smartreceive(
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
from natsbridge import NATSBridge
|
||||
|
||||
env = NATSBridge.smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=_sync_fileserver_download,
|
||||
@@ -343,7 +425,7 @@ env = NATSBridge.smartreceive(
|
||||
| `text` | `String` | `string` | `str` | `str` | Plain text strings |
|
||||
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable dictionaries |
|
||||
| `arrowtable` | `DataFrame`, `Arrow.Table` | `Array<Object>` | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) |
|
||||
| `jsontable` | `Vector{NamedTuple}` | `Array<Object>` | `list[dict]` | ❌ | Tabular data (JSON) |
|
||||
| `jsontable` | `Vector{NamedTuple}` | `Array<Object>` | `list[dict]` | ⚠️ | Tabular data (JSON) |
|
||||
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image data (PNG, JPG) |
|
||||
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio data (WAV, MP3) |
|
||||
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video data (MP4, AVI) |
|
||||
@@ -368,13 +450,13 @@ data = [
|
||||
("large_document", large_file_data, "binary")
|
||||
]
|
||||
|
||||
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; fileserver_url="http://localhost:8080")
|
||||
env, env_json_str = smartsend("/chat/room1", data; fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natsbridge');
|
||||
import NATSBridge from './src/natsbridge_ssr.js';
|
||||
|
||||
const data = [
|
||||
["message_text", "Hello!", "text"],
|
||||
@@ -389,6 +471,24 @@ const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
);
|
||||
```
|
||||
|
||||
#### JavaScript (Browser)
|
||||
|
||||
```javascript
|
||||
import NATSBridge from './src/natsbridge_csr.js';
|
||||
|
||||
const data = [
|
||||
["message_text", "Hello!", "text"],
|
||||
["user_avatar", imageData, "image"],
|
||||
["large_document", largeFileData, "binary"]
|
||||
];
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ broker_url: 'ws://localhost:4222', fileserver_url: 'http://localhost:8080' }
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
@@ -423,13 +523,13 @@ config = Dict(
|
||||
)
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env, env_json_str = NATSBridge.smartsend("/device/config", data)
|
||||
env, env_json_str = smartsend("/device/config", data)
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natsbridge');
|
||||
import NATSBridge from './src/natsbridge_ssr.js';
|
||||
|
||||
const config = {
|
||||
wifi_ssid: "MyNetwork",
|
||||
@@ -475,13 +575,13 @@ df = DataFrame(
|
||||
)
|
||||
|
||||
data = [("students", df, "arrowtable")]
|
||||
env, env_json_str = NATSBridge.smartsend("/data/analysis", data)
|
||||
env, env_json_str = smartsend("/data/analysis", data)
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natsbridge');
|
||||
import NATSBridge from './src/natsbridge_ssr.js';
|
||||
|
||||
const df = [
|
||||
{ id: 1, name: "Alice", score: 95 },
|
||||
@@ -521,18 +621,29 @@ Bi-directional communication with reply-to support.
|
||||
using NATSBridge
|
||||
|
||||
# Requester
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
env, env_json_str = smartsend(
|
||||
"/device/command",
|
||||
[("command", Dict("action" => "read_sensor"), "dictionary")];
|
||||
broker_url="nats://localhost:4222",
|
||||
reply_to="/device/response"
|
||||
)
|
||||
|
||||
# Receiver (in separate application)
|
||||
msg = NATS.subscription.next()
|
||||
env = smartreceive(msg)
|
||||
# Process request and send response
|
||||
response_env, response_json = smartsend(
|
||||
"/device/response",
|
||||
[("result", Dict("value" => 42), "dictionary")],
|
||||
reply_to="/device/command",
|
||||
reply_to_msg_id=env["msg_id"]
|
||||
)
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natsbridge');
|
||||
import NATSBridge from './src/natsbridge_ssr.js';
|
||||
|
||||
// Requester
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
@@ -540,6 +651,16 @@ const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
[["command", { action: "read_sensor" }, "dictionary"]],
|
||||
{ broker_url: 'nats://localhost:4222', reply_to: '/device/response' }
|
||||
);
|
||||
|
||||
// Receiver (in separate application)
|
||||
// const msg = await natsConsumer.next();
|
||||
// const env = await NATSBridge.smartreceive(msg);
|
||||
// Process request and send response
|
||||
// const response_env, response_json = await NATSBridge.smartsend(
|
||||
// "/device/response",
|
||||
// [["result", { value: 42 }, "dictionary"]],
|
||||
// { reply_to: '/device/command', reply_to_msg_id: env.msg_id }
|
||||
// );
|
||||
```
|
||||
|
||||
#### Python
|
||||
@@ -554,6 +675,17 @@ env, env_json_str = await NATSBridge.smartsend(
|
||||
broker_url="nats://localhost:4222",
|
||||
reply_to="/device/response"
|
||||
)
|
||||
|
||||
# Receiver (in separate application)
|
||||
# msg = await nats_consumer.next()
|
||||
# env = await NATSBridge.smartreceive(msg)
|
||||
# Process request and send response
|
||||
# response_env, response_json = await NATSBridge.smartsend(
|
||||
# "/device/response",
|
||||
# [("result", {"value": 42}, "dictionary")],
|
||||
# reply_to="/device/command",
|
||||
# reply_to_msg_id=env["msg_id"]
|
||||
# )
|
||||
```
|
||||
|
||||
---
|
||||
@@ -640,10 +772,10 @@ python3 test/test_py_table_receiver.py
|
||||
|
||||
For detailed architecture and implementation information, see:
|
||||
|
||||
- [Architecture Documentation](docs/architecture_updated.md) - Cross-platform architecture, API parity, platform-specific patterns
|
||||
- [Implementation Guide](docs/implementation_updated.md) - Detailed implementation for each platform, handler functions, testing
|
||||
- [Tutorial](docs/tutorial_updated.md) - Step-by-step getting started guide
|
||||
- [Walkthrough](docs/walkthrough_updated.md) - Real-world application building guides
|
||||
- [`docs/architecture.md`](docs/architecture.md) - Cross-platform architecture, API parity, platform-specific patterns
|
||||
- [`docs/requirements.md`](docs/requirements.md) - Business requirements and user stories
|
||||
- [`docs/spec.md`](docs/spec.md) - Technical specification and contracts
|
||||
- [`docs/walkthrough.md`](docs/walkthrough.md) - Real-world application building guides
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user