move README.md
This commit is contained in:
295
README.md
Normal file
295
README.md
Normal file
@@ -0,0 +1,295 @@
|
||||
# NATSBridge
|
||||
|
||||
A high-performance, bi-directional data bridge for **Julia**, **JavaScript**, and **Python/Micropython** using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
|
||||
## Overview
|
||||
|
||||
NATSBridge enables seamless communication between Julia, JavaScript, and Python/Micropython applications through NATS, with automatic transport selection based on payload size:
|
||||
|
||||
- **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded)
|
||||
- **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL
|
||||
|
||||
## Features
|
||||
|
||||
- ✅ Bi-directional NATS communication across Julia ↔ JavaScript ↔ Python/Micropython
|
||||
- ✅ Multi-payload support (mixed content in single message)
|
||||
- ✅ Automatic transport selection based on payload size
|
||||
- ✅ File server integration for large payloads
|
||||
- ✅ Exponential backoff for URL fetching
|
||||
- ✅ Correlation ID tracking
|
||||
- ✅ Reply-to support for request-response pattern
|
||||
|
||||
## Supported Payload Types
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `text` | Plain text strings |
|
||||
| `dictionary` | JSON-serializable dictionaries |
|
||||
| `table` | Tabular data (Arrow IPC format) |
|
||||
| `image` | Image data (PNG, JPG bytes) |
|
||||
| `audio` | Audio data (WAV, MP3 bytes) |
|
||||
| `video` | Video data (MP4, AVI bytes) |
|
||||
| `binary` | Generic binary data |
|
||||
|
||||
## Implementation Guides
|
||||
|
||||
### [Julia Implementation](../tutorial_julia.md)
|
||||
|
||||
See the [Julia tutorial](../tutorial_julia.md) for getting started with Julia.
|
||||
|
||||
### [JavaScript Implementation](#javascript-implementation)
|
||||
|
||||
See [`NATSBridge.js`](NATSBridge.js) for the JavaScript implementation.
|
||||
|
||||
### [Python/Micropython Implementation](#pythonmicropython-implementation)
|
||||
|
||||
See [`nats_bridge.py`](nats_bridge.py) for the Python/Micropython implementation.
|
||||
|
||||
## Installation
|
||||
|
||||
### Julia
|
||||
|
||||
```julia
|
||||
using Pkg
|
||||
Pkg.add("NATS")
|
||||
Pkg.add("Arrow")
|
||||
Pkg.add("JSON3")
|
||||
Pkg.add("HTTP")
|
||||
Pkg.add("UUIDs")
|
||||
Pkg.add("Dates")
|
||||
```
|
||||
|
||||
### JavaScript
|
||||
|
||||
```bash
|
||||
npm install nats.js apache-arrow uuid base64-url
|
||||
```
|
||||
|
||||
### Python/Micropython
|
||||
|
||||
1. Copy `nats_bridge.py` to your device
|
||||
2. Ensure you have the following dependencies:
|
||||
- `urequests` for HTTP requests (Micropython)
|
||||
- `requests` for HTTP requests (Python)
|
||||
- `base64` for base64 encoding
|
||||
- `json` for JSON handling
|
||||
- `socket` for networking (Micropython)
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Text Message
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend, smartreceive
|
||||
|
||||
# Sender
|
||||
data = [("message", "Hello World", "text")]
|
||||
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
||||
|
||||
# Receiver
|
||||
payloads = smartreceive(msg)
|
||||
for dataname, data, type in payloads:
|
||||
print("Received {}: {}".format(dataname, data))
|
||||
```
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Sender
|
||||
data = [("message", "Hello World", "text")]
|
||||
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
||||
|
||||
# Receiver
|
||||
envelope = smartreceive(msg, fileserverDownloadHandler)
|
||||
# envelope["payloads"] = [("message", "Hello World", "text"), ...]
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const { smartsend, smartreceive } = require('./src/NATSBridge');
|
||||
|
||||
// Sender
|
||||
await smartsend("/chat/room1", [
|
||||
{ dataname: "message", data: "Hello World", type: "text" }
|
||||
], { natsUrl: "nats://localhost:4222" });
|
||||
|
||||
// Receiver
|
||||
const envelope = await smartreceive(msg);
|
||||
// envelope.payloads = [{ dataname: "message", data: "Hello World", type: "text" }, ...]
|
||||
```
|
||||
|
||||
### Sending JSON Configuration
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
config = {
|
||||
"wifi_ssid": "MyNetwork",
|
||||
"wifi_password": "password123",
|
||||
"update_interval": 60
|
||||
}
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env = smartsend("/device/config", data, nats_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
### Mixed Content (Chat with Text + Image)
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
image_data = b"\x89PNG..." # PNG bytes
|
||||
|
||||
data = [
|
||||
("message_text", "Hello with image!", "text"),
|
||||
("user_avatar", image_data, "binary")
|
||||
]
|
||||
|
||||
env = smartsend("/chat/mixed", data, nats_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
### Request-Response Pattern
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
# Send command with reply-to
|
||||
data = [("command", {"action": "read_sensor"}, "dictionary")]
|
||||
env = smartsend(
|
||||
"/device/command",
|
||||
data,
|
||||
nats_url="nats://localhost:4222",
|
||||
reply_to="/device/response",
|
||||
reply_to_msg_id="cmd-001"
|
||||
)
|
||||
```
|
||||
|
||||
### Large Payloads (File Server)
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
# Large data (> 1MB)
|
||||
large_data = b"A" * 2000000 # 2MB
|
||||
|
||||
env = smartsend(
|
||||
"/data/large",
|
||||
[("large_file", large_data, "binary")],
|
||||
nats_url="nats://localhost:4222",
|
||||
fileserver_url="http://localhost:8080",
|
||||
size_threshold=1000000 # 1MB threshold
|
||||
)
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `smartsend(subject, data, ...)`
|
||||
|
||||
Send data via NATS with automatic transport selection.
|
||||
|
||||
**Arguments:**
|
||||
- `subject` (str): NATS subject to publish to
|
||||
- `data` (list): List of `(dataname, data, type)` tuples
|
||||
- `nats_url` (str): NATS server URL (default: `nats://localhost:4222`)
|
||||
- `fileserver_url` (str): HTTP file server URL (default: `http://localhost:8080`)
|
||||
- `size_threshold` (int): Threshold in bytes (default: 1,000,000)
|
||||
- `correlation_id` (str): Optional correlation ID for tracing
|
||||
- `msg_purpose` (str): Message purpose (default: `"chat"`)
|
||||
- `sender_name` (str): Sender name (default: `"NATSBridge"`)
|
||||
- `receiver_name` (str): Receiver name (default: `""`)
|
||||
- `receiver_id` (str): Receiver ID (default: `""`)
|
||||
- `reply_to` (str): Reply topic (default: `""`)
|
||||
- `reply_to_msg_id` (str): Reply message ID (default: `""`)
|
||||
|
||||
**Returns:** `MessageEnvelope` object
|
||||
|
||||
### `smartreceive(msg, ...)`
|
||||
|
||||
Receive and process NATS messages.
|
||||
|
||||
**Arguments:**
|
||||
- `msg`: NATS message (dict or JSON string)
|
||||
- `fileserver_download_handler` (function): Function to fetch data from URLs
|
||||
- `max_retries` (int): Maximum retry attempts (default: 5)
|
||||
- `base_delay` (int): Initial delay in ms (default: 100)
|
||||
- `max_delay` (int): Maximum delay in ms (default: 5000)
|
||||
|
||||
**Returns:** List of `(dataname, data, type)` tuples
|
||||
|
||||
### `MessageEnvelope`
|
||||
|
||||
Represents a complete NATS message envelope.
|
||||
|
||||
**Attributes:**
|
||||
- `correlation_id`: Unique identifier for tracing
|
||||
- `msg_id`: Unique message identifier
|
||||
- `timestamp`: Message publication timestamp
|
||||
- `send_to`: NATS subject
|
||||
- `msg_purpose`: Message purpose
|
||||
- `sender_name`: Sender name
|
||||
- `sender_id`: Sender UUID
|
||||
- `receiver_name`: Receiver name
|
||||
- `receiver_id`: Receiver UUID
|
||||
- `reply_to`: Reply topic
|
||||
- `reply_to_msg_id`: Reply message ID
|
||||
- `broker_url`: NATS broker URL
|
||||
- `metadata`: Message-level metadata
|
||||
- `payloads`: List of MessagePayload objects
|
||||
|
||||
### `MessagePayload`
|
||||
|
||||
Represents a single payload within a message envelope.
|
||||
|
||||
**Attributes:**
|
||||
- `id`: Unique payload identifier
|
||||
- `dataname`: Name of the payload
|
||||
- `type`: Payload type ("text", "dictionary", etc.)
|
||||
- `transport`: Transport method ("direct" or "link")
|
||||
- `encoding`: Encoding method ("none", "base64", etc.)
|
||||
- `size`: Payload size in bytes
|
||||
- `data`: Payload data (bytes for direct, URL for link)
|
||||
- `metadata`: Payload-level metadata
|
||||
|
||||
## Examples
|
||||
|
||||
See [`examples/micropython_example.py`](../examples/micropython_example.py) for more detailed examples.
|
||||
|
||||
## Testing
|
||||
|
||||
Run the test suite:
|
||||
|
||||
```bash
|
||||
# Python/Micropython
|
||||
python test/test_micropython_basic.py
|
||||
|
||||
# JavaScript
|
||||
node test/test_js_to_js_text_sender.js
|
||||
node test/test_js_to_js_text_receiver.js
|
||||
|
||||
# Julia
|
||||
julia test/test_julia_to_julia_text_sender.jl
|
||||
julia test/test_julia_to_julia_text_receiver.jl
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- **Julia**: NATS server (nats.io), HTTP file server (optional)
|
||||
- **JavaScript**: NATS server (nats.io), HTTP file server (optional)
|
||||
- **Python/Micropython**: NATS server (nats.io), HTTP file server (optional)
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
Reference in New Issue
Block a user