diff --git a/README.md b/README.md index a609308..427ec8b 100644 --- a/README.md +++ b/README.md @@ -12,14 +12,12 @@ A high-performance, bi-directional data bridge for **Julia, JavaScript, Python, - [Overview](#overview) - [Cross-Platform Support](#cross-platform-support) - [Features](#features) -- [Architecture](#architecture) -- [Installation](#installation) - [Quick Start](#quick-start) - [API Reference](#api-reference) - [Payload Types](#payload-types) -- [Transport Strategies](#transport-strategies) - [Cross-Platform Examples](#cross-platform-examples) - [Testing](#testing) +- [Documentation](#documentation) - [License](#license) --- @@ -58,7 +56,6 @@ NATSBridge enables seamless communication across multiple platforms through NATS | Multiple Dispatch | ✅ Native | ❌ | ❌ | ❌ | | Async/Await | ❌ | ✅ Native | ✅ Native | ⚠️ (uasyncio) | | Type Safety | ✅ Strong | ⚠️ (TypeScript) | ✅ (Type hints) | ❌ | -| Memory Management | ✅ GC | ✅ GC | ✅ GC | ⚠️ (Manual) | | Arrow IPC | ✅ Native | ✅ | ✅ | ❌ | | Direct Transport | ✅ | ✅ | ✅ | ✅ | | Link Transport | ✅ | ✅ | ✅ | ⚠️ (Limited) | @@ -82,171 +79,6 @@ NATSBridge enables seamless communication across multiple platforms through NATS --- -## Architecture - -### System Components - -```mermaid -flowchart TB - subgraph Sender["Application (Sender)"] - SenderApp[App Code] - NATSBridge_Send[NATSBridge] - NATS_Client[NATS.jl] - end - - subgraph Receiver["Application (Receiver)"] - ReceiverApp[App Code] - NATSBridge_Recv[NATSBridge] - NATS_Client_Recv[NATS.jl] - end - - subgraph Infrastructure["Infrastructure"] - NATS[NATS Server
Message Broker] - FileServer[HTTP File Server
Upload/Download] - end - - SenderApp --> NATSBridge_Send - NATSBridge_Send --> NATS_Client - NATS_Client --> NATS - - NATS --> NATS_Client_Recv - NATS_Client_Recv --> NATSBridge_Recv - NATSBridge_Recv --> ReceiverApp - - NATSBridge_Send -.->|HTTP POST upload| FileServer - FileServer -.->|HTTP GET download| NATSBridge_Recv - - style SenderApp fill:#e8f5e9 - style ReceiverApp fill:#e8f5e9 - style NATS fill:#fff3e0 - style FileServer fill:#f3e5f5 -``` - -### Message Flow - -1. **Sender** creates a message envelope with payloads using `smartsend()` -2. **NATSBridge** serializes and encodes each payload based on type -3. **Transport Decision**: - - **Direct** (< 1MB): Payload encoded as Base64, published to NATS - - **Link** (≥ 1MB): Payload uploaded to HTTP file server, URL published to NATS -4. **NATS** routes message envelope to subscribers -5. **Receiver** receives message via NATS subscription callback -6. **NATSBridge** processes envelope: - - Decodes Base64 payloads from NATS message - - Fetches URLs from file server with exponential backoff -7. **Receiver** deserializes payloads based on their type - -### File Server Handler Abstraction - -The system uses handler functions to abstract file server operations: - -| Handler | Purpose | -|---------|---------| -| `plik_oneshot_upload()` / `plikOneshotUpload()` | Uploads payload bytes to file server, returns URL | -| `_fetch_with_backoff()` / `fetchWithBackoff()` | Downloads data from URL with exponential backoff retry | - -This abstraction allows support for different file server implementations (Plik, AWS S3, custom HTTP server). - -### Message Envelope Schema - -All platforms use identical JSON schemas for message envelopes: - -```json -{ - "correlation_id": "uuid-v4-string", - "msg_id": "uuid-v4-string", - "timestamp": "2024-01-15T10:30:00Z", - "send_to": "topic/subject", - "msg_purpose": "ACK | NACK | updateStatus | shutdown | chat", - "sender_name": "agent-wine-web-frontend", - "sender_id": "uuid4", - "receiver_name": "agent-backend", - "receiver_id": "uuid4", - "reply_to": "topic", - "reply_to_msg_id": "uuid4", - "broker_url": "nats://localhost:4222", - "metadata": {}, - "payloads": [ - { - "id": "uuid4", - "dataname": "login_image", - "payload_type": "image", - "transport": "direct", - "encoding": "base64", - "size": 15433, - "data": "base64-encoded-string" - }, - { - "id": "uuid4", - "dataname": "large_table", - "payload_type": "table", - "transport": "link", - "encoding": "none", - "size": 524288, - "data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/data.arrow" - } - ] -} -``` - ---- - -## Installation - -### Prerequisites - -- **NATS Server** (v2.10+ recommended) -- **HTTP File Server** (optional, for payloads > 1MB) - -### Platform-Specific Dependencies - -#### Julia - -```julia -using Pkg -Pkg.add("NATS") -Pkg.add("Arrow") -Pkg.add("JSON3") -Pkg.add("HTTP") -Pkg.add("UUIDs") -Pkg.add("Dates") -``` - -#### JavaScript (Node.js) - -```bash -npm install nats uuid apache-arrow node-fetch -# or -yarn add nats uuid apache-arrow node-fetch -``` - -#### JavaScript (Browser) - -```bash -npm install nats uuid apache-arrow -# or use CDN: -# https://unpkg.com/nats-js/dist/bundle/nats.min.js -# https://unpkg.com/apache-arrow/arrow.min.js -``` - -#### Python (Desktop) - -```bash -pip install nats-py aiohttp pyarrow pandas python-dateutil -``` - -#### MicroPython - -MicroPython uses built-in modules: -- `network` - NATS connection (custom implementation) -- `time` - Timestamps -- `uos` - File operations -- `base64` - Base64 encoding -- `json` - JSON parsing -- `struct` - Binary data handling - ---- - ## Quick Start ### Step 1: Start NATS Server @@ -265,6 +97,46 @@ mkdir -p /tmp/fileserver python3 -m http.server 8080 --directory /tmp/fileserver ``` +### Step 3: Send Your First Message + +#### Julia + +```julia +using NATSBridge + +data = [("message", "Hello World", "text")] +env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222") +println("Message sent!") +``` + +#### JavaScript + +```javascript +const NATSBridge = require('./src/natsbridge.js'); + +const data = [["message", "Hello World", "text"]]; +const [env, env_json_str] = await NATSBridge.smartsend( + "/chat/room1", + data, + { broker_url: "nats://localhost:4222" } +); +console.log("Message sent!"); +``` + +#### Python + +```python +from natsbridge import smartsend + +data = [("message", "Hello World", "text")] +env, env_json_str = await smartsend( + "/chat/room1", + data, + broker_url="nats://localhost:4222" +) +print("Message sent!") +``` + --- ## API Reference @@ -308,8 +180,8 @@ Sends data either directly via NATS or via a fileserver URL, depending on payloa using NATSBridge env, env_json_str = NATSBridge.smartsend( - subject::String, # NATS subject - data::AbstractArray{Tuple{String, Any, String}}; # List of (dataname, data, type) + subject::String, + data::AbstractArray{Tuple{String, Any, String}}; broker_url::String = "nats://localhost:4222", fileserver_url = "http://localhost:8080", fileserver_upload_handler::Function = plik_oneshot_upload, @@ -468,7 +340,9 @@ env = NATSBridge.smartreceive( |------|-------|------------|--------|-------------|-------------| | `text` | `String` | `string` | `str` | `str` | Plain text strings | | `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable dictionaries | -| `table` | `DataFrame`, `Arrow.Table` | `Array` → `Arrow.Table` | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) | +| `arrowtable` | `DataFrame`, `Arrow.Table` | `Array` | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) | +| `jsontable` | `Vector{NamedTuple}` | `Array` | `list[dict]` | ❌ | Tabular data (JSON) | +| `table` | ❌ | ❌ | `pandas.DataFrame` | ❌ | Python unified table type | | `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image data (PNG, JPG) | | `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio data (WAV, MP3) | | `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video data (MP4, AVI) | @@ -476,58 +350,6 @@ env = NATSBridge.smartreceive( --- -## Transport Strategies - -### Direct Transport (Payloads < 1MB) - -Small payloads are sent directly via NATS with Base64 encoding. - -#### Cross-Platform - -```julia -# Julia -data = [("message", "Hello", "text")] -smartsend("/topic", data) -``` - -```javascript -// JavaScript -const data = [["message", "Hello", "text"]]; -smartsend("/topic", data); -``` - -```python -# Python -data = [("message", "Hello", "text")] -await smartsend("/topic", data) -``` - -### Link Transport (Payloads >= 1MB) - -Large payloads are uploaded to an HTTP file server. - -#### Cross-Platform - -```julia -# Julia -data = [("file", large_data, "binary")] -smartsend("/topic", data; fileserver_url="http://localhost:8080") -``` - -```javascript -// JavaScript -const data = [["file", largeData, "binary"]]; -smartsend("/topic", data, { fileserver_url: 'http://localhost:8080' }); -``` - -```python -# Python -data = [("file", large_data, "binary")] -await smartsend("/topic", data, fileserver_url="http://localhost:8080") -``` - ---- - ## Cross-Platform Examples ### Example 1: Chat with Mixed Content @@ -651,7 +473,7 @@ df = DataFrame( score = [95, 88, 92] ) -data = [("students", df, "table")] +data = [("students", df, "arrowtable")] env, env_json_str = NATSBridge.smartsend("/data/analysis", data) ``` @@ -668,7 +490,7 @@ const df = [ const [env, env_json_str] = await NATSBridge.smartsend( "/data/analysis", - [["students", df, "table"]] + [["students", df, "arrowtable"]] ); ``` @@ -706,32 +528,6 @@ env, env_json_str = NATSBridge.smartsend( ) ``` -```julia -# Responder -using NATS, NATSBridge - -function test_responder() - conn = NATS.connect("nats://localhost:4222") - NATS.subscribe(conn, "/device/command") do msg - env = NATSBridge.smartreceive(msg, fileserver_download_handler=_fetch_with_backoff) - - reply_to = env["reply_to"] - - for (dataname, data, type) in env["payloads"] - if dataname == "command" && data["action"] == "read_sensor" - response = Dict("sensor_id" => "sensor-001", "value" => 42.5) - if !isempty(reply_to) - smartsend(reply_to, [("data", response, "dictionary")]) - end - end - end - end - - sleep(120) - NATS.drain(conn) -end -``` - #### JavaScript ```javascript @@ -745,40 +541,6 @@ const [env, env_json_str] = await NATSBridge.smartsend( ); ``` -```javascript -// Responder -const nats = require('nats'); -const NATSBridge = require('natsbridge'); - -async function testResponder() { - const conn = await nats.connect('nats://localhost:4222'); - - const subscription = await conn.subscribe('/device/command'); - - for await (const msg of subscription) { - const env = await NATSBridge.smartreceive(msg, { - fileserver_download_handler: NATSBridge.fetchWithBackoff - }); - - const replyTo = env.reply_to; - - for (const [dataname, data, type] of env.payloads) { - if (dataname === 'command' && data.action === 'read_sensor') { - const response = { sensor_id: 'sensor-001', value: 42.5 }; - if (replyTo) { - await NATSBridge.smartsend( - replyTo, - [["data", response, "dictionary"]] - ); - } - } - } - } - - setTimeout(() => conn.close(), 120000); -} -``` - #### Python ```python @@ -793,38 +555,6 @@ env, env_json_str = await NATSBridge.smartsend( ) ``` -```python -# Responder -from natsbridge import NATSBridge -import asyncio -import nats - -async def test_responder(): - nc = await nats.connect('nats://localhost:4222') - - async def msg_handler(msg): - env = await NATSBridge.smartreceive( - msg, - fileserver_download_handler=fetch_with_backoff - ) - - reply_to = env["reply_to"] - - for dataname, data, type_ in env["payloads"]: - if dataname == "command" and data["action"] == "read_sensor": - response = {"sensor_id": "sensor-001", "value": 42.5} - if reply_to: - await NATSBridge.smartsend( - reply_to, - [("data", response, "dictionary")] - ) - - await nc.subscribe('/device/command', cb=msg_handler) - - await asyncio.sleep(120) - await nc.drain() -``` - --- ## Testing @@ -909,8 +639,10 @@ python3 test/test_py_table_receiver.py For detailed architecture and implementation information, see: -- [Architecture Documentation](docs/architecture.md) - Cross-platform architecture, API parity, platform-specific patterns -- [Implementation Guide](docs/implementation.md) - Detailed implementation for each platform, handler functions, testing +- [Architecture Documentation](docs/architecture_updated.md) - Cross-platform architecture, API parity, platform-specific patterns +- [Implementation Guide](docs/implementation_updated.md) - Detailed implementation for each platform, handler functions, testing +- [Tutorial](docs/tutorial_updated.md) - Step-by-step getting started guide +- [Walkthrough](docs/walkthrough_updated.md) - Real-world application building guides --- @@ -936,4 +668,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. \ No newline at end of file +SOFTWARE.