# msghandler - Cross-Platform Bi-Directional Data Bridge A high-performance, bi-directional data bridge for **Julia**, **JavaScript**, **Python**, and **MicroPython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads. [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT) [![NATS](https://img.shields.io/badge/NATS-Enabled-green.svg)](https://nats.io) --- ## Table of Contents - [Overview](#overview) - [Cross-Platform Support](#cross-platform-support) - [Features](#features) - [Quick Start](#quick-start) - [API Reference](#api-reference) - [Payload Types](#payload-types) - [Cross-Platform Examples](#cross-platform-examples) - [Testing](#testing) - [Documentation](#documentation) - [License](#license) --- ## Overview msghandler enables seamless communication across multiple platforms through NATS, with intelligent transport selection based on payload size: | Transport | Payload Size | Method | |-----------|--------------|--------| | **Direct** | < 500KB | Sent directly via NATS (Base64 encoded) | | **Link** | ≥ 500KB | Uploaded to HTTP file server, URL sent via NATS | ### Use Cases - **Chat Applications**: Text, images, audio, video in a single message - **File Transfer**: Efficient transfer of large files using claim-check pattern - **IoT/Embedded**: Sensor data, telemetry, and analytics pipelines (MicroPython) - **Cross-Platform Communication**: Interoperability between Julia, JavaScript, Python, and MicroPython systems --- ## Cross-Platform Support | Platform | Implementation | Features | |----------|----------------|----------| | **Julia** | [`src/msghandler.jl`](src/msghandler.jl) | Full feature set, Arrow IPC, multiple dispatch | | **JavaScript (Node.js)** | [`src/msghandler_ssr.js`](src/msghandler_ssr.js) | Node.js, async/await, Arrow IPC | | **JavaScript (Browser)** | [`src/msghandler_csr.js`](src/msghandler_csr.js) | Browser, WebSocket NATS, async/await, JSON table only | | **Python** | [`src/msghandler.py`](src/msghandler.py) | Desktop Python, asyncio, type hints, Arrow IPC | | **MicroPython** | [`src/msghandler_mpy.py`](src/msghandler_mpy.py) | Memory-constrained, synchronous API | ### Platform Comparison | Feature | Julia | JavaScript | JavaScript (Browser) | Python | MicroPython | |---------|-------|------------|----------------------|--------|-------------| | Multiple Dispatch | ✅ Native | ❌ | ❌ | ❌ | ❌ | | Async/Await | ❌ | ✅ Native | ✅ Native | ✅ Native | ⚠️ (uasyncio) | | Type Safety | ✅ Strong | ⚠️ (TypeScript) | ⚠️ (TypeScript) | ✅ (Type hints) | ❌ | | Arrow IPC | ✅ Native | ✅ Native | ❌ (Browser incompatible) | ✅ Native | ❌ | | JSON Table | ✅ | ✅ | ✅ (Only table type) | ✅ | ⚠️ (Limited) | | Direct Transport | ✅ | ✅ | ✅ | ✅ | ✅ | | Link Transport | ✅ | ✅ | ✅ | ✅ | ⚠️ (Limited) | | Handler Functions | ✅ | ✅ | ✅ | ✅ | ✅ | | Cross-Platform API | ✅ | ✅ | ✅ | ✅ | ✅ | | WebSocket NATS | ❌ | ❌ | ✅ | ❌ | ❌ | --- ## Features - ✅ **Cross-platform messaging** for Julia, JavaScript, Python, and MicroPython applications - ✅ **Bi-directional messaging** with request-reply patterns - ✅ **Multi-payload support** - send multiple payloads with different types in one message - ✅ **Automatic transport selection** - direct vs link based on payload size - ✅ **Claim-Check pattern** for payloads ≥ 500KB - ✅ **Apache Arrow IPC** support for tabular data (Desktop: Julia/Python/Node.js) - ✅ **JSON Table** support for tabular data (All platforms including Browser) - ✅ **Exponential backoff** for reliable file server downloads - ✅ **Correlation ID tracking** for message tracing - ✅ **Reply-to support** for request-response patterns - ✅ **Handler function abstraction** - pluggable file server implementations (Plik, AWS S3, custom) --- ## Quick Start ### Prerequisites 1. **NATS Server** - Install and run a NATS server: ```bash docker run -p 4222:4222 nats:latest ``` 2. **HTTP File Server** (optional, for large payloads) - Install and run a file server: ```bash # Using Plik docker run -p 8080:8080 -v /tmp/fileserver:/var/lib/plik -e PLIK_ADMIN_PASSWORD=admin plik/plik # OR using simple Python HTTP server mkdir -p /tmp/fileserver python3 -m http.server 8080 --directory /tmp/fileserver ``` ### Send Your First Message #### Julia ```julia using msghandler, NATS payload_1 = ("test_message", "Hello World", "text") file_path_large_image = "./test/large_image.png" file_data_large_image = read(file_path_large_image) filename_large_image = basename(file_path_large_image) payload_2 = (filename_large_image, file_data_large_image, "binary") payloads = [payload_1, payload_2] # payloads to be send format is [(dataname, data, type), ...] tuples. envelope, envelope_json_str = msghandler.smartpack("test.topic", payloads; broker_url="nats.yiem.cc", fileserver_url="http://192.168.88.104:8080") conn = NATS.connect("nats.yiem.cc") # Create NATS connection NATS.publish(conn, "test.topic", envelope_json_str) # Publish message to NATS NATS.drain(conn) ``` #### JavaScript (Node.js) ```javascript import msghandler from './src/msghandler_ssr.js'; const data = [["message", "Hello World", "text"]]; const [env, env_json_str] = await msghandler.smartpack( "/chat/room1", data, { broker_url: "nats://localhost:4222" } ); console.log("Message sent!"); ``` #### JavaScript (Browser) ```javascript import msghandler from './src/msghandler_csr.js'; const data = [["message", "Hello World", "text"]]; const [env, env_json_str] = await msghandler.smartpack( "/chat/room1", data, { broker_url: "ws://localhost:4222" } ); console.log("Message sent!"); ``` #### Python ```python from msghandler import smartpack data = [("message", "Hello World", "text")] env, env_json_str = await smartpack( "/chat/room1", data, broker_url="nats://localhost:4222" ) print("Message sent!") ``` #### MicroPython ```python from msghandler import smartpack data = [("message", "Hello World", "text")] env, env_json_str = smartpack( "/chat/room1", data, broker_url="nats://localhost:4222", size_threshold=100000 # 100KB for MicroPython ) print("Message sent!") ``` ### Receive Your First Message #### Julia ```julia using msghandler, NATS conn = NATS.connect("nats.yiem.cc") NATS.subscribe(conn, "test.topic") do msg println("Received message on $(msg.subject)") payloads = String(msg.payload) # Use NATSBridge.smartreceive to handle the data # API: smartreceive(msg, download_handler; max_retries, base_delay, max_delay) envelope = msghandler.smartunpack( payloads; max_retries = 5, base_delay = 100, max_delay = 5000 ) println(envelope.payloads[1]) end ``` --- ## API Reference ### Unified API Standard All platforms use the same input/output format for payloads: **Input format for `smartpack`:** ``` [(dataname1, data1, type1), (dataname2, data2, type2), ...] ``` **Output format for `smartunpack`:** ```json { "correlation_id": "...", "msg_id": "...", "timestamp": "...", "send_to": "...", "msg_purpose": "...", "sender_name": "...", "sender_id": "...", "receiver_name": "...", "receiver_id": "...", "reply_to": "...", "reply_to_msg_id": "...", "broker_url": "...", "metadata": {...}, "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...] } ``` ### smartpack Sends data either directly via NATS or via a fileserver URL, depending on payload size. #### Julia ```julia using msghandler env, env_json_str = msghandler.smartpack( subject::String, data::AbstractArray{Tuple{String, Any, String}}; broker_url::String = "nats://localhost:4222", fileserver_url = "http://localhost:8080", fileserver_upload_handler::Function = plik_oneshot_upload, size_threshold::Int = 500_000, correlation_id::String = string(uuid4()), msg_purpose::String = "chat", sender_name::String = "msghandler", receiver_name::String = "", receiver_id::String = "", reply_to::String = "", reply_to_msg_id::String = "", is_publish::Bool = true, NATS_connection::Union{NATS.Connection, Nothing} = nothing, msg_id::String = string(uuid4()), sender_id::String = string(uuid4()) ) # Returns: ::Tuple{msg_envelope_v1, String} ``` #### JavaScript (Node.js) ```javascript import msghandler from './src/msghandler_ssr.js'; const [env, env_json_str] = await msghandler.smartpack( subject, data, // Array of [dataname, data, type] tuples { broker_url: 'nats://localhost:4222', fileserver_url: 'http://localhost:8080', fileserver_upload_handler: msghandler.plikOneshotUpload, size_threshold: 500_000, correlation_id: uuidv4(), msg_purpose: 'chat', sender_name: 'msghandler', receiver_name: '', receiver_id: '', reply_to: '', reply_to_msg_id: '', is_publish: true, nats_connection: null, msg_id: uuidv4(), sender_id: uuidv4() } ); // Returns: Promise<[env, env_json_str]> ``` #### JavaScript (Browser) ```javascript import msghandler from './src/msghandler_csr.js'; const [env, env_json_str] = await msghandler.smartpack( subject, data, { broker_url: 'ws://localhost:4222', fileserver_url: 'http://localhost:8080', fileserver_upload_handler: msghandler.plikOneshotUpload, size_threshold: 500_000, correlation_id: uuidv4(), msg_purpose: 'chat', sender_name: 'msghandler', receiver_name: '', receiver_id: '', reply_to: '', reply_to_msg_id: '', is_publish: true, nats_connection: null, msg_id: uuidv4(), sender_id: uuidv4() } ); // Returns: Promise<[env, env_json_str]> ``` #### Python ```python from msghandler import msghandler env, env_json_str = await msghandler.smartpack( subject: str, data: List[Tuple[str, Any, str]], broker_url: str = "nats://localhost:4222", fileserver_url: str = "http://localhost:8080", fileserver_upload_handler: Callable = plik_oneshot_upload, size_threshold: int = 500_000, correlation_id: str = None, msg_purpose: str = "chat", sender_name: str = "msghandler", receiver_name: str = "", receiver_id: str = "", reply_to: str = "", reply_to_msg_id: str = "", is_publish: bool = True, nats_connection: Any = None, msg_id: str = None, sender_id: str = None ) # Returns: Tuple[Dict, str] ``` #### MicroPython ```python from msghandler import msghandler # Limited to direct transport (< 100KB threshold) env, env_json_str = msghandler.smartpack( subject, data, # List of (dataname, data, type) tuples broker_url="nats://localhost:4222", size_threshold=100000 # Lower threshold for memory constraints ) # Returns: Tuple[Dict, str] ``` ### smartunpack Receives and processes messages from NATS, handling both direct and link transport. #### Julia ```julia using msghandler env = msghandler.smartunpack( msg::NATS.Msg; fileserver_download_handler::Function = _fetch_with_backoff, max_retries::Int = 5, base_delay::Int = 100, max_delay::Int = 5000 ) # Returns: ::JSON.Object{String, Any} ``` #### JavaScript (Node.js) ```javascript import msghandler from './src/msghandler_ssr.js'; const env = await msghandler.smartunpack( msg, { fileserver_download_handler: msghandler.fetchWithBackoff, max_retries: 5, base_delay: 100, max_delay: 5000 } ); // Returns: Promise ``` #### JavaScript (Browser) ```javascript import msghandler from './src/msghandler_csr.js'; const env = await msghandler.smartunpack( msg, { fileserver_download_handler: msghandler.fetchWithBackoff, max_retries: 5, base_delay: 100, max_delay: 5000 } ); // Returns: Promise ``` #### Python ```python from msghandler import msghandler env = await msghandler.smartunpack( msg, fileserver_download_handler=fetch_with_backoff, max_retries=5, base_delay=100, max_delay=5000 ) # Returns: Dict with "payloads" key ``` #### MicroPython ```python from msghandler import msghandler env = msghandler.smartunpack( msg, fileserver_download_handler=_sync_fileserver_download, max_retries=3, base_delay=100, max_delay=1000 ) # Returns: Dict with "payloads" key ``` --- ## Payload Types | Type | Julia | JavaScript | Python | MicroPython | Description | |------|-------|------------|--------|-------------|-------------| | `text` | `String` | `string` | `str` | `str` | Plain text strings | | `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable dictionaries | | `arrowtable` | `DataFrame`, `Arrow.Table` | ❌ (Browser), ✅ (Node.js) | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) | | `jsontable` | `DataFrame`, `Vector{NamedTuple}` | `Array` | `list[dict]` | ⚠️ | Tabular data (JSON) - **Only table type in Browser** | | `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image data (PNG, JPG) | | `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio data (WAV, MP3) | | `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video data (MP4, AVI) | | `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `bytearray` | Generic binary data | --- ## Cross-Platform Examples ### Example 1: Chat with Mixed Content Send text, image, and large file in one message. #### Julia ```julia using msghandler data = [ ("message_text", "Hello!", "text"), ("user_avatar", image_data, "image"), ("large_document", large_file_data, "binary") ] env, env_json_str = smartpack("/chat/room1", data; fileserver_url="http://localhost:8080") ``` #### JavaScript (Node.js) ```javascript import msghandler from './src/msghandler_ssr.js'; const data = [ ["message_text", "Hello!", "text"], ["user_avatar", imageData, "image"], ["large_document", largeFileData, "binary"] ]; const [env, env_json_str] = await msghandler.smartpack( "/chat/room1", data, { fileserver_url: 'http://localhost:8080' } ); ``` #### JavaScript (Browser) ```javascript import msghandler from './src/msghandler_csr.js'; const data = [ ["message_text", "Hello!", "text"], ["user_avatar", imageData, "image"], ["large_document", largeFileData, "binary"] ]; const [env, env_json_str] = await msghandler.smartpack( "/chat/room1", data, { broker_url: 'ws://localhost:4222', fileserver_url: 'http://localhost:8080' } ); ``` #### Python ```python from msghandler import msghandler data = [ ("message_text", "Hello!", "text"), ("user_avatar", image_data, "image"), ("large_document", large_file_data, "binary") ] env, env_json_str = await msghandler.smartpack( "/chat/room1", data, fileserver_url="http://localhost:8080" ) ``` ### Example 2: Dictionary Exchange Send configuration data between platforms. #### Julia ```julia using msghandler config = Dict( "wifi_ssid" => "MyNetwork", "wifi_password" => "password123", "update_interval" => 60 ) data = [("config", config, "dictionary")] env, env_json_str = smartpack("/device/config", data) ``` #### JavaScript (Node.js) ```javascript import msghandler from './src/msghandler_ssr.js'; const config = { wifi_ssid: "MyNetwork", wifi_password: "password123", update_interval: 60 }; const [env, env_json_str] = await msghandler.smartpack( "/device/config", [["config", config, "dictionary"]] ); ``` #### Python ```python from msghandler import msghandler config = { "wifi_ssid": "MyNetwork", "wifi_password": "password123", "update_interval": 60 } data = [("config", config, "dictionary")] env, env_json_str = await msghandler.smartpack("/device/config", data) ``` ### Example 3: Table Data (Arrow IPC) Send tabular data using Apache Arrow IPC format. #### Julia ```julia using msghandler using DataFrames df = DataFrame( id = [1, 2, 3], name = ["Alice", "Bob", "Charlie"], score = [95, 88, 92] ) data = [("students", df, "arrowtable")] env, env_json_str = smartpack("/data/analysis", data) ``` #### JavaScript (Node.js) ```javascript import msghandler from './src/msghandler_ssr.js'; const df = [ { id: 1, name: "Alice", score: 95 }, { id: 2, name: "Bob", score: 88 }, { id: 3, name: "Charlie", score: 92 } ]; const [env, env_json_str] = await msghandler.smartpack( "/data/analysis", [["students", df, "arrowtable"]] ); ``` #### Python ```python from msghandler import msghandler import pandas as pd df = pd.DataFrame({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "score": [95, 88, 92] }) data = [("students", df, "arrowtable")] env, env_json_str = await msghandler.smartpack("/data/analysis", data) ``` #### JavaScript (Browser) ```javascript import msghandler from './src/msghandler_csr.js'; // Browser uses jsontable (JSON array of objects) instead of arrowtable // Apache Arrow is not compatible with browsers const df = [ { id: 1, name: "Alice", score: 95 }, { id: 2, name: "Bob", score: 88 }, { id: 3, name: "Charlie", score: 92 } ]; const [env, env_json_str] = await msghandler.smartpack( "/data/analysis", [["students", df, "jsontable"]], // Use jsontable for browser { broker_url: 'ws://localhost:4222' } ); ``` ### Example 4: Request-Response Pattern Bi-directional communication with reply-to support. #### Julia ```julia using msghandler # Requester env, env_json_str = smartpack( "/device/command", [("command", Dict("action" => "read_sensor"), "dictionary")]; broker_url="nats://localhost:4222", reply_to="/device/response" ) # Receiver (in separate application) msg = NATS.subscription.next() env = smartunpack(msg) # Process request and send response response_env, response_json = smartpack( "/device/response", [("result", Dict("value" => 42), "dictionary")], reply_to="/device/command", reply_to_msg_id=env["msg_id"] ) ``` #### JavaScript (Node.js) ```javascript import msghandler from './src/msghandler_ssr.js'; // Requester const [env, env_json_str] = await msghandler.smartpack( "/device/command", [["command", { action: "read_sensor" }, "dictionary"]], { broker_url: 'nats://localhost:4222', reply_to: '/device/response' } ); // Receiver (in separate application) // const msg = await natsConsumer.next(); // const env = await msghandler.smartunpack(msg); // Process request and send response // const response_env, response_json = await msghandler.smartpack( // "/device/response", // [["result", { value: 42 }, "dictionary"]], // { reply_to: '/device/command', reply_to_msg_id: env.msg_id } // ); ``` #### Python ```python from msghandler import msghandler # Requester env, env_json_str = await msghandler.smartpack( "/device/command", [("command", {"action": "read_sensor"}, "dictionary")], broker_url="nats://localhost:4222", reply_to="/device/response" ) # Receiver (in separate application) # msg = await nats_consumer.next() # env = await msghandler.smartunpack(msg) # Process request and send response # response_env, response_json = await msghandler.smartpack( # "/device/response", # [("result", {"value": 42}, "dictionary")], # reply_to="/device/command", # reply_to_msg_id=env["msg_id"] # ) ``` --- ## Testing ### Test File Organization | Platform | Sender Tests | Receiver Tests | |----------|--------------|----------------| | **Julia** | `test/test_julia_*_sender.jl` | `test/test_julia_*_receiver.jl` | | **JavaScript** | `test/test_js_*_sender.js` | `test/test_js_*_receiver.js` | | **Python** | `test/test_py_*_sender.py` | `test/test_py_*_receiver.py` | ### Run Tests #### Julia ```bash # Text message exchange julia test/test_julia_text_sender.jl julia test/test_julia_text_receiver.jl # Dictionary exchange julia test/test_julia_dict_sender.jl julia test/test_julia_dict_receiver.jl # File transfer julia test/test_julia_file_sender.jl julia test/test_julia_file_receiver.jl # Mixed payload types julia test/test_julia_mix_payloads_sender.jl julia test/test_julia_mix_payloads_receiver.jl # Table exchange julia test/test_julia_table_sender.jl julia test/test_julia_table_receiver.jl ``` #### JavaScript (Node.js) ```bash # Text message exchange node test/test_js_text_sender.js node test/test_js_text_receiver.js # Dictionary exchange node test/test_js_dictionary_sender.js node test/test_js_dictionary_receiver.js # Binary transfer node test/test_js_binary_sender.js node test/test_js_binary_receiver.js # Table exchange node test/test_js_table_sender.js node test/test_js_table_receiver.js ``` #### Python ```bash # Text message exchange python3 test/test_py_text_sender.py python3 test/test_py_text_receiver.py # Dictionary exchange python3 test/test_py_dictionary_sender.py python3 test/test_py_dictionary_receiver.py # Binary transfer python3 test/test_py_binary_sender.py python3 test/test_py_binary_receiver.py # Table exchange python3 test/test_py_table_sender.py python3 test/test_py_table_receiver.py ``` --- ## Browser Deployment ### Using with Node.js Build Tools The browser implementation (`src/msghandler_csr.js`) can be bundled for production deployment using modern JavaScript build tools. #### Prerequisites ```bash # Install the browser-compatible NATS client npm install nats.ws ``` #### Vite (Recommended) ```bash npm create vite@latest my-app -- --template vanilla cd my-app npm install nats.ws ``` In `vite.config.js`: ```javascript import { defineConfig } from 'vite'; export default defineConfig({ resolve: { alias: { 'nats.ws': 'nats.ws/dist/esm/browser.js' } } }); ``` Build command: ```bash npm run build # Outputs to dist/ folder ``` #### Webpack ```bash npm install webpack webpack-cli --save-dev npm install nats.ws ``` In `webpack.config.js`: ```javascript module.exports = { entry: './src/index.js', output: { filename: 'bundle.js', path: __dirname + '/dist' }, resolve: { alias: { 'nats.ws': 'nats.ws/dist/esm/browser.js' } } }; ``` Build command: ```bash npx webpack ``` #### esbuild (Simple & Fast) ```bash npm install esbuild nats.ws --save-dev ``` Create `build.js`: ```javascript import esbuild from 'esbuild'; esbuild.buildSync({ entryPoints: ['src/msghandler_csr.js'], bundle: true, outfile: 'dist/msghandler-csr-bundle.js', format: 'esm', platform: 'browser', target: 'es2020' }); ``` Build command: ```bash node build.js ``` ### Using in Your HTML ```html My App ``` --- ## Documentation For detailed architecture and implementation information, see: - [`docs/architecture.md`](docs/architecture.md) - Cross-platform architecture, API parity, platform-specific patterns - [`docs/requirements.md`](docs/requirements.md) - Business requirements and user stories - [`docs/spec.md`](docs/spec.md) - Technical specification and contracts - [`docs/walkthrough.md`](docs/walkthrough.md) - Real-world application building guides --- ## License MIT License Copyright (c) 2026 msghandler Contributors Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.