diff --git a/README.md b/README.md index e103f6f..c87a7ca 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,9 @@ # msghandler - Cross-Platform Communication Layer -A high-performance, **transport-agnostic** communication layer for **Julia**, **JavaScript**, **Python**, and **MicroPython** applications. Implements the Claim-Check pattern for efficient payload transport (direct for small payloads, URL-based for large payloads). +A high-performance, **transport-agnostic** communication layer for both **Julia** (backend) and **JavaScript** (CSR webapp) applications. Implements the Claim-Check pattern for efficient payload transport (direct for small payloads, URL-based for large payloads). + +- 🟢 **Julia Backend**: Full-featured implementation with Arrow IPC support +- 🔵 **JavaScript Frontend**: Browser-compatible implementation for CSR webapps [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT) @@ -8,12 +11,14 @@ A high-performance, **transport-agnostic** communication layer for **Julia**, ** ## Table of Contents -- [Overview](#overview) -- [Cross-Platform Support](#cross-platform-support) -- [Features](#features) - [Quick Start](#quick-start) + - [Julia Backend](#julia-backend) + - [JavaScript CSR Webapp](#javascript-csr-webapp) +- [Overview](#overview) +- [Features](#features) +- [Installation](#installation) - [API Reference](#api-reference) -- [Cross-Platform Examples](#cross-platform-examples) +- [Examples](#examples) - [Testing](#testing) - [Documentation](#documentation) - [License](#license) @@ -22,34 +27,33 @@ A high-performance, **transport-agnostic** communication layer for **Julia**, ** ## Quick Data Format Reference -All platforms use the same input format for `smartpack()`: +### Input Format for `smartpack()` -### Format: `[(dataname, data, type), ...]` +**Format:** `[(dataname, data, type), ...]` | Element | Type | Description | |---------|------|-------------| -| `dataname` | String | Name for the payload (e.g., `"message"`, `"config.json"`) | -| `data` | Any | The actual data (type depends on the `type` parameter below) | -| `type` | String | Payload type identifier | +| `dataname` | `String` | Name for the payload (e.g., `"message"`, `"config.json"`) | +| `data` | `Any` | The actual data (type depends on the `type` parameter below) | +| `type` | `String` | Payload type identifier | ### Supported Payload Types -| Type | Julia Data | JavaScript Data | Python Data | Description | -|------|------------|-----------------|-------------|-------------| -| `"text"` | `String` | `string` | `str` | Plain text | -| `"dictionary"` | `Dict` | `Object` | `dict` | JSON object | -| `"arrowtable"` | `DataFrame` | ❌ | `pandas.DataFrame` | Arrow IPC table (Desktop only) | -| `"jsontable"` | `DataFrame` | `Array` | `list[dict]` | JSON table | -| `"image"` | `Vector{UInt8}` | `Uint8Array` | `bytes` | Image data | -| `"audio"` | `Vector{UInt8}` | `Uint8Array` | `bytes` | Audio data | -| `"video"` | `Vector{UInt8}` | `Uint8Array` | `bytes` | Video data | -| `"binary"` | `Vector{UInt8}` | `Uint8Array` | `bytes` | Binary data | +| Type | Julia Data | Description | +|------|------------|-------------| +| `"text"` | `String` | Plain text | +| `"dictionary"` | `Dict` | JSON object | +| `"arrowtable"` | `DataFrame`, `Arrow.Table` | Arrow IPC table | +| `"jsontable"` | `DataFrame`, `Vector{Dict}`, `Vector{NamedTuple}` | JSON table | +| `"image"` | `Vector{UInt8}` | Image data | +| `"audio"` | `Vector{UInt8}` | Audio data | +| `"video"` | `Vector{UInt8}` | Video data | +| `"binary"` | `Vector{UInt8}`, `IOBuffer` | Binary data | ### Examples **Sending multiple payloads:** ```julia -# Julia data = [ ("message", "Hello!", "text"), ("config", Dict("key" => "value"), "dictionary"), @@ -58,38 +62,16 @@ data = [ env, json_str = msghandler.smartpack("subject", data, options...) ``` -```python -# Python -data = [ - ("message", "Hello!", "text"), - ("config", {"key": "value"}, "dictionary"), - ("file", file_bytes, "binary") -] -env, json_str = await smartpack("subject", data, **options) -``` - -```javascript -// JavaScript -const data = [ - ["message", "Hello!", "text"], - ["config", {key: "value"}, "dictionary"], - ["file", fileBytes, "binary"] -]; -const [env, json_str] = await smartpack("subject", data, options); -``` - **Important Notes:** - Always wrap payloads in a list `[]`, even for single payloads - Type must be one of the supported types above - Large payloads (≥500KB) automatically use link transport -- Browser only supports: `"text"`, `"dictionary"`, `"jsontable"`, `"image"`, `"audio"`, `"video"`, `"binary"` - (No Arrow IPC due to browser incompatibility) --- ## Overview -msghandler provides a **transport-agnostic** communication layer with intelligent payload transport selection: +msghandler provides a **transport-agnostic** communication layer with intelligent payload transport selection (same API for Julia and JavaScript): | Transport | Payload Size | Method | |-----------|--------------|--------| @@ -100,68 +82,239 @@ msghandler provides a **transport-agnostic** communication layer with intelligen - **Chat Applications**: Text, images, audio, video in a single message - **File Transfer**: Efficient transfer of large files using claim-check pattern -- **IoT/Embedded**: Sensor data, telemetry, and analytics pipelines (MicroPython) -- **Cross-Platform Communication**: Interoperability between Julia, JavaScript, Python, and MicroPython systems +- **Cross-Platform Communication**: Interoperability between Julia backend and JavaScript frontend - **Any Transport**: Works with NATS, HTTP, WebSockets, WebRTC, or any custom transport mechanism +### Platform Differences + +| Feature | Julia Backend | JavaScript Frontend | +|---------|---------------|---------------------| +| **Arrow IPC** | ✅ Supported | ❌ Not supported (use "jsontable" instead) | +| **Full Type Support** | ✅ All 8 types | ✅ 7 of 8 types | +| **Transport Support** | ✅ NATS, HTTP, WebSockets, etc. | ✅ Fetch, WebSockets, HTTP | +| **File Server** | ✅ Plik, AWS S3, custom | ✅ Plik, custom HTTP server | + +### Supported Payload Types + +| Type | Julia | JavaScript | Description | +|------|-------|------------|-------------| +| `"text"` | ✅ | ✅ | Plain text | +| `"dictionary"` | ✅ | ✅ | JSON object | +| `"arrowtable"` | ✅ | ❌ | Arrow IPC table (Julia only) | +| `"jsontable"` | ✅ | ✅ | JSON table | +| `"image"` | ✅ | ✅ | Image data | +| `"audio"` | ✅ | ✅ | Audio data | +| `"video"` | ✅ | ✅ | Video data | +| `"binary"` | ✅ | ✅ | Binary data | + ### Key Design Principles - **Transport Agnostic**: Core API (`smartpack`/`smartunpack`) is decoupled from transport - **Claim-Check Pattern**: Efficient handling of large payloads via URL references -- **Type System**: Rich payload types (text, dictionary, arrowtable, jsontable, binary) -- **Cross-Platform**: Unified API across Julia, JavaScript, Python, and MicroPython - ---- - -## Cross-Platform Support - -| Platform | Implementation | Features | -|----------|----------------|----------| -| **Julia** | [`src/msghandler.jl`](src/msghandler.jl) | Full feature set, Arrow IPC, multiple dispatch | -| **JavaScript (Node.js)** | [`src/msghandler_ssr.js`](src/msghandler_ssr.js) | Node.js, async/await, Arrow IPC | -| **JavaScript (Browser)** | [`src/msghandler_csr.js`](src/msghandler_csr.js) | Browser, async/await, JSON table only | -| **Python** | [`src/msghandler.py`](src/msghandler.py) | Desktop Python, asyncio, type hints, Arrow IPC | -| **MicroPython** | [`src/msghandler_mpy.py`](src/msghandler_mpy.py) | Memory-constrained, synchronous API | - -### Platform Comparison - -| Feature | Julia | JavaScript | JavaScript (Browser) | Python | MicroPython | -|---------|-------|------------|----------------------|--------|-------------| -| Multiple Dispatch | ✅ Native | ❌ | ❌ | ❌ | ❌ | -| Async/Await | ❌ | ✅ Native | ✅ Native | ✅ Native | ⚠️ (uasyncio) | -| Type Safety | ✅ Strong | ⚠️ (TypeScript) | ⚠️ (TypeScript) | ✅ (Type hints) | ❌ | -| Arrow IPC | ✅ Native | ✅ Native | ❌ (Browser incompatible) | ✅ Native | ❌ | -| JSON Table | ✅ | ✅ | ✅ (Only table type) | ✅ | ⚠️ (Limited) | -| Direct Transport | ✅ | ✅ | ✅ | ✅ | ✅ | -| Link Transport | ✅ | ✅ | ✅ | ✅ | ⚠️ (Limited) | -| Handler Functions | ✅ | ✅ | ✅ | ✅ | ✅ | -| Cross-Platform API | ✅ | ✅ | ✅ | ✅ | ✅ | +- **Rich Type System**: Support for Arrow IPC, JSON tables, and binary data +- **Handler Abstraction**: Pluggable file server implementations (Plik, AWS S3, custom) --- ## Features +- ✅ **Cross-platform** - Same API for Julia backend and JavaScript frontend - ✅ **Transport-agnostic** - Core API works with any communication channel (NATS, HTTP, WebSockets, etc.) -- ✅ **Cross-platform** - Unified API for Julia, JavaScript, Python, and MicroPython - ✅ **Bi-directional** - Request-reply patterns with `reply_to` support - ✅ **Multi-payload** - Send multiple payloads of different types in one message - ✅ **Automatic transport selection** - Direct vs link based on payload size (threshold: 500KB) - ✅ **Claim-Check pattern** - Efficient large payload handling via URL references -- ✅ **Rich payload types** - Text, dictionary, Arrow IPC, JSON table, image, audio, video, binary +- ✅ **Rich payload types** - Text, dictionary, Arrow IPC (Julia only), JSON table, image, audio, video, binary - ✅ **Exponential backoff** - Reliable file server downloads with retry logic - ✅ **Correlation ID** - End-to-end message tracing - ✅ **Handler abstraction** - Pluggable file server implementations (Plik, AWS S3, custom) --- +## Installation + +### Julia Backend + +```julia +using Pkg +Pkg.add("msghandler") +``` + +Add to your `Project.toml`: +```toml +[deps] +msghandler = "path-to-msghandler" +``` + +### JavaScript Frontend + +**Using NPM:** +```bash +npm install msghandler-csr +``` + +**Using local source (development):** +```bash +cp ./src/msghandler-csr.js ./public/js/msghandler.js +``` + +Then import in your browser: +```javascript +import msghandlerCSR from './js/msghandler.js'; +``` + +--- + ## Quick Start +### Julia Backend + +#### Installation + +```julia +using Pkg +Pkg.add("msghandler") +``` + +Add to your `Project.toml`: +```toml +[deps] +msghandler = "path-to-msghandler" +``` + +#### Quick Start + +```julia +using msghandler + +# Data format: [(dataname, data, type), ...] +payload_1 = ("test_message", "Hello World", "text") +file_path_large_image = "./test/large_image.png" +file_data_large_image = read(file_path_large_image) +filename_large_image = basename(file_path_large_image) +payload_2 = (filename_large_image, file_data_large_image, "binary") + +payloads = [payload_1, payload_2] # List of tuples + +# Step 1: Create the message envelope (transport-agnostic) +envelope, envelope_json_str = msghandler.smartpack("test.topic", + payloads; + broker_url="nats.yiem.cc", + fileserver_url="http://192.168.88.104:8080") + +# Step 2: Send via your chosen transport (NATS in this example) +using NATS +conn = NATS.connect("nats.yiem.cc") +NATS.publish(conn, "test.topic", envelope_json_str; reply_to="test.replytopic") +NATS.drain(conn) + +# OR using request-reply pattern +reply = NATS.request(conn, "test.topic", envelope_json_str, timeout=10) +``` + +#### Receive Messages + +```julia +using msghandler, NATS + +conn = NATS.connect("nats.yiem.cc") +NATS.subscribe(conn, "test.topic") do msg + println("Received message on $(msg.subject)") + envelope_json_str = String(msg.payload) + + # Step 2: Unpack the envelope (transport-agnostic) + envelope = msghandler.smartunpack( + envelope_json_str; + max_retries = 5, + base_delay = 100, + max_delay = 5000 + ) + println(envelope.payloads[1]) +end +``` + +### JavaScript CSR Webapp + +#### Installation + +**Using NPM:** +```bash +npm install msghandler-csr +``` + +**Using ES Modules (browser):** +```javascript +import msghandlerCSR from './src/msghandler-csr.js'; +``` + +#### Features + +- ✅ No build tools required (works directly in browser) +- ✅ Uses native browser APIs (Web Crypto, Fetch, TextEncoder/Decoder) +- ✅ Browser-compatible payload types (no Arrow IPC dependency) + +#### Quick Start + +```javascript +import msghandlerCSR from './src/msghandler-csr.js'; + +const { smartpack, smartunpack, plikOneshotUpload, fetchWithBackoff } = msghandlerCSR; + +// Data format: [[dataname, data, type], ...] +const payload1 = ["test_message", "Hello World", "text"]; +const payload2 = ["config_data", { key: "value" }, "dictionary"]; + +const payloads = [payload1, payload2]; // Array of arrays + +// Step 1: Create the message envelope (transport-agnostic) +const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack("test.topic", payloads, { + broker_url: "nats.yiem.cc", + fileserver_url: "http://192.168.88.104:8080" +}); + +// Step 2: Send via your chosen transport (NATS, WebSocket, HTTP, etc.) +// await myTransport.publish("test.topic", envelopeJsonStr); + +// OR using fetch API for HTTP transport +fetch("http://localhost:3000/api/messages", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: envelopeJsonStr +}); +``` + +#### Receive Messages + +```javascript +import msghandlerCSR from './src/msghandler-csr.js'; + +const { smartunpack } = msghandlerCSR; + +// Option 1: Direct JSON string +const envelope = await msghandlerCSR.smartunpack(jsonString, { + fileserver_download_handler: msghandlerCSR.fetchWithBackoff, + max_retries: 5, + base_delay: 100, + max_delay: 5000 +}); + +// Option 2: From transport message object (e.g., NATS, WebSocket) +const envelope = await msghandlerCSR.smartunpack(natsMessage, { + fileserver_download_handler: msghandlerCSR.fetchWithBackoff +}); + +// Process payloads +for (const [dataname, data, type] of envelope.payloads) { + console.log(`${dataname}:`, data, `(type: ${type})`); +} +``` + ### Prerequisites 1. **Transport Channel** - Choose your communication channel: - - **NATS** (recommended): `docker run -p 4222:4222 nats:latest` - - **HTTP/WebSocket**: Any HTTP server or WebSocket endpoint - - **Custom**: Implement your own transport mechanism + - **NATS** (recommended): `docker run -p 4222:4222 nats:latest` + - **HTTP/WebSocket**: Any HTTP server or WebSocket endpoint + - **Custom**: Implement your own transport mechanism 2. **HTTP File Server** (optional, for payloads ≥ 500KB) - Install and run: ```bash @@ -175,21 +328,11 @@ msghandler provides a **transport-agnostic** communication layer with intelligen ### Send Your First Message -#### Julia - ```julia using msghandler, NATS # Data format: [(dataname, data, type), ...] -# Each tuple contains: -# 1. dataname: String name for the payload (e.g., "message") -# 2. data: The actual data (String, bytes, DataFrame, etc.) -# 3. type: String type indicator (e.g., "text", "binary", "dictionary") -# Supported types: "text", "dictionary", "arrowtable", "jsontable", -# "image", "audio", "video", "binary" - payload_1 = ("test_message", "Hello World", "text") - file_path_large_image = "./test/large_image.png" file_data_large_image = read(file_path_large_image) filename_large_image = basename(file_path_large_image) @@ -208,90 +351,15 @@ conn = NATS.connect("nats.yiem.cc") NATS.publish(conn, "test.topic", envelope_json_str; reply_to="test.replytopic") NATS.drain(conn) -# OR +# OR using request-reply pattern reply = NATS.request(conn, "test.topic", envelope_json_str, timeout=10) ``` -#### JavaScript (Node.js) - -```javascript -import msghandler from './src/msghandler_ssr.js'; - -const data = [["message", "Hello World", "text"]]; -const [env, env_json_str] = await msghandler.smartpack( - "/chat/room1", - data, - { broker_url: "nats://localhost:4222" } -); -console.log("Message sent!"); -``` - -#### JavaScript (Browser) - -```javascript -import msghandler from './src/msghandler_csr.js'; - -const data = [["message", "Hello World", "text"]]; -const [env, env_json_str] = await msghandler.smartpack( - "/chat/room1", - data, - { broker_url: "ws://localhost:4222" } -); -console.log("Message sent!"); -``` - -#### Python - -```python -from msghandler import smartpack - -# Data format: [(dataname, data, type), ...] -# Each tuple contains: -# 1. dataname: String name for the payload (e.g., "message") -# 2. data: The actual data (str, bytes, dict, list, etc.) -# 3. type: String type indicator (e.g., "text", "binary", "dictionary") -# Supported types: "text", "dictionary", "jsontable", "image", "audio", "video", "binary" - -data = [("message", "Hello World", "text")] -env, env_json_str = await smartpack( - "/chat/room1", - data, - broker_url="nats://localhost:4222", - fileserver_url="http://localhost:8080" -) -print("Envelope created:", env) - -# To send via transport (e.g., NATS, HTTP, WebSocket): -# transport_send_function("/chat/room1", env_json_str) -``` - -#### MicroPython - -```python -from msghandler import smartpack - -data = [("message", "Hello World", "text")] -env, env_json_str = smartpack( - "/chat/room1", - data, - broker_url="nats://localhost:4222", - fileserver_url="http://localhost:8080", - size_threshold=100000 -) -print("Envelope created:", env) - -# To send via transport (e.g., HTTP POST, WebSocket): -# transport_send_function("/chat/room1", env_json_str) -``` - ### Receive Your First Message -#### Julia - ```julia using msghandler, NATS -# Step 1: Receive message from your transport (NATS in this example) conn = NATS.connect("nats.yiem.cc") NATS.subscribe(conn, "test.topic") do msg println("Received message on $(msg.subject)") @@ -312,103 +380,9 @@ end ## API Reference -### Unified API Standard - -All platforms use the same input/output format for payloads. - -#### Input Format for `smartpack` - -**Format:** `[(dataname1, data1, type1), (dataname2, data2, type2), ...]` - -**Each tuple contains 3 elements:** -1. **`dataname`** (String) - Name for the payload (e.g., `"message"`, `"config.json"`, `"avatar.png"`) -2. **`data`** (Any) - The actual data to send (type depends on `type`): - - `text` → String - - `dictionary` → Dict/Object - - `arrowtable` → DataFrame/Arrow.Table - - `jsontable` → List of dicts/Vector{Dict} - - `image/audio/video/binary` → bytes/Uint8Array/Vector{UInt8} -3. **`type`** (String) - Payload type (required): - - `"text"` - Plain text string - - `"dictionary"` - JSON-serializable dictionary - - `"arrowtable"` - Apache Arrow IPC table (Desktop only) - - `"jsontable"` - JSON table (all platforms) - - `"image"` - Image data (PNG, JPG) - - `"audio"` - Audio data (WAV, MP3) - - `"video"` - Video data (MP4, AVI) - - `"binary"` - Generic binary data - -**Important:** -- Always wrap payloads in a list, even for single payloads -- Type must be one of the supported types above -- Large payloads (≥500KB by default) automatically use link transport - -**Example:** -```julia -# Julia -data = [ - ("message", "Hello World", "text"), - ("config", Dict("key" => "value"), "dictionary"), - ("file", file_bytes, "binary") -] -``` - -```python -# Python -data = [ - ("message", "Hello World", "text"), - ("config", {"key": "value"}, "dictionary"), - ("file", file_bytes, "binary") -] -``` - -```javascript -// JavaScript -const data = [ - ["message", "Hello World", "text"], - ["config", {key: "value"}, "dictionary"], - ["file", fileBytes, "binary"] -]; -``` - -### Common Payload Examples - -| Use Case | Data Format | Type | -|----------|-------------|------| -| Simple text message | `("message", "Hello World", "text")` | `"text"` | -| Configuration JSON | `("config", Dict("key" => "value"), "dictionary")` | `"dictionary"` | -| Table as Arrow | `("data", dataframe, "arrowtable")` | `"arrowtable"` | -| Table as JSON | `("data", [{"col": 1}], "jsontable")` | `"jsontable"` | -| Image file | `("avatar.png", read("file.png"), "image")` | `"image"` | -| Audio file | `("audio.mp3", audio_bytes, "audio")` | `"audio"` | -| Video file | `("video.mp4", video_bytes, "video")` | `"video"` | -| Binary file | `("data.bin", file_bytes, "binary")` | `"binary"` | - -#### Output Format for `smartunpack` -```json -{ - "correlation_id": "...", - "msg_id": "...", - "timestamp": "...", - "send_to": "...", - "msg_purpose": "...", - "sender_name": "...", - "sender_id": "...", - "receiver_name": "...", - "receiver_id": "...", - "reply_to": "...", - "reply_to_msg_id": "...", - "broker_url": "...", - "metadata": {...}, - "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...] -} -``` - ### smartpack -Sends data via your chosen transport mechanism (NATS, HTTP, WebSocket, etc.) with intelligent transport selection (direct vs URL-based) based on payload size. - -#### Julia +Sends data via your chosen transport mechanism with intelligent transport selection. ```julia using msghandler @@ -427,125 +401,21 @@ env, env_json_str = msghandler.smartpack( receiver_id::String = "", reply_to::String = "", reply_to_msg_id::String = "", - is_publish::Bool = true, - NATS_connection::Union{NATS.Connection, Nothing} = nothing, msg_id::String = string(uuid4()), sender_id::String = string(uuid4()) ) # Returns: ::Tuple{msg_envelope_v1, String} ``` -#### JavaScript (Node.js) - -```javascript -import msghandler from './src/msghandler_ssr.js'; - -const [env, env_json_str] = await msghandler.smartpack( - subject, - data, // Array of [dataname, data, type] tuples - { - broker_url: 'nats://localhost:4222', - fileserver_url: 'http://localhost:8080', - fileserver_upload_handler: msghandler.plikOneshotUpload, - size_threshold: 500_000, - correlation_id: uuidv4(), - msg_purpose: 'chat', - sender_name: 'msghandler', - receiver_name: '', - receiver_id: '', - reply_to: '', - reply_to_msg_id: '', - is_publish: true, - nats_connection: null, - msg_id: uuidv4(), - sender_id: uuidv4() - } -); -// Returns: Promise<[env, env_json_str]> -``` - -#### JavaScript (Browser) - -```javascript -import msghandler from './src/msghandler_csr.js'; - -const [env, env_json_str] = await msghandler.smartpack( - subject, - data, - { - broker_url: 'ws://localhost:4222', - fileserver_url: 'http://localhost:8080', - fileserver_upload_handler: msghandler.plikOneshotUpload, - size_threshold: 500_000, - correlation_id: uuidv4(), - msg_purpose: 'chat', - sender_name: 'msghandler', - receiver_name: '', - receiver_id: '', - reply_to: '', - reply_to_msg_id: '', - is_publish: true, - nats_connection: null, - msg_id: uuidv4(), - sender_id: uuidv4() - } -); -// Returns: Promise<[env, env_json_str]> -``` - -#### Python - -```python -from msghandler import msghandler - -env, env_json_str = await msghandler.smartpack( - subject: str, - data: List[Tuple[str, Any, str]], - broker_url: str = "nats://localhost:4222", - fileserver_url: str = "http://localhost:8080", - fileserver_upload_handler: Callable = plik_oneshot_upload, - size_threshold: int = 500_000, - correlation_id: str = None, - msg_purpose: str = "chat", - sender_name: str = "msghandler", - receiver_name: str = "", - receiver_id: str = "", - reply_to: str = "", - reply_to_msg_id: str = "", - is_publish: bool = True, - nats_connection: Any = None, - msg_id: str = None, - sender_id: str = None -) -# Returns: Tuple[Dict, str] -``` - -#### MicroPython - -```python -from msghandler import smartpack - -# Limited to direct transport (< 100KB threshold) -env, env_json_str = smartpack( - "/device/config", - [("config", config, "dictionary")], - broker_url="nats://localhost:4222", - size_threshold=100000 -) -print("Message sent!") -``` - ### smartunpack -Receives and processes messages from NATS, handling both direct and link transport. - -#### Julia +Receives and processes messages from your transport. ```julia using msghandler env = msghandler.smartunpack( - msg::NATS.Msg; + msg_json_str::String; fileserver_download_handler::Function = _fetch_with_backoff, max_retries::Int = 5, base_delay::Int = 100, @@ -554,80 +424,14 @@ env = msghandler.smartunpack( # Returns: ::JSON.Object{String, Any} ``` -#### JavaScript (Node.js) - -```javascript -import msghandler from './src/msghandler_ssr.js'; - -const env = await msghandler.smartunpack( - msg, - { - fileserver_download_handler: msghandler.fetchWithBackoff, - max_retries: 5, - base_delay: 100, - max_delay: 5000 - } -); -// Returns: Promise -``` - -#### JavaScript (Browser) - -```javascript -import msghandler from './src/msghandler_csr.js'; - -const env = await msghandler.smartunpack( - msg, - { - fileserver_download_handler: msghandler.fetchWithBackoff, - max_retries: 5, - base_delay: 100, - max_delay: 5000 - } -); -// Returns: Promise -``` - -#### Python - -```python -from msghandler import msghandler - -env = await msghandler.smartunpack( - msg, - fileserver_download_handler=fetch_with_backoff, - max_retries=5, - base_delay=100, - max_delay=5000 -) -# Returns: Dict with "payloads" key -``` - -#### MicroPython - -```python -from msghandler import smartunpack - -env = smartunpack( - msg, - fileserver_download_handler=_sync_fileserver_download, - max_retries=3, - base_delay=100, - max_delay=1000 -) -print("Received payloads:", env["payloads"]) -``` - --- -## Cross-Platform Examples +## Examples ### Example 1: Chat with Mixed Content Send text, image, and large file in one message. -#### Julia - ```julia using msghandler @@ -640,95 +444,9 @@ data = [ env, env_json_str = smartpack("/chat/room1", data; fileserver_url="http://localhost:8080") ``` -#### JavaScript (Node.js) - -```javascript -import msghandler from './src/msghandler_ssr.js'; - -const data = [ - ["message_text", "Hello!", "text"], - ["user_avatar", imageData, "image"], - ["large_document", largeFileData, "binary"] -]; - -const [env, env_json_str] = await msghandler.smartpack( - "/chat/room1", - data, - { fileserver_url: 'http://localhost:8080' } -); -``` - -#### JavaScript (Browser) - -```javascript -import msghandler from './src/msghandler_csr.js'; - -const data = [ - ["message_text", "Hello!", "text"], - ["user_avatar", image_data, "image"] -]; - -const [env, env_json_str] = await msghandler.smartpack( - "/chat/room1", - data, - { broker_url: 'ws://localhost:4222' } -); -console.log("Message sent!"); -``` - -#### Python - -```python -import asyncio -from msghandler import smartpack - -async def main(): - image_data = open("user_avatar.png", "rb").read() - large_file_data = open("large_document.pdf", "rb").read() - - data = [ - ("message_text", "Hello!", "text"), - ("user_avatar", image_data, "image"), - ("large_document", large_file_data, "binary") - ] - - env, env_json_str = await smartpack( - "/chat/room1", - data, - fileserver_url="http://localhost:8080" - ) - print("Message sent!") - -if __name__ == "__main__": - asyncio.run(main()) -``` - -#### MicroPython - -```python -from msghandler import smartpack - -# Note: MicroPython only supports direct transport (< 100KB threshold) -# Large files must be uploaded via file server first -data = [ - ("message_text", "Hello!", "text"), - ("user_avatar", image_bytes, "image") -] - -env, env_json_str = smartpack( - "/chat/room1", - data, - broker_url="nats://localhost:4222", - size_threshold=100000 -) -print("Message sent!") -``` - ### Example 2: Dictionary Exchange -Send configuration data between platforms. - -#### Julia +Send configuration data. ```julia using msghandler @@ -743,72 +461,10 @@ data = [("config", config, "dictionary")] env, env_json_str = smartpack("/device/config", data) ``` -#### JavaScript (Node.js) - -```javascript -import msghandler from './src/msghandler_ssr.js'; - -const config = { - wifi_ssid: "MyNetwork", - wifi_password: "password123", - update_interval: 60 -}; - -const [env, env_json_str] = await msghandler.smartpack( - "/device/config", - [["config", config, "dictionary"]] -); -console.log("Message sent!"); -``` - -#### Python - -```python -import asyncio -from msghandler import smartpack - -async def main(): - config = { - "wifi_ssid": "MyNetwork", - "wifi_password": "password123", - "update_interval": 60 - } - - data = [("config", config, "dictionary")] - env, env_json_str = await smartpack("/device/config", data) - print("Message sent!") - -if __name__ == "__main__": - asyncio.run(main()) -``` - -#### MicroPython - -```python -from msghandler import smartpack - -config = { - "wifi_ssid": "MyNetwork", - "wifi_password": "password123", - "update_interval": 60 -} - -data = [("config", config, "dictionary")] -env, env_json_str = smartpack( - "/device/config", - data, - broker_url="nats://localhost:4222", - size_threshold=100000 -) -print("Message sent!") -``` - -### Example 3: Table Data (Arrow IPC) +### Example 3: Table Data (Arrow IPC - Julia Only) Send tabular data using Apache Arrow IPC format. -#### Julia - ```julia using msghandler using DataFrames @@ -823,87 +479,26 @@ data = [("students", df, "arrowtable")] env, env_json_str = smartpack("/data/analysis", data) ``` -#### JavaScript (Node.js) +### Example 3b: Table Data (JSON - JavaScript Compatible) + +For JavaScript frontend, use `jsontable` instead of `arrowtable`: ```javascript -import msghandler from './src/msghandler_ssr.js'; - -const df = [ +const tableData = [ { id: 1, name: "Alice", score: 95 }, { id: 2, name: "Bob", score: 88 }, { id: 3, name: "Charlie", score: 92 } ]; -const [env, env_json_str] = await msghandler.smartpack( - "/data/analysis", - [["students", df, "arrowtable"]] -); -``` - -#### Python - -```python -from msghandler import msghandler -import pandas as pd - -df = pd.DataFrame({ - "id": [1, 2, 3], - "name": ["Alice", "Bob", "Charlie"], - "score": [95, 88, 92] -}) - -data = [("students", df, "arrowtable")] -env, env_json_str = await msghandler.smartpack("/data/analysis", data) -``` - -#### JavaScript (Browser) - -```javascript -import msghandler from './src/msghandler_csr.js'; - -const df = [ - { id: 1, name: "Alice", score: 95 }, - { id: 2, name: "Bob", score: 88 }, - { id: 3, name: "Charlie", score: 92 } -]; - -const [env, env_json_str] = await msghandler.smartpack( - "/data/analysis", - [["students", df, "jsontable"]], - { broker_url: 'ws://localhost:4222' } -); -console.log("Message sent!"); -``` - -#### MicroPython - -```python -from msghandler import smartpack - -# Note: MicroPython only supports direct transport (< 100KB threshold) -# MicroPython doesn't support Arrow IPC, only jsontable -df = [ - {"id": 1, "name": "Alice", "score": 95}, - {"id": 2, "name": "Bob", "score": 88}, - {"id": 3, "name": "Charlie", "score": 92} -] - -data = [("students", df, "jsontable")] -env, env_json_str = smartpack( - "/data/analysis", - data, - broker_url="nats://localhost:4222", - size_threshold=100000 -) -print("Message sent!") +const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack("/data/analysis", [ + ["students", tableData, "jsontable"] +]); ``` ### Example 4: Request-Response Pattern Bi-directional communication with reply-to support. -#### Julia - ```julia using msghandler, NATS @@ -938,126 +533,11 @@ NATS.subscribe(conn, "/device/command") do msg end ``` -#### JavaScript (Node.js) - -```javascript -import msghandler from './src/msghandler_ssr.js'; -import { connect } from 'nats'; - -// Requester -const [env, env_json_str] = await msghandler.smartpack( - "/device/command", - [["command", { action: "read_sensor" }, "dictionary"]], - { broker_url: 'nats://localhost:4222', reply_to: '/device/response' } -); -const nc = await connect({ port: 4222 }); -nc.publish("/device/command", env_json_str); -await nc.flush(); - -// Receiver (in separate application) -const nc = await connect({ port: 4222 }); -const sub = nc.subscribe("/device/command"); -for await (const msg of sub) { - const env = await msghandler.smartunpack(msg); - console.log("Received command:", env.payloads); - - const response_env, response_json = await msghandler.smartpack( - "/device/response", - [["result", { value: 42 }, "dictionary"]], - { reply_to: '/device/command', reply_to_msg_id: env.msg_id } - ); - - nc.publish("/device/response", response_json); - await nc.flush(); -} -``` - -#### Python - -```python -import asyncio -from msghandler import smartpack - -async def main(): - # Requester - env, env_json_str = await smartpack( - "/device/command", - [("command", {"action": "read_sensor"}, "dictionary")], - broker_url="nats://localhost:4222", - reply_to="/device/response" - ) - print("Request sent!") - - # Receiver (in separate application) - # await nats_consumer.next() - # env = await smartunpack(msg) - # Process request and send response - # response_env, response_json = await smartpack( - # "/device/response", - # [("result", {"value": 42}, "dictionary")], - # reply_to="/device/command", - # reply_to_msg_id=env["msg_id"] - # ) - -if __name__ == "__main__": - asyncio.run(main()) -``` - -#### JavaScript (Browser) - -```javascript -import msghandler from './src/msghandler_csr.js'; - -// Requester -const [env, env_json_str] = await msghandler.smartpack( - "/device/command", - [["command", { action: "read_sensor" }, "dictionary"]], - { broker_url: 'ws://localhost:4222', reply_to: '/device/response' } -); -console.log("Request sent!"); - -// Receiver (in separate application) -// const msg = await natsConsumer.next(); -// const env = await msghandler.smartunpack(msg); -// console.log("Received command:", env.payloads); -// const response_env, response_json = await msghandler.smartpack( -// "/device/response", -// [["result", { value: 42 }, "dictionary"]], -// { reply_to: '/device/command', reply_to_msg_id: env.msg_id } -// ); -``` - -#### MicroPython - -```python -from msghandler import smartpack - -# Requester -env, env_json_str = smartpack( - "/device/command", - [("command", {"action": "read_sensor"}, "dictionary")], - broker_url="nats://localhost:4222", - reply_to="/device/response", - size_threshold=100000 -) -print("Request sent!") -``` - --- ## Testing -### Test File Organization - -| Platform | Sender Tests | Receiver Tests | -|----------|--------------|----------------| -| **Julia** | `test/test_julia_*_sender.jl` | `test/test_julia_*_receiver.jl` | -| **JavaScript** | `test/test_js_*_sender.js` | `test/test_js_*_receiver.js` | -| **Python** | `test/test_py_*_sender.py` | `test/test_py_*_receiver.py` | - -### Run Tests - -#### Julia +### Julia Backend ```bash # Text message exchange @@ -1081,161 +561,44 @@ julia test/test_julia_table_sender.jl julia test/test_julia_table_receiver.jl ``` -#### JavaScript (Node.js) +### JavaScript Frontend +To test the JavaScript version in a browser: + +1. Start a file server for testing: ```bash -# Text message exchange -node test/test_js_text_sender.js -node test/test_js_text_receiver.js - -# Dictionary exchange -node test/test_js_dictionary_sender.js -node test/test_js_dictionary_receiver.js - -# Binary transfer -node test/test_js_binary_sender.js -node test/test_js_binary_receiver.js - -# Table exchange -node test/test_js_table_sender.js -node test/test_js_table_receiver.js +mkdir -p /tmp/fileserver +python3 -m http.server 8080 --directory /tmp/fileserver ``` -#### Python - +2. Start a NATS server (optional): ```bash -# Text message exchange -python3 test/test_py_text_sender.py -python3 test/test_py_text_receiver.py - -# Dictionary exchange -python3 test/test_py_dictionary_sender.py -python3 test/test_py_dictionary_receiver.py - -# Binary transfer -python3 test/test_py_binary_sender.py -python3 test/test_py_binary_receiver.py - -# Table exchange -python3 test/test_py_table_sender.py -python3 test/test_py_table_receiver.py +docker run -p 4222:4222 nats:latest ``` ---- - -## Browser Deployment - -### Using with Node.js Build Tools - -The browser implementation (`src/msghandler_csr.js`) can be bundled for production deployment using modern JavaScript build tools. - -#### Prerequisites - +3. Open the test HTML file in a browser: ```bash -# Install the browser-compatible NATS client -npm install nats.ws -``` - -#### Vite (Recommended) - -```bash -npm create vite@latest my-app -- --template vanilla -cd my-app -npm install nats.ws -``` - -In `vite.config.js`: -```javascript -import { defineConfig } from 'vite'; -export default defineConfig({ - resolve: { - alias: { - 'nats.ws': 'nats.ws/dist/esm/browser.js' - } - } -}); -``` - -Build command: -```bash -npm run build # Outputs to dist/ folder -``` - -#### Webpack - -```bash -npm install webpack webpack-cli --save-dev -npm install nats.ws -``` - -In `webpack.config.js`: -```javascript -module.exports = { - entry: './src/index.js', - output: { - filename: 'bundle.js', - path: __dirname + '/dist' - }, - resolve: { - alias: { - 'nats.ws': 'nats.ws/dist/esm/browser.js' - } - } -}; -``` - -Build command: -```bash -npx webpack -``` - -#### esbuild (Simple & Fast) - -```bash -npm install esbuild nats.ws --save-dev -``` - -Create `build.js`: -```javascript -import esbuild from 'esbuild'; - -esbuild.buildSync({ - entryPoints: ['src/msghandler_csr.js'], - bundle: true, - outfile: 'dist/msghandler-csr-bundle.js', - format: 'esm', - platform: 'browser', - target: 'es2020' -}); -``` - -Build command: -```bash -node build.js -``` - -### Using in Your HTML - -```html +# Create test file (example) +cat > test-browser.html << 'EOF' - - My App - +msghandler CSR Test - - + +EOF + +python3 -m http.server 8081 +# Open http://localhost:8081/test-browser.html ``` --- @@ -1244,11 +607,56 @@ node build.js For detailed architecture and implementation information, see: -- [`docs/architecture.md`](docs/architecture.md) - Cross-platform architecture, API parity, platform-specific patterns +- [`docs/architecture.md`](docs/architecture.md) - Architecture and design patterns - [`docs/requirements.md`](docs/requirements.md) - Business requirements and user stories - [`docs/spec.md`](docs/spec.md) - Technical specification and contracts - [`docs/walkthrough.md`](docs/walkthrough.md) - Real-world application building guides +### JavaScript API Reference + +The JavaScript version provides the same core functionality as Julia, with browser-compatible APIs: + +#### smartpack + +```javascript +const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack( + subject, + data, // [[dataname, data, type], ...] + options +); +``` + +#### smartunpack + +```javascript +const envelope = await msghandlerCSR.smartunpack(msg, options); +// envelope.payloads = [[dataname, data, type], ...] +``` + +#### plikOneshotUpload + +```javascript +const result = await msghandlerCSR.plikOneshotUpload( + fileServerUrl, + dataname, + data // Uint8Array +); +// result = { status, uploadid, fileid, url } +``` + +#### fetchWithBackoff + +```javascript +const data = await msghandlerCSR.fetchWithBackoff( + url, + maxRetries, + baseDelay, + maxDelay, + correlationId +); +// data = Uint8Array +``` + --- ## License diff --git a/src/msghandler.js b/src/msghandler-ssr.js similarity index 100% rename from src/msghandler.js rename to src/msghandler-ssr.js