Files

narawat a134bd44bd update

2026-05-24 12:57:45 +07:00

20 KiB

Raw Blame History

msghandler - Cross-Platform Communication Layer

A high-performance, transport-agnostic communication layer for both Julia (backend) and JavaScript (CSR webapp) applications. Implements the Claim-Check pattern for efficient payload transport (direct for small payloads, URL-based for large payloads).

🟢 Julia Backend: Full-featured implementation with Arrow IPC support
🔵 JavaScript Frontend: Browser-compatible implementation for CSR webapps

Quick Start
- Julia Backend
- JavaScript CSR Webapp
Overview
Features
Installation
API Reference
Examples
Testing
Documentation
License

Quick Data Format Reference

Input Format for `smartpack()`

Format: [(dataname, data, type), ...]

Element	Type	Description
`dataname`	`String`	Name for the payload (e.g., `"message"`, `"config.json"`)
`data`	`Any`	The actual data (type depends on the `type` parameter below)
`type`	`String`	Payload type identifier

Supported Payload Types

Type	Julia Data	Description
`"text"`	`String`	Plain text
`"dictionary"`	`Dict`	JSON object
`"arrowtable"`	`DataFrame`, `Arrow.Table`	Arrow IPC table
`"jsontable"`	`DataFrame`, `Vector{Dict}`, `Vector{NamedTuple}`	JSON table
`"image"`	`Vector{UInt8}`	Image data
`"audio"`	`Vector{UInt8}`	Audio data
`"video"`	`Vector{UInt8}`	Video data
`"binary"`	`Vector{UInt8}`, `IOBuffer`	Binary data

Examples

Sending multiple payloads:

data = [
    ("message", "Hello!", "text"),
    ("config", Dict("key" => "value"), "dictionary"),
    ("file", file_bytes, "binary")
]
env, json_str = msghandler.smartpack("subject", data, options...)

Important Notes:

Always wrap payloads in a list [], even for single payloads
Type must be one of the supported types above
Large payloads (≥500KB) automatically use link transport

Overview

msghandler provides a transport-agnostic communication layer with intelligent payload transport selection (same API for Julia and JavaScript):

Transport	Payload Size	Method
Direct	< 500KB	Sent directly via chosen transport (Base64 encoded)
Link	≥ 500KB	Uploaded to HTTP file server, URL sent via chosen transport

Use Cases

Chat Applications: Text, images, audio, video in a single message
File Transfer: Efficient transfer of large files using claim-check pattern
Cross-Platform Communication: Interoperability between Julia backend and JavaScript frontend
Any Transport: Works with NATS, HTTP, WebSockets, WebRTC, or any custom transport mechanism

Platform Differences

Feature	Julia Backend	JavaScript Frontend
Arrow IPC	✅ Supported	❌ Not supported (use "jsontable" instead)
Full Type Support	✅ All 8 types	✅ 7 of 8 types
Transport Support	✅ NATS, HTTP, WebSockets, etc.	✅ Fetch, WebSockets, HTTP
File Server	✅ Plik, AWS S3, custom	✅ Plik, custom HTTP server

Supported Payload Types

Type	Julia	JavaScript	Description
`"text"`	✅	✅	Plain text
`"dictionary"`	✅	✅	JSON object
`"arrowtable"`	✅	❌	Arrow IPC table (Julia only)
`"jsontable"`	✅	✅	JSON table
`"image"`	✅	✅	Image data
`"audio"`	✅	✅	Audio data
`"video"`	✅	✅	Video data
`"binary"`	✅	✅	Binary data

Key Design Principles

Transport Agnostic: Core API (smartpack/smartunpack) is decoupled from transport
Claim-Check Pattern: Efficient handling of large payloads via URL references
Rich Type System: Support for Arrow IPC, JSON tables, and binary data
Handler Abstraction: Pluggable file server implementations (Plik, AWS S3, custom)

Features

✅ Cross-platform - Same API for Julia backend and JavaScript frontend
✅ Transport-agnostic - Core API works with any communication channel (NATS, HTTP, WebSockets, etc.)
✅ Bi-directional - Request-reply patterns with reply_to support
✅ Multi-payload - Send multiple payloads of different types in one message
✅ Automatic transport selection - Direct vs link based on payload size (threshold: 500KB)
✅ Claim-Check pattern - Efficient large payload handling via URL references
✅ Rich payload types - Text, dictionary, Arrow IPC (Julia only), JSON table, image, audio, video, binary
✅ Exponential backoff - Reliable file server downloads with retry logic
✅ Correlation ID - End-to-end message tracing
✅ Handler abstraction - Pluggable file server implementations (Plik, AWS S3, custom)

Installation

Julia Backend

using Pkg
Pkg.add("msghandler")

Add to your Project.toml:

[deps]
msghandler = "path-to-msghandler"

JavaScript Frontend

Using NPM:

npm install msghandler-csr

Using local source (development):

cp ./src/msghandler-csr.js ./public/js/msghandler.js

Then import in your browser:

import msghandlerCSR from './js/msghandler.js';

Quick Start

Julia Backend

Installation

using Pkg
Pkg.add("msghandler")

Add to your Project.toml:

[deps]
msghandler = "path-to-msghandler"

Quick Start

using msghandler

# Data format: [(dataname, data, type), ...]
payload_1 = ("test_message", "Hello World", "text")
file_path_large_image = "./test/large_image.png"
file_data_large_image = read(file_path_large_image)
filename_large_image = basename(file_path_large_image)
payload_2 = (filename_large_image, file_data_large_image, "binary")

payloads = [payload_1, payload_2]  # List of tuples

# Step 1: Create the message envelope (transport-agnostic)
envelope, envelope_json_str = msghandler.smartpack("test.topic", 
                                                  payloads; 
                                                  broker_url="nats.yiem.cc",
                                                  fileserver_url="http://192.168.88.104:8080")

# Step 2: Send via your chosen transport (NATS in this example)
using NATS
conn = NATS.connect("nats.yiem.cc")
NATS.publish(conn, "test.topic", envelope_json_str; reply_to="test.replytopic")
NATS.drain(conn)

# OR using request-reply pattern
reply = NATS.request(conn, "test.topic", envelope_json_str, timeout=10)

Receive Messages

using msghandler, NATS

conn = NATS.connect("nats.yiem.cc")
NATS.subscribe(conn, "test.topic") do msg
    println("Received message on $(msg.subject)")
    envelope_json_str = String(msg.payload)
    
    # Step 2: Unpack the envelope (transport-agnostic)
    envelope = msghandler.smartunpack(
        envelope_json_str;
        max_retries = 5,
        base_delay = 100,
        max_delay = 5000
    )
    println(envelope.payloads[1])
end

JavaScript CSR Webapp

Installation

Using NPM:

npm install msghandler-csr

Using ES Modules (browser):

import msghandlerCSR from './src/msghandler-csr.js';

Features

✅ No build tools required (works directly in browser)
✅ Uses native browser APIs (Web Crypto, Fetch, TextEncoder/Decoder)
✅ Browser-compatible payload types (no Arrow IPC dependency)

Quick Start

import msghandlerCSR from './src/msghandler-csr.js';

const { smartpack, smartunpack, plikOneshotUpload, fetchWithBackoff } = msghandlerCSR;

// Data format: [[dataname, data, type], ...]
const payload1 = ["test_message", "Hello World", "text"];
const payload2 = ["config_data", { key: "value" }, "dictionary"];
const payload3 = ["table_data", [{id: 1, name: "Alice"}, {id: 2, name: "Ton"}], "jsontable"];
const file_data = fs.readFileSync('./test/large_image.png');  // Read image from file
const payload4 = ["user_avatar", file_data, "binary"];

const payloads = [payload1, payload2, payload3, payload4];  // Array of arrays

// Step 1: Create the message envelope (transport-agnostic)
const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack("test.topic", payloads, {
    broker_url: "nats.yiem.cc",
    fileserver_url: "http://192.168.88.104:8080"
});

// Step 2: Send via your chosen transport (NATS, WebSocket, HTTP, etc.)
// await myTransport.publish("test.topic", envelopeJsonStr);

// OR using fetch API for HTTP transport
fetch("http://localhost:3000/api/messages", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: envelopeJsonStr
});

Receive Messages

import msghandlerCSR from './src/msghandler-csr.js';

const { smartunpack } = msghandlerCSR;

// Option 1: Direct JSON string
const envelope = await msghandlerCSR.smartunpack(jsonString, {
    fileserver_download_handler: msghandlerCSR.fetchWithBackoff,
    max_retries: 5,
    base_delay: 100,
    max_delay: 5000
});

// Option 2: From transport message object (e.g., NATS, WebSocket)
const envelope = await msghandlerCSR.smartunpack(natsMessage, {
    fileserver_download_handler: msghandlerCSR.fetchWithBackoff
});

// Process payloads
for (const [dataname, data, type] of envelope.payloads) {
    console.log(`${dataname}:`, data, `(type: ${type})`);
}

Prerequisites

Transport Channel - Choose your communication channel:
- NATS (recommended): docker run -p 4222:4222 nats:latest
- HTTP/WebSocket: Any HTTP server or WebSocket endpoint
- Custom: Implement your own transport mechanism

HTTP File Server (optional, for payloads ≥ 500KB) - Install and run:

# Using Plik
docker run -p 8080:8080 -v /tmp/fileserver:/var/lib/plik -e PLIK_ADMIN_PASSWORD=admin plik/plik

# OR using Python HTTP server
mkdir -p /tmp/fileserver
python3 -m http.server 8080 --directory /tmp/fileserver

Send Your First Message

# Julia
using msghandler, NATS

# Data format: [(dataname, data, type), ...]
payload_1 = ("test_message", "Hello World", "text")
file_path_large_image = "./test/large_image.png"
file_data_large_image = read(file_path_large_image)
filename_large_image = basename(file_path_large_image)
payload_2 = (filename_large_image, file_data_large_image, "binary")

payloads = [payload_1, payload_2]  # List of tuples

# Step 1: Create the message envelope (transport-agnostic)
envelope, envelope_json_str = msghandler.smartpack("test.topic", 
                                                  payloads; 
                                                  broker_url="nats.yiem.cc",
                                                  fileserver_url="http://192.168.88.104:8080")

# Step 2: Send via your chosen transport (NATS in this example)
conn = NATS.connect("nats.yiem.cc")
NATS.publish(conn, "test.topic", envelope_json_str; reply_to="test.replytopic")
NATS.drain(conn)

# OR using request-reply pattern
reply = NATS.request(conn, "test.topic", envelope_json_str, timeout=10)

Receive Your First Message

# Julia
using msghandler, NATS

conn = NATS.connect("nats.yiem.cc")
NATS.subscribe(conn, "test.topic") do msg
    println("Received message on $(msg.subject)")
    envelope_json_str = String(msg.payload)
    
    # Step 2: Unpack the envelope (transport-agnostic)
    envelope = msghandler.smartunpack(
        envelope_json_str;
        max_retries = 5,
        base_delay = 100,
        max_delay = 5000
    )
    println(envelope.payloads[1])
end

API Reference

smartpack

Sends data via your chosen transport mechanism with intelligent transport selection.

using msghandler

env, env_json_str = msghandler.smartpack(
    subject::String,
    data::AbstractArray{Tuple{String, Any, String}};
    broker_url::String = "nats://localhost:4222",
    fileserver_url = "http://localhost:8080",
    fileserver_upload_handler::Function = plik_oneshot_upload,
    size_threshold::Int = 500_000,
    correlation_id::String = string(uuid4()),
    msg_purpose::String = "chat",
    sender_name::String = "msghandler",
    receiver_name::String = "",
    receiver_id::String = "",
    reply_to::String = "",
    reply_to_msg_id::String = "",
    msg_id::String = string(uuid4()),
    sender_id::String = string(uuid4())
)
# Returns: ::Tuple{msg_envelope_v1, String}

smartunpack

Receives and processes messages from your transport.

using msghandler

env = msghandler.smartunpack(
    msg_json_str::String;
    fileserver_download_handler::Function = _fetch_with_backoff,
    max_retries::Int = 5,
    base_delay::Int = 100,
    max_delay::Int = 5000
)
# Returns: ::JSON.Object{String, Any}

Examples

Example 1: Chat with Mixed Content

Send text, image, and large file in one message.

using msghandler

data = [
    ("message_text", "Hello!", "text"),
    ("user_avatar", image_data, "image"),
    ("large_document", large_file_data, "binary")
]

env, env_json_str = smartpack("/chat/room1", data; fileserver_url="http://localhost:8080")

Example 2: Dictionary Exchange

Send configuration data.

using msghandler

config = Dict(
    "wifi_ssid" => "MyNetwork",
    "wifi_password" => "password123",
    "update_interval" => 60
)

data = [("config", config, "dictionary")]
env, env_json_str = smartpack("/device/config", data)

Example 3: Table Data (Arrow IPC - Julia Only)

Send tabular data using Apache Arrow IPC format for efficient binary serialization:

using msghandler
using DataFrames

df = DataFrame(
    id = [1, 2, 3],
    name = ["Alice", "Bob", "Charlie"],
    score = [95, 88, 92]
)

data = [("students", df, "arrowtable")]
env, env_json_str = smartpack("/data/analysis", data)

Example 3b: Table Data (JSON - Julia Compatible)

For cross-platform compatibility or when Arrow IPC is not needed, use jsontable:

using msghandler
using DataFrames

df = DataFrame(
    id = [1, 2, 3],
    name = ["Alice", "Bob", "Charlie"],
    score = [95, 88, 92]
)

data = [("students", df, "jsontable")]
env, env_json_str = smartpack("/data/analysis", data)

Example 3c: Table Data (JSON - JavaScript Compatible)

For JavaScript frontend, use jsontable instead of arrowtable:

const tableData = [
    { id: 1, name: "Alice", score: 95 },
    { id: 2, name: "Bob", score: 88 },
    { id: 3, name: "Charlie", score: 92 }
];

const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack("/data/analysis", [
    ["students", tableData, "jsontable"]
]);

Example 4: Image Transmission (Julia)

Send image data directly in messages:

using msghandler

# Read image file
image_path = "./test/large_image.png"
image_data = read(image_path)

data = [("user_avatar", image_data, "binary")]
env, env_json_str = smartpack("/chat/room1", data; fileserver_url="http://localhost:8080")

Example 5: Image Transmission (JavaScript)

Send image data directly in messages:

import fs from 'fs';

const file_data = fs.readFileSync('./test/large_image.png');

const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack("/chat/room1", [
    ["user_avatar", file_data, "binary"]
]);

Example 6: Table Data (JavaScript)

For JavaScript frontend, use jsontable for tabular data:

const tableData = [
    { id: 1, name: "Alice", score: 95 },
    { id: 2, name: "Bob", score: 88 },
    { id: 3, name: "Charlie", score: 92 }
];

const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack("/data/analysis", [
    ["students", tableData, "jsontable"]
]);

Example 7: Request-Response Pattern

Bi-directional communication with reply-to support.

using msghandler, NATS

# Requester
env, env_json_str = msghandler.smartpack(
    "/device/command",
    [("command", Dict("action" => "read_sensor"), "dictionary")];
    broker_url="nats://localhost:4222",
    reply_to="/device/response"
)

conn = NATS.connect("nats://localhost:4222")
NATS.publish(conn, "/device/command", env_json_str)
NATS.drain(conn)

# Receiver (in separate application)
conn = NATS.connect("nats://localhost:4222")
NATS.subscribe(conn, "/device/command") do msg
    env = msghandler.smartunpack(msg)
    println("Received command: ", env["payloads"])
    
    result = Dict("value" => 42)
    response_env, response_json = msghandler.smartpack(
        "/device/response",
        [("result", result, "dictionary")],
        reply_to="/device/command",
        reply_to_msg_id=env["msg_id"]
    )
    
    NATS.publish(conn, "/device/response", response_json)
    NATS.drain(conn)
end

Testing

Julia Backend

# Text message exchange
julia test/test_julia_text_sender.jl
julia test/test_julia_text_receiver.jl

# Dictionary exchange
julia test/test_julia_dict_sender.jl
julia test/test_julia_dict_receiver.jl

# File transfer
julia test/test_julia_file_sender.jl
julia test/test_julia_file_receiver.jl

# Mixed payload types
julia test/test_julia_mix_payloads_sender.jl
julia test/test_julia_mix_payloads_receiver.jl

# Table exchange
julia test/test_julia_table_sender.jl
julia test/test_julia_table_receiver.jl

JavaScript Frontend

To test the JavaScript version in a browser:

Start a file server for testing:

mkdir -p /tmp/fileserver
python3 -m http.server 8080 --directory /tmp/fileserver

Start a NATS server (optional):

docker run -p 4222:4222 nats:latest

Open the test HTML file in a browser:

# Create test file (example)
cat > test-browser.html << 'EOF'
<!DOCTYPE html>
<html>
<head><title>msghandler CSR Test</title></head>
<body>
<script type="module">
import msghandlerCSR from './src/msghandler-csr.js';

// Test smartpack
const [env, json] = await msghandlerCSR.smartpack("/test", [
    ["msg", "Hello Browser!", "text"]
]);
console.log("Sent:", json);
</script>
</body>
</html>
EOF

python3 -m http.server 8081
# Open http://localhost:8081/test-browser.html

Documentation

For detailed architecture and implementation information, see:

docs/architecture.md - Architecture and design patterns
docs/requirements.md - Business requirements and user stories
docs/spec.md - Technical specification and contracts
docs/walkthrough.md - Real-world application building guides

JavaScript API Reference

The JavaScript version provides the same core functionality as Julia, with browser-compatible APIs:

smartpack

const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack(
    subject,
    data,  // [[dataname, data, type], ...]
    options
);

smartunpack

const envelope = await msghandlerCSR.smartunpack(msg, options);
// envelope.payloads = [[dataname, data, type], ...]

plikOneshotUpload

const result = await msghandlerCSR.plikOneshotUpload(
    fileServerUrl,
    dataname,
    data  // Uint8Array
);
// result = { status, uploadid, fileid, url }

fetchWithBackoff

const data = await msghandlerCSR.fetchWithBackoff(
    url,
    maxRetries,
    baseDelay,
    maxDelay,
    correlationId
);
// data = Uint8Array

License

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

20 KiB Raw Blame History

msghandler - Cross-Platform Communication Layer

Table of Contents

Quick Data Format Reference

Input Format for smartpack()

Supported Payload Types

Examples

Overview

Use Cases

Platform Differences

Supported Payload Types

Key Design Principles

Features

Installation

Julia Backend

JavaScript Frontend

Quick Start

Julia Backend

Installation

Quick Start

Receive Messages

JavaScript CSR Webapp

Installation

Features

Quick Start

Receive Messages

Prerequisites

Send Your First Message

Receive Your First Message

API Reference

smartpack

smartunpack

Examples

Example 1: Chat with Mixed Content

Example 2: Dictionary Exchange

Example 3: Table Data (Arrow IPC - Julia Only)

Example 3b: Table Data (JSON - Julia Compatible)

Example 3c: Table Data (JSON - JavaScript Compatible)

Example 4: Image Transmission (Julia)

Example 5: Image Transmission (JavaScript)

Example 6: Table Data (JavaScript)

Example 7: Request-Response Pattern

Testing

Julia Backend

JavaScript Frontend

Documentation

JavaScript API Reference

smartpack

smartunpack

plikOneshotUpload

fetchWithBackoff

License

20 KiB

Raw Blame History

Input Format for `smartpack()`