msghandler - Cross-Platform Communication Layer

A high-performance, transport-agnostic communication layer for both Julia (backend) and JavaScript (CSR webapp) applications. Implements the Claim-Check pattern for efficient payload transport (direct for small payloads, URL-based for large payloads).

• 🟢 Julia Backend: Full-featured implementation with Arrow IPC support
• 🔵 JavaScript Frontend: Browser-compatible implementation for CSR webapps

---

Table of Contents

• Quick Start
  • Julia Backend
  • JavaScript CSR Webapp
• Overview
• Features
• Installation
• API Reference
• Examples
• Testing
• Documentation
• License

---

Quick Data Format Reference

Input Format for smartpack()

Format: [(dataname, data, type), ...]

Element	Type	Description
dataname	String	Name for the payload (e.g., "message", "config.json")
data	Any	The actual data (type depends on the type parameter below)
type	String	Payload type identifier

Supported Payload Types

Type	Julia Data	Description
"text"	String	Plain text
"dictionary"	Dict	JSON object
"arrowtable"	DataFrame, Arrow.Table	Arrow IPC table
"jsontable"	DataFrame, Vector{Dict}, Vector{NamedTuple}	JSON table
"image"	Vector{UInt8}	Image data
"audio"	Vector{UInt8}	Audio data
"video"	Vector{UInt8}	Video data
"binary"	Vector{UInt8}, IOBuffer	Binary data

Examples

Sending multiple payloads:

data = [
    ("message", "Hello!", "text"),
    ("config", Dict("key" => "value"), "dictionary"),
    ("file", file_bytes, "binary")
]
env, json_str = msghandler.smartpack("subject", data, options...)

Important Notes:

• Always wrap payloads in a list [], even for single payloads
• Type must be one of the supported types above
• Large payloads (≥500KB) automatically use link transport

---

Overview

msghandler provides a transport-agnostic communication layer with intelligent payload transport selection (same API for Julia and JavaScript):

Transport	Payload Size	Method
Direct	< 500KB	Sent directly via chosen transport (Base64 encoded)
Link	≥ 500KB	Uploaded to HTTP file server, URL sent via chosen transport

Use Cases

• Chat Applications: Text, images, audio, video in a single message
• File Transfer: Efficient transfer of large files using claim-check pattern
• Cross-Platform Communication: Interoperability between Julia backend and JavaScript frontend
• Any Transport: Works with NATS, HTTP, WebSockets, WebRTC, or any custom transport mechanism

Platform Differences

Feature	Julia Backend	JavaScript Frontend
Arrow IPC	✅ Supported	❌ Not supported (use "jsontable" instead)
Full Type Support	✅ All 8 types	✅ 7 of 8 types
Transport Support	✅ NATS, HTTP, WebSockets, etc.	✅ Fetch, WebSockets, HTTP
File Server	✅ Plik, AWS S3, custom	✅ Plik, custom HTTP server

Supported Payload Types

Type	Julia	JavaScript	Description
"text"	✅	✅	Plain text
"dictionary"	✅	✅	JSON object
"arrowtable"	✅	❌	Arrow IPC table (Julia only)
"jsontable"	✅	✅	JSON table
"image"	✅	✅	Image data
"audio"	✅	✅	Audio data
"video"	✅	✅	Video data
"binary"	✅	✅	Binary data

Key Design Principles

• Transport Agnostic: Core API (smartpack/smartunpack) is decoupled from transport
• Claim-Check Pattern: Efficient handling of large payloads via URL references
• Rich Type System: Support for Arrow IPC, JSON tables, and binary data
• Handler Abstraction: Pluggable file server implementations (Plik, AWS S3, custom)

---

Features

• ✅ Cross-platform - Same API for Julia backend and JavaScript frontend
• ✅ Transport-agnostic - Core API works with any communication channel (NATS, HTTP, WebSockets, etc.)
• ✅ Bi-directional - Request-reply patterns with reply_to support
• ✅ Multi-payload - Send multiple payloads of different types in one message
• ✅ Automatic transport selection - Direct vs link based on payload size (threshold: 500KB)
• ✅ Claim-Check pattern - Efficient large payload handling via URL references
• ✅ Rich payload types - Text, dictionary, Arrow IPC (Julia only), JSON table, image, audio, video, binary
• ✅ Exponential backoff - Reliable file server downloads with retry logic
• ✅ Correlation ID - End-to-end message tracing
• ✅ Handler abstraction - Pluggable file server implementations (Plik, AWS S3, custom)

---

Installation

Julia Backend

using Pkg
Pkg.add("msghandler")

Add to your Project.toml:

[deps]
msghandler = "path-to-msghandler"

JavaScript Frontend

Using NPM:

npm install msghandler-csr

Using local source (development):

cp ./src/msghandler-csr.js ./public/js/msghandler.js

Then import in your browser:

import msghandlerCSR from './js/msghandler.js';

---

Quick Start

Julia Backend

Installation

using Pkg
Pkg.add("msghandler")

Add to your Project.toml:

[deps]
msghandler = "path-to-msghandler"

Quick Start

using msghandler

# Data format: [(dataname, data, type), ...]
payload_1 = ("test_message", "Hello World", "text")
file_path_large_image = "./test/large_image.png"
file_data_large_image = read(file_path_large_image)
filename_large_image = basename(file_path_large_image)
payload_2 = (filename_large_image, file_data_large_image, "binary")

payloads = [payload_1, payload_2]  # List of tuples

# Step 1: Create the message envelope (transport-agnostic)
envelope, envelope_json_str = msghandler.smartpack("test.topic", 
                                                  payloads; 
                                                  broker_url="nats.yiem.cc",
                                                  fileserver_url="http://192.168.88.104:8080")

# Step 2: Send via your chosen transport (NATS in this example)
using NATS
conn = NATS.connect("nats.yiem.cc")
NATS.publish(conn, "test.topic", envelope_json_str; reply_to="test.replytopic")
NATS.drain(conn)

# OR using request-reply pattern
reply = NATS.request(conn, "test.topic", envelope_json_str, timeout=10)

Receive Messages

using msghandler, NATS

conn = NATS.connect("nats.yiem.cc")
NATS.subscribe(conn, "test.topic") do msg
    println("Received message on $(msg.subject)")
    envelope_json_str = String(msg.payload)
    
    # Step 2: Unpack the envelope (transport-agnostic)
    envelope = msghandler.smartunpack(
        envelope_json_str;
        max_retries = 5,
        base_delay = 100,
        max_delay = 5000
    )
    println(envelope.payloads[1])
end

JavaScript CSR Webapp

Installation

Using NPM:

npm install msghandler-csr

Using ES Modules (browser):

import msghandlerCSR from './src/msghandler-csr.js';

Features

• ✅ No build tools required (works directly in browser)
• ✅ Uses native browser APIs (Web Crypto, Fetch, TextEncoder/Decoder)
• ✅ Browser-compatible payload types (no Arrow IPC dependency)

Quick Start

import msghandlerCSR from './src/msghandler-csr.js';

const { smartpack, smartunpack, plikOneshotUpload, fetchWithBackoff } = msghandlerCSR;

// Data format: [[dataname, data, type], ...]
const payload1 = ["test_message", "Hello World", "text"];
const payload2 = ["config_data", { key: "value" }, "dictionary"];
const payload3 = ["table_data", [{id: 1, name: "Alice"}], "jsontable"];
const image_bytes = new Uint8Array([1, 2, 3, ...]); // Image data
const payload4 = ["user_avatar", image_bytes, "image"];

const payloads = [payload1, payload2, payload3, payload4];  // Array of arrays

// Step 1: Create the message envelope (transport-agnostic)
const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack("test.topic", payloads, {
    broker_url: "nats.yiem.cc",
    fileserver_url: "http://192.168.88.104:8080"
});

// Step 2: Send via your chosen transport (NATS, WebSocket, HTTP, etc.)
// await myTransport.publish("test.topic", envelopeJsonStr);

// OR using fetch API for HTTP transport
fetch("http://localhost:3000/api/messages", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: envelopeJsonStr
});

Receive Messages

import msghandlerCSR from './src/msghandler-csr.js';

const { smartunpack } = msghandlerCSR;

// Option 1: Direct JSON string
const envelope = await msghandlerCSR.smartunpack(jsonString, {
    fileserver_download_handler: msghandlerCSR.fetchWithBackoff,
    max_retries: 5,
    base_delay: 100,
    max_delay: 5000
});

// Option 2: From transport message object (e.g., NATS, WebSocket)
const envelope = await msghandlerCSR.smartunpack(natsMessage, {
    fileserver_download_handler: msghandlerCSR.fetchWithBackoff
});

// Process payloads
for (const [dataname, data, type] of envelope.payloads) {
    console.log(`${dataname}:`, data, `(type: ${type})`);
}

Prerequisites

1. Transport Channel - Choose your communication channel:

   • NATS (recommended): docker run -p 4222:4222 nats:latest
   • HTTP/WebSocket: Any HTTP server or WebSocket endpoint
   • Custom: Implement your own transport mechanism

2. HTTP File Server (optional, for payloads ≥ 500KB) - Install and run:

   # Using Plik
   docker run -p 8080:8080 -v /tmp/fileserver:/var/lib/plik -e PLIK_ADMIN_PASSWORD=admin plik/plik
   
   # OR using Python HTTP server
   mkdir -p /tmp/fileserver
   python3 -m http.server 8080 --directory /tmp/fileserver
   

Send Your First Message

using msghandler, NATS

# Data format: [(dataname, data, type), ...]
payload_1 = ("test_message", "Hello World", "text")
file_path_large_image = "./test/large_image.png"
file_data_large_image = read(file_path_large_image)
filename_large_image = basename(file_path_large_image)
payload_2 = (filename_large_image, file_data_large_image, "binary")

payloads = [payload_1, payload_2]  # List of tuples

# Step 1: Create the message envelope (transport-agnostic)
envelope, envelope_json_str = msghandler.smartpack("test.topic", 
                                                  payloads; 
                                                  broker_url="nats.yiem.cc",
                                                  fileserver_url="http://192.168.88.104:8080")

# Step 2: Send via your chosen transport (NATS in this example)
conn = NATS.connect("nats.yiem.cc")
NATS.publish(conn, "test.topic", envelope_json_str; reply_to="test.replytopic")
NATS.drain(conn)

# OR using request-reply pattern
reply = NATS.request(conn, "test.topic", envelope_json_str, timeout=10)

Receive Your First Message

using msghandler, NATS

conn = NATS.connect("nats.yiem.cc")
NATS.subscribe(conn, "test.topic") do msg
    println("Received message on $(msg.subject)")
    envelope_json_str = String(msg.payload)
    
    # Step 2: Unpack the envelope (transport-agnostic)
    envelope = msghandler.smartunpack(
        envelope_json_str;
        max_retries = 5,
        base_delay = 100,
        max_delay = 5000
    )
    println(envelope.payloads[1])
end

---

API Reference

smartpack

Sends data via your chosen transport mechanism with intelligent transport selection.

using msghandler

env, env_json_str = msghandler.smartpack(
    subject::String,
    data::AbstractArray{Tuple{String, Any, String}};
    broker_url::String = "nats://localhost:4222",
    fileserver_url = "http://localhost:8080",
    fileserver_upload_handler::Function = plik_oneshot_upload,
    size_threshold::Int = 500_000,
    correlation_id::String = string(uuid4()),
    msg_purpose::String = "chat",
    sender_name::String = "msghandler",
    receiver_name::String = "",
    receiver_id::String = "",
    reply_to::String = "",
    reply_to_msg_id::String = "",
    msg_id::String = string(uuid4()),
    sender_id::String = string(uuid4())
)
# Returns: ::Tuple{msg_envelope_v1, String}

smartunpack

Receives and processes messages from your transport.

using msghandler

env = msghandler.smartunpack(
    msg_json_str::String;
    fileserver_download_handler::Function = _fetch_with_backoff,
    max_retries::Int = 5,
    base_delay::Int = 100,
    max_delay::Int = 5000
)
# Returns: ::JSON.Object{String, Any}

---

Examples

Example 1: Chat with Mixed Content

Send text, image, and large file in one message.

using msghandler

data = [
    ("message_text", "Hello!", "text"),
    ("user_avatar", image_data, "image"),
    ("large_document", large_file_data, "binary")
]

env, env_json_str = smartpack("/chat/room1", data; fileserver_url="http://localhost:8080")

Example 2: Dictionary Exchange

Send configuration data.

using msghandler

config = Dict(
    "wifi_ssid" => "MyNetwork",
    "wifi_password" => "password123",
    "update_interval" => 60
)

data = [("config", config, "dictionary")]
env, env_json_str = smartpack("/device/config", data)

Example 3: Table Data (Arrow IPC - Julia Only)

Send tabular data using Apache Arrow IPC format for efficient binary serialization:

using msghandler
using DataFrames

df = DataFrame(
    id = [1, 2, 3],
    name = ["Alice", "Bob", "Charlie"],
    score = [95, 88, 92]
)

data = [("students", df, "arrowtable")]
env, env_json_str = smartpack("/data/analysis", data)

Example 3b: Table Data (JSON - Julia Compatible)

For cross-platform compatibility or when Arrow IPC is not needed, use jsontable:

using msghandler
using DataFrames

df = DataFrame(
    id = [1, 2, 3],
    name = ["Alice", "Bob", "Charlie"],
    score = [95, 88, 92]
)

data = [("students", df, "jsontable")]
env, env_json_str = smartpack("/data/analysis", data)

Example 3c: Table Data (JSON - JavaScript Compatible)

For JavaScript frontend, use jsontable instead of arrowtable:

const tableData = [
    { id: 1, name: "Alice", score: 95 },
    { id: 2, name: "Bob", score: 88 },
    { id: 3, name: "Charlie", score: 92 }
];

const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack("/data/analysis", [
    ["students", tableData, "jsontable"]
]);

Example 4: Image Transmission (Julia)

Send image data directly in messages:

using msghandler

# Read image file
image_path = "./path/to/image.jpg"
image_data = read(image_path)

data = [("user_avatar", image_data, "image")]
env, env_json_str = smartpack("/chat/room1", data; fileserver_url="http://localhost:8080")

Example 5: Image Transmission (JavaScript)

Send image data directly in messages:

const image_data = new Uint8Array([1, 2, 3, ...]); // Image bytes

const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack("/chat/room1", [
    ["user_avatar", image_data, "image"]
]);

Example 6: Request-Response Pattern

Bi-directional communication with reply-to support.

using msghandler, NATS

# Requester
env, env_json_str = msghandler.smartpack(
    "/device/command",
    [("command", Dict("action" => "read_sensor"), "dictionary")];
    broker_url="nats://localhost:4222",
    reply_to="/device/response"
)

conn = NATS.connect("nats://localhost:4222")
NATS.publish(conn, "/device/command", env_json_str)
NATS.drain(conn)

# Receiver (in separate application)
conn = NATS.connect("nats://localhost:4222")
NATS.subscribe(conn, "/device/command") do msg
    env = msghandler.smartunpack(msg)
    println("Received command: ", env["payloads"])
    
    result = Dict("value" => 42)
    response_env, response_json = msghandler.smartpack(
        "/device/response",
        [("result", result, "dictionary")],
        reply_to="/device/command",
        reply_to_msg_id=env["msg_id"]
    )
    
    NATS.publish(conn, "/device/response", response_json)
    NATS.drain(conn)
end

---

Testing

Julia Backend

# Text message exchange
julia test/test_julia_text_sender.jl
julia test/test_julia_text_receiver.jl

# Dictionary exchange
julia test/test_julia_dict_sender.jl
julia test/test_julia_dict_receiver.jl

# File transfer
julia test/test_julia_file_sender.jl
julia test/test_julia_file_receiver.jl

# Mixed payload types
julia test/test_julia_mix_payloads_sender.jl
julia test/test_julia_mix_payloads_receiver.jl

# Table exchange
julia test/test_julia_table_sender.jl
julia test/test_julia_table_receiver.jl

JavaScript Frontend

To test the JavaScript version in a browser:

1. Start a file server for testing:

mkdir -p /tmp/fileserver
python3 -m http.server 8080 --directory /tmp/fileserver

2. Start a NATS server (optional):

docker run -p 4222:4222 nats:latest

3. Open the test HTML file in a browser:

# Create test file (example)
cat > test-browser.html << 'EOF'
<!DOCTYPE html>
<html>
<head><title>msghandler CSR Test</title></head>
<body>
<script type="module">
import msghandlerCSR from './src/msghandler-csr.js';

// Test smartpack
const [env, json] = await msghandlerCSR.smartpack("/test", [
    ["msg", "Hello Browser!", "text"]
]);
console.log("Sent:", json);
</script>
</body>
</html>
EOF

python3 -m http.server 8081
# Open http://localhost:8081/test-browser.html

---

Documentation

For detailed architecture and implementation information, see:

• docs/architecture.md - Architecture and design patterns
• docs/requirements.md - Business requirements and user stories
• docs/spec.md - Technical specification and contracts
• docs/walkthrough.md - Real-world application building guides

JavaScript API Reference

The JavaScript version provides the same core functionality as Julia, with browser-compatible APIs:

smartpack

const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack(
    subject,
    data,  // [[dataname, data, type], ...]
    options
);

smartunpack

const envelope = await msghandlerCSR.smartunpack(msg, options);
// envelope.payloads = [[dataname, data, type], ...]

plikOneshotUpload

const result = await msghandlerCSR.plikOneshotUpload(
    fileServerUrl,
    dataname,
    data  // Uint8Array
);
// result = { status, uploadid, fileid, url }

fetchWithBackoff

const data = await msghandlerCSR.fetchWithBackoff(
    url,
    maxRetries,
    baseDelay,
    maxDelay,
    correlationId
);
// data = Uint8Array

---

License

MIT License

Copyright (c) 2026 msghandler Contributors

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.