msghandler/README.md

# msghandler - Cross-Platform Communication Layer

A high-performance, **transport-agnostic** communication layer for both **Julia** (backend) and **JavaScript** (CSR webapp) applications. Implements the Claim-Check pattern for efficient payload transport (direct for small payloads, URL-based for large payloads).

- 🟢 **Julia Backend**: Full-featured implementation with Arrow IPC support
- 🔵 **JavaScript Frontend**: Browser-compatible implementation for CSR webapps

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)

---

## Table of Contents

- [Quick Start](#quick-start)
  - [Julia Backend](#julia-backend)
  - [JavaScript CSR Webapp](#javascript-csr-webapp)
- [Overview](#overview)
- [Features](#features)
- [Installation](#installation)
- [API Reference](#api-reference)
- [Examples](#examples)
- [Testing](#testing)
- [Documentation](#documentation)
- [License](#license)

---

## Quick Data Format Reference

### Input Format for `smartpack()`

**Format:** `[(dataname, data, type), ...]`

| Element | Type | Description |
|---------|------|-------------|
| `dataname` | `String` | Name for the payload (e.g., `"message"`, `"config.json"`) |
| `data` | `Any` | The actual data (type depends on the `type` parameter below) |
| `type` | `String` | Payload type identifier |

### Supported Payload Types

| Type | Julia Data | Description |
|------|------------|-------------|
| `"text"` | `String` | Plain text |
| `"dictionary"` | `Dict` | JSON object |
| `"arrowtable"` | `DataFrame`, `Arrow.Table` | Arrow IPC table |
| `"jsontable"` | `DataFrame`, `Vector{Dict}`, `Vector{NamedTuple}` | JSON table |
| `"image"` | `Vector{UInt8}` | Image data |
| `"audio"` | `Vector{UInt8}` | Audio data |
| `"video"` | `Vector{UInt8}` | Video data |
| `"binary"` | `Vector{UInt8}`, `IOBuffer` | Binary data |

### Examples

**Sending multiple payloads:**
```julia
data = [
    ("message", "Hello!", "text"),
    ("config", Dict("key" => "value"), "dictionary"),
    ("file", file_bytes, "binary")
]
env, json_str = msghandler.smartpack("subject", data, options...)
```

**Important Notes:**
- Always wrap payloads in a list `[]`, even for single payloads
- Type must be one of the supported types above
- Large payloads (≥500KB) automatically use link transport

---

## Overview

msghandler provides a **transport-agnostic** communication layer with intelligent payload transport selection (same API for Julia and JavaScript):

| Transport | Payload Size | Method |
|-----------|--------------|--------|
| **Direct** | < 500KB | Sent directly via chosen transport (Base64 encoded) |
| **Link** | ≥ 500KB | Uploaded to HTTP file server, URL sent via chosen transport |

### Use Cases

- **Chat Applications**: Text, images, audio, video in a single message
- **File Transfer**: Efficient transfer of large files using claim-check pattern
- **Cross-Platform Communication**: Interoperability between Julia backend and JavaScript frontend
- **Any Transport**: Works with NATS, HTTP, WebSockets, WebRTC, or any custom transport mechanism

### Platform Differences

| Feature | Julia Backend | JavaScript Frontend |
|---------|---------------|---------------------|
| **Arrow IPC** | ✅ Supported | ❌ Not supported (use "jsontable" instead) |
| **Full Type Support** | ✅ All 8 types | ✅ 7 of 8 types |
| **Transport Support** | ✅ NATS, HTTP, WebSockets, etc. | ✅ Fetch, WebSockets, HTTP |
| **File Server** | ✅ Plik, AWS S3, custom | ✅ Plik, custom HTTP server |

### Supported Payload Types

| Type | Julia | JavaScript | Description |
|------|-------|------------|-------------|
| `"text"` | ✅ | ✅ | Plain text |
| `"dictionary"` | ✅ | ✅ | JSON object |
| `"arrowtable"` | ✅ | ❌ | Arrow IPC table (Julia only) |
| `"jsontable"` | ✅ | ✅ | JSON table |
| `"image"` | ✅ | ✅ | Image data |
| `"audio"` | ✅ | ✅ | Audio data |
| `"video"` | ✅ | ✅ | Video data |
| `"binary"` | ✅ | ✅ | Binary data |

### Key Design Principles

- **Transport Agnostic**: Core API (`smartpack`/`smartunpack`) is decoupled from transport
- **Claim-Check Pattern**: Efficient handling of large payloads via URL references
- **Rich Type System**: Support for Arrow IPC, JSON tables, and binary data
- **Handler Abstraction**: Pluggable file server implementations (Plik, AWS S3, custom)

---

## Features

- ✅ **Cross-platform** - Same API for Julia backend and JavaScript frontend
- ✅ **Transport-agnostic** - Core API works with any communication channel (NATS, HTTP, WebSockets, etc.)
- ✅ **Bi-directional** - Request-reply patterns with `reply_to` support
- ✅ **Multi-payload** - Send multiple payloads of different types in one message
- ✅ **Automatic transport selection** - Direct vs link based on payload size (threshold: 500KB)
- ✅ **Claim-Check pattern** - Efficient large payload handling via URL references
- ✅ **Rich payload types** - Text, dictionary, Arrow IPC (Julia only), JSON table, image, audio, video, binary
- ✅ **Exponential backoff** - Reliable file server downloads with retry logic
- ✅ **Correlation ID** - End-to-end message tracing
- ✅ **Handler abstraction** - Pluggable file server implementations (Plik, AWS S3, custom)

---

## Installation

### Julia Backend

```julia
using Pkg
Pkg.add("msghandler")
```

Add to your `Project.toml`:
```toml
[deps]
msghandler = "path-to-msghandler"
```

### JavaScript Frontend

**Using NPM:**
```bash
npm install msghandler-csr
```

**Using local source (development):**
```bash
cp ./src/msghandler-csr.js ./public/js/msghandler.js
```

Then import in your browser:
```javascript
import msghandlerCSR from './js/msghandler.js';
```

---

## Quick Start

### Julia Backend

#### Installation

```julia
using Pkg
Pkg.add("msghandler")
```

Add to your `Project.toml`:
```toml
[deps]
msghandler = "path-to-msghandler"
```

#### Quick Start

```julia
using msghandler

# Data format: [(dataname, data, type), ...]
payload_1 = ("test_message", "Hello World", "text")
file_path_large_image = "./test/large_image.png"
file_data_large_image = read(file_path_large_image)
filename_large_image = basename(file_path_large_image)
payload_2 = (filename_large_image, file_data_large_image, "binary")

payloads = [payload_1, payload_2]  # List of tuples

# Step 1: Create the message envelope (transport-agnostic)
envelope, envelope_json_str = msghandler.smartpack("test.topic", 
                                                  payloads; 
                                                  broker_url="nats.yiem.cc",
                                                  fileserver_url="http://192.168.88.104:8080")

# Step 2: Send via your chosen transport (NATS in this example)
using NATS
conn = NATS.connect("nats.yiem.cc")
NATS.publish(conn, "test.topic", envelope_json_str; reply_to="test.replytopic")
NATS.drain(conn)

# OR using request-reply pattern
reply = NATS.request(conn, "test.topic", envelope_json_str, timeout=10)
```

#### Receive Messages

```julia
using msghandler, NATS

conn = NATS.connect("nats.yiem.cc")
NATS.subscribe(conn, "test.topic") do msg
    println("Received message on $(msg.subject)")
    envelope_json_str = String(msg.payload)
    
    # Step 2: Unpack the envelope (transport-agnostic)
    envelope = msghandler.smartunpack(
        envelope_json_str;
        max_retries = 5,
        base_delay = 100,
        max_delay = 5000
    )
    println(envelope.payloads[1])
end
```

### JavaScript CSR Webapp

#### Installation

**Using NPM:**
```bash
npm install msghandler-csr
```

**Using ES Modules (browser):**
```javascript
import msghandlerCSR from './src/msghandler-csr.js';
```

#### Features

- ✅ No build tools required (works directly in browser)
- ✅ Uses native browser APIs (Web Crypto, Fetch, TextEncoder/Decoder)
- ✅ Browser-compatible payload types (no Arrow IPC dependency)

#### Quick Start

```javascript
import msghandlerCSR from './src/msghandler-csr.js';

const { smartpack, smartunpack, plikOneshotUpload, fetchWithBackoff } = msghandlerCSR;

// Data format: [[dataname, data, type], ...]
const payload1 = ["test_message", "Hello World", "text"];
const payload2 = ["config_data", { key: "value" }, "dictionary"];
const payload3 = ["table_data", [{id: 1, name: "Alice"}, {id: 2, name: "Ton"}], "jsontable"];
const file_data = fs.readFileSync('./test/large_image.png');  // Read image from file
const payload4 = ["user_avatar", file_data, "binary"];

const payloads = [payload1, payload2, payload3, payload4];  // Array of arrays

// Step 1: Create the message envelope (transport-agnostic)
const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack("test.topic", payloads, {
    broker_url: "nats.yiem.cc",
    fileserver_url: "http://192.168.88.104:8080"
});

// Step 2: Send via your chosen transport (NATS, WebSocket, HTTP, etc.)
// await myTransport.publish("test.topic", envelopeJsonStr);

// OR using fetch API for HTTP transport
fetch("http://localhost:3000/api/messages", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: envelopeJsonStr
});
```

#### Receive Messages

```javascript
import msghandlerCSR from './src/msghandler-csr.js';

const { smartunpack } = msghandlerCSR;

// Option 1: Direct JSON string
const envelope = await msghandlerCSR.smartunpack(jsonString, {
    fileserver_download_handler: msghandlerCSR.fetchWithBackoff,
    max_retries: 5,
    base_delay: 100,
    max_delay: 5000
});

// Option 2: From transport message object (e.g., NATS, WebSocket)
const envelope = await msghandlerCSR.smartunpack(natsMessage, {
    fileserver_download_handler: msghandlerCSR.fetchWithBackoff
});

// Process payloads
for (const [dataname, data, type] of envelope.payloads) {
    console.log(`${dataname}:`, data, `(type: ${type})`);
}
```

### Prerequisites

1. **Transport Channel** - Choose your communication channel:
   - **NATS** (recommended): `docker run -p 4222:4222 nats:latest`
   - **HTTP/WebSocket**: Any HTTP server or WebSocket endpoint
   - **Custom**: Implement your own transport mechanism

2. **HTTP File Server** (optional, for payloads ≥ 500KB) - Install and run:
    ```bash
    # Using Plik
    docker run -p 8080:8080 -v /tmp/fileserver:/var/lib/plik -e PLIK_ADMIN_PASSWORD=admin plik/plik
    
    # OR using Python HTTP server
    mkdir -p /tmp/fileserver
    python3 -m http.server 8080 --directory /tmp/fileserver
    ```
---

## API Reference

### smartpack

Sends data via your chosen transport mechanism with intelligent transport selection.

```julia
using msghandler

env, env_json_str = msghandler.smartpack(
    subject::String,
    data::AbstractArray{Tuple{String, Any, String}};
    broker_url::String = "nats://localhost:4222",
    fileserver_url = "http://localhost:8080",
    fileserver_upload_handler::Function = plik_oneshot_upload,
    size_threshold::Int = 500_000,
    correlation_id::String = string(uuid4()),
    msg_purpose::String = "chat",
    sender_name::String = "msghandler",
    receiver_name::String = "",
    receiver_id::String = "",
    reply_to::String = "",
    reply_to_msg_id::String = "",
    msg_id::String = string(uuid4()),
    sender_id::String = string(uuid4())
)
# Returns: ::Tuple{msg_envelope_v1, String}
```

### smartunpack

Receives and processes messages from your transport.

```julia
using msghandler

env = msghandler.smartunpack(
    msg_json_str::String;
    fileserver_download_handler::Function = _fetch_with_backoff,
    max_retries::Int = 5,
    base_delay::Int = 100,
    max_delay::Int = 5000
)
# Returns: ::JSON.Object{String, Any}
```

---

## Examples

### Example 1: Chat with Mixed Content

Send text, image, and large file in one message.

```julia
using msghandler

data = [
    ("message_text", "Hello!", "text"),
    ("user_avatar", image_data, "image"),
    ("large_document", large_file_data, "binary")
]

env, env_json_str = smartpack("/chat/room1", data; fileserver_url="http://localhost:8080")
```

### Example 2: Dictionary Exchange

Send configuration data.

```julia
using msghandler

config = Dict(
    "wifi_ssid" => "MyNetwork",
    "wifi_password" => "password123",
    "update_interval" => 60
)

data = [("config", config, "dictionary")]
env, env_json_str = smartpack("/device/config", data)
```

### Example 3: Table Data (Arrow IPC - Julia Only)

Send tabular data using Apache Arrow IPC format for efficient binary serialization:

```julia
using msghandler
using DataFrames

df = DataFrame(
    id = [1, 2, 3],
    name = ["Alice", "Bob", "Charlie"],
    score = [95, 88, 92]
)

data = [("students", df, "arrowtable")]
env, env_json_str = smartpack("/data/analysis", data)
```

### Example 3b: Table Data (JSON - Julia Compatible)

For cross-platform compatibility or when Arrow IPC is not needed, use `jsontable`:

```julia
using msghandler
using DataFrames

df = DataFrame(
    id = [1, 2, 3],
    name = ["Alice", "Bob", "Charlie"],
    score = [95, 88, 92]
)

data = [("students", df, "jsontable")]
env, env_json_str = smartpack("/data/analysis", data)
```

### Example 3c: Table Data (JSON - JavaScript Compatible)

For JavaScript frontend, use `jsontable` instead of `arrowtable`:

```javascript
const tableData = [
    { id: 1, name: "Alice", score: 95 },
    { id: 2, name: "Bob", score: 88 },
    { id: 3, name: "Charlie", score: 92 }
];

const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack("/data/analysis", [
    ["students", tableData, "jsontable"]
]);
```

### Example 4: Image Transmission (Julia)

Send image data directly in messages:

```julia
using msghandler

# Read image file
image_path = "./test/large_image.png"
image_data = read(image_path)

data = [("user_avatar", image_data, "binary")]
env, env_json_str = smartpack("/chat/room1", data; fileserver_url="http://localhost:8080")
```

### Example 5: Image Transmission (JavaScript)

Send image data directly in messages:

```javascript
import fs from 'fs';

const file_data = fs.readFileSync('./test/large_image.png');

const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack("/chat/room1", [
    ["user_avatar", file_data, "binary"]
]);
```

### Example 6: Table Data (JavaScript)

For JavaScript frontend, use `jsontable` for tabular data:

```javascript
const tableData = [
    { id: 1, name: "Alice", score: 95 },
    { id: 2, name: "Bob", score: 88 },
    { id: 3, name: "Charlie", score: 92 }
];

const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack("/data/analysis", [
    ["students", tableData, "jsontable"]
]);
```

### Example 7: Request-Response Pattern

Bi-directional communication with reply-to support.

```julia
using msghandler, NATS

# Requester
env, env_json_str = msghandler.smartpack(
    "/device/command",
    [("command", Dict("action" => "read_sensor"), "dictionary")];
    broker_url="nats://localhost:4222",
    reply_to="/device/response"
)

conn = NATS.connect("nats://localhost:4222")
NATS.publish(conn, "/device/command", env_json_str)
NATS.drain(conn)

# Receiver (in separate application)
conn = NATS.connect("nats://localhost:4222")
NATS.subscribe(conn, "/device/command") do msg
    env = msghandler.smartunpack(msg)
    println("Received command: ", env["payloads"])
    
    result = Dict("value" => 42)
    response_env, response_json = msghandler.smartpack(
        "/device/response",
        [("result", result, "dictionary")],
        reply_to="/device/command",
        reply_to_msg_id=env["msg_id"]
    )
    
    NATS.publish(conn, "/device/response", response_json)
    NATS.drain(conn)
end
```

---

## Testing

### Julia Backend

```bash
# Text message exchange
julia test/test_julia_text_sender.jl
julia test/test_julia_text_receiver.jl

# Dictionary exchange
julia test/test_julia_dict_sender.jl
julia test/test_julia_dict_receiver.jl

# File transfer
julia test/test_julia_file_sender.jl
julia test/test_julia_file_receiver.jl

# Mixed payload types
julia test/test_julia_mix_payloads_sender.jl
julia test/test_julia_mix_payloads_receiver.jl

# Table exchange
julia test/test_julia_table_sender.jl
julia test/test_julia_table_receiver.jl
```

### JavaScript Frontend

To test the JavaScript version in a browser:

1. Start a file server for testing:
```bash
mkdir -p /tmp/fileserver
python3 -m http.server 8080 --directory /tmp/fileserver
```

2. Start a NATS server (optional):
```bash
docker run -p 4222:4222 nats:latest
```

3. Open the test HTML file in a browser:
```bash
# Create test file (example)
cat > test-browser.html << 'EOF'
<!DOCTYPE html>
<html>
<head><title>msghandler CSR Test</title></head>
<body>
<script type="module">
import msghandlerCSR from './src/msghandler-csr.js';

// Test smartpack
const [env, json] = await msghandlerCSR.smartpack("/test", [
    ["msg", "Hello Browser!", "text"]
]);
console.log("Sent:", json);
</script>
</body>
</html>
EOF

python3 -m http.server 8081
# Open http://localhost:8081/test-browser.html
```

---

## Documentation

For detailed architecture and implementation information, see:

- [`docs/architecture.md`](docs/architecture.md) - Architecture and design patterns
- [`docs/requirements.md`](docs/requirements.md) - Business requirements and user stories
- [`docs/spec.md`](docs/spec.md) - Technical specification and contracts
- [`docs/walkthrough.md`](docs/walkthrough.md) - Real-world application building guides

### JavaScript API Reference

The JavaScript version provides the same core functionality as Julia, with browser-compatible APIs:

#### smartpack

```javascript
const [envelope, envelopeJsonStr] = await msghandlerCSR.smartpack(
    subject,
    data,  // [[dataname, data, type], ...]
    options
);
```

#### smartunpack

```javascript
const envelope = await msghandlerCSR.smartunpack(msg, options);
// envelope.payloads = [[dataname, data, type], ...]
```

#### plikOneshotUpload

```javascript
const result = await msghandlerCSR.plikOneshotUpload(
    fileServerUrl,
    dataname,
    data  // Uint8Array
);
// result = { status, uploadid, fileid, url }
```

#### fetchWithBackoff

```javascript
const data = await msghandlerCSR.fetchWithBackoff(
    url,
    maxRetries,
    baseDelay,
    maxDelay,
    correlationId
);
// data = Uint8Array
```

---

## License

MIT License

Copyright (c) 2026 msghandler Contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.