# NATSBridge

A high-performance, bi-directional data bridge for **Julia** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![NATS](https://img.shields.io/badge/NATS-Enabled-green.svg)](https://nats.io)

---

## Table of Contents

- [Overview](#overview)
- [Features](#features)
- [Architecture](#architecture)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [API Reference](#api-reference)
- [Payload Types](#payload-types)
- [Transport Strategies](#transport-strategies)
- [Examples](#examples)
- [Testing](#testing)
- [License](#license)

---

## Overview

NATSBridge enables seamless communication for Julia applications through NATS, with intelligent transport selection based on payload size:

| Transport | Payload Size | Method |
|-----------|--------------|--------|
| **Direct** | < 1MB | Sent directly via NATS (Base64 encoded) |
| **Link** | >= 1MB | Uploaded to HTTP file server, URL sent via NATS |

### Use Cases

- **Chat Applications**: Text, images, audio, video in a single message
- **File Transfer**: Efficient transfer of large files using claim-check pattern
- **Streaming Data**: Sensor data, telemetry, and analytics pipelines


---

## Features

- ✅ **Bi-directional messaging** for Julia applications
- ✅ **Multi-payload support** - send multiple payloads with different types in one message
- ✅ **Automatic transport selection** - direct vs link based on payload size
- ✅ **Claim-Check pattern** for payloads > 1MB
- ✅ **Apache Arrow IPC** support for tabular data (zero-copy reading)
- ✅ **Exponential backoff** for reliable file server downloads
- ✅ **Correlation ID tracking** for message tracing
- ✅ **Reply-to support** for request-response patterns
- ✅ **JetStream support** for message replay and durability


---

## Architecture

### System Components

```
┌─────────────────────────────────────────────────────────────────────┐
│                     NATSBridge Architecture                         │
├─────────────────────────────────────────────────────────────────────┤
│  ┌──────────────┐                    │                              │
│  │   Julia      │                    ▼                              │
│  │  (NATS.jl)   │    ┌─────────────────────────┐                    │
│  └──────────────┘    │    NATS                 │                    │
│                      │    (Message Broker)     │                    │
│                      └─────────────────────────┘                    │
│                              │                                      │
│                              ▼                                      │
│                    ┌──────────────────────┐                         │
│                    │  File Server         │                         │
│                    │  (HTTP Upload/Get)   │                         │
│                    └──────────────────────┘                         │
└─────────────────────────────────────────────────────────────────────┘
```

### Message Flow

1. **Sender** creates a message envelope with payloads
2. **NATSBridge** serializes and encodes payloads based on type
3. **Transport Decision**: Small payloads go directly to NATS, large payloads are uploaded to file server
4. **NATS** routes messages to subscribers
5. **Receiver** fetches payloads (from NATS or file server)
6. **NATSBridge** deserializes and decodes payloads

---

## Installation

### Prerequisites

- **NATS Server** (v2.10+ recommended)
- **HTTP File Server** (optional, for payloads > 1MB)

### Julia

```julia
using Pkg
Pkg.add("NATS")
Pkg.add("https://git.yiem.cc/ton/NATSBridge")
```

---

## Quick Start

### Step 1: Start NATS Server

```bash
docker run -p 4222:4222 nats:latest
```

### Step 2: Start HTTP File Server (Optional)

```bash
# Create a directory for file uploads
mkdir -p /tmp/fileserver

# Start HTTP file server
python3 -m http.server 8080 --directory /tmp/fileserver
```

### Step 3: Send Your First Message

#### Julia

```julia
using NATSBridge

# Send a text message
data = [("message", "Hello World", "text")]
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; broker_url="nats://localhost:4222")
println("Message sent!")
```

### Step 4: Receive Messages

#### Julia

```julia
using NATS, NATSBridge

# Configuration
const SUBJECT = "/chat/room1"
const NATS_URL = "nats://localhost:4222"

# Helper: Log with correlation ID
function log_trace(message)
    timestamp = Dates.now()
    println("[$timestamp] $message")
end

# Receiver: Listen for messages - msg comes from the callback
function test_receive()
    conn = NATS.connect(NATS_URL)
    NATS.subscribe(conn, SUBJECT) do msg
        log_trace("Received message on $(msg.subject)")
        
        # Receive and process message
        env, env_json_str = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
        for (dataname, data, type) in env["payloads"]
            println("Received $dataname: $data")
        end
    end
    
    # Keep listening for 120 seconds
    sleep(120)
    NATS.drain(conn)
end

test_receive()
```

---

## API Reference

### smartsend

Sends data either directly via NATS or via a fileserver URL, depending on payload size.

#### Julia

```julia
using NATSBridge

env, env_json_str = NATSBridge.smartsend(
    subject,                           # NATS subject
    data::AbstractArray{Tuple{String, Any, String}};  # List of (dataname, data, type)
    broker_url::String = "nats://localhost:4222",
    fileserver_url = "http://localhost:8080",
    fileserver_upload_handler::Function = plik_oneshot_upload,
    size_threshold::Int = 1_000_000,
    correlation_id::Union{String, Nothing} = nothing,
    msg_purpose::String = "chat",
    sender_name::String = "NATSBridge",
    receiver_name::String = "",
    receiver_id::String = "",
    reply_to::String = "",
    reply_to_msg_id::String = "",
    is_publish::Bool = true,          # Whether to automatically publish to NATS
    NATS_connection::Union{NATS.Connection, Nothing} = nothing  # Pre-existing NATS connection (optional, saves connection overhead)
)
# Returns: (msgEnvelope_v1, JSON string)
# - env: msgEnvelope_v1 object with all envelope metadata and payloads
# - env_json_str: JSON string representation of the envelope for publishing
```

### smartreceive

Receives and processes messages from NATS, handling both direct and link transport.

#### Julia

```julia
using NATSBridge

# Note: msg is a NATS.Msg object passed from the subscription callback
env = NATSBridge.smartreceive(
    msg::NATS.Msg;
    fileserver_download_handler::Function = _fetch_with_backoff,
    max_retries::Int = 5,
    base_delay::Int = 100,
    max_delay::Int = 5000
)
# Returns: Dict with envelope metadata and payloads array
```

### publish_message

Publish a message to a NATS subject. This function is available in Julia with two overloads:

#### Julia

**Using broker URL (creates new connection):**
```julia
using NATSBridge, NATS

# Publish with URL - creates a new connection
NATSBridge.publish_message(
    "nats://localhost:4222",  # broker_url
    "/chat/room1",            # subject
    "{\"correlation_id\":\"abc123\"}",  # message
    "abc123"                  # correlation_id
)
```

**Using pre-existing connection (saves connection overhead):**
```julia
using NATSBridge, NATS

# Create connection once and reuse
conn = NATS.connect("nats://localhost:4222")
NATSBridge.publish_message(conn, "/chat/room1", "{\"correlation_id\":\"abc123\"}", "abc123")
# Connection is automatically drained after publish
```

---

## Payload Types

| Type | Description | Serialization |
|------|-------------|---------------|
| `text` | Plain text strings | UTF-8 bytes |
| `dictionary` | JSON-serializable dictionaries | JSON |
| `table` | Tabular data (DataFrames, arrays) | Apache Arrow IPC |
| `image` | Image data (PNG, JPG) | Raw bytes |
| `audio` | Audio data (WAV, MP3) | Raw bytes |
| `video` | Video data (MP4, AVI) | Raw bytes |
| `binary` | Generic binary data | Raw bytes |

---

## Transport Strategies

### Direct Transport (Payloads < 1MB)

Small payloads are sent directly via NATS with Base64 encoding.

#### Julia
```julia
data = [("message", "Hello", "text")]
smartsend("/topic", data)
```

### Link Transport (Payloads >= 1MB)

Large payloads are uploaded to an HTTP file server.

#### Julia
```julia
data = [("file", large_data, "binary")]
smartsend("/topic", data; fileserver_url="http://localhost:8080")
```

---

## Examples

### Example 1: Chat with Mixed Content

Send text, small image, and large file in one message.

#### Julia
```julia
using NATSBridge

data = [
    ("message_text", "Hello!", "text"),
    ("user_avatar", image_data, "image"),
    ("large_document", large_file_data, "binary")
]

env, env_json_str = NATSBridge.smartsend("/chat/room1", data; fileserver_url="http://localhost:8080")
```

### Example 2: Dictionary Exchange

Send configuration data between platforms.

#### Julia
```julia
using NATSBridge

config = Dict(
    "wifi_ssid" => "MyNetwork",
    "wifi_password" => "password123",
    "update_interval" => 60
)

data = [("config", config, "dictionary")]
env, env_json_str = NATSBridge.smartsend("/device/config", data)
```

### Example 3: Table Data (Arrow IPC)

Send tabular data using Apache Arrow IPC format.

#### Julia
```julia
using NATSBridge
using DataFrames

df = DataFrame(
    id = [1, 2, 3],
    name = ["Alice", "Bob", "Charlie"],
    score = [95, 88, 92]
)

data = [("students", df, "table")]
env, env_json_str = NATSBridge.smartsend("/data/analysis", data)
```

### Example 4: Request-Response Pattern with Envelope JSON

Bi-directional communication with reply-to support. The `smartsend` function now returns both the envelope object and a JSON string that can be published directly.

#### Julia (Requester)
```julia
using NATSBridge

env, env_json_str = NATSBridge.smartsend(
    "/device/command",
    [("command", Dict("action" => "read_sensor"), "dictionary")];
    broker_url="nats://localhost:4222",
    reply_to="/device/response"
)
```

#### Julia (Responder)
```julia
using NATS, NATSBridge

# Configuration
const SUBJECT = "/device/command"
const NATS_URL = "nats://localhost:4222"

function test_responder()
    conn = NATS.connect(NATS_URL)
    NATS.subscribe(conn, SUBJECT) do msg
        env = NATSBridge.smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
        
        # Extract reply_to from the envelope metadata
        reply_to = env["reply_to"]
        
        for (dataname, data, type) in env["payloads"]
            if dataname == "command" && data["action"] == "read_sensor"
                response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
                # Send response to the reply_to subject from the request
                if !isempty(reply_to)
                    smartsend(reply_to, [("data", response, "dictionary")])
                end
            end
        end
    end
    
    sleep(120)
    NATS.drain(conn)
end

test_responder()
```

### Example 5: IoT Device Sensor Data

IoT device sending sensor data.

#### Julia (Receiver)
```julia
using NATS, NATSBridge

# Configuration
const SUBJECT = "/device/sensors"
const NATS_URL = "nats://localhost:4222"

function test_receiver()
    conn = NATS.connect(NATS_URL)
    NATS.subscribe(conn, SUBJECT) do msg
        env, env_json_str = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
        for (dataname, data, type) in env["payloads"]
            if dataname == "temperature"
                println("Temperature: $data")
            elseif dataname == "humidity"
                println("Humidity: $data")
            end
        end
    end
    
    sleep(120)
    NATS.drain(conn)
end

test_receiver()
```

---

## Testing

Run the test scripts to verify functionality:

### Julia

```julia
# Text message exchange
julia test/test_julia_text_sender.jl
julia test/test_julia_text_receiver.jl

# Dictionary exchange
julia test/test_julia_dict_sender.jl
julia test/test_julia_dict_receiver.jl

# File transfer
julia test/test_julia_file_sender.jl
julia test/test_julia_file_receiver.jl

# Mixed payload types
julia test/test_julia_mix_payloads_sender.jl
julia test/test_julia_mix_payloads_receiver.jl

# Table exchange
julia test/test_julia_table_sender.jl
julia test/test_julia_table_receiver.jl
```

---

## License

MIT License

Copyright (c) 2026 NATSBridge Contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.