ton/NATSBridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

architecture.md rev1

Browse Source
This commit is contained in:
narawat
2026-02-14 13:04:28 +07:00
parent d9fd7a61bb
commit f0df169689
3 changed files with 673 additions and 299 deletions
Download Patch File Download Diff File Expand all files Collapse all files

149
docs/IMPLEMENTATION.md
View File

@@ -4,6 +4,32 @@
This document describes the implementation of the high-performance, bi-directional data bridge between Julia and JavaScript services using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
### Multi-Payload Support
The implementation uses a **standardized list-of-tuples format** for all payload operations. **Even when sending a single payload, the user must wrap it in a list.**
**API Standard:**
```julia
# Input format for smartsend (always a list of tuples)
[(dataname1, data1), (dataname2, data2), ...]
# Output format for smartreceive (always returns a list of tuples)
[(dataname1, data1), (dataname2, data2), ...]
```
**Examples:**
```julia
# Single payload - still wrapped in a list
smartsend("/test", [(dataname1, data1)], ...)
# Multiple payloads in one message
smartsend("/test", [(dataname1, data1), (dataname2, data2)], ...)
# Receive always returns a list
payloads = smartreceive(msg, ...)
# payloads = [(dataname1, data1), (dataname2, data2), ...]
```
## Architecture
The implementation follows the Claim-Check pattern:
@@ -107,20 +133,66 @@ node test/scenario3_julia_to_julia.js
## Usage
### Scenario 0: Basic Multi-Payload Example
#### Julia (Sender)
```julia
using NATSBridge
# Send multiple payloads in one message
smartsend(
"/test",
[("dataname1", data1), ("dataname2", data2)],
nats_url="nats://localhost:4222",
fileserver_url="http://localhost:8080/upload",
metadata=Dict("custom_key" => "custom_value")
)
# Even single payload must be wrapped in a list
smartsend("/test", [("single_data", mydata)])
```
#### Julia (Receiver)
```julia
using NATSBridge
# Receive returns a list of payloads
payloads = smartreceive(msg, "http://localhost:8080/upload")
# payloads = [(dataname1, data1), (dataname2, data2), ...]
```
### Scenario 1: Command & Control (Small JSON)
#### JavaScript (Sender)
```javascript
const { SmartSend } = require('./js_bridge');
const config = {
step_size: 0.01,
iterations: 1000
};
// Single payload wrapped in a list
const config = [{
dataname: "config",
data: { step_size: 0.01, iterations: 1000 },
type: "json"
}];
await SmartSend("control", config, "json", {
correlationId: "unique-id"
});
// Multiple payloads
const configs = [
{
dataname: "config1",
data: { step_size: 0.01 },
type: "json"
},
{
dataname: "config2",
data: { iterations: 1000 },
type: "json"
}
];
await SmartSend("control", configs, "json");
```
#### Julia (Receiver)
@@ -157,8 +229,8 @@ df = DataFrame(
category = rand(["A", "B", "C"], 10_000_000)
)
# Send via SmartSend with type="table"
await SmartSend("analysis_results", df, "table");
# Send via SmartSend - wrapped in a list
await SmartSend("analysis_results", [("table_data", df)], "table");
```
#### JavaScript (Receiver)
@@ -177,8 +249,12 @@ const table = result.data;
```javascript
const { SmartSend } = require('./js_bridge');
// Capture binary chunk
const binaryData = await navigator.mediaDevices.getUserMedia({ binary: true });
// Binary data wrapped in a list
const binaryData = [{
dataname: "audio_chunk",
data: binaryBuffer,
type: "binary"
}];
await SmartSend("binary_input", binaryData, "binary", {
metadata: {
@@ -208,16 +284,13 @@ end
#### Julia (Producer)
```julia
using NATS
using NATSBridge
function publish_health_status(nats)
jetstream = JetStream(nats, "health_updates")
while true
status = Dict("cpu" => rand(), "memory" => rand())
publish(jetstream, "health", status)
sleep(5) # Every 5 seconds
end
function publish_health_status(nats_url)
# Send status wrapped in a list
status = Dict("cpu" => rand(), "memory" => rand())
smartsend("health", [("status", status)], "json", nats_url=nats_url)
sleep(5) # Every 5 seconds
end
```
@@ -238,7 +311,8 @@ const consumer = await js.pullSubscribe("health", {
// Process historical and real-time messages
for await (const msg of consumer) {
const result = await SmartReceive(msg);
// Process the data
// result.data contains the list of payloads
// result.envelope contains the message envelope
msg.ack();
}
```
@@ -257,16 +331,39 @@ for await (const msg of consumer) {
```json
{
"correlation_id": "uuid-v4-string",
"type": "json|table|binary",
"transport": "direct|link",
"payload": "base64-encoded-string", // Only if transport=direct
"url": "http://fileserver/path/to/data", // Only if transport=link
"correlationId": "uuid-v4-string",
"msgId": "uuid-v4-string",
"timestamp": "2024-01-15T10:30:00Z",
"sendTo": "topic/subject",
"msgPurpose": "ACK | NACK | updateStatus | shutdown | chat",
"senderName": "agent-wine-web-frontend",
"senderId": "uuid4",
"receiverName": "agent-backend",
"receiverId": "uuid4",
"replyTo": "topic",
"replyToMsgId": "uuid4",
"BrokerURL": "nats://localhost:4222",
"metadata": {
"content_type": "application/octet-stream",
"content_length": 123456,
"format": "arrow_ipc_stream"
}
"content_length": 123456
},
"payloads": [
{
"id": "uuid4",
"dataname": "login_image",
"type": "image",
"transport": "direct",
"encoding": "base64",
"size": 15433,
"data": "base64-encoded-string",
"metadata": {
"checksum": "sha256_hash"
}
}
]
}
```

371
docs/architecture.md
View File

@@ -4,6 +4,60 @@
This document describes the architecture for a high-performance, bi-directional data bridge between a Julia service and a JavaScript (Node.js) service using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
### File Server Handler Architecture
The system uses **handler functions** to abstract file server operations, allowing support for different file server implementations (e.g., Plik, AWS S3, custom HTTP server).
**Handler Function Signatures:**
```julia
# Upload handler - uploads data to file server and returns URL
fileserverUploadHandler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
# Download handler - fetches data from file server URL
fileserverDownloadHandler(fileserver_url::String, url::String, max_retries::Int, base_delay::Int, max_delay::Int)::Vector{UInt8}
```
This design allows the system to support multiple file server backends without changing the core messaging logic.
### Multi-Payload Support (Standard API)
The system uses a **standardized list-of-tuples format** for all payload operations. **Even when sending a single payload, the user must wrap it in a list.**
**API Standard:**
```julia
# Input format for smartsend (always a list of tuples)
[(dataname1, data1), (dataname2, data2), ...]
# Output format for smartreceive (always returns a list of tuples)
[(dataname1, data1), (dataname2, data2), ...]
```
**Examples:**
```julia
# Single payload - still wrapped in a list
smartsend(
"/test",
[("dataname1", data1)], # List with one tuple
nats_url="nats://localhost:4222",
fileserverUploadHandler=plik_oneshot_upload,
metadata=user_provided_envelope_level_metadata
)
# Multiple payloads in one message
smartsend(
"/test",
[("dataname1", data1), ("dataname2", data2)],
nats_url="nats://localhost:4222",
fileserverUploadHandler=plik_oneshot_upload
)
# Receive always returns a list
payloads = smartreceive(msg, fileserverDownloadHandler, max_retries, base_delay, max_delay)
# payloads = [("dataname1", data1), ("dataname2", data2), ...]
```
## Architecture Diagram
```mermaid
@@ -34,30 +88,113 @@ flowchart TD
## System Components
### 1. Unified JSON Envelope Schema
### 1. msgEnvelope_v1 - Message Envelope
All messages use a standardized envelope format:
The `msgEnvelope_v1` structure provides a comprehensive message format for bidirectional communication between Julia and JavaScript services.
**Julia Structure:**
```julia
struct msgEnvelope_v1
correlationId::String # Unique identifier to track messages across systems
msgId::String # This message id
timestamp::String # Message published timestamp
sendTo::String # Topic/subject the sender sends to
msgPurpose::String # Purpose of this message (ACK | NACK | updateStatus | shutdown | ...)
senderName::String # Sender name (e.g., "agent-wine-web-frontend")
senderId::String # Sender id (uuid4)
receiverName::String # Message receiver name (e.g., "agent-backend")
receiverId::String # Message receiver id (uuid4 or nothing for broadcast)
replyTo::String # Topic to reply to
replyToMsgId::String # Message id this message is replying to
brokerURL::String # NATS server address
metadata::Dict{String, Any}
payloads::AbstractArray{msgPayload_v1} # Multiple payloads stored here
end
```
**JSON Schema:**
```json
{
"correlation_id": "uuid-v4-string",
"type": "json|table|binary",
"transport": "direct|link",
"payload": "base64-encoded-string", // Only if transport=direct
"url": "http://fileserver/path/to/data", // Only if transport=link
"correlationId": "uuid-v4-string",
"msgId": "uuid-v4-string",
"timestamp": "2024-01-15T10:30:00Z",
"sendTo": "topic/subject",
"msgPurpose": "ACK | NACK | updateStatus | shutdown | chat",
"senderName": "agent-wine-web-frontend",
"senderId": "uuid4",
"receiverName": "agent-backend",
"receiverId": "uuid4",
"replyTo": "topic",
"replyToMsgId": "uuid4",
"brokerURL": "nats://localhost:4222",
"metadata": {
"content_type": "application/octet-stream",
"content_length": 123456,
"format": "arrow_ipc_stream"
}
"content_length": 123456
},
"payloads": [
{
"id": "uuid4",
"dataname": "login_image",
"type": "image",
"transport": "direct",
"encoding": "base64",
"size": 15433,
"data": "base64-encoded-string",
"metadata": {
"checksum": "sha256_hash"
}
},
{
"id": "uuid4",
"dataname": "large_data",
"type": "table",
"transport": "link",
"encoding": "none",
"size": 524288,
"data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/data.arrow",
"metadata": {
"checksum": "sha256_hash"
}
}
]
}
```
### 2. Transport Strategy Decision Logic
### 2. msgPayload_v1 - Payload Structure
The `msgPayload_v1` structure provides flexible payload handling for various data types.
**Julia Structure:**
```julia
struct msgPayload_v1
id::String # Id of this payload (e.g., "uuid4")
dataname::String # Name of this payload (e.g., "login_image")
type::String # "text | json | table | image | audio | video | binary"
transport::String # "direct | link"
encoding::String # "none | json | base64 | arrow-ipc"
size::Integer # Data size in bytes
data::Any # Payload data in case of direct transport or a URL in case of link
metadata::Dict{String, Any} # Dict("checksum" => "sha256_hash", ...)
end
```
**Key Features:**
- Supports multiple data types: text, json, table, image, audio, video, binary
- Flexible transport: "direct" (NATS) or "link" (HTTP fileserver)
- Multiple payloads per message (essential for chat with mixed content)
- Per-payload and per-envelope metadata support
### 3. Transport Strategy Decision Logic
```
┌─────────────────────────────────────────────────────────────┐
│ SmartSend Function │
│ smartsend Function │
│ Accepts: [(dataname1, data1), (dataname2, data2), ...] │
└─────────────────────────────────────────────────────────────┘
│
▼
@@ -65,34 +202,35 @@ All messages use a standardized envelope format:
│ Is payload size < 1MB? │
└─────────────────────────────────────────────────────────────┘
│
┌─────────────────┴─────────────────┐
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Direct Path │ │ Link Path │
│ (< 1MB) │ │ (> 1MB) │
│ │ │ │
│ • Serialize to │ │ • Serialize to │
│ IOBuffer │ │ IOBuffer │
│ • Base64 encode │ │ • Upload to │
│ • Publish to │ │ HTTP Server │
│ NATS │ │ • Publish to │
│ │ │ NATS with URL │
└─────────────────┘ └─────────────────┘
┌────────────────┴─-────────────────┐
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Direct Path │ │ Link Path │
│ (< 1MB) │ │ (> 1MB) │
│ │ │ │
│ • Serialize to │ │ • Serialize to │
│ IOBuffer │ │ IOBuffer │
│ • Base64 encode │ │ • Upload to │
│ • Publish to │ │ HTTP Server │
│ NATS │ │ • Publish to │
│ (with payload │ │ NATS with URL │
│ in envelope) │ │ (in envelope) │
└─────────────────┘ └─────────────────┘
```
### 3. Julia Module Architecture
### 4. Julia Module Architecture
```mermaid
graph TD
subgraph JuliaModule
SmartSendJulia[SmartSend Julia]
smartsendJulia[smartsend Julia]
SizeCheck[Size Check]
DirectPath[Direct Path]
LinkPath[Link Path]
HTTPClient[HTTP Client]
end
SmartSendJulia --> SizeCheck
smartsendJulia --> SizeCheck
SizeCheck -->|< 1MB| DirectPath
SizeCheck -->|>= 1MB| LinkPath
LinkPath --> HTTPClient
@@ -100,19 +238,19 @@ graph TD
style JuliaModule fill:#c5e1a5
```
### 4. JavaScript Module Architecture
### 5. JavaScript Module Architecture
```mermaid
graph TD
subgraph JSModule
SmartSendJS[SmartSend JS]
SmartReceiveJS[SmartReceive JS]
smartsendJS[smartsend JS]
smartreceiveJS[smartreceive JS]
JetStreamConsumer[JetStream Pull Consumer]
ApacheArrow[Apache Arrow]
end
SmartSendJS --> NATS
SmartReceiveJS --> JetStreamConsumer
smartsendJS --> NATS
smartreceiveJS --> JetStreamConsumer
JetStreamConsumer --> ApacheArrow
style JSModule fill:#f3e5f5
@@ -129,37 +267,64 @@ graph TD
- `HTTP.jl` - HTTP client for file server
- `Dates.jl` - Timestamps for logging
#### SmartSend Function
#### smartsend Function
```julia
function SmartSend(
function smartsend(
subject::String,
data::Any,
data::AbstractArray{Tuple{String, Any}},
type::String = "json";
nats_url::String = "nats://localhost:4222",
fileserver_url::String = "http://localhost:8080/upload",
fileserverUploadHandler::Function = plik_oneshot_upload,
size_threshold::Int = 1_000_000 # 1MB
)
```
**Flow:**
1. Serialize data to Arrow IPC stream (if table)
2. Check payload size
3. If < threshold: publish directly to NATS with Base64-encoded payload
4. If >= threshold: upload to HTTP server, publish NATS with URL
**Input Format:**
- `data::AbstractArray{Tuple{String, Any}}` - **Must be a list of tuples**: `[("dataname1", data1), ("dataname2", data2), ...]`
- Even for single payloads: `[(dataname1, data1)]`
#### SmartReceive Handler
**Flow:**
1. Iterate through the list of `("dataname", data)` tuples
2. For each payload: serialize to Arrow IPC stream (if table) or JSON
3. Check payload size
4. If < threshold: publish directly to NATS with Base64-encoded payload
5. If >= threshold: upload to HTTP server, publish NATS with URL
#### smartreceive Handler
```julia
function SmartReceive(msg::NATS.Message)
function smartreceive(
msg::NATS.Message;
fileserverDownloadHandler::Function,
max_retries::Int = 5,
base_delay::Int = 100,
max_delay::Int = 5000
)
# Parse envelope
# Check transport type
# If direct: decode Base64 payload
# If link: fetch from URL with exponential backoff
# Deserialize Arrow IPC to DataFrame
# Iterate through all payloads
# For each payload: check transport type
# If direct: decode Base64 payload
# If link: fetch from URL with exponential backoff using fileserverDownloadHandler
# Deserialize payload based on type
# Return list of (dataname, data) tuples
end
```
**Output Format:**
- Always returns a list of tuples: `[(dataname1, data1), (dataname2, data2), ...]`
- Even for single payloads: `[(dataname1, data1)]`
**Process Flow:**
1. Parse the JSON envelope to extract the `payloads` array
2. Iterate through each payload in `payloads`
3. For each payload:
- Determine transport type (`direct` or `link`)
- If `direct`: decode Base64 data from the message
- If `link`: fetch data from URL using exponential backoff
- Deserialize based on payload type (`json`, `table`, `binary`, etc.)
4. Return list of `(dataname, data)` tuples
### JavaScript Implementation
#### Dependencies
@@ -167,10 +332,13 @@ end
- `apache-arrow` - Arrow IPC serialization
- `uuid` - Correlation ID generation
#### SmartSend Function
#### smartsend Function
```javascript
async function SmartSend(subject, data, type = 'json', options = {})
async function smartsend(subject, data, type = 'json', options = {})
// options object should include:
// - fileserverUploadHandler: function to upload data to file server
// - fileserver_url: base URL of the file server
```
**Flow:**
@@ -179,18 +347,27 @@ async function SmartSend(subject, data, type = 'json', options = {})
3. If < threshold: publish directly to NATS
4. If >= threshold: upload to HTTP server, publish NATS with URL
#### SmartReceive Handler
#### smartreceive Handler
```javascript
async function SmartReceive(msg, options = {})
async function smartreceive(msg, options = {})
// options object should include:
// - fileserverDownloadHandler: function to fetch data from file server URL
// - fileserver_url: base URL of the file server
// - max_retries: maximum retry attempts for fetching URL
// - base_delay: initial delay for exponential backoff in ms
// - max_delay: maximum delay for exponential backoff in ms
```
**Flow:**
1. Parse envelope
2. Check transport type
3. If direct: decode Base64 payload
4. If link: fetch with exponential backoff
5. Deserialize Arrow IPC with zero-copy
**Process Flow:**
1. Parse the JSON envelope to extract the `payloads` array
2. Iterate through each payload in `payloads`
3. For each payload:
- Determine transport type (`direct` or `link`)
- If `direct`: decode Base64 data from the message
- If `link`: fetch data from URL using exponential backoff
- Deserialize based on payload type (`json`, `table`, `binary`, etc.)
4. Return list of `(dataname, data)` tuples
## Scenario Implementations
@@ -207,7 +384,7 @@ async function SmartReceive(msg, options = {})
**JavaScript (Sender):**
```javascript
// Create small JSON config
// Send via SmartSend with type="json"
// Send via smartsend with type="json"
```
### Scenario 2: Deep Dive Analysis (Large Arrow Table)
@@ -235,7 +412,7 @@ async function SmartReceive(msg, options = {})
```javascript
// Capture audio chunk
// Send as binary with metadata headers
// Use SmartSend with type="audio"
// Use smartsend with type="audio"
```
**Julia (Receiver):**
@@ -260,6 +437,76 @@ async function SmartReceive(msg, options = {})
// Process historical and real-time messages
```
### Scenario 5: Selection (Low Bandwidth)
**Focus:** Small Arrow tables, Julia to JavaScript. The Action: Julia wants to send a small DataFrame to show on a JavaScript dashboard for the user to choose.
**Julia (Sender):**
```julia
# Create small DataFrame (e.g., 50KB - 500KB)
# Convert to Arrow IPC stream
# Check payload size (< 1MB threshold)
# Publish directly to NATS with Base64-encoded payload
# Include metadata for dashboard selection context
```
**JavaScript (Receiver):**
```javascript
// Receive NATS message with direct transport
// Decode Base64 payload
// Parse Arrow IPC with zero-copy
// Load into selection UI component (e.g., dropdown, table)
// User makes selection
// Send selection back to Julia
```
**Use Case:** Julia server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to JavaScript dashboard for user selection. The selection is then sent back to Julia for processing.
### Scenario 6: Chat System
**Focus:** Every conversational message is composed of any number and any combination of components, spanning the full spectrum from small to large. This includes text, images, audio, video, tables, and files—specifically accommodating everything from brief snippets to high-resolution images, large audio files, extensive tables, and massive documents. Support for claim-check delivery and full bi-directional messaging.
**Multi-Payload Support:** The system supports mixed-payload messages where a single message can contain multiple payloads with different transport strategies. The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type.
**Julia (Sender/Receiver):**
```julia
# Build chat message with mixed payloads:
# - Text: direct transport (Base64)
# - Small images: direct transport (Base64)
# - Large images: link transport (HTTP URL)
# - Audio/video: link transport (HTTP URL)
# - Tables: direct or link depending on size
# - Files: link transport (HTTP URL)
#
# Each payload uses appropriate transport strategy:
# - Size < 1MB → direct (NATS + Base64)
# - Size >= 1MB → link (HTTP upload + NATS URL)
#
# Include claim-check metadata for delivery tracking
# Support bidirectional messaging with replyTo fields
```
**JavaScript (Sender/Receiver):**
```javascript
// Build chat message with mixed content:
// - User input text: direct transport
// - Selected image: check size, use appropriate transport
// - Audio recording: link transport for large files
// - File attachment: link transport
//
// Parse received message:
// - Direct payloads: decode Base64
// - Link payloads: fetch from HTTP with exponential backoff
// - Deserialize all payloads appropriately
//
// Render mixed content in chat interface
// Support bidirectional reply with claim-check delivery confirmation
```
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components.
**Implementation Note:** The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. See the standard API format in Section 1: `msgEnvelope_v1` supports `AbstractArray{msgPayload_v1}` for multiple payloads.
## Performance Considerations
### Zero-Copy Reading
@@ -280,8 +527,8 @@ async function SmartReceive(msg, options = {})
## Testing Strategy
### Unit Tests
- Test SmartSend with various payload sizes
- Test SmartReceive with direct and link transport
- Test smartsend with various payload sizes
- Test smartreceive with direct and link transport
- Test Arrow IPC serialization/deserialization
### Integration Tests