ton/NATSBridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

update

Browse Source
This commit is contained in:
narawat
2026-02-15 21:02:22 +07:00
parent f0df169689
commit 6b49fa68c0
5 changed files with 510 additions and 358 deletions
Download Patch File Download Diff File Expand all files Collapse all files

155
docs/architecture.md
View File

@@ -26,33 +26,55 @@ The system uses a **standardized list-of-tuples format** for all payload operati
**API Standard:**
```julia
# Input format for smartsend (always a list of tuples)
[(dataname1, data1), (dataname2, data2), ...]
# Input format for smartsend (always a list of tuples with type info)
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
# Output format for smartreceive (always returns a list of tuples)
[(dataname1, data1), (dataname2, data2), ...]
```
**Supported Types:**
- `"text"` - Plain text
- `"dictionary"` - JSON-serializable dictionaries (Dict, NamedTuple)
- `"table"` - Tabular data (DataFrame, array of structs)
- `"image"` - Image data (Bitmap, PNG/JPG bytes)
- `"audio"` - Audio data (WAV, MP3 bytes)
- `"video"` - Video data (MP4, AVI bytes)
- `"binary"` - Generic binary data (Vector{UInt8})
This design allows per-payload type specification, enabling **mixed-content messages** where different payloads can use different serialization formats in a single message.
**Examples:**
```julia
# Single payload - still wrapped in a list
smartsend(
"/test",
[("dataname1", data1)], # List with one tuple
[("dataname1", data1, "dictionary")], # List with one tuple (data, type)
nats_url="nats://localhost:4222",
fileserverUploadHandler=plik_oneshot_upload,
metadata=user_provided_envelope_level_metadata
)
# Multiple payloads in one message
# Multiple payloads in one message with different types
smartsend(
"/test",
[("dataname1", data1), ("dataname2", data2)],
[("dataname1", data1, "dictionary"), ("dataname2", data2, "table")],
nats_url="nats://localhost:4222",
fileserverUploadHandler=plik_oneshot_upload
)
# Mixed content (e.g., chat with text, image, audio)
smartsend(
"/chat",
[
("message_text", "Hello!", "text"),
("user_image", image_data, "image"),
("audio_clip", audio_data, "audio")
],
nats_url="nats://localhost:4222"
)
# Receive always returns a list
payloads = smartreceive(msg, fileserverDownloadHandler, max_retries, base_delay, max_delay)
# payloads = [("dataname1", data1), ("dataname2", data2), ...]
@@ -174,7 +196,7 @@ The `msgPayload_v1` structure provides flexible payload handling for various dat
struct msgPayload_v1
id::String # Id of this payload (e.g., "uuid4")
dataname::String # Name of this payload (e.g., "login_image")
type::String # "text | json | table | image | audio | video | binary"
type::String # "text | dictionary | table | image | audio | video | binary"
transport::String # "direct | link"
encoding::String # "none | json | base64 | arrow-ipc"
size::Integer # Data size in bytes
@@ -184,7 +206,7 @@ end
```
**Key Features:**
- Supports multiple data types: text, json, table, image, audio, video, binary
- Supports multiple data types: text, dictionary, table, image, audio, video, binary
- Flexible transport: "direct" (NATS) or "link" (HTTP fileserver)
- Multiple payloads per message (essential for chat with mixed content)
- Per-payload and per-envelope metadata support
@@ -194,28 +216,32 @@ end
```
┌─────────────────────────────────────────────────────────────┐
│ smartsend Function │
│ Accepts: [(dataname1, data1), (dataname2, data2), ...] │
│ Accepts: [(dataname1, data1, type1), ...] │
│ (No standalone type parameter - type per payload) │
└─────────────────────────────────────────────────────────────┘
│
▼
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Is payload size < 1MB? │
│ For each payload: │
│ 1. Extract type from tuple │
│ 2. Serialize based on type │
│ 3. Check payload size │
└─────────────────────────────────────────────────────────────┘
│
┌────────────────┴─-────────────────┐
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Direct Path │ │ Link Path │
│ (< 1MB) │ │ (> 1MB) │
│ │ │ │
│ • Serialize to │ │ • Serialize to │
│ IOBuffer │ │ IOBuffer │
│ • Base64 encode │ │ • Upload to │
│ • Publish to │ │ HTTP Server │
│ NATS │ │ • Publish to │
│ (with payload │ │ NATS with URL │
│ in envelope) │ │ (in envelope) │
└─────────────────┘ └─────────────────┘
│
┌────────────────┴─-────────────────┐
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Direct Path │ │ Link Path │
│ (< 1MB) │ │ (> 1MB) │
│ │ │ │
│ • Serialize to │ │ • Serialize to │
│ IOBuffer │ │ IOBuffer │
│ • Base64 encode │ │ • Upload to │
│ • Publish to │ │ HTTP Server │
│ NATS │ │ • Publish to │
│ (with payload │ │ NATS with URL │
│ in envelope) │ │ (in envelope) │
└─────────────────┘ └─────────────────┘
```
### 4. Julia Module Architecture
@@ -271,22 +297,22 @@ graph TD
```julia
function smartsend(
subject::String,
data::AbstractArray{Tuple{String, Any}},
type::String = "json";
nats_url::String = "nats://localhost:4222",
fileserverUploadHandler::Function = plik_oneshot_upload,
size_threshold::Int = 1_000_000 # 1MB
subject::String,
data::AbstractArray{Tuple{String, Any, String}}; # No standalone type parameter
nats_url::String = "nats://localhost:4222",
fileserverUploadHandler::Function = plik_oneshot_upload,
size_threshold::Int = 1_000_000 # 1MB
)
```
**Input Format:**
- `data::AbstractArray{Tuple{String, Any}}` - **Must be a list of tuples**: `[("dataname1", data1), ("dataname2", data2), ...]`
- Even for single payloads: `[(dataname1, data1)]`
- `data::AbstractArray{Tuple{String, Any, String}}` - **Must be a list of (dataname, data, type) tuples**: `[("dataname1", data1, "type1"), ("dataname2", data2, "type2"), ...]`
- Even for single payloads: `[(dataname1, data1, "type1")]`
- Each payload can have a different type, enabling mixed-content messages
**Flow:**
1. Iterate through the list of `("dataname", data)` tuples
2. For each payload: serialize to Arrow IPC stream (if table) or JSON
1. Iterate through the list of `(dataname, data, type)` tuples
2. For each payload: extract the type from the tuple and serialize accordingly
3. Check payload size
4. If < threshold: publish directly to NATS with Base64-encoded payload
5. If >= threshold: upload to HTTP server, publish NATS with URL
@@ -295,19 +321,19 @@ function smartsend(
```julia
function smartreceive(
msg::NATS.Message;
fileserverDownloadHandler::Function,
max_retries::Int = 5,
base_delay::Int = 100,
max_delay::Int = 5000
msg::NATS.Message;
fileserverDownloadHandler::Function,
max_retries::Int = 5,
base_delay::Int = 100,
max_delay::Int = 5000
)
# Parse envelope
# Iterate through all payloads
# For each payload: check transport type
# If direct: decode Base64 payload
# If link: fetch from URL with exponential backoff using fileserverDownloadHandler
# Deserialize payload based on type
# Return list of (dataname, data) tuples
# Parse envelope
# Iterate through all payloads
# For each payload: check transport type
# If direct: decode Base64 payload
# If link: fetch from URL with exponential backoff using fileserverDownloadHandler
# Deserialize payload based on type
# Return list of (dataname, data) tuples
end
```
@@ -322,7 +348,7 @@ end
- Determine transport type (`direct` or `link`)
- If `direct`: decode Base64 data from the message
- If `link`: fetch data from URL using exponential backoff
- Deserialize based on payload type (`json`, `table`, `binary`, etc.)
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
4. Return list of `(dataname, data)` tuples
### JavaScript Implementation
@@ -335,17 +361,26 @@ end
#### smartsend Function
```javascript
async function smartsend(subject, data, type = 'json', options = {})
async function smartsend(subject, data, options = {})
// data format: [(dataname, data, type), ...]
// options object should include:
// - fileserverUploadHandler: function to upload data to file server
// - fileserver_url: base URL of the file server
// - natsUrl: NATS server URL
// - fileserverUrl: base URL of the file server
// - sizeThreshold: threshold in bytes for transport selection
// - correlationId: optional correlation ID for tracing
```
**Input Format:**
- `data` - **Must be a list of (dataname, data, type) tuples**: `[(dataname1, data1, "type1"), (dataname2, data2, "type2"), ...]`
- Even for single payloads: `[(dataname1, data1, "type1")]`
- Each payload can have a different type, enabling mixed-content messages
**Flow:**
1. Serialize data to Arrow IPC buffer (if table)
2. Check payload size
3. If < threshold: publish directly to NATS
4. If >= threshold: upload to HTTP server, publish NATS with URL
1. Iterate through the list of (dataname, data, type) tuples
2. For each payload: extract the type from the tuple and serialize accordingly
3. Check payload size
4. If < threshold: publish directly to NATS
5. If >= threshold: upload to HTTP server, publish NATS with URL
#### smartreceive Handler
@@ -366,12 +401,12 @@ async function smartreceive(msg, options = {})
- Determine transport type (`direct` or `link`)
- If `direct`: decode Base64 data from the message
- If `link`: fetch data from URL using exponential backoff
- Deserialize based on payload type (`json`, `table`, `binary`, etc.)
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
4. Return list of `(dataname, data)` tuples
## Scenario Implementations
### Scenario 1: Command & Control (Small JSON)
### Scenario 1: Command & Control (Small Dictionary)
**Julia (Receiver):**
```julia
@@ -383,8 +418,8 @@ async function smartreceive(msg, options = {})
**JavaScript (Sender):**
```javascript
// Create small JSON config
// Send via smartsend with type="json"
// Create small dictionary config
// Send via smartsend with type="dictionary"
```
### Scenario 2: Deep Dive Analysis (Large Arrow Table)