ton/NATSBridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

update

Browse Source
This commit is contained in:
narawat
2026-02-22 20:43:28 +07:00
parent 0de9725ba8
commit 075d355c58
17 changed files with 147 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
docs/architecture.md
View File

@@ -35,8 +35,24 @@ The system uses a **standardized list-of-tuples format** for all payload operati
# Input format for smartsend (always a list of tuples with type info)
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
# Output format for smartreceive (always returns a list of tuples)
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
# Output format for smartreceive (returns envelope dictionary with payloads field)
# Returns: Dict with envelope metadata and payloads field containing list of tuples
# {
# "correlationId": "...",
# "msgId": "...",
# "timestamp": "...",
# "sendTo": "...",
# "msgPurpose": "...",
# "senderName": "...",
# "senderId": "...",
# "receiverName": "...",
# "receiverId": "...",
# "replyTo": "...",
# "replyToMsgId": "...",
# "brokerURL": "...",
# "metadata": {...},
# "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
# }
```
**Supported Types:**
@@ -81,9 +97,10 @@ smartsend(
nats_url="nats://localhost:4222"
)
# Receive always returns a list
payloads = smartreceive(msg, fileserverDownloadHandler, max_retries, base_delay, max_delay)
# payloads = [("dataname1", data1, type1), ("dataname2", data2, type2), ...]
# Receive returns a dictionary envelope with all metadata and deserialized payloads
envelope = smartreceive(msg, fileserverDownloadHandler, max_retries, base_delay, max_delay)
# envelope["payloads"] = [("dataname1", data1, type1), ("dataname2", data2, type2), ...]
# envelope["correlationId"], envelope["msgId"], etc.
```
## Architecture Diagram
@@ -338,23 +355,25 @@ function smartreceive(
# If direct: decode Base64 payload
# If link: fetch from URL with exponential backoff using fileserverDownloadHandler
# Deserialize payload based on type
# Return list of (dataname, data, type) tuples
# Return envelope dictionary with all metadata and deserialized payloads
end
```
**Output Format:**
- Always returns a list of tuples: `[(dataname1, data1, type1), (dataname2, data2, type2), ...]`
- Even for single payloads: `[(dataname1, data1, type1)]`
- Returns a dictionary (key-value map) containing all envelope fields:
- `correlationId`, `msgId`, `timestamp`, `sendTo`, `msgPurpose`, `senderName`, `senderId`, `receiverName`, `receiverId`, `replyTo`, `replyToMsgId`, `brokerURL`
- `metadata` - Message-level metadata dictionary
- `payloads` - List of dictionaries, each containing deserialized payload data
**Process Flow:**
1. Parse the JSON envelope to extract the `payloads` array
1. Parse the JSON envelope to extract all fields
2. Iterate through each payload in `payloads`
3. For each payload:
- Determine transport type (`direct` or `link`)
- If `direct`: decode Base64 data from the message
- If `link`: fetch data from URL using exponential backoff (via `fileserverDownloadHandler`)
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
4. Return list of `(dataname, data, type)` tuples
4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples
**Note:** The `fileserverDownloadHandler` receives `(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)` and returns `Vector{UInt8}`.
@@ -401,15 +420,21 @@ async function smartreceive(msg, options = {})
// - correlationId: optional correlation ID for tracing
```
**Output Format:**
- Returns a dictionary (key-value map) containing all envelope fields:
- `correlationId`, `msgId`, `timestamp`, `sendTo`, `msgPurpose`, `senderName`, `senderId`, `receiverName`, `receiverId`, `replyTo`, `replyToMsgId`, `brokerURL`
- `metadata` - Message-level metadata dictionary
- `payloads` - List of dictionaries, each containing deserialized payload data
**Process Flow:**
1. Parse the JSON envelope to extract the `payloads` array
1. Parse the JSON envelope to extract all fields
2. Iterate through each payload in `payloads`
3. For each payload:
- Determine transport type (`direct` or `link`)
- If `direct`: decode Base64 data from the message
- If `link`: fetch data from URL using exponential backoff
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
4. Return list of `(dataname, data, type)` tuples
4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples
## Scenario Implementations

71
docs/implementation.md
View File

@@ -13,8 +13,24 @@ The implementation uses a **standardized list-of-tuples format** for all payload
# Input format for smartsend (always a list of tuples with type info)
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
# Output format for smartreceive (always returns a list of tuples with type info)
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
# Output format for smartreceive (returns envelope dictionary with payloads field)
# Returns: Dict with envelope metadata and payloads field containing list of tuples
# {
# "correlationId": "...",
# "msgId": "...",
# "timestamp": "...",
# "sendTo": "...",
# "msgPurpose": "...",
# "senderName": "...",
# "senderId": "...",
# "receiverName": "...",
# "receiverId": "...",
# "replyTo": "...",
# "replyToMsgId": "...",
# "brokerURL": "...",
# "metadata": {...},
# "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
# }
```
Where `type` can be: `"text"`, `"dictionary"`, `"table"`, `"image"`, `"audio"`, `"video"`, `"binary"`
@@ -27,9 +43,10 @@ smartsend("/test", [(dataname1, data1, "text")], ...)
# Multiple payloads in one message (each payload has its own type)
smartsend("/test", [(dataname1, data1, "dictionary"), (dataname2, data2, "table")], ...)
# Receive always returns a list with type info
payloads = smartreceive(msg, ...)
# payloads = [(dataname1, data1, "text"), (dataname2, data2, "table"), ...]
# Receive returns a dictionary envelope with all metadata and deserialized payloads
envelope = smartreceive(msg, ...)
# envelope["payloads"] = [(dataname1, data1, "text"), (dataname2, data2, "table"), ...]
# envelope["correlationId"], envelope["msgId"], etc.
```
## Architecture
@@ -159,9 +176,10 @@ smartsend("/test", [("single_data", mydata, "dictionary")])
```julia
using NATSBridge
# Receive returns a list of payloads with type info
payloads = smartreceive(msg, "http://localhost:8080")
# payloads = [(dataname1, data1, "dictionary"), (dataname2, data2, "table"), ...]
# Receive returns a dictionary envelope with all metadata and deserialized payloads
envelope = smartreceive(msg, "http://localhost:8080")
# envelope["payloads"] = [(dataname1, data1, "dictionary"), (dataname2, data2, "table"), ...]
# envelope["correlationId"], envelope["msgId"], etc.
```
### Scenario 1: Command & Control (Small JSON)
@@ -227,13 +245,18 @@ const nc = await connect({ servers: ['nats://localhost:4222'] });
const sub = nc.subscribe("control");
for await (const msg of sub) {
const result = await smartreceive(msg);
const envelope = await smartreceive(msg);
// Process the result
for (const { dataname, data, type } of result) {
// Process the payloads from the envelope
for (const payload of envelope.payloads) {
const { dataname, data, type } = payload;
console.log(`Received ${dataname} of type ${type}`);
console.log(`Data: ${JSON.stringify(data)}`);
}
// Also access envelope metadata
console.log(`Correlation ID: ${envelope.correlationId}`);
console.log(`Message ID: ${envelope.msgId}`);
}
```
@@ -259,11 +282,11 @@ await SmartSend("analysis_results", [("table_data", df, "table")]);
```javascript
const { smartreceive } = require('./src/NATSBridge');
const result = await smartreceive(msg);
const envelope = await smartreceive(msg);
// Use table data for visualization with Perspective.js or D3
// Use table data from the payloads field
// Note: Tables are sent as arrays of objects in JavaScript
const table = result;
const table = envelope.payloads;
```
### Scenario 3: Live Binary Processing
@@ -309,13 +332,13 @@ const { smartreceive } = require('./src/NATSBridge');
// Receive binary data
function process_binary(msg) {
const result = await smartreceive(msg);
const envelope = await smartreceive(msg);
// Process the binary data
for (const { dataname, data, type } of result) {
if (type === "binary") {
// Process the binary data from envelope.payloads
for (const payload of envelope.payloads) {
if (payload.type === "binary") {
// data is an ArrayBuffer or Uint8Array
console.log(`Received binary data: ${dataname}, size: ${data.length}`);
console.log(`Received binary data: ${payload.dataname}, size: ${payload.data.length}`);
// Perform FFT or AI transcription here
}
}
@@ -353,8 +376,8 @@ const consumer = await js.pullSubscribe("health", {
// Process historical and real-time messages
for await (const msg of consumer) {
const result = await smartreceive(msg);
// result contains the list of payloads
const envelope = await smartreceive(msg);
// envelope.payloads contains the list of payloads
// Each payload has: dataname, data, type
msg.ack();
}
@@ -395,11 +418,11 @@ smartsend(
const { smartreceive, smartsend } = require('./src/NATSBridge');
// Receive NATS message with direct transport
const result = await smartreceive(msg);
const envelope = await smartreceive(msg);
// Decode Base64 payload (for direct transport)
// For tables, data is an array of objects
const table = result; // Array of objects
// For tables, data is in envelope.payloads
const table = envelope.payloads; // Array of objects
// User makes selection
const selection = uiComponent.getSelectedOption();