update
This commit is contained in:
@@ -2,12 +2,10 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the architecture for a high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
This document describes the architecture for a high-performance, bi-directional data bridge for **Julia** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
|
||||
The system enables seamless communication across all three platforms:
|
||||
- **Julia ↔ JavaScript** bi-directional messaging
|
||||
- **JavaScript ↔ Python/Micropython** bi-directional messaging
|
||||
- **Julia ↔ Python/Micropython** bi-directional messaging (via JSON serialization)
|
||||
The system enables seamless communication for Julia applications:
|
||||
- **Julia** messaging with NATS
|
||||
|
||||
### File Server Handler Architecture
|
||||
|
||||
@@ -113,8 +111,7 @@ env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff, max_ret
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph Client
|
||||
JS[JavaScript Client]
|
||||
JSApp[Application Logic]
|
||||
App[Julia Application]
|
||||
end
|
||||
|
||||
subgraph Server
|
||||
@@ -123,14 +120,12 @@ flowchart TD
|
||||
FileServer[HTTP File Server]
|
||||
end
|
||||
|
||||
JS -->|Control/Small Data| JSApp
|
||||
JSApp -->|NATS| NATS
|
||||
App -->|NATS| NATS
|
||||
NATS -->|NATS| Julia
|
||||
Julia -->|NATS| NATS
|
||||
Julia -->|HTTP POST| FileServer
|
||||
JS -->|HTTP GET| FileServer
|
||||
|
||||
style JS fill:#e1f5fe
|
||||
style App fill:#e8f5e9
|
||||
style Julia fill:#e8f5e9
|
||||
style NATS fill:#fff3e0
|
||||
style FileServer fill:#f3e5f5
|
||||
@@ -140,7 +135,7 @@ flowchart TD
|
||||
|
||||
### 1. msg_envelope_v1 - Message Envelope
|
||||
|
||||
The `msg_envelope_v1` structure provides a comprehensive message format for bidirectional communication between Julia, JavaScript, and Python/Micropython applications.
|
||||
The `msg_envelope_v1` structure provides a comprehensive message format for bidirectional communication in Julia applications.
|
||||
|
||||
**Julia Structure:**
|
||||
```julia
|
||||
@@ -216,7 +211,7 @@ end
|
||||
|
||||
### 2. msg_payload_v1 - Payload Structure
|
||||
|
||||
The `msg_payload_v1` structure provides flexible payload handling for various data types across all supported platforms.
|
||||
The `msg_payload_v1` structure provides flexible payload handling for various data types.
|
||||
|
||||
**Julia Structure:**
|
||||
```julia
|
||||
@@ -271,65 +266,7 @@ end
|
||||
└─────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
### 4. Cross-Platform Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph PythonMicropython
|
||||
Py[Python/Micropython]
|
||||
PySmartSend[smartsend]
|
||||
PySmartReceive[smartreceive]
|
||||
end
|
||||
|
||||
subgraph JavaScript
|
||||
JS[JavaScript]
|
||||
JSSmartSend[smartsend]
|
||||
JSSmartReceive[smartreceive]
|
||||
end
|
||||
|
||||
subgraph Julia
|
||||
Julia[Julia]
|
||||
JuliaSmartSend[smartsend]
|
||||
JuliaSmartReceive[smartreceive]
|
||||
end
|
||||
|
||||
subgraph NATS
|
||||
NATSServer[NATS Server]
|
||||
end
|
||||
|
||||
PySmartSend --> NATSServer
|
||||
JSSmartSend --> NATSServer
|
||||
JuliaSmartSend --> NATSServer
|
||||
NATSServer --> PySmartReceive
|
||||
NATSServer --> JSSmartReceive
|
||||
NATSServer --> JuliaSmartReceive
|
||||
|
||||
style PythonMicropython fill:#e1f5fe
|
||||
style JavaScript fill:#f3e5f5
|
||||
style Julia fill:#e8f5e9
|
||||
```
|
||||
|
||||
### 5. Python/Micropython Module Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph PyModule
|
||||
PySmartSend[smartsend]
|
||||
SizeCheck[Size Check]
|
||||
DirectPath[Direct Path]
|
||||
LinkPath[Link Path]
|
||||
HTTPClient[HTTP Client]
|
||||
end
|
||||
|
||||
PySmartSend --> SizeCheck
|
||||
SizeCheck -->|< 1MB| DirectPath
|
||||
SizeCheck -->|>= 1MB| LinkPath
|
||||
LinkPath --> HTTPClient
|
||||
|
||||
style PyModule fill:#b3e5fc
|
||||
```
|
||||
|
||||
### 6. Julia Module Architecture
|
||||
### 4. Julia Module Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
@@ -349,51 +286,8 @@ graph TD
|
||||
style JuliaModule fill:#c5e1a5
|
||||
```
|
||||
|
||||
### 7. JavaScript Module Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph JSModule
|
||||
JSSmartSend[smartsend]
|
||||
JSSmartReceive[smartreceive]
|
||||
JetStreamConsumer[JetStream Pull Consumer]
|
||||
ApacheArrow[Apache Arrow]
|
||||
end
|
||||
|
||||
JSSmartSend --> NATS
|
||||
JSSmartReceive --> JetStreamConsumer
|
||||
JetStreamConsumer --> ApacheArrow
|
||||
|
||||
style JSModule fill:#f3e5f5
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### API Consistency Across Languages
|
||||
|
||||
**High-Level API (Consistent Across All Languages):**
|
||||
- `smartsend(subject, data, ...)` - Main publishing function
|
||||
- `smartreceive(msg, ...)` - Main receiving function
|
||||
- Message envelope structure (`msg_envelope_v1` / `MessageEnvelope`)
|
||||
- Payload structure (`msg_payload_v1` / `MessagePayload`)
|
||||
- Transport strategy (direct vs link based on size threshold)
|
||||
- Supported payload types: text, dictionary, table, image, audio, video, binary
|
||||
|
||||
**Low-Level Native Functions (Language-Specific Conventions):**
|
||||
- Julia: `NATS.connect()`, `publish_message()`, function overloading
|
||||
- JavaScript: `nats.js` client, native async/await patterns
|
||||
- Python: `nats-python` client, native async/await patterns
|
||||
|
||||
**Connection Reuse Pattern:**
|
||||
- **Julia:** Uses `NATS_connection` keyword parameter with function overloading
|
||||
- **JavaScript/Python:** Achieved by creating NATS client outside the function and reusing it in custom handlers
|
||||
|
||||
**Note on `is_publish`:**
|
||||
- `is_publish` is simply a switch to control automatic publishing
|
||||
- When `true` (default): Message is published to NATS automatically
|
||||
- When `false`: Returns `(env, env_json_str)` without publishing, allowing manual publishing
|
||||
- Connection reuse is achieved separately by creating NATS client outside the function
|
||||
|
||||
### Julia Implementation
|
||||
|
||||
#### Dependencies
|
||||
@@ -546,200 +440,6 @@ env, env_json_str = smartsend(
|
||||
# Uses: publish_message(broker_url, subject, env_json_str, cid)
|
||||
```
|
||||
|
||||
**API Consistency Note:**
|
||||
- **High-level API (smartsend, smartreceive):** Uses consistent naming across all three languages (Julia, JavaScript, Python/Micropython)
|
||||
- **Low-level native functions (NATS.connect(), publish_message()):** Follow the conventions of the specific language ecosystem and do not require cross-language consistency
|
||||
|
||||
### JavaScript Implementation
|
||||
|
||||
#### Dependencies
|
||||
- `nats.js` - Core NATS functionality
|
||||
- `apache-arrow` - Arrow IPC serialization
|
||||
- `uuid` - Correlation ID and message ID generation
|
||||
- `base64-arraybuffer` - Base64 encoding/decoding
|
||||
- `node-fetch` or `fetch` - HTTP client for file server
|
||||
|
||||
#### smartsend Function
|
||||
|
||||
```javascript
|
||||
async function smartsend(
|
||||
subject,
|
||||
data, // List of (dataname, data, type) tuples: [(dataname1, data1, type1), ...]
|
||||
options = {}
|
||||
)
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- `broker_url` (String) - NATS server URL (default: `"nats://localhost:4222"`)
|
||||
- `fileserver_url` (String) - Base URL of the file server (default: `"http://localhost:8080"`)
|
||||
- `size_threshold` (Number) - Threshold in bytes for transport selection (default: `1048576` = 1MB)
|
||||
- `correlation_id` (String) - Optional correlation ID for tracing
|
||||
- `msg_purpose` (String) - Purpose of the message (default: `"chat"`)
|
||||
- `sender_name` (String) - Sender name (default: `"NATSBridge"`)
|
||||
- `receiver_name` (String) - Message receiver name (default: `""`)
|
||||
- `receiver_id` (String) - Message receiver ID (default: `""`)
|
||||
- `reply_to` (String) - Topic to reply to (default: `""`)
|
||||
- `reply_to_msg_id` (String) - Message ID this message is replying to (default: `""`)
|
||||
- `is_publish` (Boolean) - Whether to automatically publish the message to NATS (default: `true`)
|
||||
- `fileserver_upload_handler` (Function) - Custom upload handler function
|
||||
|
||||
**Note:** JavaScript uses `is_publish` option (instead of `NATS_connection` keyword) to control automatic publishing behavior. Connection reuse can be achieved by creating a NATS client outside the function and reusing it in a custom `fileserver_upload_handler` or custom publish implementation.
|
||||
|
||||
**Return Value:**
|
||||
- Returns a Promise that resolves to an object containing:
|
||||
- `env` - The envelope object containing all metadata and payloads
|
||||
- `env_json_str` - JSON string representation of the envelope for publishing
|
||||
|
||||
**Input Format:**
|
||||
- `data` - **Must be a list of (dataname, data, type) tuples**: `[(dataname1, data1, "type1"), (dataname2, data2, "type2"), ...]`
|
||||
- Even for single payloads: `[(dataname1, data1, "type1")]`
|
||||
- Each payload can have a different type, enabling mixed-content messages
|
||||
- Supported types: `"text"`, `"dictionary"`, `"table"`, `"image"`, `"audio"`, `"video"`, `"binary"`
|
||||
|
||||
**Flow:**
|
||||
1. Generate correlation ID and message ID if not provided
|
||||
2. Iterate through the list of `(dataname, data, type)` tuples
|
||||
3. For each payload:
|
||||
- Serialize based on payload type
|
||||
- Check payload size
|
||||
- If < threshold: Base64 encode and include in envelope
|
||||
- If >= threshold: Upload to HTTP server, store URL in envelope
|
||||
4. Publish the JSON envelope to NATS
|
||||
5. Return envelope object and JSON string
|
||||
|
||||
#### smartreceive Handler
|
||||
|
||||
```javascript
|
||||
async function smartreceive(msg, options = {})
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- `fileserver_download_handler` (Function) - Custom download handler function
|
||||
- `max_retries` (Number) - Maximum retry attempts for fetching URL (default: `5`)
|
||||
- `base_delay` (Number) - Initial delay for exponential backoff in ms (default: `100`)
|
||||
- `max_delay` (Number) - Maximum delay for exponential backoff in ms (default: `5000`)
|
||||
- `correlation_id` (String) - Optional correlation ID for tracing
|
||||
|
||||
**Output Format:**
|
||||
- Returns a Promise that resolves to an object containing all envelope fields:
|
||||
- `correlation_id`, `msg_id`, `timestamp`, `send_to`, `msg_purpose`, `sender_name`, `sender_id`, `receiver_name`, `receiver_id`, `reply_to`, `reply_to_msg_id`, `broker_url`
|
||||
- `metadata` - Message-level metadata dictionary
|
||||
- `payloads` - List of tuples, each containing `(dataname, data, type)` with deserialized payload data
|
||||
|
||||
**Process Flow:**
|
||||
1. Parse the JSON envelope to extract all fields
|
||||
2. Iterate through each payload in `payloads` array
|
||||
3. For each payload:
|
||||
- Determine transport type (`direct` or `link`)
|
||||
- If `direct`: Base64 decode the data from the message
|
||||
- If `link`: Fetch data from URL using exponential backoff (via `fileserver_download_handler`)
|
||||
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
|
||||
4. Return envelope object with `payloads` field containing list of `(dataname, data, type)` tuples
|
||||
|
||||
**Note:** The `fileserver_download_handler` receives `(url, max_retries, base_delay, max_delay, correlation_id)` and returns `ArrayBuffer` or `Uint8Array`.
|
||||
|
||||
### Python/Micropython Implementation
|
||||
|
||||
#### Dependencies
|
||||
- `nats-python` - Core NATS functionality
|
||||
- `pyarrow` - Arrow IPC serialization
|
||||
- `uuid` - Correlation ID and message ID generation
|
||||
- `base64` - Base64 encoding/decoding
|
||||
- `requests` or `aiohttp` - HTTP client for file server
|
||||
|
||||
#### smartsend Function
|
||||
|
||||
```python
|
||||
def smartsend(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]], # List of (dataname, data, type) tuples
|
||||
broker_url: str = DEFAULT_BROKER_URL,
|
||||
fileserver_url: str = DEFAULT_FILESERVER_URL,
|
||||
fileserver_upload_handler: Callable = plik_oneshot_upload,
|
||||
size_threshold: int = DEFAULT_SIZE_THRESHOLD,
|
||||
correlation_id: Union[str, None] = None,
|
||||
msg_purpose: str = "chat",
|
||||
sender_name: str = "NATSBridge",
|
||||
receiver_name: str = "",
|
||||
receiver_id: str = "",
|
||||
reply_to: str = "",
|
||||
reply_to_msg_id: str = "",
|
||||
is_publish: bool = True
|
||||
) -> Tuple[MessageEnvelope, str]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- `broker_url` (str) - NATS server URL (default: `"nats://localhost:4222"`)
|
||||
- `fileserver_url` (str) - Base URL of the file server (default: `"http://localhost:8080"`)
|
||||
- `size_threshold` (int) - Threshold in bytes for transport selection (default: `1048576` = 1MB)
|
||||
- `correlation_id` (str) - Optional correlation ID for tracing (auto-generated if None)
|
||||
- `msg_purpose` (str) - Purpose of the message (default: `"chat"`)
|
||||
- `sender_name` (str) - Sender name (default: `"NATSBridge"`)
|
||||
- `receiver_name` (str) - Message receiver name (default: `""`)
|
||||
- `receiver_id` (str) - Message receiver ID (default: `""`)
|
||||
- `reply_to` (str) - Topic to reply to (default: `""`)
|
||||
- `reply_to_msg_id` (str) - Message ID this message is replying to (default: `""`)
|
||||
- `is_publish` (bool) - Whether to automatically publish the message to NATS (default: `True`)
|
||||
- `fileserver_upload_handler` (Callable) - Custom upload handler function
|
||||
|
||||
**Note:** Python uses `is_publish` parameter (instead of `NATS_connection` keyword) to control automatic publishing behavior. Connection reuse can be achieved by creating a NATS client outside the function and reusing it in a custom `fileserver_upload_handler` or custom publish implementation.
|
||||
|
||||
**Return Value:**
|
||||
- Returns a tuple `(env, env_json_str)` where:
|
||||
- `env` - The envelope dictionary containing all metadata and payloads
|
||||
- `env_json_str` - JSON string representation of the envelope for publishing
|
||||
|
||||
**Input Format:**
|
||||
- `data` - **Must be a list of (dataname, data, type) tuples**: `[(dataname1, data1, "type1"), (dataname2, data2, "type2"), ...]`
|
||||
- Even for single payloads: `[(dataname1, data1, "type1")]`
|
||||
- Each payload can have a different type, enabling mixed-content messages
|
||||
- Supported types: `"text"`, `"dictionary"`, `"table"`, `"image"`, `"audio"`, `"video"`, `"binary"`
|
||||
|
||||
**Flow:**
|
||||
1. Generate correlation ID and message ID if not provided
|
||||
2. Iterate through the list of `(dataname, data, type)` tuples
|
||||
3. For each payload:
|
||||
- Serialize based on payload type
|
||||
- Check payload size
|
||||
- If < threshold: Base64 encode and include in envelope
|
||||
- If >= threshold: Upload to HTTP server, store URL in envelope
|
||||
4. Publish the JSON envelope to NATS
|
||||
5. Return envelope dictionary and JSON string
|
||||
|
||||
#### smartreceive Handler
|
||||
|
||||
```python
|
||||
async def smartreceive(
|
||||
msg: NATS.Message,
|
||||
options: Dict = {}
|
||||
)
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- `fileserver_download_handler` (Callable) - Custom download handler function
|
||||
- `max_retries` (int) - Maximum retry attempts for fetching URL (default: `5`)
|
||||
- `base_delay` (int) - Initial delay for exponential backoff in ms (default: `100`)
|
||||
- `max_delay` (int) - Maximum delay for exponential backoff in ms (default: `5000`)
|
||||
- `correlation_id` (str) - Optional correlation ID for tracing
|
||||
|
||||
**Output Format:**
|
||||
- Returns a JSON object (dictionary) containing all envelope fields:
|
||||
- `correlation_id`, `msg_id`, `timestamp`, `send_to`, `msg_purpose`, `sender_name`, `sender_id`, `receiver_name`, `receiver_id`, `reply_to`, `reply_to_msg_id`, `broker_url`
|
||||
- `metadata` - Message-level metadata dictionary
|
||||
- `payloads` - List of tuples, each containing `(dataname, data, payload_type)` with deserialized payload data
|
||||
|
||||
**Process Flow:**
|
||||
1. Parse the JSON envelope to extract all fields
|
||||
2. Iterate through each payload in `payloads` list
|
||||
3. For each payload:
|
||||
- Determine transport type (`direct` or `link`)
|
||||
- If `direct`: Base64 decode the data from the message
|
||||
- If `link`: Fetch data from URL using exponential backoff (via `fileserver_download_handler`)
|
||||
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
|
||||
4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples
|
||||
|
||||
**Note:** The `fileserver_download_handler` receives `(url: str, max_retries: int, base_delay: int, max_delay: int, correlation_id: str)` and returns `bytes`.
|
||||
|
||||
## Scenario Implementations
|
||||
|
||||
### Scenario 1: Command & Control (Small Dictionary)
|
||||
@@ -752,18 +452,6 @@ async def smartreceive(
|
||||
# Send acknowledgment
|
||||
```
|
||||
|
||||
**JavaScript (Sender/Receiver):**
|
||||
```javascript
|
||||
// Create small dictionary config
|
||||
// Send via smartsend with type="dictionary"
|
||||
```
|
||||
|
||||
**Python/Micropython (Sender/Receiver):**
|
||||
```python
|
||||
# Create small dictionary config
|
||||
# Send via smartsend with type="dictionary"
|
||||
```
|
||||
|
||||
### Scenario 2: Deep Dive Analysis (Large Arrow Table)
|
||||
|
||||
**Julia (Sender/Receiver):**
|
||||
@@ -775,32 +463,8 @@ async def smartreceive(
|
||||
# Publish NATS with URL
|
||||
```
|
||||
|
||||
**JavaScript (Sender/Receiver):**
|
||||
```javascript
|
||||
// Receive NATS message with URL
|
||||
// Fetch data from HTTP server
|
||||
// Parse Arrow IPC with zero-copy
|
||||
// Load into Perspective.js or D3
|
||||
```
|
||||
|
||||
**Python/Micropython (Sender/Receiver):**
|
||||
```python
|
||||
# Create large DataFrame
|
||||
# Convert to Arrow IPC stream
|
||||
# Check size (> 1MB)
|
||||
# Upload to HTTP server
|
||||
# Publish NATS with URL
|
||||
```
|
||||
|
||||
### Scenario 3: Live Audio Processing
|
||||
|
||||
**JavaScript (Sender/Receiver):**
|
||||
```javascript
|
||||
// Capture audio chunk
|
||||
// Send as binary with metadata headers
|
||||
// Use smartsend with type="audio"
|
||||
```
|
||||
|
||||
**Julia (Sender/Receiver):**
|
||||
```julia
|
||||
# Receive audio data
|
||||
@@ -808,13 +472,6 @@ async def smartreceive(
|
||||
# Send results back (JSON + Arrow table)
|
||||
```
|
||||
|
||||
**Python/Micropython (Sender/Receiver):**
|
||||
```python
|
||||
# Capture audio chunk
|
||||
# Send as binary with metadata headers
|
||||
# Use smartsend with type="audio"
|
||||
```
|
||||
|
||||
### Scenario 4: Catch-Up (JetStream)
|
||||
|
||||
**Julia (Producer/Consumer):**
|
||||
@@ -823,22 +480,9 @@ async def smartreceive(
|
||||
# Include metadata for temporal tracking
|
||||
```
|
||||
|
||||
**JavaScript (Producer/Consumer):**
|
||||
```javascript
|
||||
// Connect to JetStream
|
||||
// Request replay from last 10 minutes
|
||||
// Process historical and real-time messages
|
||||
```
|
||||
|
||||
**Python/Micropython (Producer/Consumer):**
|
||||
```python
|
||||
# Publish to JetStream
|
||||
# Include metadata for temporal tracking
|
||||
```
|
||||
|
||||
### Scenario 5: Selection (Low Bandwidth)
|
||||
|
||||
**Focus:** Small Arrow tables, cross-platform communication. The Action: Any platform wants to send a small DataFrame to show on any receiving application for the user to choose.
|
||||
**Focus:** Small Arrow tables. The Action: Julia wants to send a small DataFrame to show on a receiving application for the user to choose.
|
||||
|
||||
**Julia (Sender/Receiver):**
|
||||
```julia
|
||||
@@ -849,30 +493,9 @@ async def smartreceive(
|
||||
# Include metadata for dashboard selection context
|
||||
```
|
||||
|
||||
**JavaScript (Sender/Receiver):**
|
||||
```javascript
|
||||
// Receive NATS message with direct transport
|
||||
// Decode Base64 payload
|
||||
// Parse Arrow IPC with zero-copy
|
||||
// Load into selection UI component (e.g., dropdown, table)
|
||||
// User makes selection
|
||||
// Send selection back to Julia
|
||||
```
|
||||
|
||||
**Python/Micropython (Sender/Receiver):**
|
||||
```python
|
||||
# Create small DataFrame (e.g., 50KB - 500KB)
|
||||
# Convert to Arrow IPC stream
|
||||
# Check payload size (< 1MB threshold)
|
||||
# Publish directly to NATS with Base64-encoded payload
|
||||
# Include metadata for dashboard selection context
|
||||
```
|
||||
|
||||
**Use Case:** Any server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to any receiving application for user selection. The selection is then sent back to the sender for processing.
|
||||
|
||||
### Scenario 6: Chat System
|
||||
|
||||
**Focus:** Every conversational message is composed of any number and any combination of components, spanning the full spectrum from small to large. This includes text, images, audio, video, tables, and files—specifically accommodating everything from brief snippets to high-resolution images, large audio files, extensive tables, and massive documents. Support for claim-check delivery and full bi-directional messaging across all platforms.
|
||||
**Focus:** Every conversational message is composed of any number and any combination of components, spanning the full spectrum from small to large. This includes text, images, audio, video, tables, and files—specifically accommodating everything from brief snippets to high-resolution images, large audio files, extensive tables, and massive documents. Support for claim-check delivery and full bi-directional messaging.
|
||||
|
||||
**Multi-Payload Support:** The system supports mixed-payload messages where a single message can contain multiple payloads with different transport strategies. The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type.
|
||||
|
||||
@@ -894,42 +517,7 @@ async def smartreceive(
|
||||
# Support bidirectional messaging with replyTo fields
|
||||
```
|
||||
|
||||
**JavaScript (Sender/Receiver):**
|
||||
```javascript
|
||||
// Build chat message with mixed content:
|
||||
// - User input text: direct transport
|
||||
// - Selected image: check size, use appropriate transport
|
||||
// - Audio recording: link transport for large files
|
||||
// - File attachment: link transport
|
||||
//
|
||||
// Parse received message:
|
||||
// - Direct payloads: decode Base64
|
||||
// - Link payloads: fetch from HTTP with exponential backoff
|
||||
// - Deserialize all payloads appropriately
|
||||
//
|
||||
// Render mixed content in chat interface
|
||||
// Support bidirectional reply with claim-check delivery confirmation
|
||||
```
|
||||
|
||||
**Python/Micropython (Sender/Receiver):**
|
||||
```python
|
||||
# Build chat message with mixed payloads:
|
||||
# - Text: direct transport (Base64)
|
||||
# - Small images: direct transport (Base64)
|
||||
# - Large images: link transport (HTTP URL)
|
||||
# - Audio/video: link transport (HTTP URL)
|
||||
# - Tables: direct or link depending on size
|
||||
# - Files: link transport (HTTP URL)
|
||||
#
|
||||
# Each payload uses appropriate transport strategy:
|
||||
# - Size < 1MB → direct (NATS + Base64)
|
||||
# - Size >= 1MB → link (HTTP upload + NATS URL)
|
||||
#
|
||||
# Include claim-check metadata for delivery tracking
|
||||
# Support bidirectional messaging with replyTo fields
|
||||
```
|
||||
|
||||
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components across all platforms.
|
||||
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components.
|
||||
|
||||
**Implementation Note:** The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. See the standard API format in Section 1: `msg_envelope_v1` supports `Vector{msg_payload_v1}` for multiple payloads.
|
||||
|
||||
|
||||
@@ -2,22 +2,17 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the implementation of the high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
This document describes the implementation of the high-performance, bi-directional data bridge for **Julia** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
|
||||
The system enables seamless communication across all three platforms:
|
||||
- **Julia ↔ JavaScript** bi-directional messaging
|
||||
- **JavaScript ↔ Python/Micropython** bi-directional messaging
|
||||
- **Julia ↔ Python/Micropython** bi-directional messaging (via JSON serialization)
|
||||
The system enables seamless communication for Julia applications.
|
||||
|
||||
### Implementation Files
|
||||
|
||||
NATSBridge is implemented in three languages, each providing the same API:
|
||||
NATSBridge is implemented in Julia:
|
||||
|
||||
| Language | Implementation File | Description |
|
||||
|----------|---------------------|-------------|
|
||||
| **Julia** | [`src/NATSBridge.jl`](../src/NATSBridge.jl) | Full Julia implementation with Arrow IPC support |
|
||||
| **JavaScript** | [`src/NATSBridge.js`](../src/NATSBridge.js) | JavaScript implementation for Node.js and browsers |
|
||||
| **Python/Micropython** | [`src/nats_bridge.py`](../src/nats_bridge.py) | Python implementation for desktop and microcontrollers |
|
||||
|
||||
### File Server Handler Architecture
|
||||
|
||||
@@ -117,64 +112,9 @@ env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff, max_ret
|
||||
# env is a dictionary containing envelope metadata and payloads field
|
||||
```
|
||||
|
||||
## Cross-Platform Interoperability
|
||||
|
||||
NATSBridge is designed for seamless communication between Julia, JavaScript, and Python/Micropython applications. All three implementations share the same interface and data format, ensuring compatibility across platforms.
|
||||
|
||||
### Platform-Specific Features
|
||||
|
||||
| Feature | Julia | JavaScript | Python/Micropython |
|
||||
|---------|-------|------------|-------------------|
|
||||
| Direct NATS transport | ✅ | ✅ | ✅ |
|
||||
| HTTP file server (Claim-Check) | ✅ | ✅ | ✅ |
|
||||
| Arrow IPC tables | ✅ | ✅ | ✅ |
|
||||
| Base64 encoding | ✅ | ✅ | ✅ |
|
||||
| Exponential backoff | ✅ | ✅ | ✅ |
|
||||
| Correlation ID tracking | ✅ | ✅ | ✅ |
|
||||
| Reply-to support | ✅ | ✅ | ✅ |
|
||||
|
||||
### Data Type Mapping
|
||||
|
||||
| Type | Julia | JavaScript | Python/Micropython |
|
||||
|------|-------|------------|-------------------|
|
||||
| `text` | `String` | `String` | `str` |
|
||||
| `dictionary` | `Dict` | `Object` | `dict` |
|
||||
| `table` | `DataFrame` | `Array<Object>` | `DataFrame` / `list` |
|
||||
| `image` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
|
||||
| `audio` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
|
||||
| `video` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
|
||||
| `binary` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
|
||||
|
||||
### Example: Julia ↔ Python ↔ JavaScript
|
||||
|
||||
```julia
|
||||
# Julia sender - smartsend returns (env, env_json_str)
|
||||
using NATSBridge
|
||||
data = [("message", "Hello from Julia!", "text")]
|
||||
env, env_json_str = smartsend("/cross_platform", data, broker_url="nats://localhost:4222")
|
||||
# env: msg_envelope_v1 with all metadata and payloads
|
||||
# env_json_str: JSON string for publishing
|
||||
```
|
||||
|
||||
```javascript
|
||||
// JavaScript receiver
|
||||
const { smartreceive } = require('./src/NATSBridge');
|
||||
const env = await smartreceive(msg);
|
||||
// env.payloads[0].data === "Hello from Julia!"
|
||||
```
|
||||
|
||||
```python
|
||||
# Python sender
|
||||
from nats_bridge import smartsend
|
||||
data = [("response", "Hello from Python!", "text")]
|
||||
smartsend("/cross_platform", data, broker_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
All three platforms can communicate seamlessly using the same NATS subjects and data format.
|
||||
|
||||
## Architecture
|
||||
|
||||
All three implementations (Julia, JavaScript, Python/Micropython) follow the same Claim-Check pattern:
|
||||
The Julia implementation follows the Claim-Check pattern:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
@@ -227,24 +167,6 @@ The Julia implementation provides:
|
||||
- **[`smartsend()`](src/NATSBridge.jl)**: Handles transport selection based on payload size
|
||||
- **[`smartreceive()`](src/NATSBridge.jl)**: Handles both direct and link transport
|
||||
|
||||
### JavaScript Module: [`src/NATSBridge.js`](../src/NATSBridge.js)
|
||||
|
||||
The JavaScript implementation provides:
|
||||
|
||||
- **`MessageEnvelope` class**: For the unified JSON envelope
|
||||
- **`MessagePayload` class**: For individual payload representation
|
||||
- **[`smartsend()`](src/NATSBridge.js)**: Handles transport selection based on payload size
|
||||
- **[`smartreceive()`](src/NATSBridge.js)**: Handles both direct and link transport
|
||||
|
||||
### Python/Micropython Module: [`src/nats_bridge.py`](../src/nats_bridge.py)
|
||||
|
||||
The Python/Micropython implementation provides:
|
||||
|
||||
- **`MessageEnvelope` class**: For the unified JSON envelope
|
||||
- **`MessagePayload` class**: For individual payload representation
|
||||
- **[`smartsend()`](src/nats_bridge.py)**: Handles transport selection based on payload size
|
||||
- **[`smartreceive()`](src/nats_bridge.py)**: Handles both direct and link transport
|
||||
|
||||
## Installation
|
||||
|
||||
### Julia Dependencies
|
||||
@@ -259,29 +181,6 @@ Pkg.add("UUIDs")
|
||||
Pkg.add("Dates")
|
||||
```
|
||||
|
||||
### JavaScript Dependencies
|
||||
|
||||
```bash
|
||||
npm install nats.js apache-arrow uuid base64-url
|
||||
```
|
||||
|
||||
### Python/Micropython Dependencies
|
||||
|
||||
1. Copy [`src/nats_bridge.py`](../src/nats_bridge.py) to your device
|
||||
2. Ensure you have the following dependencies:
|
||||
|
||||
**For Python (desktop):**
|
||||
```bash
|
||||
pip install nats-py
|
||||
```
|
||||
|
||||
**For Micropython:**
|
||||
- `urequests` for HTTP requests
|
||||
- `base64` for base64 encoding (built-in)
|
||||
- `json` for JSON handling (built-in)
|
||||
- `socket` for networking (built-in)
|
||||
- `uuid` for UUID generation (built-in)
|
||||
|
||||
## Usage Tutorial
|
||||
|
||||
### Step 1: Start NATS Server
|
||||
@@ -304,48 +203,21 @@ python3 -m http.server 8080 --directory /tmp/fileserver
|
||||
### Step 3: Run Test Scenarios
|
||||
|
||||
```bash
|
||||
# Scenario 1: Command & Control (JavaScript sender)
|
||||
node test/scenario1_command_control.js
|
||||
# Scenario 1: Command & Control
|
||||
julia test/scenario1_command_control.jl
|
||||
|
||||
# Scenario 2: Large Arrow Table (JavaScript sender)
|
||||
node test/scenario2_large_table.js
|
||||
# Scenario 2: Large Arrow Table
|
||||
julia test/scenario2_large_table.jl
|
||||
|
||||
# Scenario 3: Julia-to-Julia communication
|
||||
# Run both Julia and JavaScript versions
|
||||
julia test/scenario3_julia_to_julia.jl
|
||||
node test/scenario3_julia_to_julia.js
|
||||
```
|
||||
|
||||
## API Consistency Across Languages
|
||||
|
||||
**High-Level API (Consistent Across All Languages):**
|
||||
- `smartsend(subject, data, ...)` - Main publishing function
|
||||
- `smartreceive(msg, ...)` - Main receiving function
|
||||
- Message envelope structure (`msg_envelope_v1` / `MessageEnvelope`)
|
||||
- Payload structure (`msg_payload_v1` / `MessagePayload`)
|
||||
- Transport strategy (direct vs link based on size threshold)
|
||||
- Supported payload types: text, dictionary, table, image, audio, video, binary
|
||||
|
||||
**Low-Level Native Functions (Language-Specific Conventions):**
|
||||
- Julia: `NATS.connect()`, `publish_message()`, function overloading
|
||||
- JavaScript: `nats.js` client, native async/await patterns
|
||||
- Python: `nats-python` client, native async/await patterns
|
||||
|
||||
**Connection Reuse Pattern - Key Differences:**
|
||||
- **Julia:** Uses `NATS_connection` keyword parameter with function overloading for automatic connection management
|
||||
- **JavaScript/Python:** Achieved by creating NATS client outside the function and reusing it in custom handlers or custom publish implementations
|
||||
|
||||
**Why the Difference?**
|
||||
- Julia supports function overloading and keyword arguments, allowing `NATS_connection` to be passed as an optional parameter
|
||||
- JavaScript/Python use a simpler `is_publish` option to control automatic publishing
|
||||
- `is_publish` is simply a switch: when `true`, publish automatically; when `false`, return `(env, env_json_str)` without publishing
|
||||
- For connection reuse in JavaScript/Python, create a NATS client once and reuse it in your custom `fileserver_upload_handler` or custom publish logic
|
||||
|
||||
## Usage
|
||||
|
||||
### Scenario 1: Command & Control (Small Dictionary)
|
||||
|
||||
**Focus:** Sending small dictionary configurations across platforms. This is the simplest use case for command and control scenarios.
|
||||
**Focus:** Sending small dictionary configurations. This is the simplest use case for command and control scenarios.
|
||||
|
||||
**Julia (Sender/Receiver):**
|
||||
```julia
|
||||
@@ -386,98 +258,11 @@ NATS.close(conn)
|
||||
|
||||
**Use Case:** High-frequency publishing scenarios where connection reuse provides performance benefits by avoiding the overhead of establishing a new NATS connection for each message.
|
||||
|
||||
**JavaScript (Sender/Receiver):**
|
||||
```javascript
|
||||
const { smartsend } = require('./src/NATSBridge');
|
||||
|
||||
// Create small dictionary config
|
||||
// Send via smartsend with type="dictionary"
|
||||
const config = {
|
||||
step_size: 0.01,
|
||||
iterations: 1000,
|
||||
threshold: 0.5
|
||||
};
|
||||
|
||||
// Use is_publish option to control automatic publishing
|
||||
await smartsend("control", [
|
||||
{ dataname: "config", data: config, type: "dictionary" }
|
||||
], {
|
||||
is_publish: true // Automatically publish to NATS
|
||||
});
|
||||
```
|
||||
|
||||
**Connection Reuse in JavaScript:**
|
||||
To achieve connection reuse in JavaScript, create a NATS client outside the function and use it in a custom `fileserver_upload_handler` or custom publish implementation:
|
||||
|
||||
```javascript
|
||||
const { connect } = require('nats');
|
||||
const { smartsend } = require('./src/NATSBridge');
|
||||
|
||||
// Create connection once
|
||||
const nc = await connect({ servers: ['nats://localhost:4222'] });
|
||||
|
||||
// Send multiple messages using the same connection
|
||||
for (let i = 0; i < 100; i++) {
|
||||
const config = { iteration: i, data: Math.random() };
|
||||
|
||||
// Option 1: Use is_publish=false and publish manually with your connection
|
||||
const { env, env_json_str } = await smartsend("control", [
|
||||
{ dataname: "config", data: config, type: "dictionary" }
|
||||
], { is_publish: false });
|
||||
|
||||
// Publish with your existing connection
|
||||
await nc.publish("control", env_json_str);
|
||||
}
|
||||
|
||||
// Close connection when done
|
||||
await nc.close();
|
||||
```
|
||||
|
||||
**Python/Micropython (Sender/Receiver):**
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
# Create small dictionary config
|
||||
# Send via smartsend with type="dictionary"
|
||||
config = {
|
||||
"step_size": 0.01,
|
||||
"iterations": 1000,
|
||||
"threshold": 0.5
|
||||
}
|
||||
|
||||
# Use is_publish parameter to control automatic publishing
|
||||
smartsend("control", [("config", config, "dictionary")], is_publish=True)
|
||||
```
|
||||
|
||||
**Connection Reuse in Python:**
|
||||
To achieve connection reuse in Python, create a NATS client outside the function and use it in a custom `fileserver_upload_handler` or custom publish implementation:
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
import nats
|
||||
|
||||
# Create connection once
|
||||
nc = await nats.connect("nats://localhost:4222")
|
||||
|
||||
# Send multiple messages using the same connection
|
||||
for i in range(100):
|
||||
config = {"iteration": i, "data": random.random()}
|
||||
|
||||
# Option 1: Use is_publish=False and publish manually with your connection
|
||||
env, env_json_str = smartsend("control", [("config", config, "dictionary")], is_publish=False)
|
||||
|
||||
# Publish with your existing connection
|
||||
await nc.publish("control", env_json_str)
|
||||
|
||||
# Close connection when done
|
||||
await nc.close()
|
||||
```
|
||||
|
||||
### Basic Multi-Payload Example
|
||||
|
||||
#### Python/Micropython (Sender)
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
#### Julia (Sender)
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Send multiple payloads in one message (type is required per payload)
|
||||
smartsend(
|
||||
@@ -491,71 +276,15 @@ smartsend(
|
||||
smartsend("/test", [("single_data", mydata, "dictionary")], broker_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
#### Python/Micropython (Receiver)
|
||||
```python
|
||||
from nats_bridge import smartreceive
|
||||
#### Julia (Receiver)
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Receive returns a dictionary with envelope metadata and payloads field
|
||||
env = smartreceive(msg)
|
||||
# env["payloads"] = [(dataname1, data1, "dictionary"), (dataname2, data2, "table"), ...]
|
||||
```
|
||||
|
||||
#### JavaScript (Sender)
|
||||
```javascript
|
||||
const { smartsend } = require('./src/NATSBridge');
|
||||
|
||||
// Single payload wrapped in a list
|
||||
const config = [{
|
||||
dataname: "config",
|
||||
data: { step_size: 0.01, iterations: 1000 },
|
||||
type: "dictionary"
|
||||
}];
|
||||
|
||||
await smartsend("control", config, {
|
||||
correlationId: "unique-id"
|
||||
});
|
||||
|
||||
// Multiple payloads
|
||||
const configs = [
|
||||
{
|
||||
dataname: "config1",
|
||||
data: { step_size: 0.01 },
|
||||
type: "dictionary"
|
||||
},
|
||||
{
|
||||
dataname: "config2",
|
||||
data: { iterations: 1000 },
|
||||
type: "dictionary"
|
||||
}
|
||||
];
|
||||
|
||||
await smartsend("control", configs);
|
||||
```
|
||||
|
||||
#### JavaScript (Receiver)
|
||||
```javascript
|
||||
const { smartreceive } = require('./src/NATSBridge');
|
||||
|
||||
// Subscribe to messages
|
||||
const nc = await connect({ servers: ['nats://localhost:4222'] });
|
||||
const sub = nc.subscribe("control");
|
||||
|
||||
for await (const msg of sub) {
|
||||
const env = await smartreceive(msg);
|
||||
|
||||
// Process the payloads from the envelope
|
||||
for (const payload of env.payloads) {
|
||||
const { dataname, data, type } = payload;
|
||||
console.log(`Received ${dataname} of type ${type}`);
|
||||
console.log(`Data: ${JSON.stringify(data)}`);
|
||||
}
|
||||
|
||||
// Also access envelope metadata
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
}
|
||||
```
|
||||
|
||||
### Scenario 2: Deep Dive Analysis (Large Arrow Table)
|
||||
|
||||
#### Julia (Sender)
|
||||
@@ -689,91 +418,25 @@ env, env_json_str = smartsend(
|
||||
|
||||
**API Consistency Note:**
|
||||
- **Julia:** Uses `NATS_connection` keyword parameter with function overloading for automatic connection management
|
||||
- **JavaScript/Python:** Use `is_publish` option and achieve connection reuse by creating NATS client outside the function and reusing it in custom handlers or custom publish implementations
|
||||
|
||||
#### JavaScript (Receiver)
|
||||
```javascript
|
||||
const { smartreceive } = require('./src/NATSBridge');
|
||||
|
||||
const env = await smartreceive(msg);
|
||||
|
||||
// Use table data from the payloads field
|
||||
// Note: Tables are sent as arrays of objects in JavaScript
|
||||
const table = env.payloads;
|
||||
```
|
||||
|
||||
### Scenario 3: Live Binary Processing
|
||||
|
||||
#### Python/Micropython (Sender)
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
**Julia (Sender/Receiver):**
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Binary data wrapped in list with type
|
||||
smartsend(
|
||||
"binary_input",
|
||||
[("audio_chunk", binary_buffer, "binary")],
|
||||
broker_url="nats://localhost:4222",
|
||||
metadata={"sample_rate": 44100, "channels": 1}
|
||||
metadata=["sample_rate" => 44100, "channels" => 1]
|
||||
)
|
||||
```
|
||||
|
||||
#### JavaScript (Sender)
|
||||
```javascript
|
||||
const { smartsend } = require('./src/NATSBridge');
|
||||
|
||||
// Binary data wrapped in a list
|
||||
const binaryData = [{
|
||||
dataname: "audio_chunk",
|
||||
data: binaryBuffer, // ArrayBuffer or Uint8Array
|
||||
type: "binary"
|
||||
}];
|
||||
|
||||
await smartsend("binary_input", binaryData, {
|
||||
metadata: {
|
||||
sample_rate: 44100,
|
||||
channels: 1
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
#### Python/Micropython (Receiver)
|
||||
```python
|
||||
from nats_bridge import smartreceive
|
||||
|
||||
# Receive binary data
|
||||
def process_binary(msg):
|
||||
env = smartreceive(msg)
|
||||
|
||||
# Process the binary data from env.payloads
|
||||
for dataname, data, type in env["payloads"]:
|
||||
if type == "binary":
|
||||
# data is bytes
|
||||
print(f"Received binary data: {dataname}, size: {len(data)}")
|
||||
# Perform FFT or AI transcription here
|
||||
```
|
||||
|
||||
#### JavaScript (Receiver)
|
||||
```javascript
|
||||
const { smartreceive } = require('./src/NATSBridge');
|
||||
|
||||
// Receive binary data
|
||||
function process_binary(msg) {
|
||||
const env = await smartreceive(msg);
|
||||
|
||||
// Process the binary data from env.payloads
|
||||
for (const payload of env.payloads) {
|
||||
if (payload.type === "binary") {
|
||||
// data is an ArrayBuffer or Uint8Array
|
||||
console.log(`Received binary data: ${payload.dataname}, size: ${payload.data.length}`);
|
||||
// Perform FFT or AI transcription here
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Scenario 4: Catch-Up (JetStream)
|
||||
|
||||
#### Julia (Producer)
|
||||
**Julia (Producer/Consumer):**
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
@@ -789,93 +452,11 @@ function publish_health_status(broker_url)
|
||||
end
|
||||
```
|
||||
|
||||
#### JavaScript (Consumer)
|
||||
```javascript
|
||||
const { connect } = require('nats');
|
||||
const { smartreceive } = require('./src/NATSBridge');
|
||||
|
||||
const nc = await connect({ servers: ['nats://localhost:4222'] });
|
||||
const js = nc.jetstream();
|
||||
|
||||
// Request replay from last 10 minutes
|
||||
const consumer = await js.pullSubscribe("health", {
|
||||
durable_name: "catchup",
|
||||
max_batch: 100,
|
||||
max_ack_wait: 30000
|
||||
});
|
||||
|
||||
// Process historical and real-time messages
|
||||
for await (const msg of consumer) {
|
||||
const env = await smartreceive(msg);
|
||||
// env.payloads contains the list of payloads
|
||||
// Each payload has: dataname, data, type
|
||||
msg.ack();
|
||||
}
|
||||
```
|
||||
|
||||
### Scenario 4: Micropython Device Control
|
||||
|
||||
**Focus:** Sending configuration to a Micropython device over NATS. This demonstrates the lightweight nature of the Python implementation suitable for microcontrollers.
|
||||
|
||||
**Python/Micropython (Receiver/Device):**
|
||||
```python
|
||||
from nats_bridge import smartsend, smartreceive
|
||||
import json
|
||||
|
||||
# Device configuration handler
|
||||
def handle_device_config(msg):
|
||||
env = smartreceive(msg)
|
||||
|
||||
# Process configuration from payloads
|
||||
for dataname, data, payload_type in env["payloads"]:
|
||||
if payload_type == "dictionary":
|
||||
print(f"Received configuration: {data}")
|
||||
# Apply configuration to device
|
||||
if "wifi_ssid" in data:
|
||||
wifi_ssid = data["wifi_ssid"]
|
||||
wifi_password = data["wifi_password"]
|
||||
update_wifi_config(wifi_ssid, wifi_password)
|
||||
|
||||
# Send confirmation back
|
||||
config = {
|
||||
"status": "configured",
|
||||
"wifi_ssid": "MyNetwork",
|
||||
"ip": get_device_ip()
|
||||
}
|
||||
smartsend(
|
||||
"device/response",
|
||||
[("config", config, "dictionary")],
|
||||
broker_url="nats://localhost:4222",
|
||||
reply_to=env.get("reply_to")
|
||||
)
|
||||
```
|
||||
|
||||
**JavaScript (Sender/Controller):**
|
||||
```javascript
|
||||
const { smartsend } = require('./src/NATSBridge');
|
||||
|
||||
// Send configuration to Micropython device
|
||||
await smartsend("device/config", [
|
||||
{
|
||||
dataname: "config",
|
||||
data: {
|
||||
wifi_ssid: "MyNetwork",
|
||||
wifi_password: "password123",
|
||||
update_interval: 60,
|
||||
temperature_threshold: 30.0
|
||||
},
|
||||
type: "dictionary"
|
||||
}
|
||||
]);
|
||||
```
|
||||
|
||||
**Use Case:** A controller sends WiFi and operational configuration to a Micropython device (e.g., ESP32). The device receives the configuration, applies it, and sends back a confirmation with its current status.
|
||||
|
||||
### Scenario 5: Selection (Low Bandwidth)
|
||||
|
||||
**Focus:** Small Arrow tables, Julia to JavaScript. The Action: Julia wants to send a small DataFrame to show on a JavaScript dashboard for the user to choose.
|
||||
**Focus:** Small Arrow tables. The Action: Julia wants to send a small DataFrame to show on a receiving application for the user to choose.
|
||||
|
||||
**Julia (Sender):**
|
||||
**Julia (Sender/Receiver):**
|
||||
```julia
|
||||
using NATSBridge
|
||||
using DataFrames
|
||||
@@ -903,27 +484,7 @@ env, env_json_str = smartsend(
|
||||
# env_json_str: JSON string for publishing
|
||||
```
|
||||
|
||||
**JavaScript (Receiver):**
|
||||
```javascript
|
||||
const { smartreceive, smartsend } = require('./src/NATSBridge');
|
||||
|
||||
// Receive NATS message with direct transport
|
||||
const env = await smartreceive(msg);
|
||||
|
||||
// Decode Base64 payload (for direct transport)
|
||||
// For tables, data is in env.payloads
|
||||
const table = env.payloads; // Array of objects
|
||||
|
||||
// User makes selection
|
||||
const selection = uiComponent.getSelectedOption();
|
||||
|
||||
// Send selection back to Julia
|
||||
await smartsend("dashboard.response", [
|
||||
{ dataname: "selected_option", data: selection, type: "dictionary" }
|
||||
]);
|
||||
```
|
||||
|
||||
**Use Case:** Julia server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to JavaScript dashboard for user selection. The selection is then sent back to Julia for processing.
|
||||
**Use Case:** Julia server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to a receiving application for user selection. The selection is then sent back to Julia for processing.
|
||||
|
||||
### Scenario 6: Chat System
|
||||
|
||||
@@ -968,46 +529,6 @@ env, env_json_str = smartsend(
|
||||
# env_json_str: JSON string for publishing
|
||||
```
|
||||
|
||||
**JavaScript (Sender/Receiver):**
|
||||
```javascript
|
||||
const { smartsend, smartreceive } = require('./src/NATSBridge');
|
||||
|
||||
// Build chat message with mixed content:
|
||||
// - User input text: direct transport
|
||||
// - Selected image: check size, use appropriate transport
|
||||
// - Audio recording: link transport for large files
|
||||
// - File attachment: link transport
|
||||
//
|
||||
// Parse received message:
|
||||
// - Direct payloads: decode Base64
|
||||
// - Link payloads: fetch from HTTP with exponential backoff
|
||||
// - Deserialize all payloads appropriately
|
||||
//
|
||||
// Render mixed content in chat interface
|
||||
// Support bidirectional reply with claim-check delivery confirmation
|
||||
|
||||
// Example: Send chat with mixed content
|
||||
const message = [
|
||||
{
|
||||
dataname: "text",
|
||||
data: "Hello from JavaScript!",
|
||||
type: "text"
|
||||
},
|
||||
{
|
||||
dataname: "image",
|
||||
data: selectedImageBuffer, // Small image (ArrayBuffer or Uint8Array)
|
||||
type: "image"
|
||||
},
|
||||
{
|
||||
dataname: "audio",
|
||||
data: audioUrl, // Large audio, link transport
|
||||
type: "audio"
|
||||
}
|
||||
];
|
||||
|
||||
await smartsend("chat.room123", message);
|
||||
```
|
||||
|
||||
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components.
|
||||
|
||||
**Implementation Note:** The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. See the standard API format in Section 1: `msg_envelope_v1` supports `Vector{msg_payload_v1}` for multiple payloads.
|
||||
@@ -1072,7 +593,6 @@ await smartsend("chat.room123", message);
|
||||
### Exponential Backoff
|
||||
- Maximum retry count: 5
|
||||
- Base delay: 100ms, max delay: 5000ms
|
||||
- Implemented in all three implementations (Julia, JavaScript, Python/Micropython)
|
||||
|
||||
### Correlation ID Logging
|
||||
- Log correlation_id at every stage
|
||||
@@ -1081,38 +601,7 @@ await smartsend("chat.room123", message);
|
||||
|
||||
## Testing
|
||||
|
||||
Run the test scripts for each platform:
|
||||
|
||||
### Python/Micropython Tests
|
||||
|
||||
```bash
|
||||
# Basic functionality test
|
||||
python test/test_micropython_basic.py
|
||||
```
|
||||
|
||||
### JavaScript Tests
|
||||
|
||||
```bash
|
||||
# Text message exchange
|
||||
node test/test_js_to_js_text_sender.js
|
||||
node test/test_js_to_js_text_receiver.js
|
||||
|
||||
# Dictionary exchange
|
||||
node test/test_js_to_js_dict_sender.js
|
||||
node test/test_js_to_js_dict_receiver.js
|
||||
|
||||
# File transfer (direct transport)
|
||||
node test/test_js_to_js_file_sender.js
|
||||
node test/test_js_to_js_file_receiver.js
|
||||
|
||||
# Mixed payload types
|
||||
node test/test_js_to_js_mix_payloads_sender.js
|
||||
node test/test_js_to_js_mix_payloads_receiver.js
|
||||
|
||||
# Table (Arrow IPC) exchange
|
||||
node test/test_js_to_js_table_sender.js
|
||||
node test/test_js_to_js_table_receiver.js
|
||||
```
|
||||
Run the test scripts for Julia:
|
||||
|
||||
### Julia Tests
|
||||
|
||||
@@ -1138,41 +627,22 @@ julia test/test_julia_to_julia_table_sender.jl
|
||||
julia test/test_julia_to_julia_table_receiver.jl
|
||||
```
|
||||
|
||||
### Cross-Platform Tests
|
||||
|
||||
```bash
|
||||
# Julia ↔ JavaScript communication
|
||||
julia test/test_julia_to_julia_text_sender.jl
|
||||
node test/test_js_to_js_text_receiver.js
|
||||
|
||||
# Python ↔ JavaScript communication
|
||||
python test/test_micropython_basic.py
|
||||
node test/test_js_to_js_text_receiver.js
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **NATS Connection Failed**
|
||||
- **Julia/JavaScript/Python**: Ensure NATS server is running
|
||||
- **Python/Micropython**: Check `nats_url` parameter and network connectivity
|
||||
- Ensure NATS server is running
|
||||
|
||||
2. **HTTP Upload Failed**
|
||||
- Ensure file server is running
|
||||
- Check `fileserver_url` configuration
|
||||
- Verify upload permissions
|
||||
- **Micropython**: Ensure `urequests` is available and network is connected
|
||||
|
||||
3. **Arrow IPC Deserialization Error**
|
||||
- Ensure data is properly serialized to Arrow format
|
||||
- Check Arrow version compatibility
|
||||
|
||||
4. **Python/Micropython Specific Issues**
|
||||
- **Import Error**: Ensure `nats_bridge.py` is in the correct path
|
||||
- **Memory Error (Micropython)**: Reduce payload size or use link transport for large payloads
|
||||
- **Unicode Error**: Ensure proper encoding when sending text data
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
Reference in New Issue
Block a user