Compare commits
20 Commits
v0.4.4
...
7205cc1ea3
| Author | SHA1 | Date | |
|---|---|---|---|
| 7205cc1ea3 | |||
| aa7cdbd36f | |||
| 1b86a9252d | |||
| e9fd148235 | |||
| 34ea1ed8ec | |||
| aa92fb6d0d | |||
| fbbea7b42b | |||
| b2859710cd | |||
| bc0ce7159c | |||
| 4614f99358 | |||
| 1ecc55f8aa | |||
| ae0f24ccb2 | |||
| 060c68cd05 | |||
| e85eba4cea | |||
| 206467e1fa | |||
| a98394b9b9 | |||
| c448811aa9 | |||
| c3225a90c7 | |||
| 89acf780bf | |||
| e5f4793370 |
@@ -51,3 +51,15 @@ Ecosystem Variance: Low-level native functions (e.g., NATS.connect(), JSON.read(
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
I'm expanding this Julia package (NATSBridge) into a cross-platform project by adding a JavaScript and Python/MicroPython implementation. To ensure accuracy, the Julia src directory will serve as the ground truth, as the documentation may be outdated.
|
||||
|
||||
My goal is to maintain interface parity at the high-level API for a consistent user experience, while ensuring the low-level implementation adheres strictly to the idiomatic conventions of each respective language (e.g., multiple dispatch in Julia vs. asynchronous, prototype, or class-based patterns in JS and Python/MicroPython)
|
||||
|
||||
Now, help me do the following:
|
||||
1) check architecture.md for any mistake.
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
name = "NATSBridge"
|
||||
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
|
||||
version = "0.4.4"
|
||||
version = "0.4.5"
|
||||
authors = ["narawat <narawat@gmail.com>"]
|
||||
|
||||
[deps]
|
||||
|
||||
717
README.md
717
README.md
@@ -1,6 +1,6 @@
|
||||
# NATSBridge
|
||||
# NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||
|
||||
A high-performance, bi-directional data bridge for **Julia** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
A high-performance, bi-directional data bridge for **Julia, JavaScript, Python, and MicroPython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://nats.io)
|
||||
@@ -10,6 +10,7 @@ A high-performance, bi-directional data bridge for **Julia** applications using
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Cross-Platform Support](#cross-platform-support)
|
||||
- [Features](#features)
|
||||
- [Architecture](#architecture)
|
||||
- [Installation](#installation)
|
||||
@@ -17,7 +18,7 @@ A high-performance, bi-directional data bridge for **Julia** applications using
|
||||
- [API Reference](#api-reference)
|
||||
- [Payload Types](#payload-types)
|
||||
- [Transport Strategies](#transport-strategies)
|
||||
- [Examples](#examples)
|
||||
- [Cross-Platform Examples](#cross-platform-examples)
|
||||
- [Testing](#testing)
|
||||
- [License](#license)
|
||||
|
||||
@@ -25,7 +26,7 @@ A high-performance, bi-directional data bridge for **Julia** applications using
|
||||
|
||||
## Overview
|
||||
|
||||
NATSBridge enables seamless communication for Julia applications through NATS, with intelligent transport selection based on payload size:
|
||||
NATSBridge enables seamless communication across multiple platforms through NATS, with intelligent transport selection based on payload size:
|
||||
|
||||
| Transport | Payload Size | Method |
|
||||
|-----------|--------------|--------|
|
||||
@@ -36,14 +37,40 @@ NATSBridge enables seamless communication for Julia applications through NATS, w
|
||||
|
||||
- **Chat Applications**: Text, images, audio, video in a single message
|
||||
- **File Transfer**: Efficient transfer of large files using claim-check pattern
|
||||
- **Streaming Data**: Sensor data, telemetry, and analytics pipelines
|
||||
- **IoT/Embedded**: Sensor data, telemetry, and analytics pipelines (MicroPython)
|
||||
- **Cross-Platform Communication**: Interoperability between Julia, JavaScript, Python, and MicroPython systems
|
||||
|
||||
---
|
||||
|
||||
## Cross-Platform Support
|
||||
|
||||
| Platform | Implementation | Features |
|
||||
|----------|----------------|----------|
|
||||
| **Julia** | [`src/NATSBridge.jl`](src/NATSBridge.jl) | Full feature set, Arrow IPC, multiple dispatch |
|
||||
| **JavaScript** | [`src/natsbridge.js`](src/natsbridge.js) | Node.js & browser, async/await |
|
||||
| **Python** | [`src/natsbridge.py`](src/natsbridge.py) | Desktop Python, asyncio, type hints |
|
||||
| **MicroPython** | [`src/natsbridge_mpy.py`](src/natsbridge_mpy.py) | Memory-constrained, synchronous API |
|
||||
|
||||
### Platform Comparison
|
||||
|
||||
| Feature | Julia | JavaScript | Python | MicroPython |
|
||||
|---------|-------|------------|--------|-------------|
|
||||
| Multiple Dispatch | ✅ Native | ❌ | ❌ | ❌ |
|
||||
| Async/Await | ❌ | ✅ Native | ✅ Native | ⚠️ (uasyncio) |
|
||||
| Type Safety | ✅ Strong | ⚠️ (TypeScript) | ✅ (Type hints) | ❌ |
|
||||
| Memory Management | ✅ GC | ✅ GC | ✅ GC | ⚠️ (Manual) |
|
||||
| Arrow IPC | ✅ Native | ✅ | ✅ | ❌ |
|
||||
| Direct Transport | ✅ | ✅ | ✅ | ✅ |
|
||||
| Link Transport | ✅ | ✅ | ✅ | ⚠️ (Limited) |
|
||||
| Handler Functions | ✅ | ✅ | ✅ | ✅ |
|
||||
| Cross-Platform API | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
- ✅ **Bi-directional messaging** for Julia applications
|
||||
- ✅ **Cross-platform messaging** for Julia, JavaScript, Python, and MicroPython applications
|
||||
- ✅ **Bi-directional messaging** with request-reply patterns
|
||||
- ✅ **Multi-payload support** - send multiple payloads with different types in one message
|
||||
- ✅ **Automatic transport selection** - direct vs link based on payload size
|
||||
- ✅ **Claim-Check pattern** for payloads > 1MB
|
||||
@@ -51,8 +78,7 @@ NATSBridge enables seamless communication for Julia applications through NATS, w
|
||||
- ✅ **Exponential backoff** for reliable file server downloads
|
||||
- ✅ **Correlation ID tracking** for message tracing
|
||||
- ✅ **Reply-to support** for request-response patterns
|
||||
- ✅ **JetStream support** for message replay and durability
|
||||
|
||||
- ✅ **Handler function abstraction** - pluggable file server implementations (Plik, AWS S3, custom)
|
||||
|
||||
---
|
||||
|
||||
@@ -62,13 +88,13 @@ NATSBridge enables seamless communication for Julia applications through NATS, w
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Sender["Julia Application (Sender)"]
|
||||
subgraph Sender["Application (Sender)"]
|
||||
SenderApp[App Code]
|
||||
NATSBridge_Send[NATSBridge]
|
||||
NATS_Client[<b>NATS.jl</b>]
|
||||
end
|
||||
|
||||
subgraph Receiver["Julia Application (Receiver)"]
|
||||
subgraph Receiver["Application (Receiver)"]
|
||||
ReceiverApp[App Code]
|
||||
NATSBridge_Recv[NATSBridge]
|
||||
NATS_Client_Recv[<b>NATS.jl</b>]
|
||||
@@ -96,14 +122,6 @@ flowchart TB
|
||||
style FileServer fill:#f3e5f5
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
| Component | Description |
|
||||
|-----------|-------------|
|
||||
| **Julia Application** | Sender and receiver applications using the NATSBridge module |
|
||||
| **NATS Server** | Message broker for transporting message envelopes |
|
||||
| **HTTP File Server** | Independent HTTP server for large payload storage (e.g., Plik) |
|
||||
|
||||
### Message Flow
|
||||
|
||||
1. **Sender** creates a message envelope with payloads using `smartsend()`
|
||||
@@ -124,11 +142,53 @@ The system uses handler functions to abstract file server operations:
|
||||
|
||||
| Handler | Purpose |
|
||||
|---------|---------|
|
||||
| `plik_oneshot_upload()` | Uploads payload bytes to file server, returns URL |
|
||||
| `_fetch_with_backoff()` | Downloads data from URL with exponential backoff retry |
|
||||
| `plik_oneshot_upload()` / `plikOneshotUpload()` | Uploads payload bytes to file server, returns URL |
|
||||
| `_fetch_with_backoff()` / `fetchWithBackoff()` | Downloads data from URL with exponential backoff retry |
|
||||
|
||||
This abstraction allows support for different file server implementations (Plik, AWS S3, custom HTTP server).
|
||||
|
||||
### Message Envelope Schema
|
||||
|
||||
All platforms use identical JSON schemas for message envelopes:
|
||||
|
||||
```json
|
||||
{
|
||||
"correlation_id": "uuid-v4-string",
|
||||
"msg_id": "uuid-v4-string",
|
||||
"timestamp": "2024-01-15T10:30:00Z",
|
||||
"send_to": "topic/subject",
|
||||
"msg_purpose": "ACK | NACK | updateStatus | shutdown | chat",
|
||||
"sender_name": "agent-wine-web-frontend",
|
||||
"sender_id": "uuid4",
|
||||
"receiver_name": "agent-backend",
|
||||
"receiver_id": "uuid4",
|
||||
"reply_to": "topic",
|
||||
"reply_to_msg_id": "uuid4",
|
||||
"broker_url": "nats://localhost:4222",
|
||||
"metadata": {},
|
||||
"payloads": [
|
||||
{
|
||||
"id": "uuid4",
|
||||
"dataname": "login_image",
|
||||
"payload_type": "image",
|
||||
"transport": "direct",
|
||||
"encoding": "base64",
|
||||
"size": 15433,
|
||||
"data": "base64-encoded-string"
|
||||
},
|
||||
{
|
||||
"id": "uuid4",
|
||||
"dataname": "large_table",
|
||||
"payload_type": "table",
|
||||
"transport": "link",
|
||||
"encoding": "none",
|
||||
"size": 524288,
|
||||
"data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/data.arrow"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
@@ -138,14 +198,53 @@ This abstraction allows support for different file server implementations (Plik,
|
||||
- **NATS Server** (v2.10+ recommended)
|
||||
- **HTTP File Server** (optional, for payloads > 1MB)
|
||||
|
||||
### Julia
|
||||
### Platform-Specific Dependencies
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using Pkg
|
||||
Pkg.add("NATS")
|
||||
Pkg.add("https://git.yiem.cc/ton/NATSBridge")
|
||||
Pkg.add("Arrow")
|
||||
Pkg.add("JSON3")
|
||||
Pkg.add("HTTP")
|
||||
Pkg.add("UUIDs")
|
||||
Pkg.add("Dates")
|
||||
```
|
||||
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```bash
|
||||
npm install nats uuid apache-arrow node-fetch
|
||||
# or
|
||||
yarn add nats uuid apache-arrow node-fetch
|
||||
```
|
||||
|
||||
#### JavaScript (Browser)
|
||||
|
||||
```bash
|
||||
npm install nats uuid apache-arrow
|
||||
# or use CDN:
|
||||
# https://unpkg.com/nats-js/dist/bundle/nats.min.js
|
||||
# https://unpkg.com/apache-arrow/arrow.min.js
|
||||
```
|
||||
|
||||
#### Python (Desktop)
|
||||
|
||||
```bash
|
||||
pip install nats-py aiohttp pyarrow pandas python-dateutil
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
MicroPython uses built-in modules:
|
||||
- `network` - NATS connection (custom implementation)
|
||||
- `time` - Timestamps
|
||||
- `uos` - File operations
|
||||
- `base64` - Base64 encoding
|
||||
- `json` - JSON parsing
|
||||
- `struct` - Binary data handling
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
@@ -166,61 +265,39 @@ mkdir -p /tmp/fileserver
|
||||
python3 -m http.server 8080 --directory /tmp/fileserver
|
||||
```
|
||||
|
||||
### Step 3: Send Your First Message
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Send a text message
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; broker_url="nats://localhost:4222")
|
||||
println("Message sent!")
|
||||
```
|
||||
|
||||
### Step 4: Receive Messages
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATS, NATSBridge
|
||||
|
||||
# Configuration
|
||||
const SUBJECT = "/chat/room1"
|
||||
const NATS_URL = "nats://localhost:4222"
|
||||
|
||||
# Helper: Log with correlation ID
|
||||
function log_trace(message)
|
||||
timestamp = Dates.now()
|
||||
println("[$timestamp] $message")
|
||||
end
|
||||
|
||||
# Receiver: Listen for messages - msg comes from the callback
|
||||
function test_receive()
|
||||
conn = NATS.connect(NATS_URL)
|
||||
NATS.subscribe(conn, SUBJECT) do msg
|
||||
log_trace("Received message on $(msg.subject)")
|
||||
|
||||
# Receive and process message
|
||||
env, env_json_str = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
println("Received $dataname: $data")
|
||||
end
|
||||
end
|
||||
|
||||
# Keep listening for 120 seconds
|
||||
sleep(120)
|
||||
NATS.drain(conn)
|
||||
end
|
||||
|
||||
test_receive()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Unified API Standard
|
||||
|
||||
All platforms use the same input/output format for payloads:
|
||||
|
||||
**Input format for smartsend:**
|
||||
```
|
||||
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||
```
|
||||
|
||||
**Output format for smartreceive:**
|
||||
```
|
||||
{
|
||||
"correlation_id": "...",
|
||||
"msg_id": "...",
|
||||
"timestamp": "...",
|
||||
"send_to": "...",
|
||||
"msg_purpose": "...",
|
||||
"sender_name": "...",
|
||||
"sender_id": "...",
|
||||
"receiver_name": "...",
|
||||
"receiver_id": "...",
|
||||
"reply_to": "...",
|
||||
"reply_to_msg_id": "...",
|
||||
"broker_url": "...",
|
||||
"metadata": {...},
|
||||
"payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||
}
|
||||
```
|
||||
|
||||
### smartsend
|
||||
|
||||
Sends data either directly via NATS or via a fileserver URL, depending on payload size.
|
||||
@@ -231,27 +308,96 @@ Sends data either directly via NATS or via a fileserver URL, depending on payloa
|
||||
using NATSBridge
|
||||
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
subject, # NATS subject
|
||||
subject::String, # NATS subject
|
||||
data::AbstractArray{Tuple{String, Any, String}}; # List of (dataname, data, type)
|
||||
broker_url::String = "nats://localhost:4222",
|
||||
fileserver_url = "http://localhost:8080",
|
||||
fileserver_upload_handler::Function = plik_oneshot_upload,
|
||||
size_threshold::Int = 1_000_000,
|
||||
correlation_id::String = string(uuid4()), # Correlation ID for tracing (auto-generated UUID)
|
||||
correlation_id::String = string(uuid4()),
|
||||
msg_purpose::String = "chat",
|
||||
sender_name::String = "NATSBridge",
|
||||
receiver_name::String = "",
|
||||
receiver_id::String = "",
|
||||
reply_to::String = "",
|
||||
reply_to_msg_id::String = "",
|
||||
is_publish::Bool = true, # Whether to automatically publish to NATS
|
||||
NATS_connection::Union{NATS.Connection, Nothing} = nothing, # Pre-existing NATS connection (optional, saves connection overhead)
|
||||
msg_id::String = string(uuid4()), # Message ID (auto-generated UUID)
|
||||
sender_id::String = string(uuid4()) # Sender ID (auto-generated UUID)
|
||||
is_publish::Bool = true,
|
||||
NATS_connection::Union{NATS.Connection, Nothing} = nothing,
|
||||
msg_id::String = string(uuid4()),
|
||||
sender_id::String = string(uuid4())
|
||||
)
|
||||
# Returns: (msgEnvelope_v1, JSON string)
|
||||
# - env: msgEnvelope_v1 object with all envelope metadata and payloads
|
||||
# - env_json_str: JSON string representation of the envelope for publishing
|
||||
# Returns: ::Tuple{msg_envelope_v1, String}
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natsbridge');
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
subject,
|
||||
data, // Array of [dataname, data, type] tuples
|
||||
{
|
||||
broker_url: 'nats://localhost:4222',
|
||||
fileserver_url: 'http://localhost:8080',
|
||||
fileserver_upload_handler: NATSBridge.plikOneshotUpload,
|
||||
size_threshold: 1_000_000,
|
||||
correlation_id: uuidv4(),
|
||||
msg_purpose: 'chat',
|
||||
sender_name: 'NATSBridge',
|
||||
receiver_name: '',
|
||||
receiver_id: '',
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
is_publish: true,
|
||||
nats_connection: null,
|
||||
msg_id: uuidv4(),
|
||||
sender_id: uuidv4()
|
||||
}
|
||||
);
|
||||
// Returns: Promise<[env, env_json_str]>
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import NATSBridge
|
||||
|
||||
env, env_json_str = await NATSBridge.smartsend(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
broker_url: str = "nats://localhost:4222",
|
||||
fileserver_url: str = "http://localhost:8080",
|
||||
fileserver_upload_handler: Callable = plik_oneshot_upload,
|
||||
size_threshold: int = 1_000_000,
|
||||
correlation_id: str = None,
|
||||
msg_purpose: str = "chat",
|
||||
sender_name: str = "NATSBridge",
|
||||
receiver_name: str = "",
|
||||
receiver_id: str = "",
|
||||
reply_to: str = "",
|
||||
reply_to_msg_id: str = "",
|
||||
is_publish: bool = True,
|
||||
nats_connection: Any = None,
|
||||
msg_id: str = None,
|
||||
sender_id: str = None
|
||||
)
|
||||
# Returns: Tuple[Dict, str]
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
from natsbridge import NATSBridge
|
||||
|
||||
# Limited to direct transport (< 100KB threshold)
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
subject,
|
||||
data, # List of (dataname, data, type) tuples
|
||||
broker_url="nats://localhost:4222",
|
||||
size_threshold=100000 # Lower threshold for memory constraints
|
||||
)
|
||||
# Returns: Tuple[Dict, str]
|
||||
```
|
||||
|
||||
### smartreceive
|
||||
@@ -263,7 +409,6 @@ Receives and processes messages from NATS, handling both direct and link transpo
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Note: msg is a NATS.Msg object passed from the subscription callback
|
||||
env = NATSBridge.smartreceive(
|
||||
msg::NATS.Msg;
|
||||
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||
@@ -271,51 +416,63 @@ env = NATSBridge.smartreceive(
|
||||
base_delay::Int = 100,
|
||||
max_delay::Int = 5000
|
||||
)
|
||||
# Returns: Dict with envelope metadata and payloads array
|
||||
# Returns: ::JSON.Object{String, Any}
|
||||
```
|
||||
|
||||
### publish_message
|
||||
#### JavaScript
|
||||
|
||||
Publish a message to a NATS subject. This function is available in Julia with two overloads:
|
||||
```javascript
|
||||
const env = await NATSBridge.smartreceive(
|
||||
msg,
|
||||
{
|
||||
fileserver_download_handler: NATSBridge.fetchWithBackoff,
|
||||
max_retries: 5,
|
||||
base_delay: 100,
|
||||
max_delay: 5000
|
||||
}
|
||||
);
|
||||
// Returns: Promise<env_object>
|
||||
```
|
||||
|
||||
#### Julia
|
||||
#### Python
|
||||
|
||||
**Using broker URL (creates new connection):**
|
||||
```julia
|
||||
using NATSBridge, NATS
|
||||
|
||||
# Publish with URL - creates a new connection
|
||||
NATSBridge.publish_message(
|
||||
"nats://localhost:4222", # broker_url
|
||||
"/chat/room1", # subject
|
||||
"{\"correlation_id\":\"abc123\"}", # message
|
||||
"abc123" # correlation_id
|
||||
```python
|
||||
env = await NATSBridge.smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=fetch_with_backoff,
|
||||
max_retries=5,
|
||||
base_delay=100,
|
||||
max_delay=5000
|
||||
)
|
||||
# Returns: Dict with "payloads" key
|
||||
```
|
||||
|
||||
**Using pre-existing connection (saves connection overhead):**
|
||||
```julia
|
||||
using NATSBridge, NATS
|
||||
#### MicroPython
|
||||
|
||||
# Create connection once and reuse
|
||||
conn = NATS.connect("nats://localhost:4222")
|
||||
NATSBridge.publish_message(conn, "/chat/room1", "{\"correlation_id\":\"abc123\"}", "abc123")
|
||||
# Connection is automatically drained after publish
|
||||
```python
|
||||
env = NATSBridge.smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=_sync_fileserver_download,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
# Returns: Dict with "payloads" key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Payload Types
|
||||
|
||||
| Type | Description | Serialization |
|
||||
|------|-------------|---------------|
|
||||
| `text` | Plain text strings | UTF-8 bytes |
|
||||
| `dictionary` | JSON-serializable dictionaries | JSON |
|
||||
| `table` | Tabular data (DataFrames, arrays) | Apache Arrow IPC |
|
||||
| `image` | Image data (PNG, JPG) | Raw bytes |
|
||||
| `audio` | Audio data (WAV, MP3) | Raw bytes |
|
||||
| `video` | Video data (MP4, AVI) | Raw bytes |
|
||||
| `binary` | Generic binary data | Raw bytes |
|
||||
| Type | Julia | JavaScript | Python | MicroPython | Description |
|
||||
|------|-------|------------|--------|-------------|-------------|
|
||||
| `text` | `String` | `string` | `str` | `str` | Plain text strings |
|
||||
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable dictionaries |
|
||||
| `table` | `DataFrame`, `Arrow.Table` | `Array<Object>` | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) |
|
||||
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image data (PNG, JPG) |
|
||||
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio data (WAV, MP3) |
|
||||
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video data (MP4, AVI) |
|
||||
| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `bytearray` | Generic binary data |
|
||||
|
||||
---
|
||||
|
||||
@@ -325,31 +482,60 @@ NATSBridge.publish_message(conn, "/chat/room1", "{\"correlation_id\":\"abc123\"}
|
||||
|
||||
Small payloads are sent directly via NATS with Base64 encoding.
|
||||
|
||||
#### Julia
|
||||
#### Cross-Platform
|
||||
|
||||
```julia
|
||||
# Julia
|
||||
data = [("message", "Hello", "text")]
|
||||
smartsend("/topic", data)
|
||||
```
|
||||
|
||||
```javascript
|
||||
// JavaScript
|
||||
const data = [["message", "Hello", "text"]];
|
||||
smartsend("/topic", data);
|
||||
```
|
||||
|
||||
```python
|
||||
# Python
|
||||
data = [("message", "Hello", "text")]
|
||||
await smartsend("/topic", data)
|
||||
```
|
||||
|
||||
### Link Transport (Payloads >= 1MB)
|
||||
|
||||
Large payloads are uploaded to an HTTP file server.
|
||||
|
||||
#### Julia
|
||||
#### Cross-Platform
|
||||
|
||||
```julia
|
||||
# Julia
|
||||
data = [("file", large_data, "binary")]
|
||||
smartsend("/topic", data; fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
```javascript
|
||||
// JavaScript
|
||||
const data = [["file", largeData, "binary"]];
|
||||
smartsend("/topic", data, { fileserver_url: 'http://localhost:8080' });
|
||||
```
|
||||
|
||||
```python
|
||||
# Python
|
||||
data = [("file", large_data, "binary")]
|
||||
await smartsend("/topic", data, fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
## Cross-Platform Examples
|
||||
|
||||
### Example 1: Chat with Mixed Content
|
||||
|
||||
Send text, small image, and large file in one message.
|
||||
Send text, image, and large file in one message.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
@@ -362,11 +548,48 @@ data = [
|
||||
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natsbridge');
|
||||
|
||||
const data = [
|
||||
["message_text", "Hello!", "text"],
|
||||
["user_avatar", imageData, "image"],
|
||||
["large_document", largeFileData, "binary"]
|
||||
];
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ fileserver_url: 'http://localhost:8080' }
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import NATSBridge
|
||||
|
||||
data = [
|
||||
("message_text", "Hello!", "text"),
|
||||
("user_avatar", image_data, "image"),
|
||||
("large_document", large_file_data, "binary")
|
||||
]
|
||||
|
||||
env, env_json_str = await NATSBridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
fileserver_url="http://localhost:8080"
|
||||
)
|
||||
```
|
||||
|
||||
### Example 2: Dictionary Exchange
|
||||
|
||||
Send configuration data between platforms.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
@@ -380,11 +603,44 @@ data = [("config", config, "dictionary")]
|
||||
env, env_json_str = NATSBridge.smartsend("/device/config", data)
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natsbridge');
|
||||
|
||||
const config = {
|
||||
wifi_ssid: "MyNetwork",
|
||||
wifi_password: "password123",
|
||||
update_interval: 60
|
||||
};
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/device/config",
|
||||
[["config", config, "dictionary"]]
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import NATSBridge
|
||||
|
||||
config = {
|
||||
"wifi_ssid": "MyNetwork",
|
||||
"wifi_password": "password123",
|
||||
"update_interval": 60
|
||||
}
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env, env_json_str = await NATSBridge.smartsend("/device/config", data)
|
||||
```
|
||||
|
||||
### Example 3: Table Data (Arrow IPC)
|
||||
|
||||
Send tabular data using Apache Arrow IPC format.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
using DataFrames
|
||||
@@ -399,14 +655,49 @@ data = [("students", df, "table")]
|
||||
env, env_json_str = NATSBridge.smartsend("/data/analysis", data)
|
||||
```
|
||||
|
||||
### Example 4: Request-Response Pattern with Envelope JSON
|
||||
#### JavaScript
|
||||
|
||||
Bi-directional communication with reply-to support. The `smartsend` function now returns both the envelope object and a JSON string that can be published directly.
|
||||
```javascript
|
||||
const NATSBridge = require('natsbridge');
|
||||
|
||||
const df = [
|
||||
{ id: 1, name: "Alice", score: 95 },
|
||||
{ id: 2, name: "Bob", score: 88 },
|
||||
{ id: 3, name: "Charlie", score: 92 }
|
||||
];
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/data/analysis",
|
||||
[["students", df, "table"]]
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import NATSBridge
|
||||
import pandas as pd
|
||||
|
||||
df = pd.DataFrame({
|
||||
"id": [1, 2, 3],
|
||||
"name": ["Alice", "Bob", "Charlie"],
|
||||
"score": [95, 88, 92]
|
||||
})
|
||||
|
||||
data = [("students", df, "table")]
|
||||
env, env_json_str = await NATSBridge.smartsend("/data/analysis", data)
|
||||
```
|
||||
|
||||
### Example 4: Request-Response Pattern
|
||||
|
||||
Bi-directional communication with reply-to support.
|
||||
|
||||
#### Julia
|
||||
|
||||
#### Julia (Requester)
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Requester
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
"/device/command",
|
||||
[("command", Dict("action" => "read_sensor"), "dictionary")];
|
||||
@@ -415,26 +706,20 @@ env, env_json_str = NATSBridge.smartsend(
|
||||
)
|
||||
```
|
||||
|
||||
#### Julia (Responder)
|
||||
```julia
|
||||
# Responder
|
||||
using NATS, NATSBridge
|
||||
|
||||
# Configuration
|
||||
const SUBJECT = "/device/command"
|
||||
const NATS_URL = "nats://localhost:4222"
|
||||
|
||||
function test_responder()
|
||||
conn = NATS.connect(NATS_URL)
|
||||
NATS.subscribe(conn, SUBJECT) do msg
|
||||
conn = NATS.connect("nats://localhost:4222")
|
||||
NATS.subscribe(conn, "/device/command") do msg
|
||||
env = NATSBridge.smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
|
||||
|
||||
# Extract reply_to from the envelope metadata
|
||||
reply_to = env["reply_to"]
|
||||
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
if dataname == "command" && data["action"] == "read_sensor"
|
||||
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
|
||||
# Send response to the reply_to subject from the request
|
||||
if !isempty(reply_to)
|
||||
smartsend(reply_to, [("data", response, "dictionary")])
|
||||
end
|
||||
@@ -445,51 +730,118 @@ function test_responder()
|
||||
sleep(120)
|
||||
NATS.drain(conn)
|
||||
end
|
||||
|
||||
test_responder()
|
||||
```
|
||||
|
||||
### Example 5: IoT Device Sensor Data
|
||||
#### JavaScript
|
||||
|
||||
IoT device sending sensor data.
|
||||
```javascript
|
||||
const NATSBridge = require('natsbridge');
|
||||
|
||||
#### Julia (Receiver)
|
||||
```julia
|
||||
using NATS, NATSBridge
|
||||
// Requester
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/device/command",
|
||||
[["command", { action: "read_sensor" }, "dictionary"]],
|
||||
{ broker_url: 'nats://localhost:4222', reply_to: '/device/response' }
|
||||
);
|
||||
```
|
||||
|
||||
# Configuration
|
||||
const SUBJECT = "/device/sensors"
|
||||
const NATS_URL = "nats://localhost:4222"
|
||||
```javascript
|
||||
// Responder
|
||||
const nats = require('nats');
|
||||
const NATSBridge = require('natsbridge');
|
||||
|
||||
function test_receiver()
|
||||
conn = NATS.connect(NATS_URL)
|
||||
NATS.subscribe(conn, SUBJECT) do msg
|
||||
env, env_json_str = NATSBridge.smartreceive(msg, fileserverDownloadHandler)
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
if dataname == "temperature"
|
||||
println("Temperature: $data")
|
||||
elseif dataname == "humidity"
|
||||
println("Humidity: $data")
|
||||
end
|
||||
end
|
||||
end
|
||||
async function testResponder() {
|
||||
const conn = await nats.connect('nats://localhost:4222');
|
||||
|
||||
sleep(120)
|
||||
NATS.drain(conn)
|
||||
end
|
||||
const subscription = await conn.subscribe('/device/command');
|
||||
|
||||
test_receiver()
|
||||
for await (const msg of subscription) {
|
||||
const env = await NATSBridge.smartreceive(msg, {
|
||||
fileserver_download_handler: NATSBridge.fetchWithBackoff
|
||||
});
|
||||
|
||||
const replyTo = env.reply_to;
|
||||
|
||||
for (const [dataname, data, type] of env.payloads) {
|
||||
if (dataname === 'command' && data.action === 'read_sensor') {
|
||||
const response = { sensor_id: 'sensor-001', value: 42.5 };
|
||||
if (replyTo) {
|
||||
await NATSBridge.smartsend(
|
||||
replyTo,
|
||||
[["data", response, "dictionary"]]
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
setTimeout(() => conn.close(), 120000);
|
||||
}
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import NATSBridge
|
||||
|
||||
# Requester
|
||||
env, env_json_str = await NATSBridge.smartsend(
|
||||
"/device/command",
|
||||
[("command", {"action": "read_sensor"}, "dictionary")],
|
||||
broker_url="nats://localhost:4222",
|
||||
reply_to="/device/response"
|
||||
)
|
||||
```
|
||||
|
||||
```python
|
||||
# Responder
|
||||
from natsbridge import NATSBridge
|
||||
import asyncio
|
||||
import nats
|
||||
|
||||
async def test_responder():
|
||||
nc = await nats.connect('nats://localhost:4222')
|
||||
|
||||
async def msg_handler(msg):
|
||||
env = await NATSBridge.smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=fetch_with_backoff
|
||||
)
|
||||
|
||||
reply_to = env["reply_to"]
|
||||
|
||||
for dataname, data, type_ in env["payloads"]:
|
||||
if dataname == "command" and data["action"] == "read_sensor":
|
||||
response = {"sensor_id": "sensor-001", "value": 42.5}
|
||||
if reply_to:
|
||||
await NATSBridge.smartsend(
|
||||
reply_to,
|
||||
[("data", response, "dictionary")]
|
||||
)
|
||||
|
||||
await nc.subscribe('/device/command', cb=msg_handler)
|
||||
|
||||
await asyncio.sleep(120)
|
||||
await nc.drain()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
Run the test scripts to verify functionality:
|
||||
### Test File Organization
|
||||
|
||||
### Julia
|
||||
| Platform | Sender Tests | Receiver Tests |
|
||||
|----------|--------------|----------------|
|
||||
| **Julia** | `test/test_julia_*_sender.jl` | `test/test_julia_*_receiver.jl` |
|
||||
| **JavaScript** | `test/test_js_*_sender.js` | `test/test_js_*_receiver.js` |
|
||||
| **Python** | `test/test_py_*_sender.py` | `test/test_py_*_receiver.py` |
|
||||
|
||||
```julia
|
||||
### Run Tests
|
||||
|
||||
#### Julia
|
||||
|
||||
```bash
|
||||
# Text message exchange
|
||||
julia test/test_julia_text_sender.jl
|
||||
julia test/test_julia_text_receiver.jl
|
||||
@@ -511,6 +863,55 @@ julia test/test_julia_table_sender.jl
|
||||
julia test/test_julia_table_receiver.jl
|
||||
```
|
||||
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```bash
|
||||
# Text message exchange
|
||||
node test/test_js_text_sender.js
|
||||
node test/test_js_text_receiver.js
|
||||
|
||||
# Dictionary exchange
|
||||
node test/test_js_dictionary_sender.js
|
||||
node test/test_js_dictionary_receiver.js
|
||||
|
||||
# Binary transfer
|
||||
node test/test_js_binary_sender.js
|
||||
node test/test_js_binary_receiver.js
|
||||
|
||||
# Table exchange
|
||||
node test/test_js_table_sender.js
|
||||
node test/test_js_table_receiver.js
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```bash
|
||||
# Text message exchange
|
||||
python3 test/test_py_text_sender.py
|
||||
python3 test/test_py_text_receiver.py
|
||||
|
||||
# Dictionary exchange
|
||||
python3 test/test_py_dictionary_sender.py
|
||||
python3 test/test_py_dictionary_receiver.py
|
||||
|
||||
# Binary transfer
|
||||
python3 test/test_py_binary_sender.py
|
||||
python3 test/test_py_binary_receiver.py
|
||||
|
||||
# Table exchange
|
||||
python3 test/test_py_table_sender.py
|
||||
python3 test/test_py_table_receiver.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
For detailed architecture and implementation information, see:
|
||||
|
||||
- [Architecture Documentation](docs/architecture.md) - Cross-platform architecture, API parity, platform-specific patterns
|
||||
- [Implementation Guide](docs/implementation.md) - Detailed implementation for each platform, handler functions, testing
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
1370
docs/architecture.md
1370
docs/architecture.md
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
29
etc.jl
29
etc.jl
@@ -7,3 +7,32 @@ Requirements:
|
||||
Source of Truth: Treat the updated NATSbridge code as the definitive source. Update README.md to align exactly with these changes.
|
||||
API Consistency: Ensure the Main Package API (e.g., smartsend(), publish_message()) uses consistent naming across all three supported languages.
|
||||
Ecosystem Variance: Low-level native functions (e.g., NATS.connect(), JSON.read()) should follow the conventions of the specific language ecosystem and do not require cross-language consistency.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Help me expands this Julia package (NATSBridge) into a cross-platform project by adding a JavaScript and Python/MicroPython implementation. To ensure accuracy, NATSBridge.jl will serve as the ground truth, as the documentation may be outdated.
|
||||
|
||||
My goal is to maintain interface parity at the high-level API for a consistent user experience, while ensuring the low-level implementation adheres strictly to the idiomatic conventions of each respective language (e.g., multiple dispatch in Julia vs. asynchronous, prototype, or class-based patterns in JS and Python/MicroPython)
|
||||
|
||||
Now do the following:
|
||||
1) check docs to see if there is any mistake.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# NATSBridge Tutorial
|
||||
# Cross-Platform NATSBridge Tutorial
|
||||
|
||||
A step-by-step guide to get started with NATSBridge - a high-performance, bi-directional data bridge for **Julia**.
|
||||
A step-by-step guide to get started with NATSBridge across **Julia**, **JavaScript**, and **Python/MicroPython**.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
@@ -15,22 +15,35 @@ A step-by-step guide to get started with NATSBridge - a high-performance, bi-dir
|
||||
|
||||
## Overview
|
||||
|
||||
NATSBridge enables seamless communication for Julia applications through NATS, with automatic transport selection based on payload size:
|
||||
NATSBridge enables seamless communication across platforms through NATS, with automatic transport selection based on payload size:
|
||||
|
||||
- **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded)
|
||||
- **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL
|
||||
|
||||
### Cross-Platform API Parity
|
||||
|
||||
All three platforms use the same high-level API:
|
||||
|
||||
```
|
||||
# Input format
|
||||
smartsend(subject, [(dataname, data, type), ...], options)
|
||||
|
||||
# Output format
|
||||
(env, env_json_str) = smartsend(...)
|
||||
env = smartreceive(msg, options)
|
||||
```
|
||||
|
||||
### Supported Payload Types
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `text` | Plain text strings |
|
||||
| `dictionary` | JSON-serializable dictionaries |
|
||||
| `table` | Tabular data (Arrow IPC format) |
|
||||
| `image` | Image data (PNG, JPG bytes) |
|
||||
| `audio` | Audio data (WAV, MP3 bytes) |
|
||||
| `video` | Video data (MP4, AVI bytes) |
|
||||
| `binary` | Generic binary data |
|
||||
| Type | Julia | JavaScript | Python | MicroPython |
|
||||
|------|-------|------------|--------|-------------|
|
||||
| `text` | `String` | `string` | `str` | `str` |
|
||||
| `dictionary` | `Dict` | `Object` | `dict` | `dict` |
|
||||
| `table` | `DataFrame` | `Array<Object>` | `DataFrame` | ❌ |
|
||||
| `image` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
|
||||
| `audio` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
|
||||
| `video` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
|
||||
| `binary` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
|
||||
|
||||
---
|
||||
|
||||
@@ -40,7 +53,7 @@ Before you begin, ensure you have:
|
||||
|
||||
1. **NATS Server** running (or accessible)
|
||||
2. **HTTP File Server** (optional, for large payloads > 1MB)
|
||||
3. **Julia** with required packages
|
||||
3. **Platform-specific packages** installed
|
||||
|
||||
---
|
||||
|
||||
@@ -58,6 +71,29 @@ Pkg.add("UUIDs")
|
||||
Pkg.add("Dates")
|
||||
```
|
||||
|
||||
### JavaScript (Node.js)
|
||||
|
||||
```bash
|
||||
npm install nats uuid apache-arrow node-fetch
|
||||
```
|
||||
|
||||
### JavaScript (Browser)
|
||||
|
||||
```html
|
||||
<script src="https://unpkg.com/nats-js/dist/bundle/nats.min.js"></script>
|
||||
<script src="https://unpkg.com/apache-arrow/arrow.min.js"></script>
|
||||
```
|
||||
|
||||
### Python (Desktop)
|
||||
|
||||
```bash
|
||||
pip install nats-py aiohttp pyarrow pandas
|
||||
```
|
||||
|
||||
### MicroPython
|
||||
|
||||
Uses built-in modules: `network`, `socket`, `time`, `json`, `base64`
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
@@ -71,10 +107,7 @@ docker run -p 4222:4222 nats:latest
|
||||
### Step 2: Start HTTP File Server (Optional)
|
||||
|
||||
```bash
|
||||
# Create a directory for file uploads
|
||||
mkdir -p /tmp/fileserver
|
||||
|
||||
# Use any HTTP server that supports POST for file uploads
|
||||
python3 -m http.server 8080 --directory /tmp/fileserver
|
||||
```
|
||||
|
||||
@@ -88,16 +121,84 @@ using NATSBridge
|
||||
# Send a text message
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222")
|
||||
# env: msg_envelope_v1 object with all metadata and payloads
|
||||
# env: msg_envelope_v1 struct with all metadata and payloads
|
||||
# env_json_str: JSON string representation of the envelope for publishing
|
||||
println("Message sent!")
|
||||
|
||||
# Or use is_publish=false to get envelope and JSON without publishing
|
||||
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222", is_publish=false)
|
||||
# env: msg_envelope_v1 object
|
||||
# env: msg_envelope_v1 struct
|
||||
# env_json_str: JSON string for publishing to NATS
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natsbridge.js');
|
||||
|
||||
// Send a text message
|
||||
const data = [["message", "Hello World", "text"]];
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222" }
|
||||
);
|
||||
// env: Object with all metadata and payloads
|
||||
// env_json_str: JSON string for publishing
|
||||
console.log("Message sent!");
|
||||
|
||||
// Or use is_publish=false
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222", is_publish: false }
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import smartsend
|
||||
|
||||
# Send a text message
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = await smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
broker_url="nats://localhost:4222"
|
||||
)
|
||||
# env: Dict with all metadata and payloads
|
||||
# env_json_str: JSON string for publishing
|
||||
print("Message sent!")
|
||||
|
||||
# Or use is_publish=False
|
||||
env, env_json_str = await smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
broker_url="nats://localhost:4222",
|
||||
is_publish=False
|
||||
)
|
||||
# env: Dict with all metadata and payloads
|
||||
# env_json_str: JSON string for publishing to NATS
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
from natsbridge_mpy import NATSBridge
|
||||
|
||||
bridge = NATSBridge()
|
||||
|
||||
# Send a text message (limited to small payloads)
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = bridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
size_threshold=100000 # Lower threshold for MicroPython
|
||||
)
|
||||
print("Message sent!")
|
||||
```
|
||||
|
||||
### Step 4: Receive Messages
|
||||
|
||||
#### Julia
|
||||
@@ -107,11 +208,43 @@ using NATSBridge
|
||||
|
||||
# Receive and process message
|
||||
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff)
|
||||
# Returns: ::JSON.Object{String, Any} with "payloads" field containing Vector{Tuple{String, Any, String}}
|
||||
# Access payloads: for (dataname, data, type) in env["payloads"]
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
println("Received $dataname: $data")
|
||||
end
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natsbridge.js');
|
||||
|
||||
// Receive and process message
|
||||
const env = await NATSBridge.smartreceive(msg, {
|
||||
fileserver_download_handler: NATSBridge.fetchWithBackoff
|
||||
});
|
||||
// env.payloads = [[dataname, data, type], ...]
|
||||
for (const [dataname, data, type] of env.payloads) {
|
||||
console.log(`Received ${dataname}:`, data);
|
||||
}
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import smartreceive, fetch_with_backoff
|
||||
|
||||
# Receive and process message
|
||||
env = await smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=fetch_with_backoff
|
||||
)
|
||||
# env["payloads"] = [(dataname, data, type), ...]
|
||||
for dataname, data, type_ in env["payloads"]:
|
||||
print(f"Received {dataname}: {data}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Basic Examples
|
||||
@@ -133,6 +266,65 @@ data = [("config", config, "dictionary")]
|
||||
env, env_json_str = smartsend("/device/config", data, broker_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natsbridge.js');
|
||||
|
||||
const config = {
|
||||
wifi_ssid: "MyNetwork",
|
||||
wifi_password: "password123",
|
||||
update_interval: 60
|
||||
};
|
||||
|
||||
const data = [["config", config, "dictionary"]];
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/device/config",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222" }
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import smartsend
|
||||
|
||||
config = {
|
||||
"wifi_ssid": "MyNetwork",
|
||||
"wifi_password": "password123",
|
||||
"update_interval": 60
|
||||
}
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env, env_json_str = await smartsend(
|
||||
"/device/config",
|
||||
data,
|
||||
broker_url="nats://localhost:4222"
|
||||
)
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
from natsbridge_mpy import NATSBridge
|
||||
|
||||
bridge = NATSBridge()
|
||||
|
||||
config = {
|
||||
"wifi_ssid": "MyNetwork",
|
||||
"wifi_password": "password123",
|
||||
"update_interval": 60
|
||||
}
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env, env_json_str = bridge.smartsend(
|
||||
"/device/config",
|
||||
data,
|
||||
size_threshold=100000
|
||||
)
|
||||
```
|
||||
|
||||
### Example 2: Sending Binary Data (Image)
|
||||
|
||||
#### Julia
|
||||
@@ -147,6 +339,59 @@ data = [("user_image", image_data, "binary")]
|
||||
env, env_json_str = smartsend("/chat/image", data, broker_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natsbridge.js');
|
||||
const fs = require('fs');
|
||||
|
||||
// Read image file
|
||||
const image_data = fs.readFileSync('image.png');
|
||||
|
||||
const data = [["user_image", image_data, "binary"]];
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/image",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222" }
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import smartsend
|
||||
|
||||
# Read image file
|
||||
with open("image.png", "rb") as f:
|
||||
image_data = f.read()
|
||||
|
||||
data = [("user_image", image_data, "binary")]
|
||||
env, env_json_str = await smartsend(
|
||||
"/chat/image",
|
||||
data,
|
||||
broker_url="nats://localhost:4222"
|
||||
)
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
from natsbridge_mpy import NATSBridge
|
||||
|
||||
bridge = NATSBridge()
|
||||
|
||||
# Read image file
|
||||
with open("image.png", "rb") as f:
|
||||
image_data = f.read()
|
||||
|
||||
data = [("user_image", image_data, "binary")]
|
||||
env, env_json_str = bridge.smartsend(
|
||||
"/chat/image",
|
||||
data,
|
||||
size_threshold=100000
|
||||
)
|
||||
```
|
||||
|
||||
### Example 3: Request-Response Pattern
|
||||
|
||||
#### Julia (Requester)
|
||||
@@ -163,16 +408,47 @@ env, env_json_str = smartsend(
|
||||
reply_to="/device/response",
|
||||
reply_to_msg_id="cmd-001"
|
||||
)
|
||||
# env: msg_envelope_v1 object
|
||||
# env_json_str: JSON string for publishing to NATS
|
||||
```
|
||||
|
||||
#### JavaScript (Requester)
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natsbridge.js');
|
||||
|
||||
// Send command with reply-to
|
||||
const data = [["command", { action: "read_sensor" }, "dictionary"]];
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/device/command",
|
||||
data,
|
||||
{
|
||||
broker_url: "nats://localhost:4222",
|
||||
reply_to: "/device/response",
|
||||
reply_to_msg_id: "cmd-001"
|
||||
}
|
||||
);
|
||||
```
|
||||
|
||||
#### Python (Requester)
|
||||
|
||||
```python
|
||||
from natsbridge import smartsend
|
||||
|
||||
# Send command with reply-to
|
||||
data = [("command", {"action": "read_sensor"}, "dictionary")]
|
||||
env, env_json_str = await smartsend(
|
||||
"/device/command",
|
||||
data,
|
||||
broker_url="nats://localhost:4222",
|
||||
reply_to="/device/response",
|
||||
reply_to_msg_id="cmd-001"
|
||||
)
|
||||
```
|
||||
|
||||
#### Julia (Responder)
|
||||
|
||||
```julia
|
||||
using NATS, NATSBridge
|
||||
using NATSBridge, NATS
|
||||
|
||||
# Configuration
|
||||
const SUBJECT = "/device/command"
|
||||
const NATS_URL = "nats://localhost:4222"
|
||||
|
||||
@@ -181,13 +457,11 @@ function test_responder()
|
||||
NATS.subscribe(conn, SUBJECT) do msg
|
||||
env = smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
|
||||
|
||||
# Extract reply_to from the envelope metadata
|
||||
reply_to = env["reply_to"]
|
||||
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
if dataname == "command" && data["action"] == "read_sensor"
|
||||
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
|
||||
# Send response to the reply_to subject from the request
|
||||
if !isempty(reply_to)
|
||||
smartsend(reply_to, [("data", response, "dictionary")])
|
||||
end
|
||||
@@ -225,8 +499,73 @@ env, env_json_str = smartsend(
|
||||
fileserver_url="http://localhost:8080"
|
||||
)
|
||||
|
||||
# The envelope will contain the download URL
|
||||
println("File uploaded to: $(env.payloads[1].data)")
|
||||
# Note: For link transport, data field contains the URL string
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natsbridge.js');
|
||||
|
||||
// Create large data (> 1MB)
|
||||
const large_data = Buffer.alloc(2_000_000);
|
||||
for (let i = 0; i < large_data.length; i++) {
|
||||
large_data[i] = Math.floor(Math.random() * 256);
|
||||
}
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/data/large",
|
||||
[["large_file", large_data, "binary"]],
|
||||
{
|
||||
broker_url: "nats://localhost:4222",
|
||||
fileserver_url: "http://localhost:8080"
|
||||
}
|
||||
);
|
||||
|
||||
console.log("File uploaded to:", env.payloads[0].data);
|
||||
// Note: For link transport, data field contains the URL string
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import smartsend
|
||||
|
||||
# Create large data (> 1MB)
|
||||
import os
|
||||
large_data = os.urandom(2_000_000)
|
||||
|
||||
env, env_json_str = await smartsend(
|
||||
"/data/large",
|
||||
[("large_file", large_data, "binary")],
|
||||
broker_url="nats://localhost:4222",
|
||||
fileserver_url="http://localhost:8080"
|
||||
)
|
||||
|
||||
print(f"File uploaded to: {env['payloads'][0]['data']}")
|
||||
# Note: For link transport, data field contains the URL string
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
MicroPython enforces a hard limit of 50KB per payload:
|
||||
|
||||
```python
|
||||
from natsbridge_mpy import NATSBridge
|
||||
|
||||
bridge = NATSBridge()
|
||||
|
||||
# MicroPython has a hard limit of 50KB per payload
|
||||
# Use streaming or chunking for larger data
|
||||
small_data = bytes(1000) # 1KB
|
||||
|
||||
data = [("small_file", small_data, "binary")]
|
||||
env, env_json_str = bridge.smartsend(
|
||||
"/data/small",
|
||||
data,
|
||||
size_threshold=100000 # Enforced max: 50000 bytes
|
||||
)
|
||||
```
|
||||
|
||||
### Example 5: Mixed Content (Chat with Text + Image)
|
||||
@@ -248,6 +587,47 @@ data = [
|
||||
env, env_json_str = smartsend("/chat/mixed", data, broker_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natsbridge.js');
|
||||
const fs = require('fs');
|
||||
|
||||
const image_data = fs.readFileSync('avatar.png');
|
||||
|
||||
const data = [
|
||||
["message_text", "Hello with image!", "text"],
|
||||
["user_avatar", image_data, "image"]
|
||||
];
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/mixed",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222" }
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import smartsend
|
||||
|
||||
with open("avatar.png", "rb") as f:
|
||||
image_data = f.read()
|
||||
|
||||
data = [
|
||||
("message_text", "Hello with image!", "text"),
|
||||
("user_avatar", image_data, "image")
|
||||
]
|
||||
|
||||
env, env_json_str = await smartsend(
|
||||
"/chat/mixed",
|
||||
data,
|
||||
broker_url="nats://localhost:4222"
|
||||
)
|
||||
# env: Dict with all metadata and payloads
|
||||
```
|
||||
|
||||
### Example 6: Table Data (Arrow IPC)
|
||||
|
||||
For tabular data, NATSBridge uses Apache Arrow IPC format:
|
||||
@@ -269,12 +649,58 @@ data = [("students", df, "table")]
|
||||
env, env_json_str = smartsend("/data/students", data, broker_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natsbridge.js');
|
||||
|
||||
// Create table data (array of objects)
|
||||
const table_data = [
|
||||
{ id: 1, name: "Alice", score: 95 },
|
||||
{ id: 2, name: "Bob", score: 88 },
|
||||
{ id: 3, name: "Charlie", score: 92 }
|
||||
];
|
||||
|
||||
const data = [["students", table_data, "table"]];
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/data/students",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222" }
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natsbridge import smartsend
|
||||
import pandas as pd
|
||||
|
||||
# Create DataFrame
|
||||
df = pd.DataFrame({
|
||||
'id': [1, 2, 3],
|
||||
'name': ['Alice', 'Bob', 'Charlie'],
|
||||
'score': [95, 88, 92]
|
||||
})
|
||||
|
||||
data = [("students", df, "table")]
|
||||
env, env_json_str = await smartsend(
|
||||
"/data/students",
|
||||
data,
|
||||
broker_url="nats://localhost:4222"
|
||||
)
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
MicroPython does not support table type due to memory constraints. Use dictionary or binary instead.
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Explore the test directory** for more examples
|
||||
2. **Check the documentation** for advanced configuration options
|
||||
3. **Read the walkthrough** for building real-world applications
|
||||
|
||||
---
|
||||
|
||||
@@ -295,7 +721,8 @@ env, env_json_str = smartsend("/data/students", data, broker_url="nats://localho
|
||||
### Serialization Errors
|
||||
|
||||
- Verify data type matches the specified type
|
||||
- Check that binary data is in the correct format (Vector{UInt8})
|
||||
- Check that binary data is in the correct format
|
||||
- MicroPython: Ensure payload size < 50KB
|
||||
|
||||
---
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
28
package.json
28
package.json
@@ -1,28 +0,0 @@
|
||||
{
|
||||
"name": "natsbridge",
|
||||
"version": "1.0.0",
|
||||
"description": "Bi-Directional Data Bridge for JavaScript using NATS",
|
||||
"main": "src/NATSBridge.js",
|
||||
"scripts": {
|
||||
"test": "echo \"Error: no test specified\" && exit 1",
|
||||
"lint": "eslint src/*.js test/*.js"
|
||||
},
|
||||
"keywords": [
|
||||
"nats",
|
||||
"message-broker",
|
||||
"bridge",
|
||||
"arrow",
|
||||
"serialization"
|
||||
],
|
||||
"author": "",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"nats": "^2.9.0",
|
||||
"apache-arrow": "^14.0.0",
|
||||
"uuid": "^9.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"eslint": "^8.0.0",
|
||||
"jest": "^29.0.0"
|
||||
}
|
||||
}
|
||||
@@ -279,42 +279,38 @@ function envelope_to_json(env::msg_envelope_v1)
|
||||
"broker_url" => env.broker_url
|
||||
)
|
||||
|
||||
if !isempty(env.metadata) # Only include metadata if it exists and is not empty
|
||||
obj["metadata"] = Dict(String(k) => v for (k, v) in env.metadata)
|
||||
end
|
||||
obj["metadata"] = Dict(String(k) => v for (k, v) in env.metadata)
|
||||
|
||||
# Convert payloads to JSON array
|
||||
if !isempty(env.payloads)
|
||||
payloads_json = []
|
||||
for payload in env.payloads
|
||||
payload_obj = Dict{String, Any}(
|
||||
"id" => payload.id,
|
||||
"dataname" => payload.dataname,
|
||||
"payload_type" => payload.payload_type,
|
||||
"transport" => payload.transport,
|
||||
"encoding" => payload.encoding,
|
||||
"size" => payload.size,
|
||||
)
|
||||
# Include data based on transport type
|
||||
if payload.transport == "direct" && payload.data !== nothing
|
||||
if payload.encoding == "base64" || payload.encoding == "json"
|
||||
payload_obj["data"] = payload.data
|
||||
else
|
||||
# For other encodings, use base64
|
||||
payload_bytes = _get_payload_bytes(payload.data)
|
||||
payload_obj["data"] = Base64.base64encode(payload_bytes)
|
||||
end
|
||||
elseif payload.transport == "link" && payload.data !== nothing
|
||||
# For link transport, data is a URL string - include directly
|
||||
payloads_json = []
|
||||
for payload in env.payloads
|
||||
payload_obj = Dict{String, Any}(
|
||||
"id" => payload.id,
|
||||
"dataname" => payload.dataname,
|
||||
"payload_type" => payload.payload_type,
|
||||
"transport" => payload.transport,
|
||||
"encoding" => payload.encoding,
|
||||
"size" => payload.size,
|
||||
)
|
||||
# Include data based on transport type
|
||||
if payload.transport == "direct" && payload.data !== nothing
|
||||
if payload.encoding == "base64" || payload.encoding == "json"
|
||||
payload_obj["data"] = payload.data
|
||||
else
|
||||
# For other encodings, use base64
|
||||
payload_bytes = _get_payload_bytes(payload.data)
|
||||
payload_obj["data"] = Base64.base64encode(payload_bytes)
|
||||
end
|
||||
if !isempty(payload.metadata)
|
||||
payload_obj["metadata"] = Dict(String(k) => v for (k, v) in payload.metadata)
|
||||
elseif payload.transport == "link" && payload.data !== nothing
|
||||
# For link transport, data is a URL string - include directly
|
||||
payload_obj["data"] = payload.data
|
||||
end
|
||||
push!(payloads_json, payload_obj)
|
||||
if !isempty(payload.metadata)
|
||||
payload_obj["metadata"] = Dict(String(k) => v for (k, v) in payload.metadata)
|
||||
end
|
||||
obj["payloads"] = payloads_json
|
||||
push!(payloads_json, payload_obj)
|
||||
end
|
||||
obj["payloads"] = payloads_json
|
||||
|
||||
JSON.json(obj)
|
||||
end
|
||||
@@ -388,7 +384,7 @@ Each payload can have a different type, enabling mixed-content messages (e.g., c
|
||||
- `sender_id::String = string(uuid4())` - Sender ID (auto-generated UUID if not provided)
|
||||
|
||||
# Return:
|
||||
- A tuple `(env, env_json_str)` where:
|
||||
- `::Tuple{msg_envelope_v1, String}` - A tuple containing:
|
||||
- `env::msg_envelope_v1` - The envelope object containing all metadata and payloads
|
||||
- `env_json_str::String` - JSON string representation of the envelope for publishing
|
||||
|
||||
@@ -447,7 +443,7 @@ function smartsend(
|
||||
NATS_connection::Union{NATS.Connection, Nothing} = nothing, # a provided connection saves establishing connection overhead.
|
||||
msg_id::String = string(uuid4()), # Message ID
|
||||
sender_id::String = string(uuid4()) # Sender ID
|
||||
) where {T1<:Any}
|
||||
)::Tuple{msg_envelope_v1, String} where {T1<:Any}
|
||||
|
||||
# Log start of send operation
|
||||
log_trace(correlation_id, "Starting smartsend for subject: $subject")
|
||||
@@ -754,14 +750,14 @@ A HTTP file server is required along with its download function.
|
||||
- `max_delay::Int = 5000` - Maximum delay for exponential backoff in ms
|
||||
|
||||
# Return:
|
||||
- JSON object of envelope with list of (dataname, data, data_type) tuples in payloads field
|
||||
- `::JSON.Object{String, Any}` - key-value structure resemble msg_envelope_v1
|
||||
|
||||
# Example
|
||||
```jldoctest
|
||||
# Receive and process message
|
||||
msg = nats_message # NATS message
|
||||
payloads = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff, max_retries=5, base_delay=100, max_delay=5000)
|
||||
# payloads = [("dataname1", data1, "type1"), ("dataname2", data2, "type2"), ...]
|
||||
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff, max_retries=5, base_delay=100, max_delay=5000)
|
||||
# env["payloads"] = [("dataname1", data1, "type1"), ("dataname2", data2, "type2"), ...]
|
||||
```
|
||||
"""
|
||||
function smartreceive(
|
||||
@@ -770,7 +766,7 @@ function smartreceive(
|
||||
max_retries::Int = 5,
|
||||
base_delay::Int = 100,
|
||||
max_delay::Int = 5000
|
||||
)
|
||||
)::JSON.Object{String, Any}
|
||||
# Parse the JSON envelope
|
||||
env_json_obj = JSON.parse(String(msg.payload))
|
||||
log_trace(env_json_obj["correlation_id"], "Processing received message") # Log message processing start
|
||||
@@ -818,7 +814,7 @@ function smartreceive(
|
||||
end
|
||||
end
|
||||
env_json_obj["payloads"] = payloads_list
|
||||
return env_json_obj # JSON object of envelope with list of (dataname, data, data_type) tuples in payloads field
|
||||
return env_json_obj # key-value structure resemble msg_envelope_v1
|
||||
end
|
||||
|
||||
|
||||
|
||||
674
src/natsbridge.js
Normal file
674
src/natsbridge.js
Normal file
@@ -0,0 +1,674 @@
|
||||
/**
|
||||
* NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||
* JavaScript/Node.js Implementation
|
||||
*
|
||||
* This module provides functionality for sending and receiving data across network boundaries
|
||||
* using NATS as the message bus, with support for both direct payload transport and
|
||||
* URL-based transport for larger payloads.
|
||||
*
|
||||
* @module NATSBridge
|
||||
*/
|
||||
|
||||
const nats = require('nats');
|
||||
const { v4: uuidv4 } = require('uuid');
|
||||
const fetch = require('node-fetch');
|
||||
const arrow = require('apache-arrow');
|
||||
|
||||
// ---------------------------------------------- Constants ---------------------------------------------- //
|
||||
|
||||
/**
|
||||
* Default size threshold for switching from direct to link transport (1MB)
|
||||
*/
|
||||
const DEFAULT_SIZE_THRESHOLD = 1_000_000;
|
||||
|
||||
/**
|
||||
* Default NATS server URL
|
||||
*/
|
||||
const DEFAULT_BROKER_URL = 'nats://localhost:4222';
|
||||
|
||||
/**
|
||||
* Default HTTP file server URL for link transport
|
||||
*/
|
||||
const DEFAULT_FILESERVER_URL = 'http://localhost:8080';
|
||||
|
||||
// ---------------------------------------------- Utility Functions ---------------------------------------------- //
|
||||
|
||||
/**
|
||||
* Convert Buffer to Base64 string
|
||||
* @param {Buffer} buffer - Buffer to encode
|
||||
* @returns {string} Base64 encoded string
|
||||
*/
|
||||
function bufferToBase64(buffer) {
|
||||
return buffer.toString('base64');
|
||||
}
|
||||
|
||||
/**
|
||||
* Log a trace message with correlation ID and timestamp
|
||||
* @param {string} correlationId - Correlation ID for tracing
|
||||
* @param {string} message - Message content to log
|
||||
*/
|
||||
function logTrace(correlationId, message) {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`[${timestamp}] [Correlation: ${correlationId}] ${message}`);
|
||||
}
|
||||
|
||||
// ---------------------------------------------- Serialization Functions ---------------------------------------------- //
|
||||
|
||||
/**
|
||||
* Serialize data according to specified format
|
||||
* @param {any} data - Data to serialize
|
||||
* @param {string} payloadType - Target format: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||
* @returns {Buffer} Binary representation of the serialized data
|
||||
*/
|
||||
async function serializeData(data, payloadType) {
|
||||
if (payloadType === 'text') {
|
||||
if (typeof data === 'string') {
|
||||
return Buffer.from(data, 'utf8');
|
||||
} else {
|
||||
throw new Error('Text data must be a string');
|
||||
}
|
||||
} else if (payloadType === 'dictionary') {
|
||||
const jsonStr = JSON.stringify(data);
|
||||
return Buffer.from(jsonStr, 'utf8');
|
||||
} else if (payloadType === 'table') {
|
||||
// Convert array of objects to Arrow IPC format
|
||||
if (!Array.isArray(data) || data.length === 0) {
|
||||
throw new Error('Table data must be a non-empty array of objects');
|
||||
}
|
||||
|
||||
return serializeArrowTable(data);
|
||||
} else if (payloadType === 'image') {
|
||||
if (data instanceof Uint8Array || Buffer.isBuffer(data)) {
|
||||
return Buffer.from(data);
|
||||
} else {
|
||||
throw new Error('Image data must be Uint8Array or Buffer');
|
||||
}
|
||||
} else if (payloadType === 'audio') {
|
||||
if (data instanceof Uint8Array || Buffer.isBuffer(data)) {
|
||||
return Buffer.from(data);
|
||||
} else {
|
||||
throw new Error('Audio data must be Uint8Array or Buffer');
|
||||
}
|
||||
} else if (payloadType === 'video') {
|
||||
if (data instanceof Uint8Array || Buffer.isBuffer(data)) {
|
||||
return Buffer.from(data);
|
||||
} else {
|
||||
throw new Error('Video data must be Uint8Array or Buffer');
|
||||
}
|
||||
} else if (payloadType === 'binary') {
|
||||
if (data instanceof Uint8Array || Buffer.isBuffer(data)) {
|
||||
return Buffer.from(data);
|
||||
} else {
|
||||
throw new Error('Binary data must be Uint8Array or Buffer');
|
||||
}
|
||||
} else {
|
||||
throw new Error(`Unknown payload_type: ${payloadType}`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper function to properly serialize table data to Arrow IPC
|
||||
* @param {Array<Object>} data - Array of objects representing table rows
|
||||
* @returns {Buffer} Arrow IPC formatted buffer
|
||||
*/
|
||||
function serializeArrowTable(data) {
|
||||
if (!Array.isArray(data) || data.length === 0) {
|
||||
throw new Error('Table data must be a non-empty array of objects');
|
||||
}
|
||||
|
||||
// Build schema from first row
|
||||
const fields = Object.keys(data[0]).map(key => {
|
||||
const value = data[0][key];
|
||||
let arrowType;
|
||||
if (typeof value === 'number') {
|
||||
arrowType = Number.isInteger(value) ? arrow.Int64 : arrow.Float64;
|
||||
} else if (typeof value === 'boolean') {
|
||||
arrowType = arrow.Boolean;
|
||||
} else if (value instanceof Date) {
|
||||
arrowType = arrow.Date;
|
||||
} else {
|
||||
arrowType = arrow.Utf8;
|
||||
}
|
||||
return new arrow.Field(key, arrowType, true);
|
||||
});
|
||||
|
||||
const schema = new arrow.Schema(fields);
|
||||
const batches = [];
|
||||
|
||||
// Create record batches
|
||||
for (const row of data) {
|
||||
const batch = arrow.recordBatch.fromObjects([row], schema);
|
||||
batches.push(batch);
|
||||
}
|
||||
|
||||
// Write to buffer using IPC format
|
||||
const buffers = arrow.ipc.recordBatchesToMessage(batches, schema).buffers;
|
||||
const combined = new Uint8Array(buffers.reduce((acc, b) => acc + b.byteLength, 0));
|
||||
let offset = 0;
|
||||
for (const buf of buffers) {
|
||||
combined.set(new Uint8Array(buf), offset);
|
||||
offset += buf.byteLength;
|
||||
}
|
||||
|
||||
return Buffer.from(combined);
|
||||
}
|
||||
|
||||
/**
|
||||
* Deserialize bytes to data based on type
|
||||
* @param {Buffer|Uint8Array} data - Serialized data as bytes
|
||||
* @param {string} payloadType - Data type
|
||||
* @param {string} correlationId - Correlation ID for logging
|
||||
* @returns {any} Deserialized data
|
||||
*/
|
||||
async function deserializeData(data, payloadType, correlationId) {
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data);
|
||||
|
||||
if (payloadType === 'text') {
|
||||
return buffer.toString('utf8');
|
||||
} else if (payloadType === 'dictionary') {
|
||||
const jsonStr = buffer.toString('utf8');
|
||||
return JSON.parse(jsonStr);
|
||||
} else if (payloadType === 'table') {
|
||||
const table = arrow.tableFromRawBytes(buffer);
|
||||
return table;
|
||||
} else if (payloadType === 'image') {
|
||||
return buffer;
|
||||
} else if (payloadType === 'audio') {
|
||||
return buffer;
|
||||
} else if (payloadType === 'video') {
|
||||
return buffer;
|
||||
} else if (payloadType === 'binary') {
|
||||
return buffer;
|
||||
} else {
|
||||
throw new Error(`Unknown payload_type: ${payloadType}`);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------- File Server Handlers ---------------------------------------------- //
|
||||
|
||||
/**
|
||||
* Upload data to plik server in one-shot mode
|
||||
* @param {string} fileServerUrl - Base URL of the plik server
|
||||
* @param {string} dataname - Name of the file being uploaded
|
||||
* @param {Buffer|Uint8Array} data - Raw byte data of the file content
|
||||
* @returns {Promise<{status: number, uploadid: string, fileid: string, url: string}>}
|
||||
*/
|
||||
async function plikOneshotUpload(fileServerUrl, dataname, data) {
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data);
|
||||
|
||||
// Get upload id
|
||||
const urlGetUploadID = `${fileServerUrl}/upload`;
|
||||
const headers = { 'Content-Type': 'application/json' };
|
||||
const body = JSON.stringify({ OneShot: true });
|
||||
|
||||
const httpResponse = await fetch(urlGetUploadID, {
|
||||
method: 'POST',
|
||||
headers,
|
||||
body
|
||||
});
|
||||
|
||||
const responseJson = await httpResponse.json();
|
||||
const uploadid = responseJson.id;
|
||||
const uploadtoken = responseJson.uploadToken;
|
||||
|
||||
// Upload file
|
||||
const urlUpload = `${fileServerUrl}/file/${uploadid}`;
|
||||
const form = new FormData();
|
||||
const blob = new Blob([buffer], { type: 'application/octet-stream' });
|
||||
form.append('file', blob, dataname);
|
||||
|
||||
const uploadHeaders = {
|
||||
'X-UploadToken': uploadtoken
|
||||
};
|
||||
|
||||
const uploadResponse = await fetch(urlUpload, {
|
||||
method: 'POST',
|
||||
headers: uploadHeaders,
|
||||
body: form
|
||||
});
|
||||
|
||||
const uploadJson = await uploadResponse.json();
|
||||
const fileid = uploadJson.id;
|
||||
|
||||
const url = `${fileServerUrl}/file/${uploadid}/${fileid}/${dataname}`;
|
||||
|
||||
return {
|
||||
status: uploadResponse.status,
|
||||
uploadid,
|
||||
fileid,
|
||||
url
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch data from URL with exponential backoff
|
||||
* @param {string} url - URL to fetch from
|
||||
* @param {number} maxRetries - Maximum number of retry attempts
|
||||
* @param {number} baseDelay - Initial delay in milliseconds
|
||||
* @param {number} maxDelay - Maximum delay in milliseconds
|
||||
* @param {string} correlationId - Correlation ID for logging
|
||||
* @returns {Promise<Uint8Array>} Fetched data as bytes
|
||||
*/
|
||||
async function fetchWithBackoff(url, maxRetries, baseDelay, maxDelay, correlationId) {
|
||||
let delay = baseDelay;
|
||||
|
||||
for (let attempt = 1; attempt <= maxRetries; attempt++) {
|
||||
try {
|
||||
const response = await fetch(url);
|
||||
|
||||
if (response.status === 200) {
|
||||
logTrace(correlationId, `Successfully fetched data from ${url} on attempt ${attempt}`);
|
||||
const arrayBuffer = await response.arrayBuffer();
|
||||
return new Uint8Array(arrayBuffer);
|
||||
} else {
|
||||
throw new Error(`Failed to fetch: ${response.status}`);
|
||||
}
|
||||
} catch (e) {
|
||||
logTrace(correlationId, `Attempt ${attempt} failed: ${e.constructor.name}`);
|
||||
|
||||
if (attempt < maxRetries) {
|
||||
await new Promise(resolve => setTimeout(resolve, delay));
|
||||
delay = Math.min(delay * 2, maxDelay);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error(`Failed to fetch data after ${maxRetries} attempts`);
|
||||
}
|
||||
|
||||
// ---------------------------------------------- NATS Client ---------------------------------------------- //
|
||||
|
||||
/**
|
||||
* NATS client wrapper for connection management
|
||||
*/
|
||||
class NATSClient {
|
||||
/**
|
||||
* Create a new NATS client
|
||||
* @param {string} url - NATS server URL
|
||||
*/
|
||||
constructor(url) {
|
||||
this.url = url;
|
||||
this.connection = null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Connect to NATS server
|
||||
* @returns {Promise<NATS.Connection>}
|
||||
*/
|
||||
async connect() {
|
||||
this.connection = await nats.connect({ servers: this.url });
|
||||
return this.connection;
|
||||
}
|
||||
|
||||
/**
|
||||
* Publish message to NATS subject
|
||||
* @param {string} subject - NATS subject to publish to
|
||||
* @param {string} message - Message to publish
|
||||
* @param {string} correlationId - Correlation ID for logging
|
||||
*/
|
||||
async publish(subject, message, correlationId) {
|
||||
if (!this.connection) {
|
||||
await this.connect();
|
||||
}
|
||||
await this.connection.publish(subject, message);
|
||||
logTrace(correlationId, `Message published to ${subject}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Close the NATS connection
|
||||
*/
|
||||
async close() {
|
||||
if (this.connection) {
|
||||
this.connection.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------- Core Functions ---------------------------------------------- //
|
||||
|
||||
/**
|
||||
* Publish message to NATS
|
||||
* @param {string|NATSClient|NATS.Connection} brokerUrlOrClient - NATS URL, client, or connection
|
||||
* @param {string} subject - NATS subject to publish to
|
||||
* @param {string} message - JSON message to publish
|
||||
* @param {string} correlationId - Correlation ID for tracing
|
||||
*/
|
||||
async function publishMessage(brokerUrlOrClient, subject, message, correlationId) {
|
||||
let conn;
|
||||
|
||||
if (brokerUrlOrClient instanceof NATSClient) {
|
||||
conn = brokerUrlOrClient;
|
||||
} else if (brokerUrlOrClient instanceof nats.Connection) {
|
||||
// Create a wrapper for direct connection
|
||||
conn = {
|
||||
async publish(subj, msg) {
|
||||
await brokerUrlOrClient.publish(subj, msg);
|
||||
},
|
||||
async close() {
|
||||
await brokerUrlOrClient.close();
|
||||
}
|
||||
};
|
||||
} else {
|
||||
// String URL - create new client
|
||||
const client = new NATSClient(brokerUrlOrClient);
|
||||
conn = client;
|
||||
}
|
||||
|
||||
await conn.publish(subject, message, correlationId);
|
||||
|
||||
if (conn instanceof NATSClient) {
|
||||
await conn.close();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build message envelope from payloads and metadata
|
||||
* @param {string} subject - NATS subject
|
||||
* @param {Array} payloads - Array of payload objects
|
||||
* @param {Object} options - Envelope metadata options
|
||||
* @returns {Object} Envelope object
|
||||
*/
|
||||
function buildEnvelope(subject, payloads, options) {
|
||||
return {
|
||||
correlation_id: options.correlation_id,
|
||||
msg_id: options.msg_id,
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: subject,
|
||||
msg_purpose: options.msg_purpose,
|
||||
sender_name: options.sender_name,
|
||||
sender_id: options.sender_id,
|
||||
receiver_name: options.receiver_name,
|
||||
receiver_id: options.receiver_id,
|
||||
reply_to: options.reply_to,
|
||||
reply_to_msg_id: options.reply_to_msg_id,
|
||||
broker_url: options.broker_url,
|
||||
metadata: options.metadata || {},
|
||||
payloads: payloads
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Build payload object from serialized data
|
||||
* @param {string} dataname - Name of the payload
|
||||
* @param {string} payloadType - Type of the payload
|
||||
* @param {Buffer} payloadBytes - Serialized payload bytes
|
||||
* @param {string} transport - Transport type ("direct" or "link")
|
||||
* @param {string} data - Data (base64 for direct, URL for link)
|
||||
* @returns {Object} Payload object
|
||||
*/
|
||||
function buildPayload(dataname, payloadType, payloadBytes, transport, data) {
|
||||
return {
|
||||
id: uuidv4(),
|
||||
dataname,
|
||||
payload_type: payloadType,
|
||||
transport,
|
||||
encoding: transport === 'direct' ? 'base64' : 'none',
|
||||
size: payloadBytes.byteLength,
|
||||
data,
|
||||
metadata: transport === 'direct' ? { payload_bytes: payloadBytes.byteLength } : {}
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Send data via NATS with automatic transport selection
|
||||
*
|
||||
* This function intelligently routes data delivery based on payload size.
|
||||
* If the serialized payload is smaller than size_threshold, it encodes the data as Base64
|
||||
* and publishes directly over NATS. Otherwise, it uploads the data to a fileserver
|
||||
* and publishes only the download URL over NATS.
|
||||
*
|
||||
* @param {string} subject - NATS subject to publish the message to
|
||||
* @param {Array} data - List of [dataname, data, type] tuples to send
|
||||
* @param {Object} options - Optional configuration
|
||||
* @param {string} [options.broker_url=DEFAULT_BROKER_URL] - URL of the NATS server
|
||||
* @param {string} [options.fileserver_url=DEFAULT_FILESERVER_URL] - URL of the HTTP file server
|
||||
* @param {Function} [options.fileserver_upload_handler=plikOneshotUpload] - Function to handle fileserver uploads
|
||||
* @param {number} [options.size_threshold=DEFAULT_SIZE_THRESHOLD] - Threshold separating direct vs link transport
|
||||
* @param {string} [options.correlation_id=uuidv4()] - Correlation ID for tracing
|
||||
* @param {string} [options.msg_purpose="chat"] - Purpose of the message
|
||||
* @param {string} [options.sender_name="NATSBridge"] - Name of the sender
|
||||
* @param {string} [options.receiver_name=""] - Name of the receiver (empty means broadcast)
|
||||
* @param {string} [options.receiver_id=""] - UUID of the receiver (empty means broadcast)
|
||||
* @param {string} [options.reply_to=""] - Topic to reply to
|
||||
* @param {string} [options.reply_to_msg_id=""] - Message ID this message is replying to
|
||||
* @param {boolean} [options.is_publish=true] - Whether to automatically publish the message
|
||||
* @param {NATSClient|NATS.Connection} [options.nats_connection=null] - Pre-existing NATS connection
|
||||
* @param {string} [options.msg_id=uuidv4()] - Message ID
|
||||
* @param {string} [options.sender_id=uuidv4()] - Sender ID
|
||||
* @returns {Promise<[Object, string]>} Tuple of [env, env_json_str]
|
||||
*
|
||||
* @example
|
||||
* // Send a single payload
|
||||
* const [env, envJsonStr] = await smartsend(
|
||||
* "/test",
|
||||
* [["dataname1", data1, "dictionary"]],
|
||||
* { broker_url: "nats://localhost:4222" }
|
||||
* );
|
||||
*
|
||||
* // Send multiple payloads
|
||||
* const [env, envJsonStr] = await smartsend(
|
||||
* "/test",
|
||||
* [
|
||||
* ["dataname1", data1, "dictionary"],
|
||||
* ["dataname2", data2, "table"]
|
||||
* ],
|
||||
* { broker_url: "nats://localhost:4222" }
|
||||
* );
|
||||
*
|
||||
* // Send with pre-existing connection
|
||||
* const client = await NATSBridge.NATSClient.connect("nats://localhost:4222");
|
||||
* const [env, envJsonStr] = await smartsend(
|
||||
* "/test",
|
||||
* [["data", myData, "text"]],
|
||||
* { nats_connection: client }
|
||||
* );
|
||||
*/
|
||||
async function smartsend(subject, data, options = {}) {
|
||||
const {
|
||||
broker_url = DEFAULT_BROKER_URL,
|
||||
fileserver_url = DEFAULT_FILESERVER_URL,
|
||||
fileserver_upload_handler = plikOneshotUpload,
|
||||
size_threshold = DEFAULT_SIZE_THRESHOLD,
|
||||
correlation_id = uuidv4(),
|
||||
msg_purpose = 'chat',
|
||||
sender_name = 'NATSBridge',
|
||||
receiver_name = '',
|
||||
receiver_id = '',
|
||||
reply_to = '',
|
||||
reply_to_msg_id = '',
|
||||
is_publish = true,
|
||||
nats_connection = null,
|
||||
msg_id = uuidv4(),
|
||||
sender_id = uuidv4()
|
||||
} = options;
|
||||
|
||||
logTrace(correlation_id, `Starting smartsend for subject: ${subject}`);
|
||||
|
||||
// Process payloads
|
||||
const payloads = [];
|
||||
for (const [dataname, payloadData, payloadType] of data) {
|
||||
const payloadBytes = await serializeData(payloadData, payloadType);
|
||||
const payloadSize = payloadBytes.byteLength;
|
||||
|
||||
logTrace(correlation_id, `Serialized payload '${dataname}' (type: ${payloadType}) size: ${payloadSize} bytes`);
|
||||
|
||||
if (payloadSize < size_threshold) {
|
||||
// Direct path
|
||||
const payloadB64 = bufferToBase64(payloadBytes);
|
||||
logTrace(correlation_id, `Using direct transport for ${payloadSize} bytes`);
|
||||
|
||||
const payload = buildPayload(dataname, payloadType, payloadBytes, 'direct', payloadB64);
|
||||
payloads.push(payload);
|
||||
} else {
|
||||
// Link path
|
||||
logTrace(correlation_id, `Using link transport, uploading to fileserver`);
|
||||
|
||||
const response = await fileserver_upload_handler(fileserver_url, dataname, payloadBytes);
|
||||
|
||||
if (response.status !== 200) {
|
||||
throw new Error(`Failed to upload data to fileserver: ${response.status}`);
|
||||
}
|
||||
|
||||
logTrace(correlation_id, `Uploaded to URL: ${response.url}`);
|
||||
|
||||
const payload = buildPayload(dataname, payloadType, payloadBytes, 'link', response.url);
|
||||
payloads.push(payload);
|
||||
}
|
||||
}
|
||||
|
||||
// Build envelope
|
||||
const env = buildEnvelope(subject, payloads, {
|
||||
correlation_id,
|
||||
msg_id,
|
||||
msg_purpose,
|
||||
sender_name,
|
||||
sender_id,
|
||||
receiver_name,
|
||||
receiver_id,
|
||||
reply_to,
|
||||
reply_to_msg_id,
|
||||
broker_url
|
||||
});
|
||||
|
||||
const env_json_str = JSON.stringify(env);
|
||||
|
||||
if (is_publish) {
|
||||
if (nats_connection) {
|
||||
await publishMessage(nats_connection, subject, env_json_str, correlation_id);
|
||||
} else {
|
||||
await publishMessage(broker_url, subject, env_json_str, correlation_id);
|
||||
}
|
||||
}
|
||||
|
||||
return [env, env_json_str];
|
||||
}
|
||||
|
||||
/**
|
||||
* Receive and process NATS message
|
||||
*
|
||||
* This function processes incoming NATS messages, handling both direct transport
|
||||
* (base64 decoded payloads) and link transport (URL-based payloads).
|
||||
* It deserializes the data based on the transport type and returns the result.
|
||||
*
|
||||
* @param {Object} msg - NATS message object with payload property
|
||||
* @param {Object} options - Optional configuration
|
||||
* @param {Function} [options.fileserver_download_handler=fetchWithBackoff] - Function to handle fileserver downloads
|
||||
* @param {number} [options.max_retries=5] - Maximum retry attempts for fetching URL
|
||||
* @param {number} [options.base_delay=100] - Initial delay for exponential backoff in ms
|
||||
* @param {number} [options.max_delay=5000] - Maximum delay for exponential backoff in ms
|
||||
* @returns {Promise<Object>} Envelope object with processed payloads
|
||||
*
|
||||
* @example
|
||||
* // Receive and process message
|
||||
* const env = await smartreceive(msg, {
|
||||
* fileserver_download_handler: fetchWithBackoff,
|
||||
* max_retries: 5,
|
||||
* base_delay: 100,
|
||||
* max_delay: 5000
|
||||
* });
|
||||
* // env.payloads is an Array of [dataname, data, type] arrays
|
||||
* for (const [dataname, data, type] of env.payloads) {
|
||||
* console.log(`${dataname}: ${data} (type: ${type})`);
|
||||
* }
|
||||
*/
|
||||
async function smartreceive(msg, options = {}) {
|
||||
const {
|
||||
fileserver_download_handler = fetchWithBackoff,
|
||||
max_retries = 5,
|
||||
base_delay = 100,
|
||||
max_delay = 5000
|
||||
} = options;
|
||||
|
||||
// Parse the JSON envelope
|
||||
const payload = typeof msg.payload === 'string' ? msg.payload : Buffer.from(msg.payload).toString('utf8');
|
||||
const envJsonObj = JSON.parse(payload);
|
||||
logTrace(envJsonObj.correlation_id, 'Processing received message');
|
||||
|
||||
// Process all payloads in the envelope
|
||||
const payloadsList = [];
|
||||
const numPayloads = envJsonObj.payloads.length;
|
||||
|
||||
for (let i = 0; i < numPayloads; i++) {
|
||||
const payloadObj = envJsonObj.payloads[i];
|
||||
const transport = payloadObj.transport;
|
||||
const dataname = payloadObj.dataname;
|
||||
|
||||
if (transport === 'direct') {
|
||||
logTrace(envJsonObj.correlation_id, `Direct transport - decoding payload '${dataname}'`);
|
||||
|
||||
// Extract base64 payload from the payload
|
||||
const payloadB64 = payloadObj.data;
|
||||
|
||||
// Decode Base64 payload
|
||||
const payloadBytes = Buffer.from(payloadB64, 'base64');
|
||||
|
||||
// Deserialize based on type
|
||||
const dataType = payloadObj.payload_type;
|
||||
const data = await deserializeData(payloadBytes, dataType, envJsonObj.correlation_id);
|
||||
|
||||
payloadsList.push([dataname, data, dataType]);
|
||||
} else if (transport === 'link') {
|
||||
// Extract download URL from the payload
|
||||
const url = payloadObj.data;
|
||||
logTrace(envJsonObj.correlation_id, `Link transport - fetching '${dataname}' from URL: ${url}`);
|
||||
|
||||
// Fetch with exponential backoff using the download handler
|
||||
const downloadedData = await fileserver_download_handler(
|
||||
url,
|
||||
max_retries,
|
||||
base_delay,
|
||||
max_delay,
|
||||
envJsonObj.correlation_id
|
||||
);
|
||||
|
||||
// Deserialize based on type
|
||||
const dataType = payloadObj.payload_type;
|
||||
const data = await deserializeData(downloadedData, dataType, envJsonObj.correlation_id);
|
||||
|
||||
payloadsList.push([dataname, data, dataType]);
|
||||
} else {
|
||||
throw new Error(`Unknown transport type for payload '${dataname}': ${transport}`);
|
||||
}
|
||||
}
|
||||
|
||||
envJsonObj.payloads = payloadsList;
|
||||
return envJsonObj;
|
||||
}
|
||||
|
||||
// ---------------------------------------------- Module Exports ---------------------------------------------- //
|
||||
|
||||
const NATSBridge = {
|
||||
/**
|
||||
* NATS client class for connection management
|
||||
*/
|
||||
NATSClient,
|
||||
|
||||
/**
|
||||
* Send data via NATS with automatic transport selection
|
||||
*/
|
||||
smartsend,
|
||||
|
||||
/**
|
||||
* Receive and process NATS message
|
||||
*/
|
||||
smartreceive,
|
||||
|
||||
/**
|
||||
* Upload data to plik server in one-shot mode
|
||||
*/
|
||||
plikOneshotUpload,
|
||||
|
||||
/**
|
||||
* Fetch data from URL with exponential backoff
|
||||
*/
|
||||
fetchWithBackoff,
|
||||
|
||||
/**
|
||||
* Default constants
|
||||
*/
|
||||
DEFAULT_SIZE_THRESHOLD,
|
||||
DEFAULT_BROKER_URL,
|
||||
DEFAULT_FILESERVER_URL
|
||||
};
|
||||
|
||||
module.exports = NATSBridge;
|
||||
815
src/natsbridge.py
Normal file
815
src/natsbridge.py
Normal file
@@ -0,0 +1,815 @@
|
||||
"""
|
||||
NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||
Python Desktop Implementation
|
||||
|
||||
This module provides functionality for sending and receiving data across network boundaries
|
||||
using NATS as the message bus, with support for both direct payload transport and
|
||||
URL-based transport for larger payloads.
|
||||
|
||||
@package natsbridge
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from typing import Any, Callable, Dict, List, Tuple, Union
|
||||
import aiohttp
|
||||
|
||||
try:
|
||||
import pyarrow as arrow
|
||||
import pyarrow.ipc as ipc
|
||||
ARROW_AVAILABLE = True
|
||||
except ImportError:
|
||||
ARROW_AVAILABLE = False
|
||||
|
||||
try:
|
||||
import nats
|
||||
from nats.aio.client import Client as NATSClient
|
||||
NATS_AVAILABLE = True
|
||||
except ImportError:
|
||||
NATS_AVAILABLE = False
|
||||
|
||||
# ---------------------------------------------- Constants ---------------------------------------------- #
|
||||
|
||||
"""
|
||||
Default size threshold for switching from direct to link transport (1MB)
|
||||
"""
|
||||
DEFAULT_SIZE_THRESHOLD = 1_000_000
|
||||
|
||||
"""
|
||||
Default NATS server URL
|
||||
"""
|
||||
DEFAULT_BROKER_URL = "nats://localhost:4222"
|
||||
|
||||
"""
|
||||
Default HTTP file server URL for link transport
|
||||
"""
|
||||
DEFAULT_FILESERVER_URL = "http://localhost:8080"
|
||||
|
||||
|
||||
# ---------------------------------------------- Utility Functions ---------------------------------------------- #
|
||||
|
||||
def log_trace(correlation_id: str, message: str) -> None:
|
||||
"""
|
||||
Log a trace message with correlation ID and timestamp.
|
||||
|
||||
Args:
|
||||
correlation_id: Correlation ID for tracing
|
||||
message: Message content to log
|
||||
"""
|
||||
timestamp = datetime.utcnow().isoformat() + 'Z'
|
||||
print(f"[{timestamp}] [Correlation: {correlation_id}] {message}")
|
||||
|
||||
|
||||
# ---------------------------------------------- Serialization Functions ---------------------------------------------- #
|
||||
|
||||
def _serialize_data(data: Any, payload_type: str) -> bytes:
|
||||
"""
|
||||
Serialize data according to specified format.
|
||||
|
||||
Args:
|
||||
data: Data to serialize (string for "text", JSON-serializable for "dictionary",
|
||||
table-like for "table", binary for "image", "audio", "video", "binary")
|
||||
payload_type: Target format: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||
|
||||
Returns:
|
||||
Binary representation of the serialized data
|
||||
|
||||
Raises:
|
||||
Error: If payload_type is not one of the supported types
|
||||
Error: If payload_type is "image", "audio", or "video" but data is not bytes
|
||||
Error: If payload_type is "table" but data is not a pandas DataFrame or pyarrow Table
|
||||
"""
|
||||
if payload_type == 'text':
|
||||
if isinstance(data, str):
|
||||
return data.encode('utf-8')
|
||||
else:
|
||||
raise ValueError('Text data must be a string')
|
||||
elif payload_type == 'dictionary':
|
||||
json_str = json.dumps(data)
|
||||
return json_str.encode('utf-8')
|
||||
elif payload_type == 'table':
|
||||
if not ARROW_AVAILABLE:
|
||||
raise RuntimeError('pyarrow not available for table serialization')
|
||||
|
||||
import io
|
||||
buf = io.BytesIO()
|
||||
|
||||
import pandas as pd
|
||||
if isinstance(data, pd.DataFrame):
|
||||
table = arrow.Table.from_pandas(data)
|
||||
sink = ipc.new_file(buf, table.schema)
|
||||
ipc.write_table(table, sink)
|
||||
sink.close()
|
||||
return buf.getvalue()
|
||||
elif isinstance(data, arrow.Table):
|
||||
sink = ipc.new_file(buf, data.schema)
|
||||
ipc.write_table(data, sink)
|
||||
sink.close()
|
||||
return buf.getvalue()
|
||||
else:
|
||||
raise ValueError('Table data must be a pandas DataFrame or pyarrow Table')
|
||||
elif payload_type == 'image':
|
||||
if isinstance(data, (bytes, bytearray)):
|
||||
return bytes(data)
|
||||
else:
|
||||
raise ValueError('Image data must be bytes')
|
||||
elif payload_type == 'audio':
|
||||
if isinstance(data, (bytes, bytearray)):
|
||||
return bytes(data)
|
||||
else:
|
||||
raise ValueError('Audio data must be bytes')
|
||||
elif payload_type == 'video':
|
||||
if isinstance(data, (bytes, bytearray)):
|
||||
return bytes(data)
|
||||
else:
|
||||
raise ValueError('Video data must be bytes')
|
||||
elif payload_type == 'binary':
|
||||
if isinstance(data, (bytes, bytearray)):
|
||||
return bytes(data)
|
||||
else:
|
||||
raise ValueError('Binary data must be bytes')
|
||||
else:
|
||||
raise ValueError(f'Unknown payload_type: {payload_type}')
|
||||
|
||||
|
||||
def _deserialize_data(data: bytes, payload_type: str, correlation_id: str) -> Any:
|
||||
"""
|
||||
Deserialize bytes to data based on type.
|
||||
|
||||
Args:
|
||||
data: Serialized data as bytes
|
||||
payload_type: Data type ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
||||
correlation_id: Correlation ID for logging
|
||||
|
||||
Returns:
|
||||
Deserialized data (String for "text", DataFrame for "table", JSON data for "dictionary",
|
||||
bytes for "image", "audio", "video", "binary")
|
||||
|
||||
Raises:
|
||||
Error: If payload_type is not one of the supported types
|
||||
"""
|
||||
if payload_type == 'text':
|
||||
return data.decode('utf-8')
|
||||
elif payload_type == 'dictionary':
|
||||
json_str = data.decode('utf-8')
|
||||
return json.loads(json_str)
|
||||
elif payload_type == 'table':
|
||||
if not ARROW_AVAILABLE:
|
||||
raise RuntimeError('pyarrow not available for table deserialization')
|
||||
|
||||
import io
|
||||
buf = io.BytesIO(data)
|
||||
reader = ipc.open_file(buf)
|
||||
return reader.read_all().to_pandas()
|
||||
elif payload_type == 'image':
|
||||
return data
|
||||
elif payload_type == 'audio':
|
||||
return data
|
||||
elif payload_type == 'video':
|
||||
return data
|
||||
elif payload_type == 'binary':
|
||||
return data
|
||||
else:
|
||||
raise ValueError(f'Unknown payload_type: {payload_type}')
|
||||
|
||||
|
||||
# ---------------------------------------------- File Server Handlers ---------------------------------------------- #
|
||||
|
||||
async def plik_oneshot_upload(
|
||||
file_server_url: str,
|
||||
dataname: str,
|
||||
data: bytes
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Upload data to plik server in one-shot mode.
|
||||
|
||||
This function uploads a raw byte array to a plik server in one-shot mode (no upload session).
|
||||
It first creates a one-shot upload session by sending a POST request with {"OneShot": true},
|
||||
retrieves an upload ID and token, then uploads the file data as multipart form data using the token.
|
||||
|
||||
Args:
|
||||
file_server_url: Base URL of the plik server (e.g., "http://localhost:8080")
|
||||
dataname: Name of the file being uploaded
|
||||
data: Raw byte data of the file content
|
||||
|
||||
Returns:
|
||||
Dict with keys:
|
||||
- "status": HTTP server response status
|
||||
- "uploadid": ID of the one-shot upload session
|
||||
- "fileid": ID of the uploaded file within the session
|
||||
- "url": Full URL to download the uploaded file
|
||||
|
||||
Example:
|
||||
>>> fileserver_url = "http://localhost:8080"
|
||||
>>> dataname = "test.txt"
|
||||
>>> data = b"hello world"
|
||||
>>> result = await plik_oneshot_upload(file_server_url, dataname, data)
|
||||
>>> result["status"], result["uploadid"], result["fileid"], result["url"]
|
||||
"""
|
||||
async with aiohttp.ClientSession() as session:
|
||||
# Get upload id
|
||||
url_getUploadID = f"{file_server_url}/upload"
|
||||
headers = {'Content-Type': 'application/json'}
|
||||
body = json.dumps({"OneShot": True})
|
||||
|
||||
async with session.post(url_getUploadID, headers=headers, data=body) as response:
|
||||
response_json = await response.json()
|
||||
uploadid = response_json['id']
|
||||
uploadtoken = response_json['uploadToken']
|
||||
|
||||
# Upload file
|
||||
url_upload = f"{file_server_url}/file/{uploadid}"
|
||||
headers = {'X-UploadToken': uploadtoken}
|
||||
|
||||
form = aiohttp.FormData()
|
||||
form.add_field('file', data, filename=dataname, content_type='application/octet-stream')
|
||||
|
||||
async with session.post(url_upload, headers=headers, data=form) as upload_response:
|
||||
upload_json = await upload_response.json()
|
||||
fileid = upload_json['id']
|
||||
|
||||
url = f"{file_server_url}/file/{uploadid}/{fileid}/{dataname}"
|
||||
|
||||
return {
|
||||
'status': upload_response.status,
|
||||
'uploadid': uploadid,
|
||||
'fileid': fileid,
|
||||
'url': url
|
||||
}
|
||||
|
||||
|
||||
async def fetch_with_backoff(
|
||||
url: str,
|
||||
max_retries: int,
|
||||
base_delay: int,
|
||||
max_delay: int,
|
||||
correlation_id: str
|
||||
) -> bytes:
|
||||
"""
|
||||
Fetch data from URL with exponential backoff.
|
||||
|
||||
This internal function retrieves data from a URL with retry logic using
|
||||
exponential backoff to handle transient failures.
|
||||
|
||||
Args:
|
||||
url: URL to fetch from
|
||||
max_retries: Maximum number of retry attempts
|
||||
base_delay: Initial delay in milliseconds
|
||||
max_delay: Maximum delay in milliseconds
|
||||
correlation_id: Correlation ID for logging
|
||||
|
||||
Returns:
|
||||
Fetched data as bytes
|
||||
|
||||
Raises:
|
||||
Error: If all retry attempts fail
|
||||
|
||||
Example:
|
||||
>>> data = await fetch_with_backoff("http://example.com/file.zip", 5, 100, 5000, "correlation123")
|
||||
"""
|
||||
delay = base_delay
|
||||
|
||||
for attempt in range(1, max_retries + 1):
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url) as response:
|
||||
if response.status == 200:
|
||||
log_trace(correlation_id, f"Successfully fetched data from {url} on attempt {attempt}")
|
||||
return await response.read()
|
||||
else:
|
||||
raise Exception(f"Failed to fetch: {response.status}")
|
||||
except Exception as e:
|
||||
log_trace(correlation_id, f"Attempt {attempt} failed: {type(e).__name__}")
|
||||
|
||||
if attempt < max_retries:
|
||||
await asyncio.sleep(delay / 1000.0)
|
||||
delay = min(delay * 2, max_delay)
|
||||
|
||||
raise Exception(f"Failed to fetch data after {max_retries} attempts")
|
||||
|
||||
|
||||
# ---------------------------------------------- NATS Client ---------------------------------------------- #
|
||||
|
||||
class NATSClient:
|
||||
"""NATS client wrapper for connection management."""
|
||||
|
||||
def __init__(self, url: str = DEFAULT_BROKER_URL):
|
||||
"""
|
||||
Create a new NATS client.
|
||||
|
||||
Args:
|
||||
url: NATS server URL
|
||||
"""
|
||||
self.url = url
|
||||
self._client: NATSClient = None
|
||||
|
||||
async def connect(self) -> NATSClient:
|
||||
"""
|
||||
Connect to NATS server.
|
||||
|
||||
Returns:
|
||||
NATS client instance
|
||||
"""
|
||||
if NATS_AVAILABLE:
|
||||
self._client = nats.connect(self.url)
|
||||
await self._client
|
||||
else:
|
||||
raise Error('nats-py not available')
|
||||
return self._client
|
||||
|
||||
async def publish(self, subject: str, message: str, correlation_id: str = "") -> None:
|
||||
"""
|
||||
Publish message to NATS subject.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
message: Message to publish
|
||||
correlation_id: Correlation ID for logging
|
||||
"""
|
||||
if self._client:
|
||||
await self._client.publish(subject, message)
|
||||
if correlation_id:
|
||||
log_trace(correlation_id, f"Message published to {subject}")
|
||||
|
||||
async def close(self) -> None:
|
||||
"""Close the NATS connection."""
|
||||
if self._client:
|
||||
await self._client.drain()
|
||||
await self._client.close()
|
||||
|
||||
|
||||
# ---------------------------------------------- Core Functions ---------------------------------------------- #
|
||||
|
||||
def _build_envelope(
|
||||
subject: str,
|
||||
payloads: List[Dict[str, Any]],
|
||||
options: Dict[str, Any]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Build message envelope from payloads and metadata.
|
||||
|
||||
Args:
|
||||
subject: NATS subject
|
||||
payloads: Array of payload objects
|
||||
options: Envelope metadata options
|
||||
|
||||
Returns:
|
||||
Envelope object
|
||||
"""
|
||||
return {
|
||||
'correlation_id': options['correlation_id'],
|
||||
'msg_id': options['msg_id'],
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'send_to': subject,
|
||||
'msg_purpose': options['msg_purpose'],
|
||||
'sender_name': options['sender_name'],
|
||||
'sender_id': options['sender_id'],
|
||||
'receiver_name': options['receiver_name'],
|
||||
'receiver_id': options['receiver_id'],
|
||||
'reply_to': options['reply_to'],
|
||||
'reply_to_msg_id': options['reply_to_msg_id'],
|
||||
'broker_url': options['broker_url'],
|
||||
'metadata': options.get('metadata', {}),
|
||||
'payloads': payloads
|
||||
}
|
||||
|
||||
|
||||
def _build_payload(
|
||||
dataname: str,
|
||||
payload_type: str,
|
||||
payload_bytes: bytes,
|
||||
transport: str,
|
||||
data: Union[str, bytes]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Build payload object from serialized data.
|
||||
|
||||
Args:
|
||||
dataname: Name of the payload
|
||||
payload_type: Type of the payload
|
||||
payload_bytes: Serialized payload bytes
|
||||
transport: Transport type ("direct" or "link")
|
||||
data: Data (base64 for direct, URL for link)
|
||||
|
||||
Returns:
|
||||
Payload object
|
||||
"""
|
||||
return {
|
||||
'id': str(uuid.uuid4()),
|
||||
'dataname': dataname,
|
||||
'payload_type': payload_type,
|
||||
'transport': transport,
|
||||
'encoding': 'base64' if transport == 'direct' else 'none',
|
||||
'size': len(payload_bytes),
|
||||
'data': data,
|
||||
'metadata': {'payload_bytes': len(payload_bytes)} if transport == 'direct' else {}
|
||||
}
|
||||
|
||||
|
||||
async def publish_message(
|
||||
broker_url_or_client: Union[str, NATSClient, Any],
|
||||
subject: str,
|
||||
message: str,
|
||||
correlation_id: str
|
||||
) -> None:
|
||||
"""
|
||||
Publish message to NATS.
|
||||
|
||||
Args:
|
||||
broker_url_or_client: NATS URL, client, or connection
|
||||
subject: NATS subject to publish to
|
||||
message: JSON message to publish
|
||||
correlation_id: Correlation ID for tracing
|
||||
"""
|
||||
if isinstance(broker_url_or_client, NATSClient):
|
||||
client = broker_url_or_client
|
||||
elif NATS_AVAILABLE and hasattr(broker_url_or_client, 'publish'):
|
||||
# Direct NATS client connection
|
||||
await broker_url_or_client.publish(subject, message)
|
||||
log_trace(correlation_id, f"Message published to {subject}")
|
||||
return
|
||||
else:
|
||||
# String URL - create new client
|
||||
client = NATSClient(broker_url_or_client)
|
||||
await client.connect()
|
||||
|
||||
await client.publish(subject, message, correlation_id)
|
||||
|
||||
if isinstance(broker_url_or_client, NATSClient):
|
||||
await broker_url_or_client.close()
|
||||
elif not (NATS_AVAILABLE and hasattr(broker_url_or_client, 'publish')):
|
||||
await client.close()
|
||||
|
||||
|
||||
async def smartsend(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
broker_url: str = DEFAULT_BROKER_URL,
|
||||
fileserver_url: str = DEFAULT_FILESERVER_URL,
|
||||
fileserver_upload_handler: Callable = plik_oneshot_upload,
|
||||
size_threshold: int = DEFAULT_SIZE_THRESHOLD,
|
||||
correlation_id: str = None,
|
||||
msg_purpose: str = "chat",
|
||||
sender_name: str = "NATSBridge",
|
||||
receiver_name: str = "",
|
||||
receiver_id: str = "",
|
||||
reply_to: str = "",
|
||||
reply_to_msg_id: str = "",
|
||||
is_publish: bool = True,
|
||||
nats_connection: Any = None,
|
||||
msg_id: str = None,
|
||||
sender_id: str = None
|
||||
) -> Tuple[Dict, str]:
|
||||
"""
|
||||
Send data via NATS with automatic transport selection.
|
||||
|
||||
This function intelligently routes data delivery based on payload size.
|
||||
If the serialized payload is smaller than size_threshold, it encodes the data as Base64
|
||||
and publishes directly over NATS. Otherwise, it uploads the data to a fileserver
|
||||
and publishes only the download URL over NATS.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish the message to
|
||||
data: List of (dataname, data, type) tuples to send
|
||||
- dataname: Name of the payload
|
||||
- data: The actual data to send
|
||||
- type: Payload type: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||
broker_url: URL of the NATS server
|
||||
fileserver_url: URL of the HTTP file server for large payloads
|
||||
fileserver_upload_handler: Function to handle fileserver uploads (must return Dict with "status",
|
||||
"uploadid", "fileid", "url" keys)
|
||||
size_threshold: Threshold in bytes separating direct vs link transport
|
||||
correlation_id: Correlation ID for tracing (auto-generated UUID if not provided)
|
||||
msg_purpose: Purpose of the message: "ACK", "NACK", "updateStatus", "shutdown", "chat", etc.
|
||||
sender_name: Name of the sender
|
||||
receiver_name: Name of the receiver (empty string means broadcast)
|
||||
receiver_id: UUID of the receiver (empty string means broadcast)
|
||||
reply_to: Topic to reply to (empty string if no reply expected)
|
||||
reply_to_msg_id: Message ID this message is replying to
|
||||
is_publish: Whether to automatically publish the message to NATS
|
||||
nats_connection: Pre-existing NATS connection (if provided, uses this connection instead of
|
||||
creating a new one; saves connection establishment overhead)
|
||||
msg_id: Message ID (auto-generated UUID if not provided)
|
||||
sender_id: Sender ID (auto-generated UUID if not provided)
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str) where:
|
||||
- env: Dict containing all metadata and payloads
|
||||
- env_json_str: JSON string for publishing to NATS
|
||||
|
||||
Example:
|
||||
>>> # Send a single payload (still wrapped in a list)
|
||||
>>> data = {"key": "value"}
|
||||
>>> env, env_json_str = await smartsend(
|
||||
... "my.subject",
|
||||
... [("dataname1", data, "dictionary")],
|
||||
... broker_url="nats://localhost:4222"
|
||||
... )
|
||||
>>>
|
||||
>>> # Send multiple payloads with different types
|
||||
>>> data1 = {"key1": "value1"}
|
||||
>>> data2 = [1, 2, 3, 4, 5]
|
||||
>>> env, env_json_str = await smartsend(
|
||||
... "my.subject",
|
||||
... [("dataname1", data1, "dictionary"), ("dataname2", data2, "table")]
|
||||
... )
|
||||
>>>
|
||||
>>> # Send a large array using fileserver upload
|
||||
>>> data = list(range(10_000_000)) # ~80 MB
|
||||
>>> env, env_json_str = await smartsend(
|
||||
... "large.data",
|
||||
... [("large_table", data, "table")]
|
||||
... )
|
||||
>>>
|
||||
>>> # Mixed content (e.g., chat with text and image)
|
||||
>>> env, env_json_str = await smartsend(
|
||||
... "chat.subject",
|
||||
... [
|
||||
... ("message_text", "Hello!", "text"),
|
||||
... ("user_image", image_data, "image"),
|
||||
... ("audio_clip", audio_data, "audio")
|
||||
... ]
|
||||
... )
|
||||
>>>
|
||||
>>> # Publish the JSON string directly using NATS request-reply pattern
|
||||
>>> # reply = await nats.request(broker_url, subject, env_json_str, reply_to=reply_to_topic)
|
||||
"""
|
||||
if correlation_id is None:
|
||||
correlation_id = str(uuid.uuid4())
|
||||
if msg_id is None:
|
||||
msg_id = str(uuid.uuid4())
|
||||
if sender_id is None:
|
||||
sender_id = str(uuid.uuid4())
|
||||
|
||||
log_trace(correlation_id, f"Starting smartsend for subject: {subject}")
|
||||
|
||||
# Process payloads
|
||||
payloads = []
|
||||
for dataname, payload_data, payload_type in data:
|
||||
payload_bytes = _serialize_data(payload_data, payload_type)
|
||||
payload_size = len(payload_bytes)
|
||||
|
||||
log_trace(correlation_id, f"Serialized payload '{dataname}' (type: {payload_type}) size: {payload_size} bytes")
|
||||
|
||||
if payload_size < size_threshold:
|
||||
# Direct path
|
||||
payload_b64 = base64.b64encode(payload_bytes).decode('utf-8')
|
||||
log_trace(correlation_id, f"Using direct transport for {payload_size} bytes")
|
||||
|
||||
payload = _build_payload(dataname, payload_type, payload_bytes, 'direct', payload_b64)
|
||||
payloads.append(payload)
|
||||
else:
|
||||
# Link path
|
||||
log_trace(correlation_id, "Using link transport, uploading to fileserver")
|
||||
|
||||
response = await fileserver_upload_handler(fileserver_url, dataname, payload_bytes)
|
||||
|
||||
if response['status'] != 200:
|
||||
raise Exception(f"Failed to upload data to fileserver: {response['status']}")
|
||||
|
||||
log_trace(correlation_id, f"Uploaded to URL: {response['url']}")
|
||||
|
||||
payload = _build_payload(dataname, payload_type, payload_bytes, 'link', response['url'])
|
||||
payloads.append(payload)
|
||||
|
||||
# Build envelope
|
||||
env = _build_envelope(subject, payloads, {
|
||||
'correlation_id': correlation_id,
|
||||
'msg_id': msg_id,
|
||||
'msg_purpose': msg_purpose,
|
||||
'sender_name': sender_name,
|
||||
'sender_id': sender_id,
|
||||
'receiver_name': receiver_name,
|
||||
'receiver_id': receiver_id,
|
||||
'reply_to': reply_to,
|
||||
'reply_to_msg_id': reply_to_msg_id,
|
||||
'broker_url': broker_url
|
||||
})
|
||||
|
||||
env_json_str = json.dumps(env)
|
||||
|
||||
if is_publish:
|
||||
if nats_connection:
|
||||
await publish_message(nats_connection, subject, env_json_str, correlation_id)
|
||||
else:
|
||||
await publish_message(broker_url, subject, env_json_str, correlation_id)
|
||||
|
||||
return env, env_json_str
|
||||
|
||||
|
||||
async def smartreceive(
|
||||
msg: Any,
|
||||
fileserver_download_handler: Callable = fetch_with_backoff,
|
||||
max_retries: int = 5,
|
||||
base_delay: int = 100,
|
||||
max_delay: int = 5000
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Receive and process NATS messages.
|
||||
|
||||
This function processes incoming NATS messages, handling both direct transport
|
||||
(base64 decoded payloads) and link transport (URL-based payloads).
|
||||
It deserializes the data based on the transport type and returns the result.
|
||||
|
||||
Args:
|
||||
msg: NATS message to process
|
||||
fileserver_download_handler: Function to handle downloading data from file server URLs
|
||||
max_retries: Maximum retry attempts for fetching URL
|
||||
base_delay: Initial delay for exponential backoff in ms
|
||||
max_delay: Maximum delay for exponential backoff in ms
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads field containing List[Tuple[str, Any, str]]
|
||||
|
||||
Example:
|
||||
>>> # Receive and process message
|
||||
>>> env = await smartreceive(msg, fileserver_download_handler=fetch_with_backoff)
|
||||
>>> # env is a Dict with "payloads" key containing List[Tuple[str, Any, str]]
|
||||
>>> # Access payloads: for dataname, data, type_ in env["payloads"]
|
||||
>>> for dataname, data, type_ in env["payloads"]:
|
||||
>>> print(f"{dataname}: {data} (type: {type_})")
|
||||
"""
|
||||
# Parse the JSON envelope
|
||||
if isinstance(msg, dict):
|
||||
# Already parsed
|
||||
env_json_obj = msg
|
||||
elif hasattr(msg, 'payload'):
|
||||
# NATS message object
|
||||
payload = msg.payload if isinstance(msg.payload, str) else msg.payload.decode('utf-8')
|
||||
env_json_obj = json.loads(payload)
|
||||
else:
|
||||
# Assume it's already a JSON string or dict
|
||||
env_json_obj = json.loads(msg) if isinstance(msg, str) else msg
|
||||
|
||||
log_trace(env_json_obj['correlation_id'], "Processing received message")
|
||||
|
||||
# Process all payloads in the envelope
|
||||
payloads_list = []
|
||||
num_payloads = len(env_json_obj['payloads'])
|
||||
|
||||
for i in range(num_payloads):
|
||||
payload_obj = env_json_obj['payloads'][i]
|
||||
transport = payload_obj['transport']
|
||||
dataname = payload_obj['dataname']
|
||||
|
||||
if transport == 'direct':
|
||||
log_trace(env_json_obj['correlation_id'], f"Direct transport - decoding payload '{dataname}'")
|
||||
|
||||
# Extract base64 payload from the payload
|
||||
payload_b64 = payload_obj['data']
|
||||
|
||||
# Decode Base64 payload
|
||||
payload_bytes = base64.b64decode(payload_b64)
|
||||
|
||||
# Deserialize based on type
|
||||
data_type = payload_obj['payload_type']
|
||||
data = _deserialize_data(payload_bytes, data_type, env_json_obj['correlation_id'])
|
||||
|
||||
payloads_list.append((dataname, data, data_type))
|
||||
elif transport == 'link':
|
||||
# Extract download URL from the payload
|
||||
url = payload_obj['data']
|
||||
log_trace(env_json_obj['correlation_id'], f"Link transport - fetching '{dataname}' from URL: {url}")
|
||||
|
||||
# Fetch with exponential backoff using the download handler
|
||||
downloaded_data = await fileserver_download_handler(
|
||||
url,
|
||||
max_retries,
|
||||
base_delay,
|
||||
max_delay,
|
||||
env_json_obj['correlation_id']
|
||||
)
|
||||
|
||||
# Deserialize based on type
|
||||
data_type = payload_obj['payload_type']
|
||||
data = _deserialize_data(downloaded_data, data_type, env_json_obj['correlation_id'])
|
||||
|
||||
payloads_list.append((dataname, data, data_type))
|
||||
else:
|
||||
raise Exception(f"Unknown transport type for payload '{dataname}': {transport}")
|
||||
|
||||
env_json_obj['payloads'] = payloads_list
|
||||
return env_json_obj
|
||||
|
||||
|
||||
# ---------------------------------------------- Module Exports ---------------------------------------------- #
|
||||
|
||||
class NATSBridge:
|
||||
"""
|
||||
Cross-platform NATS bridge implementation.
|
||||
|
||||
This class provides a convenient interface for NATSBridge functionality,
|
||||
encapsulating the main functions and providing a class-based API.
|
||||
"""
|
||||
|
||||
DEFAULT_SIZE_THRESHOLD = DEFAULT_SIZE_THRESHOLD
|
||||
DEFAULT_BROKER_URL = DEFAULT_BROKER_URL
|
||||
DEFAULT_FILESERVER_URL = DEFAULT_FILESERVER_URL
|
||||
|
||||
def __init__(self, broker_url: str = None, fileserver_url: str = None):
|
||||
"""
|
||||
Initialize NATSBridge.
|
||||
|
||||
Args:
|
||||
broker_url: NATS server URL (defaults to DEFAULT_BROKER_URL)
|
||||
fileserver_url: HTTP file server URL (defaults to DEFAULT_FILESERVER_URL)
|
||||
"""
|
||||
self.broker_url = broker_url or self.DEFAULT_BROKER_URL
|
||||
self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL
|
||||
|
||||
async def smartsend(
|
||||
self,
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
**kwargs
|
||||
) -> Tuple[Dict, str]:
|
||||
"""
|
||||
Send data via NATS.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
data: List of (dataname, data, type) tuples
|
||||
**kwargs: Additional options passed to smartsend
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str)
|
||||
"""
|
||||
kwargs['broker_url'] = kwargs.get('broker_url', self.broker_url)
|
||||
kwargs['fileserver_url'] = kwargs.get('fileserver_url', self.fileserver_url)
|
||||
return await smartsend(subject, data, **kwargs)
|
||||
|
||||
async def smartreceive(
|
||||
self,
|
||||
msg: Any,
|
||||
**kwargs
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Receive and process NATS message.
|
||||
|
||||
Args:
|
||||
msg: NATS message to process
|
||||
**kwargs: Additional options passed to smartreceive
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads
|
||||
"""
|
||||
return await smartreceive(msg, **kwargs)
|
||||
|
||||
|
||||
# Convenience functions for module-level usage
|
||||
def send(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
**kwargs
|
||||
) -> Tuple[Dict, str]:
|
||||
"""
|
||||
Convenience function for sending data.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
data: List of (dataname, data, type) tuples
|
||||
**kwargs: Additional options
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str)
|
||||
"""
|
||||
return asyncio.run(smartsend(subject, data, **kwargs))
|
||||
|
||||
|
||||
def receive(
|
||||
msg: Any,
|
||||
**kwargs
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Convenience function for receiving messages.
|
||||
|
||||
Args:
|
||||
msg: NATS message to process
|
||||
**kwargs: Additional options
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads
|
||||
"""
|
||||
return asyncio.run(smartreceive(msg, **kwargs))
|
||||
|
||||
|
||||
__all__ = [
|
||||
'smartsend',
|
||||
'smartreceive',
|
||||
'plik_oneshot_upload',
|
||||
'fetch_with_backoff',
|
||||
'NATSBridge',
|
||||
'send',
|
||||
'receive',
|
||||
'DEFAULT_SIZE_THRESHOLD',
|
||||
'DEFAULT_BROKER_URL',
|
||||
'DEFAULT_FILESERVER_URL',
|
||||
'NATSClient',
|
||||
'_serialize_data',
|
||||
'_deserialize_data',
|
||||
'log_trace',
|
||||
'publish_message'
|
||||
]
|
||||
673
src/natsbridge_mpy.py
Normal file
673
src/natsbridge_mpy.py
Normal file
@@ -0,0 +1,673 @@
|
||||
"""
|
||||
NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||
MicroPython Implementation
|
||||
|
||||
This module provides functionality for sending and receiving data across network boundaries
|
||||
using NATS as the message bus, with support for both direct payload transport and
|
||||
URL-based transport for larger payloads.
|
||||
|
||||
Note: MicroPython has significant constraints compared to desktop implementations:
|
||||
- Limited memory (~256KB - 1MB)
|
||||
- No Arrow IPC support (memory constraints)
|
||||
- Synchronous API (no async/await)
|
||||
- Lower size threshold for direct transport
|
||||
"""
|
||||
|
||||
import network
|
||||
import time
|
||||
import json
|
||||
import base64
|
||||
import uos
|
||||
import struct
|
||||
import random
|
||||
|
||||
# ---------------------------------------------- Constants ---------------------------------------------- #
|
||||
|
||||
"""
|
||||
Default size threshold for switching from direct to link transport (100KB for MicroPython)
|
||||
"""
|
||||
DEFAULT_SIZE_THRESHOLD = 100000
|
||||
|
||||
"""
|
||||
Default NATS server URL
|
||||
"""
|
||||
DEFAULT_BROKER_URL = "nats://localhost:4222"
|
||||
|
||||
"""
|
||||
Default HTTP file server URL for link transport
|
||||
"""
|
||||
DEFAULT_FILESERVER_URL = "http://localhost:8080"
|
||||
|
||||
"""
|
||||
Hard limit for payload size in MicroPython (50KB)
|
||||
"""
|
||||
MAX_PAYLOAD_SIZE = 50000
|
||||
|
||||
|
||||
# ---------------------------------------------- Utility Functions ---------------------------------------------- #
|
||||
|
||||
def log_trace(correlation_id, message):
|
||||
"""
|
||||
Log a trace message with correlation ID and timestamp.
|
||||
|
||||
Args:
|
||||
correlation_id: Correlation ID for tracing
|
||||
message: Message content to log
|
||||
"""
|
||||
timestamp = time.strftime('%Y-%m-%dT%H:%M:%SZ', time.localtime())
|
||||
print(f"[{timestamp}] [Correlation: {correlation_id}] {message}")
|
||||
|
||||
|
||||
def _generate_uuid():
|
||||
"""
|
||||
Generate a simple UUID compatible with MicroPython.
|
||||
|
||||
Returns:
|
||||
UUID string
|
||||
"""
|
||||
# Generate a simple UUID-like string
|
||||
# Format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
|
||||
hex_chars = '0123456789abcdef'
|
||||
uuid_str = ''.join([random.choice(hex_chars) for _ in range(32)])
|
||||
# Insert hyphens at proper positions
|
||||
return f"{uuid_str[:8]}-{uuid_str[8:12]}-{uuid_str[12:16]}-{uuid_str[16:20]}-{uuid_str[20:]}"
|
||||
|
||||
|
||||
# ---------------------------------------------- Serialization Functions ---------------------------------------------- #
|
||||
|
||||
def _serialize_data(data, payload_type):
|
||||
"""
|
||||
Serialize data according to specified format.
|
||||
|
||||
Args:
|
||||
data: Data to serialize (string for "text", dict for "dictionary",
|
||||
bytes for "image", "audio", "video", "binary")
|
||||
payload_type: Target format: "text", "dictionary", "image", "audio", "video", "binary"
|
||||
|
||||
Returns:
|
||||
Binary representation of the serialized data
|
||||
|
||||
Note:
|
||||
MicroPython does not support "table" type due to memory constraints.
|
||||
|
||||
Raises:
|
||||
ValueError: If payload_type is not one of the supported types
|
||||
"""
|
||||
if payload_type == 'text':
|
||||
if isinstance(data, str):
|
||||
return data.encode('utf-8')
|
||||
else:
|
||||
raise ValueError('Text data must be a string')
|
||||
elif payload_type == 'dictionary':
|
||||
json_str = json.dumps(data)
|
||||
return json_str.encode('utf-8')
|
||||
elif payload_type in ('image', 'audio', 'video', 'binary'):
|
||||
if isinstance(data, (bytes, bytearray, memoryview)):
|
||||
return bytes(data)
|
||||
else:
|
||||
raise ValueError(f'{payload_type} data must be bytes')
|
||||
else:
|
||||
raise ValueError(f'Unknown payload_type: {payload_type}')
|
||||
|
||||
|
||||
def _deserialize_data(data, payload_type):
|
||||
"""
|
||||
Deserialize bytes to data based on type.
|
||||
|
||||
Args:
|
||||
data: Serialized data as bytes
|
||||
payload_type: Data type ("text", "dictionary", "image", "audio", "video", "binary")
|
||||
|
||||
Returns:
|
||||
Deserialized data (String for "text", dict for "dictionary", bytes for others)
|
||||
|
||||
Note:
|
||||
MicroPython does not support "table" type due to memory constraints.
|
||||
|
||||
Raises:
|
||||
ValueError: If payload_type is not one of the supported types
|
||||
"""
|
||||
if payload_type == 'text':
|
||||
return data.decode('utf-8')
|
||||
elif payload_type == 'dictionary':
|
||||
json_str = data.decode('utf-8')
|
||||
return json.loads(json_str)
|
||||
elif payload_type in ('image', 'audio', 'video', 'binary'):
|
||||
return data
|
||||
else:
|
||||
raise ValueError(f'Unknown payload_type: {payload_type}')
|
||||
|
||||
|
||||
# ---------------------------------------------- File Server Handlers ---------------------------------------------- #
|
||||
|
||||
def _sync_fileserver_upload(file_server_url, dataname, data):
|
||||
"""
|
||||
Synchronous file upload to HTTP server.
|
||||
|
||||
Note:
|
||||
This is a simplified implementation for MicroPython.
|
||||
In practice, would use network.HTTP or similar.
|
||||
Currently raises NotImplementedError as file upload is not fully supported.
|
||||
|
||||
Args:
|
||||
file_server_url: Base URL of the file server
|
||||
dataname: Name of the file being uploaded
|
||||
data: Raw byte data of the file content
|
||||
|
||||
Returns:
|
||||
Dict with keys: 'status', 'url'
|
||||
|
||||
Raises:
|
||||
NotImplementedError: File upload is not implemented in MicroPython
|
||||
"""
|
||||
raise NotImplementedError("File upload not fully implemented in MicroPython. "
|
||||
"Use direct transport only for memory-constrained devices.")
|
||||
|
||||
|
||||
def _sync_fileserver_download(url, max_retries, base_delay, max_delay, correlation_id):
|
||||
"""
|
||||
Synchronous file download with exponential backoff.
|
||||
|
||||
Note:
|
||||
This is a simplified implementation for MicroPython.
|
||||
In practice, would use network.HTTP or similar.
|
||||
Currently raises NotImplementedError as file download is not fully supported.
|
||||
|
||||
Args:
|
||||
url: URL to download from
|
||||
max_retries: Maximum retry attempts
|
||||
base_delay: Initial delay in ms
|
||||
max_delay: Maximum delay in ms
|
||||
correlation_id: Correlation ID for logging
|
||||
|
||||
Returns:
|
||||
Downloaded bytes
|
||||
|
||||
Raises:
|
||||
NotImplementedError: File download is not implemented in MicroPython
|
||||
"""
|
||||
raise NotImplementedError("File download not fully implemented in MicroPython. "
|
||||
"Use direct transport only for memory-constrained devices.")
|
||||
|
||||
|
||||
# ---------------------------------------------- NATS Client ---------------------------------------------- #
|
||||
|
||||
class NATSClient:
|
||||
"""
|
||||
NATS client wrapper for MicroPython.
|
||||
|
||||
Note:
|
||||
This is a simplified implementation for MicroPython.
|
||||
Full NATS client implementation would require additional network stack support.
|
||||
"""
|
||||
|
||||
def __init__(self, url=DEFAULT_BROKER_URL):
|
||||
"""
|
||||
Initialize NATS client.
|
||||
|
||||
Args:
|
||||
url: NATS server URL
|
||||
"""
|
||||
self.url = url
|
||||
self._connected = False
|
||||
|
||||
def connect(self):
|
||||
"""
|
||||
Connect to NATS server.
|
||||
|
||||
Note:
|
||||
This is a placeholder implementation.
|
||||
Actual NATS client would require network stack support.
|
||||
|
||||
Returns:
|
||||
True if connected, False otherwise
|
||||
"""
|
||||
# Placeholder - actual implementation would connect to NATS server
|
||||
self._connected = True
|
||||
return self._connected
|
||||
|
||||
def publish(self, subject, message):
|
||||
"""
|
||||
Publish message to NATS subject.
|
||||
|
||||
Note:
|
||||
This is a placeholder implementation.
|
||||
Actual NATS client would require network stack support.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
message: Message to publish
|
||||
"""
|
||||
if not self._connected:
|
||||
raise RuntimeError("Not connected to NATS server")
|
||||
# Placeholder - actual implementation would publish to NATS
|
||||
print(f"[NATS] Publish to {subject}: {message[:50]}...")
|
||||
|
||||
def close(self):
|
||||
"""Close the NATS connection."""
|
||||
self._connected = False
|
||||
|
||||
|
||||
# ---------------------------------------------- Core Functions ---------------------------------------------- #
|
||||
|
||||
def _build_envelope(subject, payloads, options):
|
||||
"""
|
||||
Build message envelope from payloads and metadata.
|
||||
|
||||
Args:
|
||||
subject: NATS subject
|
||||
payloads: Array of payload objects
|
||||
options: Envelope metadata options
|
||||
|
||||
Returns:
|
||||
Envelope dict
|
||||
"""
|
||||
return {
|
||||
'correlation_id': options['correlation_id'],
|
||||
'msg_id': options['msg_id'],
|
||||
'timestamp': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.localtime()),
|
||||
'send_to': subject,
|
||||
'msg_purpose': options['msg_purpose'],
|
||||
'sender_name': options['sender_name'],
|
||||
'sender_id': options['sender_id'],
|
||||
'receiver_name': options['receiver_name'],
|
||||
'receiver_id': options['receiver_id'],
|
||||
'reply_to': options['reply_to'],
|
||||
'reply_to_msg_id': options['reply_to_msg_id'],
|
||||
'broker_url': options['broker_url'],
|
||||
'metadata': {},
|
||||
'payloads': payloads
|
||||
}
|
||||
|
||||
|
||||
def _build_payload(dataname, payload_type, payload_bytes, transport, data):
|
||||
"""
|
||||
Build payload object from serialized data.
|
||||
|
||||
Args:
|
||||
dataname: Name of the payload
|
||||
payload_type: Type of the payload
|
||||
payload_bytes: Serialized payload bytes
|
||||
transport: Transport type ("direct" or "link")
|
||||
data: Data (base64 for direct, URL for link)
|
||||
|
||||
Returns:
|
||||
Payload dict
|
||||
"""
|
||||
return {
|
||||
'id': _generate_uuid(),
|
||||
'dataname': dataname,
|
||||
'payload_type': payload_type,
|
||||
'transport': transport,
|
||||
'encoding': 'base64' if transport == 'direct' else 'none',
|
||||
'size': len(payload_bytes),
|
||||
'data': data,
|
||||
'metadata': {'payload_bytes': len(payload_bytes)} if transport == 'direct' else {}
|
||||
}
|
||||
|
||||
|
||||
def _publish(subject, message, correlation_id):
|
||||
"""
|
||||
Publish message to NATS.
|
||||
|
||||
Note:
|
||||
This is a simplified implementation for MicroPython.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
message: JSON message to publish
|
||||
correlation_id: Correlation ID for logging
|
||||
"""
|
||||
log_trace(correlation_id, f"Publishing to {subject}")
|
||||
# Placeholder - actual implementation would use NATSClient
|
||||
# client = NATSClient()
|
||||
# client.connect()
|
||||
# client.publish(subject, message)
|
||||
# client.close()
|
||||
|
||||
|
||||
def smartsend(subject, data, **kwargs):
|
||||
"""
|
||||
Send data via NATS with automatic transport selection.
|
||||
|
||||
This function intelligently routes data delivery based on payload size.
|
||||
If the serialized payload is smaller than size_threshold, it encodes the data as Base64
|
||||
and publishes directly over NATS. Otherwise, it uploads the data to a fileserver
|
||||
and publishes only the download URL over NATS.
|
||||
|
||||
Note:
|
||||
MicroPython has memory constraints, so the default size_threshold is lower (100KB).
|
||||
Table type is not supported due to memory constraints.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish the message to
|
||||
data: List of (dataname, data, type) tuples to send
|
||||
- dataname: Name of the payload
|
||||
- data: The actual data to send
|
||||
- type: Payload type: "text", "dictionary", "image", "audio", "video", "binary"
|
||||
broker_url: NATS server URL (default: DEFAULT_BROKER_URL)
|
||||
fileserver_url: HTTP file server URL (default: DEFAULT_FILESERVER_URL)
|
||||
fileserver_upload_handler: Function to handle fileserver uploads (default: _sync_fileserver_upload)
|
||||
size_threshold: Threshold in bytes separating direct vs link transport (default: 100000)
|
||||
correlation_id: Correlation ID for tracing (auto-generated if not provided)
|
||||
msg_purpose: Purpose of the message (default: "chat")
|
||||
sender_name: Name of the sender (default: "NATSBridge")
|
||||
receiver_name: Name of the receiver (empty means broadcast)
|
||||
receiver_id: UUID of the receiver (empty means broadcast)
|
||||
reply_to: Topic to reply to (empty if no reply expected)
|
||||
reply_to_msg_id: Message ID this message is replying to
|
||||
is_publish: Whether to automatically publish the message (default: True)
|
||||
msg_id: Message ID (auto-generated if not provided)
|
||||
sender_id: Sender ID (auto-generated if not provided)
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str) where:
|
||||
- env: Dict containing all metadata and payloads
|
||||
- env_json_str: JSON string for publishing to NATS
|
||||
|
||||
Example:
|
||||
>>> # Send text payload
|
||||
>>> env, env_json_str = NATSBridge.smartsend(
|
||||
... "/chat",
|
||||
... [("message", "Hello!", "text")],
|
||||
... broker_url="nats://localhost:4222"
|
||||
... )
|
||||
>>>
|
||||
>>> # Send dictionary payload
|
||||
>>> env, env_json_str = NATSBridge.smartsend(
|
||||
... "/config",
|
||||
... [("config", {"key": "value"}, "dictionary")],
|
||||
... broker_url="nats://localhost:4222"
|
||||
... )
|
||||
>>>
|
||||
>>> # Send binary payload (image, audio, video)
|
||||
>>> env, env_json_str = NATSBridge.smartsend(
|
||||
... "/media",
|
||||
... [("image", image_bytes, "image")],
|
||||
... broker_url="nats://localhost:4222"
|
||||
... )
|
||||
"""
|
||||
# Extract options with defaults
|
||||
correlation_id = kwargs.get('correlation_id', _generate_uuid())
|
||||
msg_id = kwargs.get('msg_id', _generate_uuid())
|
||||
sender_id = kwargs.get('sender_id', _generate_uuid())
|
||||
broker_url = kwargs.get('broker_url', DEFAULT_BROKER_URL)
|
||||
fileserver_url = kwargs.get('fileserver_url', DEFAULT_FILESERVER_URL)
|
||||
size_threshold = kwargs.get('size_threshold', DEFAULT_SIZE_THRESHOLD)
|
||||
msg_purpose = kwargs.get('msg_purpose', 'chat')
|
||||
sender_name = kwargs.get('sender_name', 'NATSBridge')
|
||||
receiver_name = kwargs.get('receiver_name', '')
|
||||
receiver_id = kwargs.get('receiver_id', '')
|
||||
reply_to = kwargs.get('reply_to', '')
|
||||
reply_to_msg_id = kwargs.get('reply_to_msg_id', '')
|
||||
is_publish = kwargs.get('is_publish', True)
|
||||
fileserver_upload_handler = kwargs.get('fileserver_upload_handler', _sync_fileserver_upload)
|
||||
|
||||
log_trace(correlation_id, f"Starting smartsend for subject: {subject}")
|
||||
|
||||
# Process payloads
|
||||
payloads = []
|
||||
for dataname, payload_data, payload_type in data:
|
||||
payload_bytes = _serialize_data(payload_data, payload_type)
|
||||
payload_size = len(payload_bytes)
|
||||
|
||||
# Check against hard limit for MicroPython
|
||||
if payload_size > MAX_PAYLOAD_SIZE:
|
||||
raise MemoryError(f"Payload '{dataname}' exceeds max size {MAX_PAYLOAD_SIZE} bytes")
|
||||
|
||||
log_trace(correlation_id, f"Serialized payload '{dataname}' (type: {payload_type}) size: {payload_size} bytes")
|
||||
|
||||
if payload_size < size_threshold:
|
||||
# Direct path
|
||||
payload_b64 = base64.b64encode(payload_bytes).decode('ascii')
|
||||
log_trace(correlation_id, f"Using direct transport for {payload_size} bytes")
|
||||
|
||||
payload = _build_payload(dataname, payload_type, payload_bytes, 'direct', payload_b64)
|
||||
payloads.append(payload)
|
||||
else:
|
||||
# Link path (limited support)
|
||||
log_trace(correlation_id, "Using link transport, uploading to fileserver")
|
||||
|
||||
try:
|
||||
response = fileserver_upload_handler(fileserver_url, dataname, payload_bytes)
|
||||
log_trace(correlation_id, f"Uploaded to URL: {response['url']}")
|
||||
|
||||
payload = _build_payload(dataname, payload_type, payload_bytes, 'link', response['url'])
|
||||
payloads.append(payload)
|
||||
except NotImplementedError:
|
||||
# Fall back to direct transport if file upload not available
|
||||
log_trace(correlation_id, "File upload not available, using direct transport")
|
||||
payload_b64 = base64.b64encode(payload_bytes).decode('ascii')
|
||||
payload = _build_payload(dataname, payload_type, payload_bytes, 'direct', payload_b64)
|
||||
payloads.append(payload)
|
||||
|
||||
# Build envelope
|
||||
env = _build_envelope(subject, payloads, {
|
||||
'correlation_id': correlation_id,
|
||||
'msg_id': msg_id,
|
||||
'msg_purpose': msg_purpose,
|
||||
'sender_name': sender_name,
|
||||
'sender_id': sender_id,
|
||||
'receiver_name': receiver_name,
|
||||
'receiver_id': receiver_id,
|
||||
'reply_to': reply_to,
|
||||
'reply_to_msg_id': reply_to_msg_id,
|
||||
'broker_url': broker_url
|
||||
})
|
||||
|
||||
env_json_str = json.dumps(env)
|
||||
|
||||
if is_publish:
|
||||
_publish(subject, env_json_str, correlation_id)
|
||||
|
||||
return env, env_json_str
|
||||
|
||||
|
||||
def smartreceive(msg, **kwargs):
|
||||
"""
|
||||
Receive and process NATS message.
|
||||
|
||||
This function processes incoming NATS messages, handling both direct transport
|
||||
(base64 decoded payloads) and link transport (URL-based payloads).
|
||||
It deserializes the data based on the transport type and returns the result.
|
||||
|
||||
Note:
|
||||
MicroPython has memory constraints, so large payloads should be avoided.
|
||||
Table type is not supported due to memory constraints.
|
||||
|
||||
Args:
|
||||
msg: NATS message to process (can be string, dict, or object with 'payload' attribute)
|
||||
fileserver_download_handler: Function to handle downloading data from file server URLs
|
||||
max_retries: Maximum retry attempts (default: 3)
|
||||
base_delay: Initial delay in ms (default: 100)
|
||||
max_delay: Maximum delay in ms (default: 1000)
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads field containing List[Tuple[str, Any, str]]
|
||||
|
||||
Example:
|
||||
>>> # Receive and process message
|
||||
>>> env = NATSBridge.smartreceive(msg, fileserver_download_handler=_sync_fileserver_download)
|
||||
>>> # env is a Dict with "payloads" key containing List[Tuple[str, Any, str]]
|
||||
>>> for dataname, data, type_ in env["payloads"]:
|
||||
... print(f"{dataname}: {data} (type: {type_})")
|
||||
"""
|
||||
# Parse the JSON envelope
|
||||
if isinstance(msg, dict):
|
||||
# Already parsed
|
||||
env_json_obj = msg
|
||||
elif hasattr(msg, 'payload'):
|
||||
# Object with payload attribute
|
||||
payload = msg.payload if isinstance(msg.payload, str) else msg.payload.decode('utf-8')
|
||||
env_json_obj = json.loads(payload)
|
||||
else:
|
||||
# Assume it's already a JSON string or dict
|
||||
env_json_obj = json.loads(msg) if isinstance(msg, str) else msg
|
||||
|
||||
correlation_id = env_json_obj['correlation_id']
|
||||
log_trace(correlation_id, "Processing received message")
|
||||
|
||||
# Process all payloads in the envelope
|
||||
payloads_list = []
|
||||
num_payloads = len(env_json_obj['payloads'])
|
||||
|
||||
for i in range(num_payloads):
|
||||
payload_obj = env_json_obj['payloads'][i]
|
||||
transport = payload_obj['transport']
|
||||
dataname = payload_obj['dataname']
|
||||
|
||||
if transport == 'direct':
|
||||
log_trace(correlation_id, f"Direct transport - decoding payload '{dataname}'")
|
||||
|
||||
# Extract base64 payload from the payload
|
||||
payload_b64 = payload_obj['data']
|
||||
|
||||
# Decode Base64 payload
|
||||
payload_bytes = base64.b64decode(payload_b64)
|
||||
|
||||
# Deserialize based on type
|
||||
data_type = payload_obj['payload_type']
|
||||
data = _deserialize_data(payload_bytes, data_type)
|
||||
|
||||
payloads_list.append((dataname, data, data_type))
|
||||
elif transport == 'link':
|
||||
# Extract download URL from the payload
|
||||
url = payload_obj['data']
|
||||
log_trace(correlation_id, f"Link transport - fetching '{dataname}' from URL: {url}")
|
||||
|
||||
# Fetch with exponential backoff using the download handler
|
||||
fileserver_download_handler = kwargs.get('fileserver_download_handler', _sync_fileserver_download)
|
||||
max_retries = kwargs.get('max_retries', 3)
|
||||
base_delay = kwargs.get('base_delay', 100)
|
||||
max_delay = kwargs.get('max_delay', 1000)
|
||||
|
||||
downloaded_data = fileserver_download_handler(
|
||||
url,
|
||||
max_retries,
|
||||
base_delay,
|
||||
max_delay,
|
||||
correlation_id
|
||||
)
|
||||
|
||||
# Deserialize based on type
|
||||
data_type = payload_obj['payload_type']
|
||||
data = _deserialize_data(downloaded_data, data_type)
|
||||
|
||||
payloads_list.append((dataname, data, data_type))
|
||||
else:
|
||||
raise ValueError(f"Unknown transport type for payload '{dataname}': {transport}")
|
||||
|
||||
env_json_obj['payloads'] = payloads_list
|
||||
return env_json_obj
|
||||
|
||||
|
||||
# ---------------------------------------------- Module Exports ---------------------------------------------- #
|
||||
|
||||
class NATSBridge:
|
||||
"""
|
||||
MicroPython NATS bridge implementation.
|
||||
|
||||
This class provides a convenient interface for NATSBridge functionality,
|
||||
encapsulating the main functions and providing a class-based API.
|
||||
|
||||
Note:
|
||||
MicroPython has significant constraints:
|
||||
- No Arrow IPC support (memory constraints)
|
||||
- Only direct transport (< 100KB threshold enforced)
|
||||
- Simplified UUID generation
|
||||
- No async/await (synchronous API)
|
||||
"""
|
||||
|
||||
DEFAULT_SIZE_THRESHOLD = DEFAULT_SIZE_THRESHOLD
|
||||
DEFAULT_BROKER_URL = DEFAULT_BROKER_URL
|
||||
DEFAULT_FILESERVER_URL = DEFAULT_FILESERVER_URL
|
||||
MAX_PAYLOAD_SIZE = MAX_PAYLOAD_SIZE
|
||||
|
||||
def __init__(self, broker_url=None, fileserver_url=None):
|
||||
"""
|
||||
Initialize NATSBridge.
|
||||
|
||||
Args:
|
||||
broker_url: NATS server URL (defaults to DEFAULT_BROKER_URL)
|
||||
fileserver_url: HTTP file server URL (defaults to DEFAULT_FILESERVER_URL)
|
||||
"""
|
||||
self.broker_url = broker_url or self.DEFAULT_BROKER_URL
|
||||
self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL
|
||||
|
||||
def smartsend(self, subject, data, **kwargs):
|
||||
"""
|
||||
Send data via NATS.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
data: List of (dataname, data, type) tuples
|
||||
**kwargs: Additional options passed to smartsend
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str)
|
||||
"""
|
||||
kwargs['broker_url'] = kwargs.get('broker_url', self.broker_url)
|
||||
kwargs['fileserver_url'] = kwargs.get('fileserver_url', self.fileserver_url)
|
||||
return smartsend(subject, data, **kwargs)
|
||||
|
||||
def smartreceive(self, msg, **kwargs):
|
||||
"""
|
||||
Receive and process NATS message.
|
||||
|
||||
Args:
|
||||
msg: NATS message to process
|
||||
**kwargs: Additional options passed to smartreceive
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads
|
||||
"""
|
||||
return smartreceive(msg, **kwargs)
|
||||
|
||||
|
||||
# Convenience functions for module-level usage
|
||||
def send(subject, data, **kwargs):
|
||||
"""
|
||||
Convenience function for sending data.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
data: List of (dataname, data, type) tuples
|
||||
**kwargs: Additional options
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str)
|
||||
"""
|
||||
return smartsend(subject, data, **kwargs)
|
||||
|
||||
|
||||
def receive(msg, **kwargs):
|
||||
"""
|
||||
Convenience function for receiving messages.
|
||||
|
||||
Args:
|
||||
msg: NATS message to process
|
||||
**kwargs: Additional options
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads
|
||||
"""
|
||||
return smartreceive(msg, **kwargs)
|
||||
|
||||
|
||||
__all__ = [
|
||||
'smartsend',
|
||||
'smartreceive',
|
||||
'NATSBridge',
|
||||
'send',
|
||||
'receive',
|
||||
'DEFAULT_SIZE_THRESHOLD',
|
||||
'DEFAULT_BROKER_URL',
|
||||
'DEFAULT_FILESERVER_URL',
|
||||
'MAX_PAYLOAD_SIZE',
|
||||
'NATSClient',
|
||||
'_serialize_data',
|
||||
'_deserialize_data',
|
||||
'log_trace',
|
||||
'_sync_fileserver_upload',
|
||||
'_sync_fileserver_download'
|
||||
]
|
||||
216
test/test_js_binary_receiver.js
Normal file
216
test/test_js_binary_receiver.js
Normal file
@@ -0,0 +1,216 @@
|
||||
/**
|
||||
* JavaScript Binary Receiver Test
|
||||
* Tests the smartreceive function with binary/image/audio/video payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natsbridge.js');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Binary Receiver Test ===\n');
|
||||
|
||||
// Create mock NATS message with binary payloads
|
||||
const binaryData = Buffer.from([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]); // PNG header
|
||||
const audioData = Buffer.from([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]); // FLAC header
|
||||
const videoData = Buffer.from([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]); // MP4 header
|
||||
const genericBinary = Buffer.from([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE]);
|
||||
|
||||
const testData = {
|
||||
correlation_id: 'js-binary-receiver-' + crypto.randomUUID(),
|
||||
msg_id: 'msg-' + crypto.randomUUID(),
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: '/test/binary',
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-binary-test',
|
||||
sender_id: 'sender-' + crypto.randomUUID(),
|
||||
receiver_name: 'js-receiver',
|
||||
receiver_id: 'receiver-' + crypto.randomUUID(),
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
broker_url: TEST_BROKER_URL,
|
||||
metadata: {},
|
||||
payloads: [
|
||||
{
|
||||
id: 'payload-1',
|
||||
dataname: 'image',
|
||||
payload_type: 'image',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: binaryData.length,
|
||||
data: binaryData.toString('base64'),
|
||||
metadata: { payload_bytes: binaryData.length }
|
||||
},
|
||||
{
|
||||
id: 'payload-2',
|
||||
dataname: 'audio',
|
||||
payload_type: 'audio',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: audioData.length,
|
||||
data: audioData.toString('base64'),
|
||||
metadata: { payload_bytes: audioData.length }
|
||||
},
|
||||
{
|
||||
id: 'payload-3',
|
||||
dataname: 'video',
|
||||
payload_type: 'video',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: videoData.length,
|
||||
data: videoData.toString('base64'),
|
||||
metadata: { payload_bytes: videoData.length }
|
||||
},
|
||||
{
|
||||
id: 'payload-4',
|
||||
dataname: 'binary',
|
||||
payload_type: 'binary',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: genericBinary.length,
|
||||
data: genericBinary.toString('base64'),
|
||||
metadata: { payload_bytes: genericBinary.length }
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const mockMsg = {
|
||||
payload: JSON.stringify(testData)
|
||||
};
|
||||
|
||||
console.log('Mock Message Created:');
|
||||
console.log(` Correlation ID: ${testData.correlation_id}`);
|
||||
console.log(` Payloads: ${testData.payloads.length}`);
|
||||
console.log(` Payload types: ${testData.payloads.map(p => p.payload_type).join(', ')}\n`);
|
||||
|
||||
try {
|
||||
// Receive and process the message
|
||||
console.log('Receiving and processing message...');
|
||||
const env = await NATSBridge.smartreceive(
|
||||
mockMsg,
|
||||
{
|
||||
max_retries: 3,
|
||||
base_delay: 100,
|
||||
max_delay: 1000
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Received Envelope ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate received data
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (!env.correlation_id) {
|
||||
console.log('❌ correlation_id is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ correlation_id present');
|
||||
}
|
||||
|
||||
if (env.payloads.length !== 4) {
|
||||
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
// Expected data
|
||||
const expectedData = [
|
||||
['image', binaryData, 'image'],
|
||||
['audio', audioData, 'audio'],
|
||||
['video', videoData, 'video'],
|
||||
['binary', genericBinary, 'binary']
|
||||
];
|
||||
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
const expected = expectedData[i];
|
||||
|
||||
if (payload[0] !== expected[0]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected dataname '${expected[0]}', got '${payload[0]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct dataname`);
|
||||
}
|
||||
|
||||
if (payload[2] !== expected[2]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected type '${expected[2]}', got '${payload[2]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct type`);
|
||||
}
|
||||
|
||||
// Verify binary data integrity
|
||||
const receivedData = payload[1];
|
||||
if (!(receivedData instanceof Buffer || receivedData instanceof Uint8Array)) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected Buffer/Uint8Array, got ${typeof receivedData}`);
|
||||
passed = false;
|
||||
} else if (Buffer.isBuffer(receivedData)) {
|
||||
if (receivedData.length !== expected[1].length) {
|
||||
console.log(`❌ Payload ${i + 1}: Length mismatch`);
|
||||
passed = false;
|
||||
} else {
|
||||
let dataMatch = true;
|
||||
for (let j = 0; j < expected[1].length; j++) {
|
||||
if (receivedData[j] !== expected[1][j]) {
|
||||
dataMatch = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (dataMatch) {
|
||||
console.log(`✅ Payload ${i + 1}: Data correctly deserialized`);
|
||||
} else {
|
||||
console.log(`❌ Payload ${i + 1}: Data mismatch`);
|
||||
passed = false;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Uint8Array comparison
|
||||
const receivedBuffer = Buffer.from(receivedData);
|
||||
if (receivedBuffer.length !== expected[1].length) {
|
||||
console.log(`❌ Payload ${i + 1}: Length mismatch`);
|
||||
passed = false;
|
||||
} else {
|
||||
let dataMatch = true;
|
||||
for (let j = 0; j < expected[1].length; j++) {
|
||||
if (receivedBuffer[j] !== expected[1][j]) {
|
||||
dataMatch = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (dataMatch) {
|
||||
console.log(`✅ Payload ${i + 1}: Data correctly deserialized`);
|
||||
} else {
|
||||
console.log(`❌ Payload ${i + 1}: Data mismatch`);
|
||||
passed = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
174
test/test_js_binary_sender.js
Normal file
174
test/test_js_binary_sender.js
Normal file
@@ -0,0 +1,174 @@
|
||||
/**
|
||||
* JavaScript Binary Sender Test
|
||||
* Tests the smartsend function with binary/image/audio/video payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natsbridge.js');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const TEST_SUBJECT = '/test/binary';
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Binary Sender Test ===\n');
|
||||
|
||||
const correlationId = crypto.randomUUID();
|
||||
console.log(`Correlation ID: ${correlationId}`);
|
||||
console.log(`Subject: ${TEST_SUBJECT}`);
|
||||
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
|
||||
|
||||
// Test data - binary data for different types
|
||||
const binaryData = Buffer.from([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]); // PNG header
|
||||
const audioData = Buffer.from([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]); // FLAC header
|
||||
const videoData = Buffer.from([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]); // MP4 header
|
||||
const genericBinary = Buffer.from([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE]);
|
||||
|
||||
const testData = [
|
||||
['image', binaryData, 'image'],
|
||||
['audio', audioData, 'audio'],
|
||||
['video', videoData, 'video'],
|
||||
['binary', genericBinary, 'binary']
|
||||
];
|
||||
|
||||
try {
|
||||
// Send the message
|
||||
console.log('Sending binary payloads...');
|
||||
const [env, envJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
testData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: correlationId,
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-binary-test',
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Envelope Created ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Purpose: ${env.msg_purpose}`);
|
||||
console.log(`Sender: ${env.sender_name}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate envelope structure
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (env.payloads.length !== 4) {
|
||||
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
// Test each payload
|
||||
const expectedDatanames = ['image', 'audio', 'video', 'binary'];
|
||||
const expectedTypes = ['image', 'audio', 'video', 'binary'];
|
||||
const expectedData = [binaryData, audioData, videoData, genericBinary];
|
||||
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
|
||||
if (payload.dataname !== expectedDatanames[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected dataname '${expectedDatanames[i]}', got '${payload.dataname}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct dataname`);
|
||||
}
|
||||
|
||||
if (payload.payload_type !== expectedTypes[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected type '${expectedTypes[i]}', got '${payload.payload_type}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct type`);
|
||||
}
|
||||
|
||||
if (payload.transport !== 'direct') {
|
||||
console.log(`❌ Payload ${i + 1}: Expected transport 'direct', got '${payload.transport}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct transport`);
|
||||
}
|
||||
|
||||
if (payload.encoding !== 'base64') {
|
||||
console.log(`❌ Payload ${i + 1}: Expected encoding 'base64', got '${payload.encoding}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct encoding`);
|
||||
}
|
||||
|
||||
// Decode and verify the data
|
||||
const decodedData = Buffer.from(payload.data, 'base64');
|
||||
const originalData = expectedData[i];
|
||||
|
||||
if (decodedData.length !== originalData.length) {
|
||||
console.log(`❌ Payload ${i + 1}: Length mismatch (${decodedData.length} vs ${originalData.length})`);
|
||||
passed = false;
|
||||
} else {
|
||||
let dataMatch = true;
|
||||
for (let j = 0; j < originalData.length; j++) {
|
||||
if (decodedData[j] !== originalData[j]) {
|
||||
dataMatch = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (dataMatch) {
|
||||
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
|
||||
} else {
|
||||
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
|
||||
passed = false;
|
||||
}
|
||||
}
|
||||
|
||||
console.log(` Size: ${payload.size} bytes\n`);
|
||||
}
|
||||
|
||||
// Test with larger binary data (simulating file upload scenario)
|
||||
console.log('=== Large Binary Data Test ===');
|
||||
const largeData = Buffer.alloc(10000, 0xFF); // 10KB of binary data
|
||||
const largeTestData = [
|
||||
['large_binary', largeData, 'binary']
|
||||
];
|
||||
|
||||
const [largeEnv, _] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
largeTestData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: 'large-' + correlationId,
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
if (largeEnv.payloads.length === 1 && largeEnv.payloads[0].size === 10000) {
|
||||
console.log('✅ Large binary data handled correctly');
|
||||
} else {
|
||||
console.log('❌ Large binary data handling failed');
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
221
test/test_js_dictionary_receiver.js
Normal file
221
test/test_js_dictionary_receiver.js
Normal file
@@ -0,0 +1,221 @@
|
||||
/**
|
||||
* JavaScript Dictionary Receiver Test
|
||||
* Tests the smartreceive function with dictionary payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natsbridge.js');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Dictionary Receiver Test ===\n');
|
||||
|
||||
// Create a mock NATS message with dictionary payloads
|
||||
const simpleDict = { key1: 'value1', key2: 'value2' };
|
||||
const nestedDict = { outer: { inner: 'value', number: 42 } };
|
||||
const arrayDict = { items: [1, 2, 3, 'four', 'five'] };
|
||||
const mixedDict = { string: 'text', number: 123, boolean: true, null_val: null };
|
||||
|
||||
const testData = {
|
||||
correlation_id: 'test-receiver-dict-' + crypto.randomUUID(),
|
||||
msg_id: 'msg-' + crypto.randomUUID(),
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: '/test/dictionary',
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-dict-test',
|
||||
sender_id: 'sender-' + crypto.randomUUID(),
|
||||
receiver_name: 'js-receiver',
|
||||
receiver_id: 'receiver-' + crypto.randomUUID(),
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
broker_url: TEST_BROKER_URL,
|
||||
metadata: {},
|
||||
payloads: [
|
||||
{
|
||||
id: 'payload-1',
|
||||
dataname: 'simple_dict',
|
||||
payload_type: 'dictionary',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: Buffer.from(JSON.stringify(simpleDict)).length,
|
||||
data: Buffer.from(JSON.stringify(simpleDict)).toString('base64'),
|
||||
metadata: { payload_bytes: Buffer.from(JSON.stringify(simpleDict)).length }
|
||||
},
|
||||
{
|
||||
id: 'payload-2',
|
||||
dataname: 'nested_dict',
|
||||
payload_type: 'dictionary',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: Buffer.from(JSON.stringify(nestedDict)).length,
|
||||
data: Buffer.from(JSON.stringify(nestedDict)).toString('base64'),
|
||||
metadata: { payload_bytes: Buffer.from(JSON.stringify(nestedDict)).length }
|
||||
},
|
||||
{
|
||||
id: 'payload-3',
|
||||
dataname: 'array_dict',
|
||||
payload_type: 'dictionary',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: Buffer.from(JSON.stringify(arrayDict)).length,
|
||||
data: Buffer.from(JSON.stringify(arrayDict)).toString('base64'),
|
||||
metadata: { payload_bytes: Buffer.from(JSON.stringify(arrayDict)).length }
|
||||
},
|
||||
{
|
||||
id: 'payload-4',
|
||||
dataname: 'mixed_dict',
|
||||
payload_type: 'dictionary',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: Buffer.from(JSON.stringify(mixedDict)).length,
|
||||
data: Buffer.from(JSON.stringify(mixedDict)).toString('base64'),
|
||||
metadata: { payload_bytes: Buffer.from(JSON.stringify(mixedDict)).length }
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const mockMsg = {
|
||||
payload: JSON.stringify(testData)
|
||||
};
|
||||
|
||||
console.log('Mock Message Created:');
|
||||
console.log(` Correlation ID: ${testData.correlation_id}`);
|
||||
console.log(` Payloads: ${testData.payloads.length}`);
|
||||
console.log(` Payload types: ${testData.payloads.map(p => p.payload_type).join(', ')}\n`);
|
||||
|
||||
try {
|
||||
// Receive and process the message
|
||||
console.log('Receiving and processing message...');
|
||||
const env = await NATSBridge.smartreceive(
|
||||
mockMsg,
|
||||
{
|
||||
max_retries: 3,
|
||||
base_delay: 100,
|
||||
max_delay: 1000
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Received Envelope ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate received data
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (!env.correlation_id) {
|
||||
console.log('❌ correlation_id is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ correlation_id present');
|
||||
}
|
||||
|
||||
if (env.payloads.length !== 4) {
|
||||
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
// Expected data
|
||||
const expectedData = [
|
||||
['simple_dict', simpleDict, 'dictionary'],
|
||||
['nested_dict', nestedDict, 'dictionary'],
|
||||
['array_dict', arrayDict, 'dictionary'],
|
||||
['mixed_dict', mixedDict, 'dictionary']
|
||||
];
|
||||
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
const expected = expectedData[i];
|
||||
|
||||
if (payload[0] !== expected[0]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected dataname '${expected[0]}', got '${payload[0]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct dataname`);
|
||||
}
|
||||
|
||||
if (payload[2] !== expected[2]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected type '${expected[2]}', got '${payload[2]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct type`);
|
||||
}
|
||||
|
||||
const dataMatch = JSON.stringify(payload[1]) === JSON.stringify(expected[1]);
|
||||
if (!dataMatch) {
|
||||
console.log(`❌ Payload ${i + 1}: Data mismatch`);
|
||||
console.log(` Expected: ${JSON.stringify(expected[1])}`);
|
||||
console.log(` Got: ${JSON.stringify(payload[1])}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Data correctly deserialized`);
|
||||
}
|
||||
}
|
||||
|
||||
// Test round-trip with receive
|
||||
console.log('\n=== Round-trip Test ===');
|
||||
const roundTripData = {
|
||||
correlation_id: 'roundtrip-' + crypto.randomUUID(),
|
||||
msg_id: 'msg-' + crypto.randomUUID(),
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: '/test/dictionary',
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-test',
|
||||
sender_id: 'sender-' + crypto.randomUUID(),
|
||||
receiver_name: 'js-receiver',
|
||||
receiver_id: 'receiver-' + crypto.randomUUID(),
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
broker_url: TEST_BROKER_URL,
|
||||
metadata: {},
|
||||
payloads: [
|
||||
{
|
||||
id: 'payload-rt',
|
||||
dataname: 'roundtrip',
|
||||
payload_type: 'dictionary',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: Buffer.from(JSON.stringify({ test: 'data', nested: { a: 1, b: 2 } })).length,
|
||||
data: Buffer.from(JSON.stringify({ test: 'data', nested: { a: 1, b: 2 } })).toString('base64'),
|
||||
metadata: { payload_bytes: Buffer.from(JSON.stringify({ test: 'data', nested: { a: 1, b: 2 } })).length }
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const mockRtMsg = { payload: JSON.stringify(roundTripData) };
|
||||
const rtEnv = await NATSBridge.smartreceive(mockRtMsg);
|
||||
|
||||
if (rtEnv.payloads.length === 1 &&
|
||||
rtEnv.payloads[0][0] === 'roundtrip' &&
|
||||
rtEnv.payloads[0][2] === 'dictionary') {
|
||||
console.log('✅ Round-trip test successful');
|
||||
} else {
|
||||
console.log('❌ Round-trip test failed');
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
179
test/test_js_dictionary_sender.js
Normal file
179
test/test_js_dictionary_sender.js
Normal file
@@ -0,0 +1,179 @@
|
||||
/**
|
||||
* JavaScript Dictionary Sender Test
|
||||
* Tests the smartsend function with dictionary payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natsbridge.js');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const TEST_SUBJECT = '/test/dictionary';
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Dictionary Sender Test ===\n');
|
||||
|
||||
const correlationId = crypto.randomUUID();
|
||||
console.log(`Correlation ID: ${correlationId}`);
|
||||
console.log(`Subject: ${TEST_SUBJECT}`);
|
||||
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
|
||||
|
||||
// Test data - various dictionary structures
|
||||
const testData = [
|
||||
['simple_dict', { key1: 'value1', key2: 'value2' }, 'dictionary'],
|
||||
['nested_dict', { outer: { inner: 'value', number: 42 } }, 'dictionary'],
|
||||
['array_dict', { items: [1, 2, 3, 'four', 'five'] }, 'dictionary'],
|
||||
['mixed_dict', { string: 'text', number: 123, boolean: true, null_val: null }, 'dictionary']
|
||||
];
|
||||
|
||||
try {
|
||||
// Send the message
|
||||
console.log('Sending dictionary payloads...');
|
||||
const [env, envJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
testData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: correlationId,
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-dict-test',
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Envelope Created ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Purpose: ${env.msg_purpose}`);
|
||||
console.log(`Sender: ${env.sender_name}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate envelope structure
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (env.payloads.length !== 4) {
|
||||
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
// Test each payload
|
||||
const expectedDatanames = ['simple_dict', 'nested_dict', 'array_dict', 'mixed_dict'];
|
||||
const expectedTypes = ['dictionary', 'dictionary', 'dictionary', 'dictionary'];
|
||||
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
|
||||
if (payload.dataname !== expectedDatanames[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected dataname '${expectedDatanames[i]}', got '${payload.dataname}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct dataname`);
|
||||
}
|
||||
|
||||
if (payload.payload_type !== expectedTypes[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected type '${expectedTypes[i]}', got '${payload.payload_type}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct type`);
|
||||
}
|
||||
|
||||
if (payload.transport !== 'direct') {
|
||||
console.log(`❌ Payload ${i + 1}: Expected transport 'direct', got '${payload.transport}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct transport`);
|
||||
}
|
||||
|
||||
if (payload.encoding !== 'base64') {
|
||||
console.log(`❌ Payload ${i + 1}: Expected encoding 'base64', got '${payload.encoding}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct encoding`);
|
||||
}
|
||||
|
||||
// Decode and verify the data
|
||||
const decodedData = JSON.parse(Buffer.from(payload.data, 'base64').toString('utf8'));
|
||||
const originalData = testData[i][1];
|
||||
|
||||
const originalJson = JSON.stringify(originalData);
|
||||
const decodedJson = JSON.stringify(decodedData);
|
||||
|
||||
if (originalJson !== decodedJson) {
|
||||
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
|
||||
console.log(` Expected: ${originalJson}`);
|
||||
console.log(` Got: ${decodedJson}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
|
||||
}
|
||||
|
||||
console.log(` Size: ${payload.size} bytes\n`);
|
||||
}
|
||||
|
||||
// Test round-trip serialization
|
||||
console.log('=== Round-trip Serialization Test ===');
|
||||
const roundTripTestData = [
|
||||
['roundtrip', { test: 'data', numbers: [1, 2, 3], nested: { a: 1, b: 2 } }, 'dictionary']
|
||||
];
|
||||
|
||||
const [rtEnv, rtEnvJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
roundTripTestData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: 'roundtrip-' + correlationId,
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
const rtPayload = rtEnv.payloads[0];
|
||||
const rtDecoded = JSON.parse(Buffer.from(rtPayload.data, 'base64').toString('utf8'));
|
||||
|
||||
if (JSON.stringify(rtDecoded) === JSON.stringify(roundTripTestData[0][1])) {
|
||||
console.log('✅ Round-trip serialization successful');
|
||||
} else {
|
||||
console.log('❌ Round-trip serialization failed');
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Test JSON string output
|
||||
console.log('\n=== JSON String Output Test ===');
|
||||
try {
|
||||
const parsed = JSON.parse(envJsonStr);
|
||||
if (parsed.correlation_id === env.correlation_id &&
|
||||
parsed.payloads.length === env.payloads.length) {
|
||||
console.log('✅ JSON string is valid and matches envelope');
|
||||
} else {
|
||||
console.log('❌ JSON string does not match envelope');
|
||||
passed = false;
|
||||
}
|
||||
} catch (e) {
|
||||
console.log('❌ JSON string is invalid:', e.message);
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
216
test/test_js_mix_payloads_receiver.js
Normal file
216
test/test_js_mix_payloads_receiver.js
Normal file
@@ -0,0 +1,216 @@
|
||||
/**
|
||||
* JavaScript Mix Payloads Receiver Test
|
||||
* Tests the smartreceive function with mixed payload types
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natsbridge.js');
|
||||
const nats = require('nats');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const TEST_SUBJECT = '/test/mix';
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Mix Payloads Receiver Test ===\n');
|
||||
|
||||
const correlationId = crypto.randomUUID();
|
||||
console.log(`Correlation ID: ${correlationId}`);
|
||||
console.log(`Subject: ${TEST_SUBJECT}`);
|
||||
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
|
||||
|
||||
// Expected test data - same as sender
|
||||
const expectedTextData = 'Hello, NATSBridge!';
|
||||
const expectedDictData = { key1: 'value1', key2: 42, nested: { a: 1, b: 2 } };
|
||||
const expectedBinaryData = Buffer.from([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]); // PNG header
|
||||
const expectedTableData = [
|
||||
{ id: 1, name: 'Alice', age: 30 },
|
||||
{ id: 2, name: 'Bob', age: 25 },
|
||||
{ id: 3, name: 'Charlie', age: 35 }
|
||||
];
|
||||
|
||||
const expectedDatanames = ['message', 'config', 'image', 'users'];
|
||||
const expectedTypes = ['text', 'dictionary', 'image', 'table'];
|
||||
|
||||
let testPassed = true;
|
||||
let messagesReceived = 0;
|
||||
const receivedPayloads = [];
|
||||
|
||||
try {
|
||||
// Connect to NATS
|
||||
console.log('Connecting to NATS server...');
|
||||
const nc = await nats.connect({ servers: TEST_BROKER_URL });
|
||||
console.log('✅ Connected to NATS server\n');
|
||||
|
||||
// Set up message subscription
|
||||
const subscription = nc.subscribe(TEST_SUBJECT);
|
||||
|
||||
// Wait for messages with timeout
|
||||
const messagePromise = new Promise(async (resolve, reject) => {
|
||||
const timeout = setTimeout(() => {
|
||||
resolve('timeout');
|
||||
}, 10000); // 10 second timeout
|
||||
|
||||
(async () => {
|
||||
for await (const msg of subscription) {
|
||||
clearTimeout(timeout);
|
||||
messagesReceived++;
|
||||
console.log(`\n=== Message ${messagesReceived} Received ===`);
|
||||
console.log(`Raw payload length: ${msg.payload.length} bytes`);
|
||||
|
||||
try {
|
||||
// Process the message using smartreceive
|
||||
const envelope = await NATSBridge.smartreceive(msg, {
|
||||
fileserver_download_handler: NATSBridge.fetchWithBackoff,
|
||||
max_retries: 5,
|
||||
base_delay: 100,
|
||||
max_delay: 5000
|
||||
});
|
||||
|
||||
console.log(`Correlation ID: ${envelope.correlation_id}`);
|
||||
console.log(`Message ID: ${envelope.msg_id}`);
|
||||
console.log(`Number of payloads: ${envelope.payloads.length}`);
|
||||
|
||||
receivedPayloads.push(envelope);
|
||||
|
||||
// Validate envelope structure
|
||||
console.log('\n=== Envelope Validation ===');
|
||||
|
||||
if (envelope.payloads.length < 4) {
|
||||
console.log(`❌ Expected at least 4 payloads, got ${envelope.payloads.length}`);
|
||||
testPassed = false;
|
||||
} else {
|
||||
console.log(`✅ Correct number of payloads: ${envelope.payloads.length}`);
|
||||
}
|
||||
|
||||
// Validate each payload
|
||||
for (let i = 0; i < envelope.payloads.length; i++) {
|
||||
const [dataname, data, dataType] = envelope.payloads[i];
|
||||
|
||||
console.log(`\n--- Payload ${i + 1}: ${dataname} (type: ${dataType}) ---`);
|
||||
|
||||
// Check dataname
|
||||
if (i < expectedDatanames.length && dataname !== expectedDatanames[i]) {
|
||||
console.log(`❌ Expected dataname '${expectedDatanames[i]}', got '${dataname}'`);
|
||||
testPassed = false;
|
||||
} else {
|
||||
console.log(`✅ Correct dataname: ${dataname}`);
|
||||
}
|
||||
|
||||
// Check data type
|
||||
if (i < expectedTypes.length && dataType !== expectedTypes[i]) {
|
||||
console.log(`❌ Expected type '${expectedTypes[i]}', got '${dataType}'`);
|
||||
testPassed = false;
|
||||
} else {
|
||||
console.log(`✅ Correct type: ${dataType}`);
|
||||
}
|
||||
|
||||
// Validate data based on type
|
||||
if (dataType === 'text') {
|
||||
if (typeof data === 'string' && data === expectedTextData) {
|
||||
console.log(`✅ Text data verified: "${data}"`);
|
||||
} else {
|
||||
console.log(`❌ Text data mismatch. Expected: "${expectedTextData}", Got: "${data}"`);
|
||||
testPassed = false;
|
||||
}
|
||||
} else if (dataType === 'dictionary') {
|
||||
if (typeof data === 'object' && JSON.stringify(data) === JSON.stringify(expectedDictData)) {
|
||||
console.log(`✅ Dictionary data verified`);
|
||||
console.log(` Keys: ${Object.keys(data).join(', ')}`);
|
||||
} else {
|
||||
console.log(`❌ Dictionary data mismatch`);
|
||||
console.log(` Expected: ${JSON.stringify(expectedDictData)}`);
|
||||
console.log(` Got: ${JSON.stringify(data)}`);
|
||||
testPassed = false;
|
||||
}
|
||||
} else if (dataType === 'image') {
|
||||
if (data instanceof Buffer || data instanceof Uint8Array) {
|
||||
const dataBuffer = Buffer.isBuffer(data) ? data : Buffer.from(data);
|
||||
if (dataBuffer.length === expectedBinaryData.length) {
|
||||
let dataMatch = true;
|
||||
for (let j = 0; j < expectedBinaryData.length; j++) {
|
||||
if (dataBuffer[j] !== expectedBinaryData[j]) {
|
||||
dataMatch = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (dataMatch) {
|
||||
console.log(`✅ Image data verified (${dataBuffer.length} bytes)`);
|
||||
} else {
|
||||
console.log(`❌ Image data mismatch`);
|
||||
testPassed = false;
|
||||
}
|
||||
} else {
|
||||
console.log(`❌ Image data length mismatch. Expected: ${expectedBinaryData.length}, Got: ${dataBuffer.length}`);
|
||||
testPassed = false;
|
||||
}
|
||||
} else {
|
||||
console.log(`❌ Image data is not a Buffer or Uint8Array`);
|
||||
testPassed = false;
|
||||
}
|
||||
} else if (dataType === 'table') {
|
||||
// For table data, check if it's an Arrow table-like object
|
||||
if (data && typeof data === 'object') {
|
||||
// Arrow tables have specific properties
|
||||
if (data.numRows !== undefined && data.numCols !== undefined) {
|
||||
console.log(`✅ Table data verified`);
|
||||
console.log(` Rows: ${data.numRows}, Columns: ${data.numCols}`);
|
||||
} else {
|
||||
console.log(`⚠️ Table data received but not standard Arrow format`);
|
||||
console.log(` Keys: ${Object.keys(data).join(', ')}`);
|
||||
}
|
||||
} else {
|
||||
console.log(`❌ Table data is not a valid object`);
|
||||
testPassed = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Stop after receiving at least one valid message
|
||||
if (messagesReceived >= 1) {
|
||||
resolve('done');
|
||||
}
|
||||
} catch (error) {
|
||||
console.error(`❌ Error processing message: ${error.message}`);
|
||||
console.error(error.stack);
|
||||
testPassed = false;
|
||||
resolve('error');
|
||||
}
|
||||
}
|
||||
})();
|
||||
});
|
||||
|
||||
console.log('Waiting for messages...\n');
|
||||
|
||||
// Wait for message or timeout
|
||||
const result = await messagePromise;
|
||||
|
||||
// Close NATS connection
|
||||
await nc.close();
|
||||
console.log('\n✅ NATS connection closed');
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (messagesReceived === 0) {
|
||||
console.log('❌ NO MESSAGES RECEIVED');
|
||||
console.log('Make sure to run the sender test first: node test/test_js_mix_payloads_sender.js');
|
||||
process.exit(1);
|
||||
} else if (result === 'error') {
|
||||
console.log('❌ ERROR PROCESSING MESSAGES');
|
||||
process.exit(1);
|
||||
} else if (testPassed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
205
test/test_js_mix_payloads_sender.js
Normal file
205
test/test_js_mix_payloads_sender.js
Normal file
@@ -0,0 +1,205 @@
|
||||
/**
|
||||
* JavaScript Mix Payloads Sender Test
|
||||
* Tests the smartsend function with mixed payload types
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natsbridge.js');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const TEST_SUBJECT = '/test/mix';
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Mix Payloads Sender Test ===\n');
|
||||
|
||||
const correlationId = crypto.randomUUID();
|
||||
console.log(`Correlation ID: ${correlationId}`);
|
||||
console.log(`Subject: ${TEST_SUBJECT}`);
|
||||
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
|
||||
|
||||
// Test data - mixed payload types
|
||||
const textData = 'Hello, NATSBridge!';
|
||||
const dictData = { key1: 'value1', key2: 42, nested: { a: 1, b: 2 } };
|
||||
const binaryData = Buffer.from([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]); // PNG header
|
||||
|
||||
// Table data
|
||||
const tableData = [
|
||||
{ id: 1, name: 'Alice', age: 30 },
|
||||
{ id: 2, name: 'Bob', age: 25 },
|
||||
{ id: 3, name: 'Charlie', age: 35 }
|
||||
];
|
||||
|
||||
const testData = [
|
||||
['message', textData, 'text'],
|
||||
['config', dictData, 'dictionary'],
|
||||
['image', binaryData, 'image'],
|
||||
['users', tableData, 'table']
|
||||
];
|
||||
|
||||
try {
|
||||
// Send the message
|
||||
console.log('Sending mixed payloads...');
|
||||
const [env, envJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
testData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: correlationId,
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-mix-test',
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Envelope Created ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Purpose: ${env.msg_purpose}`);
|
||||
console.log(`Sender: ${env.sender_name}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate envelope structure
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (env.payloads.length !== 4) {
|
||||
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
// Test each payload
|
||||
const expectedDatanames = ['message', 'config', 'image', 'users'];
|
||||
const expectedTypes = ['text', 'dictionary', 'image', 'table'];
|
||||
const expectedData = [textData, dictData, binaryData, tableData];
|
||||
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
|
||||
if (payload.dataname !== expectedDatanames[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected dataname '${expectedDatanames[i]}', got '${payload.dataname}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct dataname`);
|
||||
}
|
||||
|
||||
if (payload.payload_type !== expectedTypes[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected type '${expectedTypes[i]}', got '${payload.payload_type}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct type`);
|
||||
}
|
||||
|
||||
if (payload.transport !== 'direct') {
|
||||
console.log(`❌ Payload ${i + 1}: Expected transport 'direct', got '${payload.transport}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct transport`);
|
||||
}
|
||||
|
||||
if (payload.encoding !== 'base64') {
|
||||
console.log(`❌ Payload ${i + 1}: Expected encoding 'base64', got '${payload.encoding}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct encoding`);
|
||||
}
|
||||
|
||||
// Verify data integrity based on type
|
||||
if (expectedTypes[i] === 'text') {
|
||||
const decodedData = Buffer.from(payload.data, 'base64').toString('utf8');
|
||||
if (decodedData !== expectedData[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
|
||||
}
|
||||
} else if (expectedTypes[i] === 'dictionary') {
|
||||
const decodedData = JSON.parse(Buffer.from(payload.data, 'base64').toString('utf8'));
|
||||
if (JSON.stringify(decodedData) !== JSON.stringify(expectedData[i])) {
|
||||
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
|
||||
}
|
||||
} else if (expectedTypes[i] === 'image') {
|
||||
const decodedData = Buffer.from(payload.data, 'base64');
|
||||
if (decodedData.length !== expectedData[i].length) {
|
||||
console.log(`❌ Payload ${i + 1}: Length mismatch`);
|
||||
passed = false;
|
||||
} else {
|
||||
let dataMatch = true;
|
||||
for (let j = 0; j < expectedData[i].length; j++) {
|
||||
if (decodedData[j] !== expectedData[i][j]) {
|
||||
dataMatch = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (dataMatch) {
|
||||
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
|
||||
} else {
|
||||
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
|
||||
passed = false;
|
||||
}
|
||||
}
|
||||
} else if (expectedTypes[i] === 'table') {
|
||||
const decodedData = Buffer.from(payload.data, 'base64');
|
||||
if (decodedData.length > 0) {
|
||||
console.log(`✅ Payload ${i + 1}: Arrow IPC data present (${decodedData.length} bytes)`);
|
||||
} else {
|
||||
console.log(`❌ Payload ${i + 1}: Arrow IPC data is empty`);
|
||||
passed = false;
|
||||
}
|
||||
}
|
||||
|
||||
console.log(` Size: ${payload.size} bytes\n`);
|
||||
}
|
||||
|
||||
// Test with chat-like payload (text + image + audio)
|
||||
console.log('=== Chat-like Payload Test ===');
|
||||
const chatData = [
|
||||
['text', 'Hello!', 'text'],
|
||||
['image', Buffer.from([0xFF, 0xD8, 0xFF, 0xE0]), 'image'],
|
||||
['audio', Buffer.from([0x46, 0x4C, 0x41, 0x43]), 'audio']
|
||||
];
|
||||
|
||||
const [chatEnv, _] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
chatData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: 'chat-' + correlationId,
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
if (chatEnv.payloads.length === 3) {
|
||||
console.log('✅ Chat-like payloads handled correctly');
|
||||
} else {
|
||||
console.log('❌ Chat-like payloads handling failed');
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
173
test/test_js_table_receiver.js
Normal file
173
test/test_js_table_receiver.js
Normal file
@@ -0,0 +1,173 @@
|
||||
/**
|
||||
* JavaScript Table Receiver Test
|
||||
* Tests the smartreceive function with table (Arrow IPC) payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natsbridge.js');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Table Receiver Test ===\n');
|
||||
|
||||
// Create a mock NATS message with table payload
|
||||
const tableData = [
|
||||
{ id: 1, name: 'Alice', age: 30, active: true },
|
||||
{ id: 2, name: 'Bob', age: 25, active: false },
|
||||
{ id: 3, name: 'Charlie', age: 35, active: true }
|
||||
];
|
||||
|
||||
// Convert to Arrow IPC format
|
||||
const arrow = require('apache-arrow');
|
||||
const fields = [
|
||||
new arrow.Field('id', arrow.Int64, true),
|
||||
new arrow.Field('name', arrow.Utf8, true),
|
||||
new arrow.Field('age', arrow.Int64, true),
|
||||
new arrow.Field('active', arrow.Boolean, true)
|
||||
];
|
||||
const schema = new arrow.Schema(fields);
|
||||
const batches = [];
|
||||
for (const row of tableData) {
|
||||
const batch = arrow.recordBatch.fromObjects([row], schema);
|
||||
batches.push(batch);
|
||||
}
|
||||
const buffers = arrow.ipc.recordBatchesToMessage(batches, schema).buffers;
|
||||
const combined = new Uint8Array(buffers.reduce((acc, b) => acc + b.byteLength, 0));
|
||||
let offset = 0;
|
||||
for (const buf of buffers) {
|
||||
combined.set(new Uint8Array(buf), offset);
|
||||
offset += buf.byteLength;
|
||||
}
|
||||
const arrowBuffer = Buffer.from(combined);
|
||||
|
||||
const testData = {
|
||||
correlation_id: 'js-table-receiver-' + crypto.randomUUID(),
|
||||
msg_id: 'msg-' + crypto.randomUUID(),
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: '/test/table',
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-table-test',
|
||||
sender_id: 'sender-' + crypto.randomUUID(),
|
||||
receiver_name: 'js-receiver',
|
||||
receiver_id: 'receiver-' + crypto.randomUUID(),
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
broker_url: TEST_BROKER_URL,
|
||||
metadata: {},
|
||||
payloads: [
|
||||
{
|
||||
id: 'payload-1',
|
||||
dataname: 'users_table',
|
||||
payload_type: 'table',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: arrowBuffer.length,
|
||||
data: arrowBuffer.toString('base64'),
|
||||
metadata: { payload_bytes: arrowBuffer.length }
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const mockMsg = {
|
||||
payload: JSON.stringify(testData)
|
||||
};
|
||||
|
||||
console.log('Mock Message Created:');
|
||||
console.log(` Correlation ID: ${testData.correlation_id}`);
|
||||
console.log(` Payloads: ${testData.payloads.length}`);
|
||||
console.log(` Payload type: ${testData.payloads[0].payload_type}`);
|
||||
console.log(` Transport: ${testData.payloads[0].transport}\n`);
|
||||
|
||||
try {
|
||||
// Receive and process the message
|
||||
console.log('Receiving and processing message...');
|
||||
const env = await NATSBridge.smartreceive(
|
||||
mockMsg,
|
||||
{
|
||||
max_retries: 3,
|
||||
base_delay: 100,
|
||||
max_delay: 1000
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Received Envelope ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate received data
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (!env.correlation_id) {
|
||||
console.log('❌ correlation_id is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ correlation_id present');
|
||||
}
|
||||
|
||||
if (env.payloads.length !== 1) {
|
||||
console.log(`❌ Expected 1 payload, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
const payload = env.payloads[0];
|
||||
if (payload[0] !== 'users_table') {
|
||||
console.log(`❌ Expected dataname 'users_table', got '${payload[0]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct dataname');
|
||||
}
|
||||
|
||||
if (payload[2] !== 'table') {
|
||||
console.log(`❌ Expected type 'table', got '${payload[2]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct type');
|
||||
}
|
||||
|
||||
// Verify table data is a Buffer (Arrow IPC format)
|
||||
if (payload[1] instanceof Buffer || payload[1] instanceof Uint8Array) {
|
||||
console.log('✅ Table data is Arrow IPC buffer');
|
||||
console.log(` Buffer size: ${payload[1].length} bytes`);
|
||||
} else {
|
||||
console.log(`❌ Expected Buffer/Uint8Array, got ${typeof payload[1]}`);
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Test round-trip with Arrow deserialization
|
||||
console.log('\n=== Arrow Deserialization Test ===');
|
||||
try {
|
||||
const table = arrow.tableFromRawBytes(payload[1]);
|
||||
console.log(`✅ Arrow table deserialized successfully`);
|
||||
console.log(` Schema: ${table.schema.fields.map(f => f.name).join(', ')}`);
|
||||
console.log(` Num rows: ${table.numRows}`);
|
||||
} catch (e) {
|
||||
console.log('❌ Arrow deserialization failed:', e.message);
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
180
test/test_js_table_sender.js
Normal file
180
test/test_js_table_sender.js
Normal file
@@ -0,0 +1,180 @@
|
||||
/**
|
||||
* JavaScript Table Sender Test
|
||||
* Tests the smartsend function with table (Arrow IPC) payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natsbridge.js');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const TEST_SUBJECT = '/test/table';
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Table Sender Test ===\n');
|
||||
|
||||
const correlationId = crypto.randomUUID();
|
||||
console.log(`Correlation ID: ${correlationId}`);
|
||||
console.log(`Subject: ${TEST_SUBJECT}`);
|
||||
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
|
||||
|
||||
// Test data - table data as array of objects
|
||||
const tableData = [
|
||||
{ id: 1, name: 'Alice', age: 30, active: true },
|
||||
{ id: 2, name: 'Bob', age: 25, active: false },
|
||||
{ id: 3, name: 'Charlie', age: 35, active: true },
|
||||
{ id: 4, name: 'Diana', age: 28, active: true },
|
||||
{ id: 5, name: 'Eve', age: 32, active: false }
|
||||
];
|
||||
|
||||
const testData = [
|
||||
['users_table', tableData, 'table']
|
||||
];
|
||||
|
||||
try {
|
||||
// Send the message
|
||||
console.log('Sending table payload...');
|
||||
const [env, envJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
testData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: correlationId,
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-table-test',
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Envelope Created ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Purpose: ${env.msg_purpose}`);
|
||||
console.log(`Sender: ${env.sender_name}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate envelope structure
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (env.payloads.length !== 1) {
|
||||
console.log(`❌ Expected 1 payload, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
const payload = env.payloads[0];
|
||||
if (payload.dataname !== 'users_table') {
|
||||
console.log(`❌ Expected dataname 'users_table', got '${payload.dataname}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct dataname');
|
||||
}
|
||||
|
||||
if (payload.payload_type !== 'table') {
|
||||
console.log(`❌ Expected payload_type 'table', got '${payload.payload_type}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct payload_type');
|
||||
}
|
||||
|
||||
if (payload.transport !== 'direct') {
|
||||
console.log(`❌ Expected transport 'direct', got '${payload.transport}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct transport');
|
||||
}
|
||||
|
||||
if (payload.encoding !== 'base64') {
|
||||
console.log(`❌ Expected encoding 'base64', got '${payload.encoding}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct encoding');
|
||||
}
|
||||
|
||||
// Verify Arrow IPC data can be decoded
|
||||
console.log('\n=== Arrow IPC Verification ===');
|
||||
const decodedData = Buffer.from(payload.data, 'base64');
|
||||
console.log(`Arrow IPC buffer size: ${decodedData.length} bytes`);
|
||||
|
||||
if (decodedData.length > 0) {
|
||||
console.log('✅ Arrow IPC data present');
|
||||
} else {
|
||||
console.log('❌ Arrow IPC data is empty');
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Test with larger table
|
||||
console.log('\n=== Larger Table Test ===');
|
||||
const largeTableData = [];
|
||||
for (let i = 1; i <= 100; i++) {
|
||||
largeTableData.push({
|
||||
id: i,
|
||||
name: `User${i}`,
|
||||
age: Math.floor(Math.random() * 100),
|
||||
active: Math.random() > 0.5,
|
||||
score: Math.random() * 100
|
||||
});
|
||||
}
|
||||
|
||||
const largeTestData = [
|
||||
['large_table', largeTableData, 'table']
|
||||
];
|
||||
|
||||
const [largeEnv, _] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
largeTestData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: 'large-' + correlationId,
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
if (largeEnv.payloads.length === 1) {
|
||||
console.log('✅ Large table handled correctly');
|
||||
console.log(` Size: ${largeEnv.payloads[0].size} bytes`);
|
||||
} else {
|
||||
console.log('❌ Large table handling failed');
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Test JSON string output
|
||||
console.log('\n=== JSON String Output Test ===');
|
||||
try {
|
||||
const parsed = JSON.parse(envJsonStr);
|
||||
if (parsed.correlation_id === env.correlation_id &&
|
||||
parsed.payloads.length === env.payloads.length) {
|
||||
console.log('✅ JSON string is valid and matches envelope');
|
||||
} else {
|
||||
console.log('❌ JSON string does not match envelope');
|
||||
passed = false;
|
||||
}
|
||||
} catch (e) {
|
||||
console.log('❌ JSON string is invalid:', e.message);
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
207
test/test_js_text_receiver.js
Normal file
207
test/test_js_text_receiver.js
Normal file
@@ -0,0 +1,207 @@
|
||||
/**
|
||||
* JavaScript Text Receiver Test
|
||||
* Tests the smartreceive function with text payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natsbridge.js');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Text Receiver Test ===\n');
|
||||
|
||||
// Create a mock NATS message with text payload
|
||||
const testData = {
|
||||
correlation_id: 'test-receiver-' + crypto.randomUUID(),
|
||||
msg_id: 'msg-' + crypto.randomUUID(),
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: '/test/text',
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-text-test',
|
||||
sender_id: 'sender-' + crypto.randomUUID(),
|
||||
receiver_name: 'js-receiver',
|
||||
receiver_id: 'receiver-' + crypto.randomUUID(),
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
broker_url: TEST_BROKER_URL,
|
||||
metadata: {},
|
||||
payloads: [
|
||||
{
|
||||
id: 'payload-' + crypto.randomUUID(),
|
||||
dataname: 'message',
|
||||
payload_type: 'text',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: 38,
|
||||
data: Buffer.from('Hello, NATSBridge! This is a test message.').toString('base64'),
|
||||
metadata: { payload_bytes: 38 }
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const mockMsg = {
|
||||
payload: JSON.stringify(testData)
|
||||
};
|
||||
|
||||
console.log('Mock Message Created:');
|
||||
console.log(` Correlation ID: ${testData.correlation_id}`);
|
||||
console.log(` Payloads: ${testData.payloads.length}`);
|
||||
console.log(` Payload dataname: ${testData.payloads[0].dataname}`);
|
||||
console.log(` Payload type: ${testData.payloads[0].payload_type}`);
|
||||
console.log(` Transport: ${testData.payloads[0].transport}\n`);
|
||||
|
||||
try {
|
||||
// Receive and process the message
|
||||
console.log('Receiving and processing message...');
|
||||
const env = await NATSBridge.smartreceive(
|
||||
mockMsg,
|
||||
{
|
||||
max_retries: 3,
|
||||
base_delay: 100,
|
||||
max_delay: 1000
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Received Envelope ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate received data
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (!env.correlation_id) {
|
||||
console.log('❌ correlation_id is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ correlation_id present');
|
||||
}
|
||||
|
||||
if (env.payloads.length !== 1) {
|
||||
console.log(`❌ Expected 1 payload, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
const payload = env.payloads[0];
|
||||
if (payload[0] !== 'message') {
|
||||
console.log(`❌ Expected dataname 'message', got '${payload[0]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct dataname');
|
||||
}
|
||||
|
||||
if (payload[2] !== 'text') {
|
||||
console.log(`❌ Expected type 'text', got '${payload[2]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct type');
|
||||
}
|
||||
|
||||
if (payload[1] !== 'Hello, NATSBridge! This is a test message.') {
|
||||
console.log(`❌ Data mismatch`);
|
||||
console.log(` Expected: Hello, NATSBridge! This is a test message.`);
|
||||
console.log(` Got: ${payload[1]}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Data correctly deserialized');
|
||||
}
|
||||
|
||||
// Test with multiple text payloads
|
||||
console.log('\n=== Multiple Text Payloads Test ===');
|
||||
const multiTestData = {
|
||||
correlation_id: 'multi-receiver-' + crypto.randomUUID(),
|
||||
msg_id: 'msg-' + crypto.randomUUID(),
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: '/test/text',
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-text-test',
|
||||
sender_id: 'sender-' + crypto.randomUUID(),
|
||||
receiver_name: 'js-receiver',
|
||||
receiver_id: 'receiver-' + crypto.randomUUID(),
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
broker_url: TEST_BROKER_URL,
|
||||
metadata: {},
|
||||
payloads: [
|
||||
{
|
||||
id: 'payload-1',
|
||||
dataname: 'msg1',
|
||||
payload_type: 'text',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: 16,
|
||||
data: Buffer.from('First message').toString('base64'),
|
||||
metadata: { payload_bytes: 16 }
|
||||
},
|
||||
{
|
||||
id: 'payload-2',
|
||||
dataname: 'msg2',
|
||||
payload_type: 'text',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: 16,
|
||||
data: Buffer.from('Second message').toString('base64'),
|
||||
metadata: { payload_bytes: 16 }
|
||||
},
|
||||
{
|
||||
id: 'payload-3',
|
||||
dataname: 'msg3',
|
||||
payload_type: 'text',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: 16,
|
||||
data: Buffer.from('Third message').toString('base64'),
|
||||
metadata: { payload_bytes: 16 }
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const mockMultiMsg = {
|
||||
payload: JSON.stringify(multiTestData)
|
||||
};
|
||||
|
||||
const multiEnv = await NATSBridge.smartreceive(mockMultiMsg);
|
||||
|
||||
if (multiEnv.payloads.length === 3) {
|
||||
console.log('✅ Multiple payloads handled correctly');
|
||||
|
||||
// Verify each payload
|
||||
const expectedMessages = ['First message', 'Second message', 'Third message'];
|
||||
for (let i = 0; i < 3; i++) {
|
||||
if (multiEnv.payloads[i][1] === expectedMessages[i]) {
|
||||
console.log(`✅ Payload ${i + 1} correctly deserialized`);
|
||||
} else {
|
||||
console.log(`❌ Payload ${i + 1} mismatch`);
|
||||
passed = false;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
console.log(`❌ Expected 3 payloads, got ${multiEnv.payloads.length}`);
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
170
test/test_js_text_sender.js
Normal file
170
test/test_js_text_sender.js
Normal file
@@ -0,0 +1,170 @@
|
||||
/**
|
||||
* JavaScript Text Sender Test
|
||||
* Tests the smartsend function with text payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natsbridge.js');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const TEST_SUBJECT = '/test/text';
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Text Sender Test ===\n');
|
||||
|
||||
const correlationId = crypto.randomUUID();
|
||||
console.log(`Correlation ID: ${correlationId}`);
|
||||
console.log(`Subject: ${TEST_SUBJECT}`);
|
||||
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
|
||||
|
||||
// Test data
|
||||
const textData = 'Hello, NATSBridge! This is a test message.';
|
||||
const testData = [
|
||||
['message', textData, 'text']
|
||||
];
|
||||
|
||||
try {
|
||||
// Send the message
|
||||
console.log('Sending text payload...');
|
||||
const [env, envJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
testData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: correlationId,
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-text-test',
|
||||
is_publish: false // Don't actually publish for this test
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Envelope Created ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Purpose: ${env.msg_purpose}`);
|
||||
console.log(`Sender: ${env.sender_name}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate envelope structure
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (!env.correlation_id) {
|
||||
console.log('❌ correlation_id is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ correlation_id present');
|
||||
}
|
||||
|
||||
if (!env.msg_id) {
|
||||
console.log('❌ msg_id is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ msg_id present');
|
||||
}
|
||||
|
||||
if (!env.timestamp) {
|
||||
console.log('❌ timestamp is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ timestamp present');
|
||||
}
|
||||
|
||||
if (env.payloads.length !== 1) {
|
||||
console.log(`❌ Expected 1 payload, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
const payload = env.payloads[0];
|
||||
if (payload.dataname !== 'message') {
|
||||
console.log(`❌ Expected dataname 'message', got '${payload.dataname}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct dataname');
|
||||
}
|
||||
|
||||
if (payload.payload_type !== 'text') {
|
||||
console.log(`❌ Expected payload_type 'text', got '${payload.payload_type}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct payload_type');
|
||||
}
|
||||
|
||||
if (payload.transport !== 'direct') {
|
||||
console.log(`❌ Expected transport 'direct', got '${payload.transport}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct transport');
|
||||
}
|
||||
|
||||
if (payload.encoding !== 'base64') {
|
||||
console.log(`❌ Expected encoding 'base64', got '${payload.encoding}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct encoding');
|
||||
}
|
||||
|
||||
// Decode and verify the data
|
||||
const decodedData = Buffer.from(payload.data, 'base64').toString('utf8');
|
||||
if (decodedData !== textData) {
|
||||
console.log(`❌ Decoded data mismatch`);
|
||||
console.log(` Expected: ${textData}`);
|
||||
console.log(` Got: ${decodedData}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Data integrity verified');
|
||||
}
|
||||
|
||||
console.log(`\nPayload size: ${payload.size} bytes`);
|
||||
console.log(`Base64 data length: ${payload.data.length} chars`);
|
||||
|
||||
// Test with multiple text payloads
|
||||
console.log('\n=== Multiple Text Payloads Test ===');
|
||||
const multiTestData = [
|
||||
['msg1', 'First message', 'text'],
|
||||
['msg2', 'Second message', 'text'],
|
||||
['msg3', 'Third message', 'text']
|
||||
];
|
||||
|
||||
const [multiEnv, multiEnvJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
multiTestData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: 'multi-test-' + correlationId,
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
if (multiEnv.payloads.length === 3) {
|
||||
console.log('✅ Multiple payloads handled correctly');
|
||||
} else {
|
||||
console.log(`❌ Expected 3 payloads, got ${multiEnv.payloads.length}`);
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
@@ -186,7 +186,7 @@ function test_mix_send()
|
||||
]
|
||||
|
||||
# Use smartsend with mixed content
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
sendinfo = NATSBridge.smartsend(
|
||||
SUBJECT,
|
||||
payloads; # List of (dataname, data, type) tuples
|
||||
broker_url = NATS_URL,
|
||||
@@ -203,6 +203,7 @@ function test_mix_send()
|
||||
is_publish = true # Publish the message to NATS
|
||||
)
|
||||
|
||||
env, env_json_str = sendinfo
|
||||
log_trace("Sent message with $(length(env.payloads)) payloads")
|
||||
|
||||
# Log transport type for each payload
|
||||
|
||||
185
test/test_mpy_binary_receiver.py
Normal file
185
test/test_mpy_binary_receiver.py
Normal file
@@ -0,0 +1,185 @@
|
||||
"""
|
||||
MicroPython Binary Receiver Test
|
||||
Tests the smartreceive function with binary/image/audio/video payloads
|
||||
|
||||
Note: This test is designed for both MicroPython and desktop Python
|
||||
for compatibility testing.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import base64
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge_mpy import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
def run_test():
|
||||
print('=== MicroPython Binary Receiver Test ===\n')
|
||||
|
||||
from natsbridge_mpy import _generate_uuid
|
||||
|
||||
# Create mock NATS message with binary payloads
|
||||
image_data = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
|
||||
audio_data = bytes([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]) # FLAC header
|
||||
video_data = bytes([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]) # MP4 header
|
||||
generic_binary = bytes([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE])
|
||||
|
||||
test_data = {
|
||||
'correlation_id': 'mpy-binary-receiver-' + _generate_uuid(),
|
||||
'msg_id': _generate_uuid(),
|
||||
'timestamp': '2024-01-15T10:30:00Z',
|
||||
'send_to': '/test/binary',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'mpy-binary-test',
|
||||
'sender_id': _generate_uuid(),
|
||||
'receiver_name': 'mpy-receiver',
|
||||
'receiver_id': _generate_uuid(),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'image',
|
||||
'payload_type': 'image',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(image_data),
|
||||
'data': base64.b64encode(image_data).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(image_data)}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'audio',
|
||||
'payload_type': 'audio',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(audio_data),
|
||||
'data': base64.b64encode(audio_data).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(audio_data)}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'video',
|
||||
'payload_type': 'video',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(video_data),
|
||||
'data': base64.b64encode(video_data).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(video_data)}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'binary',
|
||||
'payload_type': 'binary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(generic_binary),
|
||||
'data': base64.b64encode(generic_binary).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(generic_binary)}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_msg = {
|
||||
'payload': json.dumps(test_data)
|
||||
}
|
||||
|
||||
print('Mock Message Created:')
|
||||
print(f' Correlation ID: {test_data["correlation_id"]}')
|
||||
print(f' Payloads: {len(test_data["payloads"])}')
|
||||
print(f' Payload types: {", ".join(p["payload_type"] for p in test_data["payloads"])}\n')
|
||||
|
||||
try:
|
||||
# Receive and process the message
|
||||
print('Receiving and processing message...')
|
||||
env = smartreceive(
|
||||
mock_msg,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
|
||||
print('\n=== Received Envelope ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate received data
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Expected data
|
||||
expected_data = [
|
||||
('image', image_data, 'image'),
|
||||
('audio', audio_data, 'audio'),
|
||||
('video', video_data, 'video'),
|
||||
('binary', generic_binary, 'binary')
|
||||
]
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
expected = expected_data[i]
|
||||
|
||||
if payload[0] != expected[0]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected[0]}', got '{payload[0]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload[2] != expected[2]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected[2]}', got '{payload[2]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
# Verify binary data integrity
|
||||
received_data = payload[1]
|
||||
if received_data != expected[1]:
|
||||
print(f'❌ Payload {i + 1}: Data mismatch')
|
||||
print(f' Expected: {expected[1]}')
|
||||
print(f' Got: {received_data}')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data correctly deserialized')
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_test()
|
||||
163
test/test_mpy_binary_sender.py
Normal file
163
test/test_mpy_binary_sender.py
Normal file
@@ -0,0 +1,163 @@
|
||||
"""
|
||||
MicroPython Binary Sender Test
|
||||
Tests the smartsend function with binary/image/audio/video payloads
|
||||
|
||||
Note: This test is designed for both MicroPython and desktop Python
|
||||
for compatibility testing.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import base64
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge_mpy import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL, DEFAULT_SIZE_THRESHOLD, MAX_PAYLOAD_SIZE
|
||||
|
||||
TEST_SUBJECT = '/test/binary'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
def run_test():
|
||||
print('=== MicroPython Binary Sender Test ===\n')
|
||||
|
||||
from natsbridge_mpy import _generate_uuid
|
||||
correlation_id = 'mpy-binary-test-' + _generate_uuid()
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}')
|
||||
print(f'Default Size Threshold: {DEFAULT_SIZE_THRESHOLD} bytes')
|
||||
print(f'Max Payload Size: {MAX_PAYLOAD_SIZE} bytes\n')
|
||||
|
||||
# Test data - binary data for different types
|
||||
image_data = bytearray([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
|
||||
audio_data = bytearray([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]) # FLAC header
|
||||
video_data = bytearray([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]) # MP4 header
|
||||
generic_binary = bytearray([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE])
|
||||
|
||||
test_data = [
|
||||
('image', bytes(image_data), 'image'),
|
||||
('audio', bytes(audio_data), 'audio'),
|
||||
('video', bytes(video_data), 'video'),
|
||||
('binary', bytes(generic_binary), 'binary')
|
||||
]
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending binary payloads...')
|
||||
env, env_json_str = smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='mpy-binary-test',
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Test each payload
|
||||
expected_datanames = ['image', 'audio', 'video', 'binary']
|
||||
expected_types = ['image', 'audio', 'video', 'binary']
|
||||
expected_data = [bytes(image_data), bytes(audio_data), bytes(video_data), bytes(generic_binary)]
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
|
||||
if payload['dataname'] != expected_datanames[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload['payload_type'] != expected_types[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct encoding')
|
||||
|
||||
# Decode and verify the data
|
||||
decoded_data = base64.b64decode(payload['data'])
|
||||
original_data = expected_data[i]
|
||||
|
||||
if decoded_data != original_data:
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
|
||||
print(f' Size: {payload["size"]} bytes\n')
|
||||
|
||||
# Test with larger binary data
|
||||
print('=== Large Binary Data Test ===')
|
||||
large_data = bytes([0xFF] * 1000) # 1KB of binary data
|
||||
large_test_data = [
|
||||
('large_binary', large_data, 'binary')
|
||||
]
|
||||
|
||||
large_env, _ = smartsend(
|
||||
TEST_SUBJECT,
|
||||
large_test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='large-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if len(large_env['payloads']) == 1 and large_env['payloads'][0]['size'] == 1000:
|
||||
print('✅ Large binary data handled correctly')
|
||||
else:
|
||||
print('❌ Large binary data handling failed')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_test()
|
||||
224
test/test_mpy_dictionary_receiver.py
Normal file
224
test/test_mpy_dictionary_receiver.py
Normal file
@@ -0,0 +1,224 @@
|
||||
"""
|
||||
MicroPython Dictionary Receiver Test
|
||||
Tests the smartreceive function with dictionary payloads
|
||||
|
||||
Note: This test is designed for both MicroPython and desktop Python
|
||||
for compatibility testing.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge_mpy import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
def run_test():
|
||||
print('=== MicroPython Dictionary Receiver Test ===\n')
|
||||
|
||||
from natsbridge_mpy import _generate_uuid
|
||||
|
||||
# Create a mock NATS message with dictionary payloads
|
||||
import base64
|
||||
|
||||
simple_dict = {'key1': 'value1', 'key2': 'value2'}
|
||||
nested_dict = {'outer': {'inner': 'value', 'number': 42}}
|
||||
array_dict = {'items': [1, 2, 3, 'four', 'five']}
|
||||
mixed_dict = {'string': 'text', 'number': 123, 'boolean': True, 'null_val': None}
|
||||
|
||||
test_data = {
|
||||
'correlation_id': 'mpy-receiver-dict-' + _generate_uuid(),
|
||||
'msg_id': _generate_uuid(),
|
||||
'timestamp': '2024-01-15T10:30:00Z',
|
||||
'send_to': '/test/dictionary',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'mpy-dict-test',
|
||||
'sender_id': _generate_uuid(),
|
||||
'receiver_name': 'mpy-receiver',
|
||||
'receiver_id': _generate_uuid(),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'simple_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(simple_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(simple_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(simple_dict).encode('utf8'))}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'nested_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(nested_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(nested_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(nested_dict).encode('utf8'))}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'array_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(array_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(array_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(array_dict).encode('utf8'))}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'mixed_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(mixed_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(mixed_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(mixed_dict).encode('utf8'))}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_msg = {
|
||||
'payload': json.dumps(test_data)
|
||||
}
|
||||
|
||||
print('Mock Message Created:')
|
||||
print(f' Correlation ID: {test_data["correlation_id"]}')
|
||||
print(f' Payloads: {len(test_data["payloads"])}')
|
||||
print(f' Payload types: {", ".join(p["payload_type"] for p in test_data["payloads"])}\n')
|
||||
|
||||
try:
|
||||
# Receive and process the message
|
||||
print('Receiving and processing message...')
|
||||
env = smartreceive(
|
||||
mock_msg,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
|
||||
print('\n=== Received Envelope ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate received data
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Expected data
|
||||
expected_data = [
|
||||
('simple_dict', simple_dict, 'dictionary'),
|
||||
('nested_dict', nested_dict, 'dictionary'),
|
||||
('array_dict', array_dict, 'dictionary'),
|
||||
('mixed_dict', mixed_dict, 'dictionary')
|
||||
]
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
expected = expected_data[i]
|
||||
|
||||
if payload[0] != expected[0]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected[0]}', got '{payload[0]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload[2] != expected[2]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected[2]}', got '{payload[2]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
data_match = json.dumps(payload[1], sort_keys=True) == json.dumps(expected[1], sort_keys=True)
|
||||
if not data_match:
|
||||
print(f'❌ Payload {i + 1}: Data mismatch')
|
||||
print(f' Expected: {json.dumps(expected[1], sort_keys=True)}')
|
||||
print(f' Got: {json.dumps(payload[1], sort_keys=True)}')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data correctly deserialized')
|
||||
|
||||
# Test round-trip with receive
|
||||
print('\n=== Round-trip Test ===')
|
||||
round_trip_data = {
|
||||
'correlation_id': 'roundtrip-' + _generate_uuid(),
|
||||
'msg_id': _generate_uuid(),
|
||||
'timestamp': '2024-01-15T10:30:00Z',
|
||||
'send_to': '/test/dictionary',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'mpy-test',
|
||||
'sender_id': _generate_uuid(),
|
||||
'receiver_name': 'mpy-receiver',
|
||||
'receiver_id': _generate_uuid(),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'roundtrip',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8'))}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_rt_msg = {'payload': json.dumps(round_trip_data)}
|
||||
rt_env = smartreceive(mock_rt_msg)
|
||||
|
||||
if rt_env['payloads'][0][0] == 'roundtrip' and rt_env['payloads'][0][2] == 'dictionary':
|
||||
print('✅ Round-trip test successful')
|
||||
else:
|
||||
print('❌ Round-trip test failed')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_test()
|
||||
177
test/test_mpy_dictionary_sender.py
Normal file
177
test/test_mpy_dictionary_sender.py
Normal file
@@ -0,0 +1,177 @@
|
||||
"""
|
||||
MicroPython Dictionary Sender Test
|
||||
Tests the smartsend function with dictionary payloads
|
||||
|
||||
Note: This test is designed for both MicroPython and desktop Python
|
||||
for compatibility testing.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge_mpy import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL, DEFAULT_SIZE_THRESHOLD, MAX_PAYLOAD_SIZE
|
||||
|
||||
TEST_SUBJECT = '/test/dictionary'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
def run_test():
|
||||
print('=== MicroPython Dictionary Sender Test ===\n')
|
||||
|
||||
from natsbridge_mpy import _generate_uuid
|
||||
correlation_id = 'mpy-dict-test-' + _generate_uuid()
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}')
|
||||
print(f'Default Size Threshold: {DEFAULT_SIZE_THRESHOLD} bytes')
|
||||
print(f'Max Payload Size: {MAX_PAYLOAD_SIZE} bytes\n')
|
||||
|
||||
# Test data - various dictionary structures
|
||||
test_data = [
|
||||
('simple_dict', {'key1': 'value1', 'key2': 'value2'}, 'dictionary'),
|
||||
('nested_dict', {'outer': {'inner': 'value', 'number': 42}}, 'dictionary'),
|
||||
('array_dict', {'items': [1, 2, 3, 'four', 'five']}, 'dictionary'),
|
||||
('mixed_dict', {'string': 'text', 'number': 123, 'boolean': True, 'null_val': None}, 'dictionary')
|
||||
]
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending dictionary payloads...')
|
||||
env, env_json_str = smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='mpy-dict-test',
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Test each payload
|
||||
expected_datanames = ['simple_dict', 'nested_dict', 'array_dict', 'mixed_dict']
|
||||
expected_types = ['dictionary', 'dictionary', 'dictionary', 'dictionary']
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
|
||||
if payload['dataname'] != expected_datanames[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload['payload_type'] != expected_types[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct encoding')
|
||||
|
||||
# Decode and verify the data
|
||||
import base64
|
||||
decoded_data = json.loads(base64.b64decode(payload['data']).decode('ascii'))
|
||||
original_data = test_data[i][1]
|
||||
|
||||
# Normalize for comparison
|
||||
if json.dumps(decoded_data, sort_keys=True) != json.dumps(original_data, sort_keys=True):
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
print(f' Expected: {json.dumps(original_data)}')
|
||||
print(f' Got: {json.dumps(decoded_data)}')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
|
||||
print(f' Size: {payload["size"]} bytes\n')
|
||||
|
||||
# Test round-trip serialization
|
||||
print('=== Round-trip Serialization Test ===')
|
||||
round_trip_data = [
|
||||
('roundtrip', {'test': 'data', 'numbers': [1, 2, 3], 'nested': {'a': 1, 'b': 2}}, 'dictionary')
|
||||
]
|
||||
|
||||
rt_env, _ = smartsend(
|
||||
TEST_SUBJECT,
|
||||
round_trip_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='roundtrip-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
rt_payload = rt_env['payloads'][0]
|
||||
rt_decoded = json.loads(base64.b64decode(rt_payload['data']).decode('ascii'))
|
||||
|
||||
if json.dumps(rt_decoded, sort_keys=True) == json.dumps(round_trip_data[0][1], sort_keys=True):
|
||||
print('✅ Round-trip serialization successful')
|
||||
else:
|
||||
print('❌ Round-trip serialization failed')
|
||||
passed = False
|
||||
|
||||
# Test JSON string output
|
||||
print('\n=== JSON String Output Test ===')
|
||||
try:
|
||||
parsed = json.loads(env_json_str)
|
||||
if parsed['correlation_id'] == env['correlation_id'] and \
|
||||
len(parsed['payloads']) == len(env['payloads']):
|
||||
print('✅ JSON string is valid and matches envelope')
|
||||
else:
|
||||
print('❌ JSON string does not match envelope')
|
||||
passed = False
|
||||
except json.JSONDecodeError as e:
|
||||
print(f'❌ JSON string is invalid: {e}')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_test()
|
||||
209
test/test_mpy_text_receiver.py
Normal file
209
test/test_mpy_text_receiver.py
Normal file
@@ -0,0 +1,209 @@
|
||||
"""
|
||||
MicroPython Text Receiver Test
|
||||
Tests the smartreceive function with text payloads
|
||||
|
||||
Note: This test is designed for both MicroPython and desktop Python
|
||||
for compatibility testing.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge_mpy import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
def run_test():
|
||||
print('=== MicroPython Text Receiver Test ===\n')
|
||||
|
||||
from natsbridge_mpy import _generate_uuid
|
||||
|
||||
# Create a mock NATS message with text payload
|
||||
test_text = 'Hello, NATSBridge! This is a test message.'
|
||||
import base64
|
||||
|
||||
test_data = {
|
||||
'correlation_id': 'mpy-receiver-test-' + _generate_uuid(),
|
||||
'msg_id': _generate_uuid(),
|
||||
'timestamp': '2024-01-15T10:30:00Z',
|
||||
'send_to': '/test/text',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'mpy-text-test',
|
||||
'sender_id': _generate_uuid(),
|
||||
'receiver_name': 'mpy-receiver',
|
||||
'receiver_id': _generate_uuid(),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'message',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(test_text.encode('utf8')),
|
||||
'data': base64.b64encode(test_text.encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(test_text.encode('utf8'))}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_msg = {
|
||||
'payload': json.dumps(test_data)
|
||||
}
|
||||
|
||||
print('Mock Message Created:')
|
||||
print(f' Correlation ID: {test_data["correlation_id"]}')
|
||||
print(f' Payloads: {len(test_data["payloads"])}')
|
||||
print(f' Payload dataname: {test_data["payloads"][0]["dataname"]}')
|
||||
print(f' Payload type: {test_data["payloads"][0]["payload_type"]}')
|
||||
print(f' Transport: {test_data["payloads"][0]["transport"]}\n')
|
||||
|
||||
try:
|
||||
# Receive and process the message
|
||||
print('Receiving and processing message...')
|
||||
env = smartreceive(
|
||||
mock_msg,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
|
||||
print('\n=== Received Envelope ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate received data
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if len(env['payloads']) != 1:
|
||||
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
payload = env['payloads'][0]
|
||||
if payload[0] != 'message':
|
||||
print(f"❌ Expected dataname 'message', got '{payload[0]}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct dataname')
|
||||
|
||||
if payload[2] != 'text':
|
||||
print(f"❌ Expected type 'text', got '{payload[2]}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct type')
|
||||
|
||||
if payload[1] != test_text:
|
||||
print('❌ Data mismatch')
|
||||
print(f' Expected: {test_text}')
|
||||
print(f' Got: {payload[1]}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Data correctly deserialized')
|
||||
|
||||
# Test with multiple text payloads
|
||||
print('\n=== Multiple Text Payloads Test ===')
|
||||
multi_test_data = {
|
||||
'correlation_id': 'multi-receiver-' + _generate_uuid(),
|
||||
'msg_id': _generate_uuid(),
|
||||
'timestamp': '2024-01-15T10:30:00Z',
|
||||
'send_to': '/test/text',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'mpy-text-test',
|
||||
'sender_id': _generate_uuid(),
|
||||
'receiver_name': 'mpy-receiver',
|
||||
'receiver_id': _generate_uuid(),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'msg1',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': 16,
|
||||
'data': base64.b64encode(b'First message').decode('ascii'),
|
||||
'metadata': {'payload_bytes': 16}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'msg2',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': 16,
|
||||
'data': base64.b64encode(b'Second message').decode('ascii'),
|
||||
'metadata': {'payload_bytes': 16}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'msg3',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': 16,
|
||||
'data': base64.b64encode(b'Third message').decode('ascii'),
|
||||
'metadata': {'payload_bytes': 16}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_multi_msg = {'payload': json.dumps(multi_test_data)}
|
||||
multi_env = smartreceive(mock_multi_msg)
|
||||
|
||||
if len(multi_env['payloads']) == 3:
|
||||
print('✅ Multiple payloads handled correctly')
|
||||
|
||||
# Verify each payload
|
||||
expected_messages = ['First message', 'Second message', 'Third message']
|
||||
for i in range(3):
|
||||
if multi_env['payloads'][i][1] == expected_messages[i]:
|
||||
print(f'✅ Payload {i + 1} correctly deserialized')
|
||||
else:
|
||||
print(f'❌ Payload {i + 1} mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f'❌ Expected 3 payloads, got {len(multi_env["payloads"])}')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_test()
|
||||
205
test/test_mpy_text_sender.py
Normal file
205
test/test_mpy_text_sender.py
Normal file
@@ -0,0 +1,205 @@
|
||||
"""
|
||||
MicroPython Text Sender Test
|
||||
Tests the smartsend function with text payloads
|
||||
|
||||
Note: This test is designed for both MicroPython and desktop Python
|
||||
for compatibility testing.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge_mpy import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL, DEFAULT_SIZE_THRESHOLD, MAX_PAYLOAD_SIZE
|
||||
|
||||
TEST_SUBJECT = '/test/text'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
def run_test():
|
||||
print('=== MicroPython Text Sender Test ===\n')
|
||||
|
||||
from natsbridge_mpy import _generate_uuid
|
||||
correlation_id = 'mpy-text-test-' + _generate_uuid()
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}')
|
||||
print(f'Default Size Threshold: {DEFAULT_SIZE_THRESHOLD} bytes')
|
||||
print(f'Max Payload Size: {MAX_PAYLOAD_SIZE} bytes\n')
|
||||
|
||||
# Test data
|
||||
text_data = 'Hello, NATSBridge! This is a test message.'
|
||||
test_data = [
|
||||
('message', text_data, 'text')
|
||||
]
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending text payload...')
|
||||
env, env_json_str = smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='mpy-text-test',
|
||||
is_publish=False # Don't actually publish for this test
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if not env.get('msg_id'):
|
||||
print('❌ msg_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ msg_id present')
|
||||
|
||||
if not env.get('timestamp'):
|
||||
print('❌ timestamp is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ timestamp present')
|
||||
|
||||
if len(env['payloads']) != 1:
|
||||
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
payload = env['payloads'][0]
|
||||
if payload['dataname'] != 'message':
|
||||
print(f"❌ Expected dataname 'message', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct dataname')
|
||||
|
||||
if payload['payload_type'] != 'text':
|
||||
print(f"❌ Expected payload_type 'text', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct payload_type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct encoding')
|
||||
|
||||
# Decode and verify the data
|
||||
import base64
|
||||
decoded_data = base64.b64decode(payload['data']).decode('ascii')
|
||||
if decoded_data != text_data:
|
||||
print('❌ Decoded data mismatch')
|
||||
print(f' Expected: {text_data}')
|
||||
print(f' Got: {decoded_data}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Data integrity verified')
|
||||
|
||||
print(f'\nPayload size: {payload["size"]} bytes')
|
||||
print(f'Base64 data length: {len(payload["data"])} chars')
|
||||
|
||||
# Test with multiple text payloads
|
||||
print('\n=== Multiple Text Payloads Test ===')
|
||||
multi_test_data = [
|
||||
('msg1', 'First message', 'text'),
|
||||
('msg2', 'Second message', 'text'),
|
||||
('msg3', 'Third message', 'text')
|
||||
]
|
||||
|
||||
multi_env, _ = smartsend(
|
||||
TEST_SUBJECT,
|
||||
multi_test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='multi-test-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if len(multi_env['payloads']) == 3:
|
||||
print('✅ Multiple payloads handled correctly')
|
||||
else:
|
||||
print(f'❌ Expected 3 payloads, got {len(multi_env["payloads"])}')
|
||||
passed = False
|
||||
|
||||
# Test size threshold enforcement
|
||||
print('\n=== Size Threshold Test ===')
|
||||
small_text = 'small'
|
||||
large_text = 'x' * (DEFAULT_SIZE_THRESHOLD - 100) # Just under threshold
|
||||
|
||||
small_env, _ = smartsend(
|
||||
TEST_SUBJECT,
|
||||
[('small', small_text, 'text')],
|
||||
broker_url=TEST_BROKER_URL,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if small_env['payloads'][0]['transport'] == 'direct':
|
||||
print('✅ Small payload uses direct transport')
|
||||
else:
|
||||
print('❌ Small payload should use direct transport')
|
||||
passed = False
|
||||
|
||||
# Test that large text (> MAX_PAYLOAD_SIZE) raises error
|
||||
print('\n=== Max Payload Size Test ===')
|
||||
try:
|
||||
too_large_text = 'x' * (MAX_PAYLOAD_SIZE + 1000)
|
||||
large_env, _ = smartsend(
|
||||
TEST_SUBJECT,
|
||||
[('large', too_large_text, 'text')],
|
||||
broker_url=TEST_BROKER_URL,
|
||||
is_publish=False
|
||||
)
|
||||
print('❌ Should have raised MemoryError for payload exceeding MAX_PAYLOAD_SIZE')
|
||||
passed = False
|
||||
except MemoryError as e:
|
||||
print(f'✅ Correctly raised MemoryError: {e}')
|
||||
except Exception as e:
|
||||
print(f'❌ Unexpected error: {type(e).__name__}: {e}')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_test()
|
||||
184
test/test_py_binary_receiver.py
Normal file
184
test/test_py_binary_receiver.py
Normal file
@@ -0,0 +1,184 @@
|
||||
"""
|
||||
Python Binary Receiver Test
|
||||
Tests the smartreceive function with binary/image/audio/video/table payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import base64
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Binary Receiver Test ===\n')
|
||||
|
||||
# Create mock NATS message with binary payloads
|
||||
image_data = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
|
||||
audio_data = bytes([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]) # FLAC header
|
||||
video_data = bytes([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]) # MP4 header
|
||||
generic_binary = bytes([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE])
|
||||
|
||||
test_data = {
|
||||
'correlation_id': 'py-binary-receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'timestamp': asyncio.get_event_loop().time().isoformat(),
|
||||
'send_to': '/test/binary',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'py-binary-test',
|
||||
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'receiver_name': 'py-receiver',
|
||||
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': 'payload-1',
|
||||
'dataname': 'image',
|
||||
'payload_type': 'image',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(image_data),
|
||||
'data': base64.b64encode(image_data).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(image_data)}
|
||||
},
|
||||
{
|
||||
'id': 'payload-2',
|
||||
'dataname': 'audio',
|
||||
'payload_type': 'audio',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(audio_data),
|
||||
'data': base64.b64encode(audio_data).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(audio_data)}
|
||||
},
|
||||
{
|
||||
'id': 'payload-3',
|
||||
'dataname': 'video',
|
||||
'payload_type': 'video',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(video_data),
|
||||
'data': base64.b64encode(video_data).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(video_data)}
|
||||
},
|
||||
{
|
||||
'id': 'payload-4',
|
||||
'dataname': 'binary',
|
||||
'payload_type': 'binary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(generic_binary),
|
||||
'data': base64.b64encode(generic_binary).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(generic_binary)}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_msg = {
|
||||
'payload': json.dumps(test_data)
|
||||
}
|
||||
|
||||
print('Mock Message Created:')
|
||||
print(f' Correlation ID: {test_data["correlation_id"]}')
|
||||
print(f' Payloads: {len(test_data["payloads"])}')
|
||||
print(f' Payload types: {", ".join(p["payload_type"] for p in test_data["payloads"])}\n')
|
||||
|
||||
try:
|
||||
# Receive and process the message
|
||||
print('Receiving and processing message...')
|
||||
env = await smartreceive(
|
||||
mock_msg,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
|
||||
print('\n=== Received Envelope ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate received data
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Expected data
|
||||
expected_data = [
|
||||
('image', image_data, 'image'),
|
||||
('audio', audio_data, 'audio'),
|
||||
('video', video_data, 'video'),
|
||||
('binary', generic_binary, 'binary')
|
||||
]
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
expected = expected_data[i]
|
||||
|
||||
if payload[0] != expected[0]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected[0]}', got '{payload[0]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload[2] != expected[2]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected[2]}', got '{payload[2]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
# Verify binary data integrity
|
||||
received_data = payload[1]
|
||||
if not isinstance(received_data, (bytes, bytearray)):
|
||||
print(f'❌ Payload {i + 1}: Expected bytes/bytearray, got {type(received_data)}')
|
||||
passed = False
|
||||
elif received_data != expected[1]:
|
||||
print(f'❌ Payload {i + 1}: Data mismatch')
|
||||
print(f' Expected: {expected[1]}')
|
||||
print(f' Got: {received_data}')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data correctly deserialized')
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
183
test/test_py_binary_sender.py
Normal file
183
test/test_py_binary_sender.py
Normal file
@@ -0,0 +1,183 @@
|
||||
"""
|
||||
Python Binary Sender Test
|
||||
Tests the smartsend function with binary/image/audio/video/table payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
import base64
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_SUBJECT = '/test/binary'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Binary Sender Test ===\n')
|
||||
|
||||
correlation_id = 'py-binary-test-' + str(asyncio.get_event_loop().time() * 1000000)
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}\n')
|
||||
|
||||
# Test data - binary data for different types
|
||||
image_data = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
|
||||
audio_data = bytes([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]) # FLAC header
|
||||
video_data = bytes([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]) # MP4 header
|
||||
generic_binary = bytes([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE])
|
||||
|
||||
# Test table data
|
||||
try:
|
||||
import pandas as pd
|
||||
table_data = pd.DataFrame({
|
||||
'id': [1, 2, 3, 4, 5],
|
||||
'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
|
||||
'value': [10.5, 20.3, 30.1, 40.9, 50.7]
|
||||
})
|
||||
table_available = True
|
||||
except ImportError:
|
||||
table_available = False
|
||||
table_data = None
|
||||
|
||||
test_data = [
|
||||
('image', image_data, 'image'),
|
||||
('audio', audio_data, 'audio'),
|
||||
('video', video_data, 'video'),
|
||||
('binary', generic_binary, 'binary')
|
||||
]
|
||||
|
||||
if table_available:
|
||||
test_data.append(('table', table_data, 'table'))
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending binary payloads...')
|
||||
env, env_json_str = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='py-binary-test',
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
expected_count = 5 if table_available else 4
|
||||
if len(env['payloads']) != expected_count:
|
||||
print(f'❌ Expected {expected_count} payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Test each payload
|
||||
expected_datanames = ['image', 'audio', 'video', 'binary']
|
||||
expected_types = ['image', 'audio', 'video', 'binary']
|
||||
expected_data = [image_data, audio_data, video_data, generic_binary]
|
||||
|
||||
if table_available:
|
||||
expected_datanames.append('table')
|
||||
expected_types.append('table')
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
|
||||
if payload['dataname'] != expected_datanames[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload['payload_type'] != expected_types[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct encoding')
|
||||
|
||||
# Decode and verify the data
|
||||
decoded_data = base64.b64decode(payload['data'])
|
||||
|
||||
if i < len(expected_data):
|
||||
original_data = expected_data[i]
|
||||
if decoded_data != original_data:
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
else:
|
||||
# Table payload - just verify it's present
|
||||
print(f'✅ Payload {i + 1}: Table data present (size: {payload["size"]} bytes)')
|
||||
|
||||
print(f' Size: {payload["size"]} bytes\n')
|
||||
|
||||
# Test with larger binary data
|
||||
print('=== Large Binary Data Test ===')
|
||||
large_data = bytes([0xFF] * 10000) # 10KB of binary data
|
||||
large_test_data = [
|
||||
('large_binary', large_data, 'binary')
|
||||
]
|
||||
|
||||
large_env, _ = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
large_test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='large-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if len(large_env['payloads']) == 1 and large_env['payloads'][0]['size'] == 10000:
|
||||
print('✅ Large binary data handled correctly')
|
||||
else:
|
||||
print('❌ Large binary data handling failed')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
220
test/test_py_dictionary_receiver.py
Normal file
220
test/test_py_dictionary_receiver.py
Normal file
@@ -0,0 +1,220 @@
|
||||
"""
|
||||
Python Dictionary Receiver Test
|
||||
Tests the smartreceive function with dictionary payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Dictionary Receiver Test ===\n')
|
||||
|
||||
# Create a mock NATS message with dictionary payloads
|
||||
import base64
|
||||
|
||||
simple_dict = {'key1': 'value1', 'key2': 'value2'}
|
||||
nested_dict = {'outer': {'inner': 'value', 'number': 42}}
|
||||
array_dict = {'items': [1, 2, 3, 'four', 'five']}
|
||||
mixed_dict = {'string': 'text', 'number': 123, 'boolean': True, 'null_val': None}
|
||||
|
||||
test_data = {
|
||||
'correlation_id': 'py-receiver-dict-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'timestamp': asyncio.get_event_loop().time().isoformat(),
|
||||
'send_to': '/test/dictionary',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'py-dict-test',
|
||||
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'receiver_name': 'py-receiver',
|
||||
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': 'payload-1',
|
||||
'dataname': 'simple_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(simple_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(simple_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(simple_dict).encode('utf8'))}
|
||||
},
|
||||
{
|
||||
'id': 'payload-2',
|
||||
'dataname': 'nested_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(nested_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(nested_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(nested_dict).encode('utf8'))}
|
||||
},
|
||||
{
|
||||
'id': 'payload-3',
|
||||
'dataname': 'array_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(array_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(array_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(array_dict).encode('utf8'))}
|
||||
},
|
||||
{
|
||||
'id': 'payload-4',
|
||||
'dataname': 'mixed_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(mixed_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(mixed_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(mixed_dict).encode('utf8'))}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_msg = {
|
||||
'payload': json.dumps(test_data)
|
||||
}
|
||||
|
||||
print('Mock Message Created:')
|
||||
print(f' Correlation ID: {test_data["correlation_id"]}')
|
||||
print(f' Payloads: {len(test_data["payloads"])}')
|
||||
print(f' Payload types: {", ".join(p["payload_type"] for p in test_data["payloads"])}\n')
|
||||
|
||||
try:
|
||||
# Receive and process the message
|
||||
print('Receiving and processing message...')
|
||||
env = await smartreceive(
|
||||
mock_msg,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
|
||||
print('\n=== Received Envelope ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate received data
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Expected data
|
||||
expected_data = [
|
||||
('simple_dict', simple_dict, 'dictionary'),
|
||||
('nested_dict', nested_dict, 'dictionary'),
|
||||
('array_dict', array_dict, 'dictionary'),
|
||||
('mixed_dict', mixed_dict, 'dictionary')
|
||||
]
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
expected = expected_data[i]
|
||||
|
||||
if payload[0] != expected[0]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected[0]}', got '{payload[0]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload[2] != expected[2]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected[2]}', got '{payload[2]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
data_match = json.dumps(payload[1], sort_keys=True) == json.dumps(expected[1], sort_keys=True)
|
||||
if not data_match:
|
||||
print(f'❌ Payload {i + 1}: Data mismatch')
|
||||
print(f' Expected: {json.dumps(expected[1], sort_keys=True)}')
|
||||
print(f' Got: {json.dumps(payload[1], sort_keys=True)}')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data correctly deserialized')
|
||||
|
||||
# Test round-trip with receive
|
||||
print('\n=== Round-trip Test ===')
|
||||
round_trip_data = {
|
||||
'correlation_id': 'roundtrip-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'timestamp': asyncio.get_event_loop().time().isoformat(),
|
||||
'send_to': '/test/dictionary',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'py-test',
|
||||
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'receiver_name': 'py-receiver',
|
||||
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': 'payload-rt',
|
||||
'dataname': 'roundtrip',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8'))}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_rt_msg = {'payload': json.dumps(round_trip_data)}
|
||||
rt_env = await smartreceive(mock_rt_msg)
|
||||
|
||||
if rt_env['payloads'][0][0] == 'roundtrip' and rt_env['payloads'][0][2] == 'dictionary':
|
||||
print('✅ Round-trip test successful')
|
||||
else:
|
||||
print('❌ Round-trip test failed')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
172
test/test_py_dictionary_sender.py
Normal file
172
test/test_py_dictionary_sender.py
Normal file
@@ -0,0 +1,172 @@
|
||||
"""
|
||||
Python Dictionary Sender Test
|
||||
Tests the smartsend function with dictionary payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_SUBJECT = '/test/dictionary'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Dictionary Sender Test ===\n')
|
||||
|
||||
correlation_id = 'py-dict-test-' + str(asyncio.get_event_loop().time() * 1000000)
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}\n')
|
||||
|
||||
# Test data - various dictionary structures
|
||||
test_data = [
|
||||
('simple_dict', {'key1': 'value1', 'key2': 'value2'}, 'dictionary'),
|
||||
('nested_dict', {'outer': {'inner': 'value', 'number': 42}}, 'dictionary'),
|
||||
('array_dict', {'items': [1, 2, 3, 'four', 'five']}, 'dictionary'),
|
||||
('mixed_dict', {'string': 'text', 'number': 123, 'boolean': True, 'null_val': None}, 'dictionary')
|
||||
]
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending dictionary payloads...')
|
||||
env, env_json_str = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='py-dict-test',
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Test each payload
|
||||
expected_datanames = ['simple_dict', 'nested_dict', 'array_dict', 'mixed_dict']
|
||||
expected_types = ['dictionary', 'dictionary', 'dictionary', 'dictionary']
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
|
||||
if payload['dataname'] != expected_datanames[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload['payload_type'] != expected_types[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct encoding')
|
||||
|
||||
# Decode and verify the data
|
||||
import base64
|
||||
decoded_data = json.loads(base64.b64decode(payload['data']).decode('utf8'))
|
||||
original_data = test_data[i][1]
|
||||
|
||||
# Normalize for comparison (None vs null, True vs true, etc.)
|
||||
if json.dumps(decoded_data, sort_keys=True) != json.dumps(original_data, sort_keys=True):
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
print(f' Expected: {json.dumps(original_data)}')
|
||||
print(f' Got: {json.dumps(decoded_data)}')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
|
||||
print(f' Size: {payload["size"]} bytes\n')
|
||||
|
||||
# Test round-trip serialization
|
||||
print('=== Round-trip Serialization Test ===')
|
||||
round_trip_data = [
|
||||
('roundtrip', {'test': 'data', 'numbers': [1, 2, 3], 'nested': {'a': 1, 'b': 2}}, 'dictionary')
|
||||
]
|
||||
|
||||
rt_env, _ = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
round_trip_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='roundtrip-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
rt_payload = rt_env['payloads'][0]
|
||||
rt_decoded = json.loads(base64.b64decode(rt_payload['data']).decode('utf8'))
|
||||
|
||||
if json.dumps(rt_decoded, sort_keys=True) == json.dumps(round_trip_data[0][1], sort_keys=True):
|
||||
print('✅ Round-trip serialization successful')
|
||||
else:
|
||||
print('❌ Round-trip serialization failed')
|
||||
passed = False
|
||||
|
||||
# Test JSON string output
|
||||
print('\n=== JSON String Output Test ===')
|
||||
try:
|
||||
parsed = json.loads(env_json_str)
|
||||
if parsed['correlation_id'] == env['correlation_id'] and \
|
||||
len(parsed['payloads']) == len(env['payloads']):
|
||||
print('✅ JSON string is valid and matches envelope')
|
||||
else:
|
||||
print('❌ JSON string does not match envelope')
|
||||
passed = False
|
||||
except json.JSONDecodeError as e:
|
||||
print(f'❌ JSON string is invalid: {e}')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
199
test/test_py_mix_payloads_sender.py
Normal file
199
test/test_py_mix_payloads_sender.py
Normal file
@@ -0,0 +1,199 @@
|
||||
"""
|
||||
Python Mix Payloads Sender Test
|
||||
Tests the smartsend function with mixed payload types
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
import base64
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_SUBJECT = '/test/mix'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Mix Payloads Sender Test ===\n')
|
||||
|
||||
correlation_id = 'py-mix-test-' + str(asyncio.get_event_loop().time() * 1000000)
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}\n')
|
||||
|
||||
# Test data - mixed payload types
|
||||
text_data = 'Hello, NATSBridge!'
|
||||
dict_data = {'key1': 'value1', 'key2': 42, 'nested': {'a': 1, 'b': 2}}
|
||||
image_data = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
|
||||
|
||||
# Table data
|
||||
try:
|
||||
import pandas as pd
|
||||
table_data = pd.DataFrame({
|
||||
'id': [1, 2, 3],
|
||||
'name': ['Alice', 'Bob', 'Charlie'],
|
||||
'age': [30, 25, 35]
|
||||
})
|
||||
table_available = True
|
||||
except ImportError:
|
||||
table_available = False
|
||||
table_data = None
|
||||
|
||||
test_data = [
|
||||
('message', text_data, 'text'),
|
||||
('config', dict_data, 'dictionary'),
|
||||
('image', image_data, 'image')
|
||||
]
|
||||
|
||||
if table_available:
|
||||
test_data.append(('users', table_data, 'table'))
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending mixed payloads...')
|
||||
env, env_json_str = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='py-mix-test',
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
expected_count = 4 if table_available else 3
|
||||
if len(env['payloads']) != expected_count:
|
||||
print(f'❌ Expected {expected_count} payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Test each payload
|
||||
expected_datanames = ['message', 'config', 'image']
|
||||
expected_types = ['text', 'dictionary', 'image']
|
||||
expected_data = [text_data, dict_data, image_data]
|
||||
|
||||
if table_available:
|
||||
expected_datanames.append('users')
|
||||
expected_types.append('table')
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
|
||||
if payload['dataname'] != expected_datanames[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload['payload_type'] != expected_types[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct encoding')
|
||||
|
||||
# Verify data integrity based on type
|
||||
decoded_data = base64.b64decode(payload['data'])
|
||||
|
||||
if expected_types[i] == 'text':
|
||||
decoded_text = decoded_data.decode('utf8')
|
||||
if decoded_text != expected_data[i]:
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
elif expected_types[i] == 'dictionary':
|
||||
import json
|
||||
decoded_dict = json.loads(decoded_data.decode('utf8'))
|
||||
if json.dumps(decoded_dict, sort_keys=True) != json.dumps(expected_data[i], sort_keys=True):
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
elif expected_types[i] == 'image':
|
||||
if decoded_data != expected_data[i]:
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
elif expected_types[i] == 'table':
|
||||
if len(decoded_data) > 0:
|
||||
print(f'✅ Payload {i + 1}: Arrow IPC data present ({len(decoded_data)} bytes)')
|
||||
else:
|
||||
print(f'❌ Payload {i + 1}: Arrow IPC data is empty')
|
||||
passed = False
|
||||
|
||||
print(f' Size: {payload["size"]} bytes\n')
|
||||
|
||||
# Test with chat-like payload (text + image + audio)
|
||||
print('=== Chat-like Payload Test ===')
|
||||
chat_data = [
|
||||
('text', 'Hello!', 'text'),
|
||||
('image', bytes([0xFF, 0xD8, 0xFF, 0xE0]), 'image'),
|
||||
('audio', bytes([0x46, 0x4C, 0x41, 0x43]), 'audio')
|
||||
]
|
||||
|
||||
chat_env, _ = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
chat_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='chat-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if len(chat_env['payloads']) == 3:
|
||||
print('✅ Chat-like payloads handled correctly')
|
||||
else:
|
||||
print('❌ Chat-like payloads handling failed')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
167
test/test_py_table_sender.py
Normal file
167
test/test_py_table_sender.py
Normal file
@@ -0,0 +1,167 @@
|
||||
"""
|
||||
Python Table Sender Test
|
||||
Tests the smartsend function with table (Arrow IPC) payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_SUBJECT = '/test/table'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Table Sender Test ===\n')
|
||||
|
||||
correlation_id = 'py-table-test-' + str(asyncio.get_event_loop().time() * 1000000)
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}\n')
|
||||
|
||||
# Test data - pandas DataFrame
|
||||
try:
|
||||
import pandas as pd
|
||||
table_data = pd.DataFrame({
|
||||
'id': [1, 2, 3, 4, 5],
|
||||
'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
|
||||
'age': [30, 25, 35, 28, 32],
|
||||
'active': [True, False, True, True, False]
|
||||
})
|
||||
table_available = True
|
||||
except ImportError:
|
||||
print('❌ pandas not available - skipping table tests')
|
||||
sys.exit(0)
|
||||
|
||||
test_data = [
|
||||
('users_table', table_data, 'table')
|
||||
]
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending table payload...')
|
||||
env, env_json_str = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='py-table-test',
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if len(env['payloads']) != 1:
|
||||
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
payload = env['payloads'][0]
|
||||
if payload['dataname'] != 'users_table':
|
||||
print(f"❌ Expected dataname 'users_table', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct dataname')
|
||||
|
||||
if payload['payload_type'] != 'table':
|
||||
print(f"❌ Expected payload_type 'table', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct payload_type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct encoding')
|
||||
|
||||
print(f'\nPayload size: {payload["size"]} bytes')
|
||||
|
||||
# Test with larger table
|
||||
print('\n=== Larger Table Test ===')
|
||||
large_table_data = pd.DataFrame({
|
||||
'id': range(100),
|
||||
'name': [f'User{i}' for i in range(100)],
|
||||
'age': [20 + (i % 50) for i in range(100)],
|
||||
'active': [i % 2 == 0 for i in range(100)]
|
||||
})
|
||||
|
||||
large_test_data = [
|
||||
('large_table', large_table_data, 'table')
|
||||
]
|
||||
|
||||
large_env, _ = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
large_test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='large-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if len(large_env['payloads']) == 1:
|
||||
print('✅ Large table handled correctly')
|
||||
print(f' Size: {large_env["payloads"][0]["size"]} bytes')
|
||||
else:
|
||||
print('❌ Large table handling failed')
|
||||
passed = False
|
||||
|
||||
# Test JSON string output
|
||||
print('\n=== JSON String Output Test ===')
|
||||
import json
|
||||
try:
|
||||
parsed = json.loads(env_json_str)
|
||||
if parsed['correlation_id'] == env['correlation_id'] and \
|
||||
len(parsed['payloads']) == len(env['payloads']):
|
||||
print('✅ JSON string is valid and matches envelope')
|
||||
else:
|
||||
print('❌ JSON string does not match envelope')
|
||||
passed = False
|
||||
except json.JSONDecodeError as e:
|
||||
print(f'❌ JSON string is invalid: {e}')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
205
test/test_py_text_receiver.py
Normal file
205
test/test_py_text_receiver.py
Normal file
@@ -0,0 +1,205 @@
|
||||
"""
|
||||
Python Text Receiver Test
|
||||
Tests the smartreceive function with text payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Text Receiver Test ===\n')
|
||||
|
||||
# Create a mock NATS message with text payload
|
||||
test_text = 'Hello, NATSBridge! This is a test message.'
|
||||
import base64
|
||||
|
||||
test_data = {
|
||||
'correlation_id': 'py-receiver-test-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'timestamp': asyncio.get_event_loop().time().isoformat(),
|
||||
'send_to': '/test/text',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'py-text-test',
|
||||
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'receiver_name': 'py-receiver',
|
||||
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': 'payload-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'dataname': 'message',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(test_text.encode('utf8')),
|
||||
'data': base64.b64encode(test_text.encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(test_text.encode('utf8'))}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_msg = {
|
||||
'payload': json.dumps(test_data)
|
||||
}
|
||||
|
||||
print('Mock Message Created:')
|
||||
print(f' Correlation ID: {test_data["correlation_id"]}')
|
||||
print(f' Payloads: {len(test_data["payloads"])}')
|
||||
print(f' Payload dataname: {test_data["payloads"][0]["dataname"]}')
|
||||
print(f' Payload type: {test_data["payloads"][0]["payload_type"]}')
|
||||
print(f' Transport: {test_data["payloads"][0]["transport"]}\n')
|
||||
|
||||
try:
|
||||
# Receive and process the message
|
||||
print('Receiving and processing message...')
|
||||
env = await smartreceive(
|
||||
mock_msg,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
|
||||
print('\n=== Received Envelope ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate received data
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if len(env['payloads']) != 1:
|
||||
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
payload = env['payloads'][0]
|
||||
if payload[0] != 'message':
|
||||
print(f"❌ Expected dataname 'message', got '{payload[0]}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct dataname')
|
||||
|
||||
if payload[2] != 'text':
|
||||
print(f"❌ Expected type 'text', got '{payload[2]}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct type')
|
||||
|
||||
if payload[1] != test_text:
|
||||
print('❌ Data mismatch')
|
||||
print(f' Expected: {test_text}')
|
||||
print(f' Got: {payload[1]}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Data correctly deserialized')
|
||||
|
||||
# Test with multiple text payloads
|
||||
print('\n=== Multiple Text Payloads Test ===')
|
||||
multi_test_data = {
|
||||
'correlation_id': 'multi-receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'timestamp': asyncio.get_event_loop().time().isoformat(),
|
||||
'send_to': '/test/text',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'py-text-test',
|
||||
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'receiver_name': 'py-receiver',
|
||||
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': 'payload-1',
|
||||
'dataname': 'msg1',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': 16,
|
||||
'data': base64.b64encode(b'First message').decode('ascii'),
|
||||
'metadata': {'payload_bytes': 16}
|
||||
},
|
||||
{
|
||||
'id': 'payload-2',
|
||||
'dataname': 'msg2',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': 16,
|
||||
'data': base64.b64encode(b'Second message').decode('ascii'),
|
||||
'metadata': {'payload_bytes': 16}
|
||||
},
|
||||
{
|
||||
'id': 'payload-3',
|
||||
'dataname': 'msg3',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': 16,
|
||||
'data': base64.b64encode(b'Third message').decode('ascii'),
|
||||
'metadata': {'payload_bytes': 16}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_multi_msg = {'payload': json.dumps(multi_test_data)}
|
||||
multi_env = await smartreceive(mock_multi_msg)
|
||||
|
||||
if len(multi_env['payloads']) == 3:
|
||||
print('✅ Multiple payloads handled correctly')
|
||||
|
||||
# Verify each payload
|
||||
expected_messages = ['First message', 'Second message', 'Third message']
|
||||
for i in range(3):
|
||||
if multi_env['payloads'][i][1] == expected_messages[i]:
|
||||
print(f'✅ Payload {i + 1} correctly deserialized')
|
||||
else:
|
||||
print(f'❌ Payload {i + 1} mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f"❌ Expected 3 payloads, got {len(multi_env['payloads'])}")
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
164
test/test_py_text_sender.py
Normal file
164
test/test_py_text_sender.py
Normal file
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
Python Text Sender Test
|
||||
Tests the smartsend function with text payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natsbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_SUBJECT = '/test/text'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Text Sender Test ===\n')
|
||||
|
||||
correlation_id = 'py-text-test-' + str(asyncio.get_event_loop().time() * 1000000)
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}\n')
|
||||
|
||||
# Test data
|
||||
text_data = 'Hello, NATSBridge! This is a test message.'
|
||||
test_data = [
|
||||
('message', text_data, 'text')
|
||||
]
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending text payload...')
|
||||
env, env_json_str = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='py-text-test',
|
||||
is_publish=False # Don't actually publish for this test
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if not env.get('msg_id'):
|
||||
print('❌ msg_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ msg_id present')
|
||||
|
||||
if not env.get('timestamp'):
|
||||
print('❌ timestamp is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ timestamp present')
|
||||
|
||||
if len(env['payloads']) != 1:
|
||||
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
payload = env['payloads'][0]
|
||||
if payload['dataname'] != 'message':
|
||||
print(f"❌ Expected dataname 'message', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct dataname')
|
||||
|
||||
if payload['payload_type'] != 'text':
|
||||
print(f"❌ Expected payload_type 'text', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct payload_type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct encoding')
|
||||
|
||||
# Decode and verify the data
|
||||
import base64
|
||||
decoded_data = base64.b64decode(payload['data']).decode('utf8')
|
||||
if decoded_data != text_data:
|
||||
print('❌ Decoded data mismatch')
|
||||
print(f' Expected: {text_data}')
|
||||
print(f' Got: {decoded_data}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Data integrity verified')
|
||||
|
||||
print(f"\nPayload size: {payload['size']} bytes")
|
||||
print(f'Base64 data length: {len(payload["data"])} chars')
|
||||
|
||||
# Test with multiple text payloads
|
||||
print('\n=== Multiple Text Payloads Test ===')
|
||||
multi_test_data = [
|
||||
('msg1', 'First message', 'text'),
|
||||
('msg2', 'Second message', 'text'),
|
||||
('msg3', 'Third message', 'text')
|
||||
]
|
||||
|
||||
multi_env, _ = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
multi_test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='multi-test-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if len(multi_env['payloads']) == 3:
|
||||
print('✅ Multiple payloads handled correctly')
|
||||
else:
|
||||
print(f"❌ Expected 3 payloads, got {len(multi_env['payloads'])}")
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
Binary file not shown.
Binary file not shown.
Reference in New Issue
Block a user