Compare commits
65 Commits
v0.4.2
...
1b86a9252d
| Author | SHA1 | Date | |
|---|---|---|---|
| 1b86a9252d | |||
| e9fd148235 | |||
| 34ea1ed8ec | |||
| aa92fb6d0d | |||
| fbbea7b42b | |||
| b2859710cd | |||
| bc0ce7159c | |||
| 4614f99358 | |||
| 1ecc55f8aa | |||
| ae0f24ccb2 | |||
| 060c68cd05 | |||
| e85eba4cea | |||
| 206467e1fa | |||
| a98394b9b9 | |||
| c448811aa9 | |||
| c3225a90c7 | |||
| 89acf780bf | |||
| e5f4793370 | |||
| 95fe697501 | |||
| ee2d2c7238 | |||
| 1dfa277279 | |||
| 78a8952383 | |||
| fcc50847e4 | |||
| f8d93991f5 | |||
| bee9f783d9 | |||
| 3e1c8d563e | |||
| 1299febcdc | |||
| be94c62760 | |||
| 6a862ef243 | |||
| ae2de5fc62 | |||
| df0bbc7327 | |||
| d94761c866 | |||
| f8235e1a59 | |||
| 647cadf497 | |||
| 8c793a81b6 | |||
| 6a42ba7e43 | |||
| 14b3790251 | |||
| 61d81bed62 | |||
| 1a10bc1a5f | |||
| 7f68d08134 | |||
| ab20cd896f | |||
| 5a9e93d6e7 | |||
| b51641dc7e | |||
| 45f1257896 | |||
| 3e2b8b1e3a | |||
| 90d81617ef | |||
| 64c62e616b | |||
| 2c340e37c7 | |||
| 7853e94d2e | |||
| 99bf57b154 | |||
| 0fa6eaf95b | |||
| 76f42be740 | |||
| d99dc41be9 | |||
| 263508b8f7 | |||
| 0c2cca30ed | |||
| 46fdf668c6 | |||
| f8a92a45a0 | |||
| cec70e6036 | |||
| f9e08ba628 | |||
| c12a078149 | |||
| dedd803dc3 | |||
| e8e927a491 | |||
| d950bbac23 | |||
| fc8da2ebf5 | |||
| f6e50c405f |
@@ -13,3 +13,53 @@ Role: Principal Systems Architect & Lead Software Engineer.Objective: Implement
|
||||
|
||||
|
||||
Create a walkthrough for Julia service-A service sending a mix-content chat message to Julia service-B. the chat message must includes
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
I updated the following:
|
||||
- NATSBridge.jl. Essentially I add NATS_connection keyword and new publish_message function to support the keyword.
|
||||
|
||||
Use them and ONLY them as ground truth.
|
||||
|
||||
Then update the following files accordingly:
|
||||
- architecture.md
|
||||
- implementation.md
|
||||
|
||||
All API should be semantically consistent and naming should be consistent across the board.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Task: Update NATSBridge.js to reflect recent changes in NATSBridge.jl and docs
|
||||
|
||||
Context: NATSBridge.jl and docs has been updated.
|
||||
|
||||
Requirements:
|
||||
|
||||
Source of Truth: Treat the updated NATSBridge.jl and docs as the definitive source.
|
||||
API Consistency: Ensure the Main Package API (e.g., smartsend(), publish_message()) uses consistent naming across all three supported languages.
|
||||
Ecosystem Variance: Low-level native functions (e.g., NATS.connect(), JSON.read()) should follow the conventions of the specific language ecosystem and do not require cross-language consistency.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
I'm expanding this Julia package (NATSBridge) into a cross-platform project by adding a JavaScript and Python/MicroPython implementation. To ensure accuracy, the Julia src directory will serve as the ground truth, as the documentation may be outdated.
|
||||
|
||||
My goal is to maintain interface parity at the high-level API for a consistent user experience, while ensuring the low-level implementation adheres strictly to the idiomatic conventions of each respective language (e.g., multiple dispatch in Julia vs. asynchronous, prototype, or class-based patterns in JS and Python/MicroPython)
|
||||
|
||||
Now, help me do the following:
|
||||
1) check architecture.md for any mistake.
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
name = "NATSBridge"
|
||||
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
|
||||
version = "0.4.2"
|
||||
version = "0.4.5"
|
||||
authors = ["narawat <narawat@gmail.com>"]
|
||||
|
||||
[deps]
|
||||
|
||||
939
README.md
Normal file
939
README.md
Normal file
@@ -0,0 +1,939 @@
|
||||
# NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||
|
||||
A high-performance, bi-directional data bridge for **Julia, JavaScript, Python, and MicroPython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://nats.io)
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Cross-Platform Support](#cross-platform-support)
|
||||
- [Features](#features)
|
||||
- [Architecture](#architecture)
|
||||
- [Installation](#installation)
|
||||
- [Quick Start](#quick-start)
|
||||
- [API Reference](#api-reference)
|
||||
- [Payload Types](#payload-types)
|
||||
- [Transport Strategies](#transport-strategies)
|
||||
- [Cross-Platform Examples](#cross-platform-examples)
|
||||
- [Testing](#testing)
|
||||
- [License](#license)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
NATSBridge enables seamless communication across multiple platforms through NATS, with intelligent transport selection based on payload size:
|
||||
|
||||
| Transport | Payload Size | Method |
|
||||
|-----------|--------------|--------|
|
||||
| **Direct** | < 1MB | Sent directly via NATS (Base64 encoded) |
|
||||
| **Link** | >= 1MB | Uploaded to HTTP file server, URL sent via NATS |
|
||||
|
||||
### Use Cases
|
||||
|
||||
- **Chat Applications**: Text, images, audio, video in a single message
|
||||
- **File Transfer**: Efficient transfer of large files using claim-check pattern
|
||||
- **IoT/Embedded**: Sensor data, telemetry, and analytics pipelines (MicroPython)
|
||||
- **Cross-Platform Communication**: Interoperability between Julia, JavaScript, Python, and MicroPython systems
|
||||
|
||||
---
|
||||
|
||||
## Cross-Platform Support
|
||||
|
||||
| Platform | Implementation | Features |
|
||||
|----------|----------------|----------|
|
||||
| **Julia** | [`src/NATSBridge.jl`](src/NATSBridge.jl) | Full feature set, Arrow IPC, multiple dispatch |
|
||||
| **JavaScript** | [`src/natbridge.js`](src/natbridge.js) | Node.js & browser, async/await |
|
||||
| **Python** | [`src/natbridge.py`](src/natbridge.py) | Desktop Python, asyncio, type hints |
|
||||
| **MicroPython** | [`src/natbridge_mpy.py`](src/natbridge_mpy.py) | Memory-constrained, synchronous API |
|
||||
|
||||
### Platform Comparison
|
||||
|
||||
| Feature | Julia | JavaScript | Python | MicroPython |
|
||||
|---------|-------|------------|--------|-------------|
|
||||
| Multiple Dispatch | ✅ Native | ❌ | ❌ | ❌ |
|
||||
| Async/Await | ❌ | ✅ Native | ✅ Native | ⚠️ (uasyncio) |
|
||||
| Type Safety | ✅ Strong | ⚠️ (TypeScript) | ✅ (Type hints) | ❌ |
|
||||
| Memory Management | ✅ GC | ✅ GC | ✅ GC | ⚠️ (Manual) |
|
||||
| Arrow IPC | ✅ Native | ✅ | ✅ | ❌ |
|
||||
| Direct Transport | ✅ | ✅ | ✅ | ✅ |
|
||||
| Link Transport | ✅ | ✅ | ✅ | ⚠️ (Limited) |
|
||||
| Handler Functions | ✅ | ✅ | ✅ | ✅ |
|
||||
| Cross-Platform API | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
- ✅ **Cross-platform messaging** for Julia, JavaScript, Python, and MicroPython applications
|
||||
- ✅ **Bi-directional messaging** with request-reply patterns
|
||||
- ✅ **Multi-payload support** - send multiple payloads with different types in one message
|
||||
- ✅ **Automatic transport selection** - direct vs link based on payload size
|
||||
- ✅ **Claim-Check pattern** for payloads > 1MB
|
||||
- ✅ **Apache Arrow IPC** support for tabular data (zero-copy reading)
|
||||
- ✅ **Exponential backoff** for reliable file server downloads
|
||||
- ✅ **Correlation ID tracking** for message tracing
|
||||
- ✅ **Reply-to support** for request-response patterns
|
||||
- ✅ **Handler function abstraction** - pluggable file server implementations (Plik, AWS S3, custom)
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### System Components
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Sender["Application (Sender)"]
|
||||
SenderApp[App Code]
|
||||
NATSBridge_Send[NATSBridge]
|
||||
NATS_Client[<b>NATS.jl</b>]
|
||||
end
|
||||
|
||||
subgraph Receiver["Application (Receiver)"]
|
||||
ReceiverApp[App Code]
|
||||
NATSBridge_Recv[NATSBridge]
|
||||
NATS_Client_Recv[<b>NATS.jl</b>]
|
||||
end
|
||||
|
||||
subgraph Infrastructure["Infrastructure"]
|
||||
NATS[<b>NATS Server</b><br/>Message Broker]
|
||||
FileServer[<b>HTTP File Server</b><br/>Upload/Download]
|
||||
end
|
||||
|
||||
SenderApp --> NATSBridge_Send
|
||||
NATSBridge_Send --> NATS_Client
|
||||
NATS_Client --> NATS
|
||||
|
||||
NATS --> NATS_Client_Recv
|
||||
NATS_Client_Recv --> NATSBridge_Recv
|
||||
NATSBridge_Recv --> ReceiverApp
|
||||
|
||||
NATSBridge_Send -.->|HTTP POST upload| FileServer
|
||||
FileServer -.->|HTTP GET download| NATSBridge_Recv
|
||||
|
||||
style SenderApp fill:#e8f5e9
|
||||
style ReceiverApp fill:#e8f5e9
|
||||
style NATS fill:#fff3e0
|
||||
style FileServer fill:#f3e5f5
|
||||
```
|
||||
|
||||
### Message Flow
|
||||
|
||||
1. **Sender** creates a message envelope with payloads using `smartsend()`
|
||||
2. **NATSBridge** serializes and encodes each payload based on type
|
||||
3. **Transport Decision**:
|
||||
- **Direct** (< 1MB): Payload encoded as Base64, published to NATS
|
||||
- **Link** (≥ 1MB): Payload uploaded to HTTP file server, URL published to NATS
|
||||
4. **NATS** routes message envelope to subscribers
|
||||
5. **Receiver** receives message via NATS subscription callback
|
||||
6. **NATSBridge** processes envelope:
|
||||
- Decodes Base64 payloads from NATS message
|
||||
- Fetches URLs from file server with exponential backoff
|
||||
7. **Receiver** deserializes payloads based on their type
|
||||
|
||||
### File Server Handler Abstraction
|
||||
|
||||
The system uses handler functions to abstract file server operations:
|
||||
|
||||
| Handler | Purpose |
|
||||
|---------|---------|
|
||||
| `plik_oneshot_upload()` / `plikOneshotUpload()` | Uploads payload bytes to file server, returns URL |
|
||||
| `_fetch_with_backoff()` / `fetchWithBackoff()` | Downloads data from URL with exponential backoff retry |
|
||||
|
||||
This abstraction allows support for different file server implementations (Plik, AWS S3, custom HTTP server).
|
||||
|
||||
### Message Envelope Schema
|
||||
|
||||
All platforms use identical JSON schemas for message envelopes:
|
||||
|
||||
```json
|
||||
{
|
||||
"correlation_id": "uuid-v4-string",
|
||||
"msg_id": "uuid-v4-string",
|
||||
"timestamp": "2024-01-15T10:30:00Z",
|
||||
"send_to": "topic/subject",
|
||||
"msg_purpose": "ACK | NACK | updateStatus | shutdown | chat",
|
||||
"sender_name": "agent-wine-web-frontend",
|
||||
"sender_id": "uuid4",
|
||||
"receiver_name": "agent-backend",
|
||||
"receiver_id": "uuid4",
|
||||
"reply_to": "topic",
|
||||
"reply_to_msg_id": "uuid4",
|
||||
"broker_url": "nats://localhost:4222",
|
||||
"metadata": {},
|
||||
"payloads": [
|
||||
{
|
||||
"id": "uuid4",
|
||||
"dataname": "login_image",
|
||||
"payload_type": "image",
|
||||
"transport": "direct",
|
||||
"encoding": "base64",
|
||||
"size": 15433,
|
||||
"data": "base64-encoded-string"
|
||||
},
|
||||
{
|
||||
"id": "uuid4",
|
||||
"dataname": "large_table",
|
||||
"payload_type": "table",
|
||||
"transport": "link",
|
||||
"encoding": "none",
|
||||
"size": 524288,
|
||||
"data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/data.arrow"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **NATS Server** (v2.10+ recommended)
|
||||
- **HTTP File Server** (optional, for payloads > 1MB)
|
||||
|
||||
### Platform-Specific Dependencies
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using Pkg
|
||||
Pkg.add("NATS")
|
||||
Pkg.add("Arrow")
|
||||
Pkg.add("JSON3")
|
||||
Pkg.add("HTTP")
|
||||
Pkg.add("UUIDs")
|
||||
Pkg.add("Dates")
|
||||
```
|
||||
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```bash
|
||||
npm install nats uuid apache-arrow node-fetch
|
||||
# or
|
||||
yarn add nats uuid apache-arrow node-fetch
|
||||
```
|
||||
|
||||
#### JavaScript (Browser)
|
||||
|
||||
```bash
|
||||
npm install nats uuid apache-arrow
|
||||
# or use CDN:
|
||||
# https://unpkg.com/nats-js/dist/bundle/nats.min.js
|
||||
# https://unpkg.com/apache-arrow/arrow.min.js
|
||||
```
|
||||
|
||||
#### Python (Desktop)
|
||||
|
||||
```bash
|
||||
pip install nats-py aiohttp pyarrow pandas python-dateutil
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
MicroPython uses built-in modules:
|
||||
- `network` - NATS connection (custom implementation)
|
||||
- `time` - Timestamps
|
||||
- `uos` - File operations
|
||||
- `base64` - Base64 encoding
|
||||
- `json` - JSON parsing
|
||||
- `struct` - Binary data handling
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Step 1: Start NATS Server
|
||||
|
||||
```bash
|
||||
docker run -p 4222:4222 nats:latest
|
||||
```
|
||||
|
||||
### Step 2: Start HTTP File Server (Optional)
|
||||
|
||||
```bash
|
||||
# Create a directory for file uploads
|
||||
mkdir -p /tmp/fileserver
|
||||
|
||||
# Start HTTP file server
|
||||
python3 -m http.server 8080 --directory /tmp/fileserver
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Unified API Standard
|
||||
|
||||
All platforms use the same input/output format for payloads:
|
||||
|
||||
**Input format for smartsend:**
|
||||
```
|
||||
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||
```
|
||||
|
||||
**Output format for smartreceive:**
|
||||
```
|
||||
{
|
||||
"correlation_id": "...",
|
||||
"msg_id": "...",
|
||||
"timestamp": "...",
|
||||
"send_to": "...",
|
||||
"msg_purpose": "...",
|
||||
"sender_name": "...",
|
||||
"sender_id": "...",
|
||||
"receiver_name": "...",
|
||||
"receiver_id": "...",
|
||||
"reply_to": "...",
|
||||
"reply_to_msg_id": "...",
|
||||
"broker_url": "...",
|
||||
"metadata": {...},
|
||||
"payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||
}
|
||||
```
|
||||
|
||||
### smartsend
|
||||
|
||||
Sends data either directly via NATS or via a fileserver URL, depending on payload size.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
subject::String, # NATS subject
|
||||
data::AbstractArray{Tuple{String, Any, String}}; # List of (dataname, data, type)
|
||||
broker_url::String = "nats://localhost:4222",
|
||||
fileserver_url = "http://localhost:8080",
|
||||
fileserver_upload_handler::Function = plik_oneshot_upload,
|
||||
size_threshold::Int = 1_000_000,
|
||||
correlation_id::String = string(uuid4()),
|
||||
msg_purpose::String = "chat",
|
||||
sender_name::String = "NATSBridge",
|
||||
receiver_name::String = "",
|
||||
receiver_id::String = "",
|
||||
reply_to::String = "",
|
||||
reply_to_msg_id::String = "",
|
||||
is_publish::Bool = true,
|
||||
NATS_connection::Union{NATS.Connection, Nothing} = nothing,
|
||||
msg_id::String = string(uuid4()),
|
||||
sender_id::String = string(uuid4())
|
||||
)
|
||||
# Returns: ::Tuple{msg_envelope_v1, String}
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natbridge');
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
subject,
|
||||
data, // Array of [dataname, data, type] tuples
|
||||
{
|
||||
broker_url: 'nats://localhost:4222',
|
||||
fileserver_url: 'http://localhost:8080',
|
||||
fileserver_upload_handler: NATSBridge.plikOneshotUpload,
|
||||
size_threshold: 1_000_000,
|
||||
correlation_id: uuidv4(),
|
||||
msg_purpose: 'chat',
|
||||
sender_name: 'NATSBridge',
|
||||
receiver_name: '',
|
||||
receiver_id: '',
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
is_publish: true,
|
||||
nats_connection: null,
|
||||
msg_id: uuidv4(),
|
||||
sender_id: uuidv4()
|
||||
}
|
||||
);
|
||||
// Returns: Promise<[env, env_json_str]>
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import NATSBridge
|
||||
|
||||
env, env_json_str = await NATSBridge.smartsend(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
broker_url: str = "nats://localhost:4222",
|
||||
fileserver_url: str = "http://localhost:8080",
|
||||
fileserver_upload_handler: Callable = plik_oneshot_upload,
|
||||
size_threshold: int = 1_000_000,
|
||||
correlation_id: str = None,
|
||||
msg_purpose: str = "chat",
|
||||
sender_name: str = "NATSBridge",
|
||||
receiver_name: str = "",
|
||||
receiver_id: str = "",
|
||||
reply_to: str = "",
|
||||
reply_to_msg_id: str = "",
|
||||
is_publish: bool = True,
|
||||
nats_connection: Any = None,
|
||||
msg_id: str = None,
|
||||
sender_id: str = None
|
||||
)
|
||||
# Returns: Tuple[Dict, str]
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
from natbridge import NATSBridge
|
||||
|
||||
# Limited to direct transport (< 100KB threshold)
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
subject,
|
||||
data, # List of (dataname, data, type) tuples
|
||||
broker_url="nats://localhost:4222",
|
||||
size_threshold=100000 # Lower threshold for memory constraints
|
||||
)
|
||||
# Returns: Tuple[Dict, str]
|
||||
```
|
||||
|
||||
### smartreceive
|
||||
|
||||
Receives and processes messages from NATS, handling both direct and link transport.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
env = NATSBridge.smartreceive(
|
||||
msg::NATS.Msg;
|
||||
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||
max_retries::Int = 5,
|
||||
base_delay::Int = 100,
|
||||
max_delay::Int = 5000
|
||||
)
|
||||
# Returns: ::JSON.Object{String, Any}
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const env = await NATSBridge.smartreceive(
|
||||
msg,
|
||||
{
|
||||
fileserver_download_handler: NATSBridge.fetchWithBackoff,
|
||||
max_retries: 5,
|
||||
base_delay: 100,
|
||||
max_delay: 5000
|
||||
}
|
||||
);
|
||||
// Returns: Promise<env_object>
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
env = await NATSBridge.smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=fetch_with_backoff,
|
||||
max_retries=5,
|
||||
base_delay=100,
|
||||
max_delay=5000
|
||||
)
|
||||
# Returns: Dict with "payloads" key
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
env = NATSBridge.smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=_sync_fileserver_download,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
# Returns: Dict with "payloads" key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Payload Types
|
||||
|
||||
| Type | Julia | JavaScript | Python | MicroPython | Description |
|
||||
|------|-------|------------|--------|-------------|-------------|
|
||||
| `text` | `String` | `string` | `str` | `str` | Plain text strings |
|
||||
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable dictionaries |
|
||||
| `table` | `DataFrame`, `Arrow.Table` | `Array<Object>` | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) |
|
||||
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image data (PNG, JPG) |
|
||||
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio data (WAV, MP3) |
|
||||
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video data (MP4, AVI) |
|
||||
| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `bytearray` | Generic binary data |
|
||||
|
||||
---
|
||||
|
||||
## Transport Strategies
|
||||
|
||||
### Direct Transport (Payloads < 1MB)
|
||||
|
||||
Small payloads are sent directly via NATS with Base64 encoding.
|
||||
|
||||
#### Cross-Platform
|
||||
|
||||
```julia
|
||||
# Julia
|
||||
data = [("message", "Hello", "text")]
|
||||
smartsend("/topic", data)
|
||||
```
|
||||
|
||||
```javascript
|
||||
// JavaScript
|
||||
const data = [["message", "Hello", "text"]];
|
||||
smartsend("/topic", data);
|
||||
```
|
||||
|
||||
```python
|
||||
# Python
|
||||
data = [("message", "Hello", "text")]
|
||||
await smartsend("/topic", data)
|
||||
```
|
||||
|
||||
### Link Transport (Payloads >= 1MB)
|
||||
|
||||
Large payloads are uploaded to an HTTP file server.
|
||||
|
||||
#### Cross-Platform
|
||||
|
||||
```julia
|
||||
# Julia
|
||||
data = [("file", large_data, "binary")]
|
||||
smartsend("/topic", data; fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
```javascript
|
||||
// JavaScript
|
||||
const data = [["file", largeData, "binary"]];
|
||||
smartsend("/topic", data, { fileserver_url: 'http://localhost:8080' });
|
||||
```
|
||||
|
||||
```python
|
||||
# Python
|
||||
data = [("file", large_data, "binary")]
|
||||
await smartsend("/topic", data, fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cross-Platform Examples
|
||||
|
||||
### Example 1: Chat with Mixed Content
|
||||
|
||||
Send text, image, and large file in one message.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
data = [
|
||||
("message_text", "Hello!", "text"),
|
||||
("user_avatar", image_data, "image"),
|
||||
("large_document", large_file_data, "binary")
|
||||
]
|
||||
|
||||
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; fileserver_url="http://localhost:8080")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natbridge');
|
||||
|
||||
const data = [
|
||||
["message_text", "Hello!", "text"],
|
||||
["user_avatar", imageData, "image"],
|
||||
["large_document", largeFileData, "binary"]
|
||||
];
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ fileserver_url: 'http://localhost:8080' }
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import NATSBridge
|
||||
|
||||
data = [
|
||||
("message_text", "Hello!", "text"),
|
||||
("user_avatar", image_data, "image"),
|
||||
("large_document", large_file_data, "binary")
|
||||
]
|
||||
|
||||
env, env_json_str = await NATSBridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
fileserver_url="http://localhost:8080"
|
||||
)
|
||||
```
|
||||
|
||||
### Example 2: Dictionary Exchange
|
||||
|
||||
Send configuration data between platforms.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
config = Dict(
|
||||
"wifi_ssid" => "MyNetwork",
|
||||
"wifi_password" => "password123",
|
||||
"update_interval" => 60
|
||||
)
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env, env_json_str = NATSBridge.smartsend("/device/config", data)
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natbridge');
|
||||
|
||||
const config = {
|
||||
wifi_ssid: "MyNetwork",
|
||||
wifi_password: "password123",
|
||||
update_interval: 60
|
||||
};
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/device/config",
|
||||
[["config", config, "dictionary"]]
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import NATSBridge
|
||||
|
||||
config = {
|
||||
"wifi_ssid": "MyNetwork",
|
||||
"wifi_password": "password123",
|
||||
"update_interval": 60
|
||||
}
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env, env_json_str = await NATSBridge.smartsend("/device/config", data)
|
||||
```
|
||||
|
||||
### Example 3: Table Data (Arrow IPC)
|
||||
|
||||
Send tabular data using Apache Arrow IPC format.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
using DataFrames
|
||||
|
||||
df = DataFrame(
|
||||
id = [1, 2, 3],
|
||||
name = ["Alice", "Bob", "Charlie"],
|
||||
score = [95, 88, 92]
|
||||
)
|
||||
|
||||
data = [("students", df, "table")]
|
||||
env, env_json_str = NATSBridge.smartsend("/data/analysis", data)
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natbridge');
|
||||
|
||||
const df = [
|
||||
{ id: 1, name: "Alice", score: 95 },
|
||||
{ id: 2, name: "Bob", score: 88 },
|
||||
{ id: 3, name: "Charlie", score: 92 }
|
||||
];
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/data/analysis",
|
||||
[["students", df, "table"]]
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import NATSBridge
|
||||
import pandas as pd
|
||||
|
||||
df = pd.DataFrame({
|
||||
"id": [1, 2, 3],
|
||||
"name": ["Alice", "Bob", "Charlie"],
|
||||
"score": [95, 88, 92]
|
||||
})
|
||||
|
||||
data = [("students", df, "table")]
|
||||
env, env_json_str = await NATSBridge.smartsend("/data/analysis", data)
|
||||
```
|
||||
|
||||
### Example 4: Request-Response Pattern
|
||||
|
||||
Bi-directional communication with reply-to support.
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Requester
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
"/device/command",
|
||||
[("command", Dict("action" => "read_sensor"), "dictionary")];
|
||||
broker_url="nats://localhost:4222",
|
||||
reply_to="/device/response"
|
||||
)
|
||||
```
|
||||
|
||||
```julia
|
||||
# Responder
|
||||
using NATS, NATSBridge
|
||||
|
||||
function test_responder()
|
||||
conn = NATS.connect("nats://localhost:4222")
|
||||
NATS.subscribe(conn, "/device/command") do msg
|
||||
env = NATSBridge.smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
|
||||
|
||||
reply_to = env["reply_to"]
|
||||
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
if dataname == "command" && data["action"] == "read_sensor"
|
||||
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
|
||||
if !isempty(reply_to)
|
||||
smartsend(reply_to, [("data", response, "dictionary")])
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
sleep(120)
|
||||
NATS.drain(conn)
|
||||
end
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('natbridge');
|
||||
|
||||
// Requester
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/device/command",
|
||||
[["command", { action: "read_sensor" }, "dictionary"]],
|
||||
{ broker_url: 'nats://localhost:4222', reply_to: '/device/response' }
|
||||
);
|
||||
```
|
||||
|
||||
```javascript
|
||||
// Responder
|
||||
const nats = require('nats');
|
||||
const NATSBridge = require('natbridge');
|
||||
|
||||
async function testResponder() {
|
||||
const conn = await nats.connect('nats://localhost:4222');
|
||||
|
||||
const subscription = await conn.subscribe('/device/command');
|
||||
|
||||
for await (const msg of subscription) {
|
||||
const env = await NATSBridge.smartreceive(msg, {
|
||||
fileserver_download_handler: NATSBridge.fetchWithBackoff
|
||||
});
|
||||
|
||||
const replyTo = env.reply_to;
|
||||
|
||||
for (const [dataname, data, type] of env.payloads) {
|
||||
if (dataname === 'command' && data.action === 'read_sensor') {
|
||||
const response = { sensor_id: 'sensor-001', value: 42.5 };
|
||||
if (replyTo) {
|
||||
await NATSBridge.smartsend(
|
||||
replyTo,
|
||||
[["data", response, "dictionary"]]
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
setTimeout(() => conn.close(), 120000);
|
||||
}
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import NATSBridge
|
||||
|
||||
# Requester
|
||||
env, env_json_str = await NATSBridge.smartsend(
|
||||
"/device/command",
|
||||
[("command", {"action": "read_sensor"}, "dictionary")],
|
||||
broker_url="nats://localhost:4222",
|
||||
reply_to="/device/response"
|
||||
)
|
||||
```
|
||||
|
||||
```python
|
||||
# Responder
|
||||
from natbridge import NATSBridge
|
||||
import asyncio
|
||||
import nats
|
||||
|
||||
async def test_responder():
|
||||
nc = await nats.connect('nats://localhost:4222')
|
||||
|
||||
async def msg_handler(msg):
|
||||
env = await NATSBridge.smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=fetch_with_backoff
|
||||
)
|
||||
|
||||
reply_to = env["reply_to"]
|
||||
|
||||
for dataname, data, type_ in env["payloads"]:
|
||||
if dataname == "command" and data["action"] == "read_sensor":
|
||||
response = {"sensor_id": "sensor-001", "value": 42.5}
|
||||
if reply_to:
|
||||
await NATSBridge.smartsend(
|
||||
reply_to,
|
||||
[("data", response, "dictionary")]
|
||||
)
|
||||
|
||||
await nc.subscribe('/device/command', cb=msg_handler)
|
||||
|
||||
await asyncio.sleep(120)
|
||||
await nc.drain()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Test File Organization
|
||||
|
||||
| Platform | Sender Tests | Receiver Tests |
|
||||
|----------|--------------|----------------|
|
||||
| **Julia** | `test/test_julia_*_sender.jl` | `test/test_julia_*_receiver.jl` |
|
||||
| **JavaScript** | `test/test_js_*_sender.js` | `test/test_js_*_receiver.js` |
|
||||
| **Python** | `test/test_py_*_sender.py` | `test/test_py_*_receiver.py` |
|
||||
|
||||
### Run Tests
|
||||
|
||||
#### Julia
|
||||
|
||||
```bash
|
||||
# Text message exchange
|
||||
julia test/test_julia_text_sender.jl
|
||||
julia test/test_julia_text_receiver.jl
|
||||
|
||||
# Dictionary exchange
|
||||
julia test/test_julia_dict_sender.jl
|
||||
julia test/test_julia_dict_receiver.jl
|
||||
|
||||
# File transfer
|
||||
julia test/test_julia_file_sender.jl
|
||||
julia test/test_julia_file_receiver.jl
|
||||
|
||||
# Mixed payload types
|
||||
julia test/test_julia_mix_payloads_sender.jl
|
||||
julia test/test_julia_mix_payloads_receiver.jl
|
||||
|
||||
# Table exchange
|
||||
julia test/test_julia_table_sender.jl
|
||||
julia test/test_julia_table_receiver.jl
|
||||
```
|
||||
|
||||
#### JavaScript (Node.js)
|
||||
|
||||
```bash
|
||||
# Text message exchange
|
||||
node test/test_js_text_sender.js
|
||||
node test/test_js_text_receiver.js
|
||||
|
||||
# Dictionary exchange
|
||||
node test/test_js_dictionary_sender.js
|
||||
node test/test_js_dictionary_receiver.js
|
||||
|
||||
# Binary transfer
|
||||
node test/test_js_binary_sender.js
|
||||
node test/test_js_binary_receiver.js
|
||||
|
||||
# Table exchange
|
||||
node test/test_js_table_sender.js
|
||||
node test/test_js_table_receiver.js
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```bash
|
||||
# Text message exchange
|
||||
python3 test/test_py_text_sender.py
|
||||
python3 test/test_py_text_receiver.py
|
||||
|
||||
# Dictionary exchange
|
||||
python3 test/test_py_dictionary_sender.py
|
||||
python3 test/test_py_dictionary_receiver.py
|
||||
|
||||
# Binary transfer
|
||||
python3 test/test_py_binary_sender.py
|
||||
python3 test/test_py_binary_receiver.py
|
||||
|
||||
# Table exchange
|
||||
python3 test/test_py_table_sender.py
|
||||
python3 test/test_py_table_receiver.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
For detailed architecture and implementation information, see:
|
||||
|
||||
- [Architecture Documentation](docs/architecture.md) - Cross-platform architecture, API parity, platform-specific patterns
|
||||
- [Implementation Guide](docs/implementation.md) - Detailed implementation for each platform, handler functions, testing
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 NATSBridge Contributors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
1579
docs/architecture.md
1579
docs/architecture.md
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
38
etc.jl
38
etc.jl
@@ -0,0 +1,38 @@
|
||||
Task: Update README.md to reflect recent changes in NATSbridge package.
|
||||
|
||||
Context: the package has been updated with the NATS_connection keyword and the publish_message function.
|
||||
|
||||
Requirements:
|
||||
|
||||
Source of Truth: Treat the updated NATSbridge code as the definitive source. Update README.md to align exactly with these changes.
|
||||
API Consistency: Ensure the Main Package API (e.g., smartsend(), publish_message()) uses consistent naming across all three supported languages.
|
||||
Ecosystem Variance: Low-level native functions (e.g., NATS.connect(), JSON.read()) should follow the conventions of the specific language ecosystem and do not require cross-language consistency.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Help me expands this Julia package (NATSBridge) into a cross-platform project by adding a JavaScript and Python/MicroPython implementation. To ensure accuracy, NATSBridge.jl will serve as the ground truth, as the documentation may be outdated.
|
||||
|
||||
My goal is to maintain interface parity at the high-level API for a consistent user experience, while ensuring the low-level implementation adheres strictly to the idiomatic conventions of each respective language (e.g., multiple dispatch in Julia vs. asynchronous, prototype, or class-based patterns in JS and Python/MicroPython)
|
||||
|
||||
Now do the following:
|
||||
1) check docs to see if there is any mistake.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# NATSBridge Tutorial
|
||||
# Cross-Platform NATSBridge Tutorial
|
||||
|
||||
A step-by-step guide to get started with NATSBridge - a high-performance, bi-directional data bridge for **Julia**, **JavaScript**, and **Python/Micropython**.
|
||||
A step-by-step guide to get started with NATSBridge across **Julia**, **JavaScript**, and **Python/MicroPython**.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
@@ -10,28 +10,40 @@ A step-by-step guide to get started with NATSBridge - a high-performance, bi-dir
|
||||
4. [Quick Start](#quick-start)
|
||||
5. [Basic Examples](#basic-examples)
|
||||
6. [Advanced Usage](#advanced-usage)
|
||||
7. [Cross-Platform Communication](#cross-platform-communication)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
NATSBridge enables seamless communication between Julia, JavaScript, and Python/Micropython applications through NATS, with automatic transport selection based on payload size:
|
||||
NATSBridge enables seamless communication across platforms through NATS, with automatic transport selection based on payload size:
|
||||
|
||||
- **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded)
|
||||
- **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL
|
||||
|
||||
### Cross-Platform API Parity
|
||||
|
||||
All three platforms use the same high-level API:
|
||||
|
||||
```
|
||||
# Input format
|
||||
smartsend(subject, [(dataname, data, type), ...], options)
|
||||
|
||||
# Output format
|
||||
(env, env_json_str) = smartsend(...)
|
||||
env = smartreceive(msg, options)
|
||||
```
|
||||
|
||||
### Supported Payload Types
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `text` | Plain text strings |
|
||||
| `dictionary` | JSON-serializable dictionaries |
|
||||
| `table` | Tabular data (Arrow IPC format) |
|
||||
| `image` | Image data (PNG, JPG bytes) |
|
||||
| `audio` | Audio data (WAV, MP3 bytes) |
|
||||
| `video` | Video data (MP4, AVI bytes) |
|
||||
| `binary` | Generic binary data |
|
||||
| Type | Julia | JavaScript | Python | MicroPython |
|
||||
|------|-------|------------|--------|-------------|
|
||||
| `text` | `String` | `string` | `str` | `str` |
|
||||
| `dictionary` | `Dict` | `Object` | `dict` | `dict` |
|
||||
| `table` | `DataFrame` | `Array<Object>` | `DataFrame` | ❌ |
|
||||
| `image` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
|
||||
| `audio` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
|
||||
| `video` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
|
||||
| `binary` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
|
||||
|
||||
---
|
||||
|
||||
@@ -41,7 +53,7 @@ Before you begin, ensure you have:
|
||||
|
||||
1. **NATS Server** running (or accessible)
|
||||
2. **HTTP File Server** (optional, for large payloads > 1MB)
|
||||
3. **One of the supported platforms**: Julia, JavaScript (Node.js), or Python/Micropython
|
||||
3. **Platform-specific packages** installed
|
||||
|
||||
---
|
||||
|
||||
@@ -59,26 +71,28 @@ Pkg.add("UUIDs")
|
||||
Pkg.add("Dates")
|
||||
```
|
||||
|
||||
### JavaScript
|
||||
### JavaScript (Node.js)
|
||||
|
||||
```bash
|
||||
npm install nats.js apache-arrow uuid base64-url
|
||||
npm install nats uuid apache-arrow node-fetch
|
||||
```
|
||||
|
||||
### Python/Micropython
|
||||
### JavaScript (Browser)
|
||||
|
||||
1. Copy `src/nats_bridge.py` to your device
|
||||
2. Install dependencies:
|
||||
```html
|
||||
<script src="https://unpkg.com/nats-js/dist/bundle/nats.min.js"></script>
|
||||
<script src="https://unpkg.com/apache-arrow/arrow.min.js"></script>
|
||||
```
|
||||
|
||||
### Python (Desktop)
|
||||
|
||||
**For Python (desktop):**
|
||||
```bash
|
||||
pip install nats-py
|
||||
pip install nats-py aiohttp pyarrow pandas
|
||||
```
|
||||
|
||||
**For Micropython:**
|
||||
- `urequests` for HTTP requests
|
||||
- `base64` for base64 encoding (built-in)
|
||||
- `json` for JSON handling (built-in)
|
||||
### MicroPython
|
||||
|
||||
Uses built-in modules: `network`, `socket`, `time`, `json`, `base64`
|
||||
|
||||
---
|
||||
|
||||
@@ -93,39 +107,12 @@ docker run -p 4222:4222 nats:latest
|
||||
### Step 2: Start HTTP File Server (Optional)
|
||||
|
||||
```bash
|
||||
# Create a directory for file uploads
|
||||
mkdir -p /tmp/fileserver
|
||||
|
||||
# Use Python's built-in server
|
||||
python3 -m http.server 8080 --directory /tmp/fileserver
|
||||
```
|
||||
|
||||
### Step 3: Send Your First Message
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
# Send a text message
|
||||
data = [("message", "Hello World", "text")]
|
||||
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
||||
print("Message sent!")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const { smartsend } = require('./src/NATSBridge');
|
||||
|
||||
// Send a text message
|
||||
await smartsend("/chat/room1", [
|
||||
{ dataname: "message", data: "Hello World", type: "text" }
|
||||
], { natsUrl: "nats://localhost:4222" });
|
||||
|
||||
console.log("Message sent!");
|
||||
```
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
@@ -133,86 +120,137 @@ using NATSBridge
|
||||
|
||||
# Send a text message
|
||||
data = [("message", "Hello World", "text")]
|
||||
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
||||
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222")
|
||||
# env: msg_envelope_v1 struct with all metadata and payloads
|
||||
# env_json_str: JSON string representation of the envelope for publishing
|
||||
println("Message sent!")
|
||||
|
||||
# Or use is_publish=false to get envelope and JSON without publishing
|
||||
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222", is_publish=false)
|
||||
# env: msg_envelope_v1 struct
|
||||
# env_json_str: JSON string for publishing to NATS
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natbridge.js');
|
||||
|
||||
// Send a text message
|
||||
const data = [["message", "Hello World", "text"]];
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222" }
|
||||
);
|
||||
// env: Object with all metadata and payloads
|
||||
// env_json_str: JSON string for publishing
|
||||
console.log("Message sent!");
|
||||
|
||||
// Or use is_publish=false
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222", is_publish: false }
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import smartsend
|
||||
|
||||
# Send a text message
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = await smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
broker_url="nats://localhost:4222"
|
||||
)
|
||||
# env: Dict with all metadata and payloads
|
||||
# env_json_str: JSON string for publishing
|
||||
print("Message sent!")
|
||||
|
||||
# Or use is_publish=False
|
||||
env, env_json_str = await smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
broker_url="nats://localhost:4222",
|
||||
is_publish=False
|
||||
)
|
||||
# env: Dict with all metadata and payloads
|
||||
# env_json_str: JSON string for publishing to NATS
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
from natbridge_mpy import NATSBridge
|
||||
|
||||
bridge = NATSBridge()
|
||||
|
||||
# Send a text message (limited to small payloads)
|
||||
data = [("message", "Hello World", "text")]
|
||||
env, env_json_str = bridge.smartsend(
|
||||
"/chat/room1",
|
||||
data,
|
||||
size_threshold=100000 # Lower threshold for MicroPython
|
||||
)
|
||||
print("Message sent!")
|
||||
```
|
||||
|
||||
### Step 4: Receive Messages
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartreceive
|
||||
|
||||
# Receive and process message
|
||||
envelope = smartreceive(msg)
|
||||
for dataname, data, type in envelope["payloads"]:
|
||||
print(f"Received {dataname}: {data}")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const { smartreceive } = require('./src/NATSBridge');
|
||||
|
||||
// Receive and process message
|
||||
const envelope = await smartreceive(msg);
|
||||
for (const payload of envelope.payloads) {
|
||||
console.log(`Received ${payload.dataname}: ${payload.data}`);
|
||||
}
|
||||
```
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Receive and process message
|
||||
envelope = smartreceive(msg, fileserverDownloadHandler)
|
||||
for (dataname, data, type) in envelope["payloads"]
|
||||
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff)
|
||||
# Returns: ::JSON.Object{String, Any} with "payloads" field containing Vector{Tuple{String, Any, String}}
|
||||
# Access payloads: for (dataname, data, type) in env["payloads"]
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
println("Received $dataname: $data")
|
||||
end
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natbridge.js');
|
||||
|
||||
// Receive and process message
|
||||
const env = await NATSBridge.smartreceive(msg, {
|
||||
fileserver_download_handler: NATSBridge.fetchWithBackoff
|
||||
});
|
||||
// env.payloads = [[dataname, data, type], ...]
|
||||
for (const [dataname, data, type] of env.payloads) {
|
||||
console.log(`Received ${dataname}:`, data);
|
||||
}
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import smartreceive, fetch_with_backoff
|
||||
|
||||
# Receive and process message
|
||||
env = await smartreceive(
|
||||
msg,
|
||||
fileserver_download_handler=fetch_with_backoff
|
||||
)
|
||||
# env["payloads"] = [(dataname, data, type), ...]
|
||||
for dataname, data, type_ in env["payloads"]:
|
||||
print(f"Received {dataname}: {data}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Basic Examples
|
||||
|
||||
### Example 1: Sending a Dictionary
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
# Create configuration dictionary
|
||||
config = {
|
||||
"wifi_ssid": "MyNetwork",
|
||||
"wifi_password": "password123",
|
||||
"update_interval": 60
|
||||
}
|
||||
|
||||
# Send as dictionary type
|
||||
data = [("config", config, "dictionary")]
|
||||
env = smartsend("/device/config", data, nats_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const { smartsend } = require('./src/NATSBridge');
|
||||
|
||||
const config = {
|
||||
wifi_ssid: "MyNetwork",
|
||||
wifi_password: "password123",
|
||||
update_interval: 60
|
||||
};
|
||||
|
||||
await smartsend("/device/config", [
|
||||
{ dataname: "config", data: config, type: "dictionary" }
|
||||
]);
|
||||
```
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
@@ -225,39 +263,70 @@ config = Dict(
|
||||
)
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
smartsend("/device/config", data)
|
||||
```
|
||||
|
||||
### Example 2: Sending Binary Data (Image)
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
# Read image file
|
||||
with open("image.png", "rb") as f:
|
||||
image_data = f.read()
|
||||
|
||||
# Send as binary type
|
||||
data = [("user_image", image_data, "binary")]
|
||||
env = smartsend("/chat/image", data, nats_url="nats://localhost:4222")
|
||||
env, env_json_str = smartsend("/device/config", data, broker_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const { smartsend } = require('./src/NATSBridge');
|
||||
const NATSBridge = require('./src/natbridge.js');
|
||||
|
||||
// Read image file (Node.js)
|
||||
const fs = require('fs');
|
||||
const image_data = fs.readFileSync('image.png');
|
||||
const config = {
|
||||
wifi_ssid: "MyNetwork",
|
||||
wifi_password: "password123",
|
||||
update_interval: 60
|
||||
};
|
||||
|
||||
await smartsend("/chat/image", [
|
||||
{ dataname: "user_image", data: image_data, type: "binary" }
|
||||
]);
|
||||
const data = [["config", config, "dictionary"]];
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/device/config",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222" }
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import smartsend
|
||||
|
||||
config = {
|
||||
"wifi_ssid": "MyNetwork",
|
||||
"wifi_password": "password123",
|
||||
"update_interval": 60
|
||||
}
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env, env_json_str = await smartsend(
|
||||
"/device/config",
|
||||
data,
|
||||
broker_url="nats://localhost:4222"
|
||||
)
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
from natbridge_mpy import NATSBridge
|
||||
|
||||
bridge = NATSBridge()
|
||||
|
||||
config = {
|
||||
"wifi_ssid": "MyNetwork",
|
||||
"wifi_password": "password123",
|
||||
"update_interval": 60
|
||||
}
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env, env_json_str = bridge.smartsend(
|
||||
"/device/config",
|
||||
data,
|
||||
size_threshold=100000
|
||||
)
|
||||
```
|
||||
|
||||
### Example 2: Sending Binary Data (Image)
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
@@ -267,61 +336,144 @@ using NATSBridge
|
||||
image_data = read("image.png")
|
||||
|
||||
data = [("user_image", image_data, "binary")]
|
||||
smartsend("/chat/image", data)
|
||||
env, env_json_str = smartsend("/chat/image", data, broker_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natbridge.js');
|
||||
const fs = require('fs');
|
||||
|
||||
// Read image file
|
||||
const image_data = fs.readFileSync('image.png');
|
||||
|
||||
const data = [["user_image", image_data, "binary"]];
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/image",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222" }
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import smartsend
|
||||
|
||||
# Read image file
|
||||
with open("image.png", "rb") as f:
|
||||
image_data = f.read()
|
||||
|
||||
data = [("user_image", image_data, "binary")]
|
||||
env, env_json_str = await smartsend(
|
||||
"/chat/image",
|
||||
data,
|
||||
broker_url="nats://localhost:4222"
|
||||
)
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
```python
|
||||
from natbridge_mpy import NATSBridge
|
||||
|
||||
bridge = NATSBridge()
|
||||
|
||||
# Read image file
|
||||
with open("image.png", "rb") as f:
|
||||
image_data = f.read()
|
||||
|
||||
data = [("user_image", image_data, "binary")]
|
||||
env, env_json_str = bridge.smartsend(
|
||||
"/chat/image",
|
||||
data,
|
||||
size_threshold=100000
|
||||
)
|
||||
```
|
||||
|
||||
### Example 3: Request-Response Pattern
|
||||
|
||||
#### Python/Micropython (Requester)
|
||||
#### Julia (Requester)
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Send command with reply-to
|
||||
data = [("command", {"action": "read_sensor"}, "dictionary")]
|
||||
env = smartsend(
|
||||
data = [("command", Dict("action" => "read_sensor"), "dictionary")]
|
||||
env, env_json_str = smartsend(
|
||||
"/device/command",
|
||||
data,
|
||||
nats_url="nats://localhost:4222",
|
||||
broker_url="nats://localhost:4222",
|
||||
reply_to="/device/response",
|
||||
reply_to_msg_id="cmd-001"
|
||||
)
|
||||
```
|
||||
|
||||
#### JavaScript (Responder)
|
||||
#### JavaScript (Requester)
|
||||
|
||||
```javascript
|
||||
const { smartreceive, smartsend } = require('./src/NATSBridge');
|
||||
const NATSBridge = require('./src/natbridge.js');
|
||||
|
||||
// Subscribe to command topic
|
||||
const sub = nc.subscribe("/device/command");
|
||||
|
||||
for await (const msg of sub) {
|
||||
const envelope = await smartreceive(msg);
|
||||
|
||||
// Process command
|
||||
for (const payload of envelope.payloads) {
|
||||
if (payload.dataname === "command") {
|
||||
const command = payload.data;
|
||||
|
||||
if (command.action === "read_sensor") {
|
||||
// Read sensor and send response
|
||||
const response = {
|
||||
sensor_id: "sensor-001",
|
||||
value: 42.5,
|
||||
timestamp: new Date().toISOString()
|
||||
};
|
||||
|
||||
await smartsend("/device/response", [
|
||||
{ dataname: "sensor_data", data: response, type: "dictionary" }
|
||||
], {
|
||||
reply_to: envelope.replyTo,
|
||||
reply_to_msg_id: envelope.msgId
|
||||
});
|
||||
// Send command with reply-to
|
||||
const data = [["command", { action: "read_sensor" }, "dictionary"]];
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/device/command",
|
||||
data,
|
||||
{
|
||||
broker_url: "nats://localhost:4222",
|
||||
reply_to: "/device/response",
|
||||
reply_to_msg_id: "cmd-001"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
);
|
||||
```
|
||||
|
||||
#### Python (Requester)
|
||||
|
||||
```python
|
||||
from natbridge import smartsend
|
||||
|
||||
# Send command with reply-to
|
||||
data = [("command", {"action": "read_sensor"}, "dictionary")]
|
||||
env, env_json_str = await smartsend(
|
||||
"/device/command",
|
||||
data,
|
||||
broker_url="nats://localhost:4222",
|
||||
reply_to="/device/response",
|
||||
reply_to_msg_id="cmd-001"
|
||||
)
|
||||
```
|
||||
|
||||
#### Julia (Responder)
|
||||
|
||||
```julia
|
||||
using NATSBridge, NATS
|
||||
|
||||
const SUBJECT = "/device/command"
|
||||
const NATS_URL = "nats://localhost:4222"
|
||||
|
||||
function test_responder()
|
||||
conn = NATS.connect(NATS_URL)
|
||||
NATS.subscribe(conn, SUBJECT) do msg
|
||||
env = smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
|
||||
|
||||
reply_to = env["reply_to"]
|
||||
|
||||
for (dataname, data, type) in env["payloads"]
|
||||
if dataname == "command" && data["action"] == "read_sensor"
|
||||
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
|
||||
if !isempty(reply_to)
|
||||
smartsend(reply_to, [("data", response, "dictionary")])
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
sleep(120)
|
||||
NATS.drain(conn)
|
||||
end
|
||||
|
||||
test_responder()
|
||||
```
|
||||
|
||||
---
|
||||
@@ -332,46 +484,6 @@ for await (const msg of sub) {
|
||||
|
||||
For payloads larger than 1MB, NATSBridge automatically uses the file server:
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
import os
|
||||
|
||||
# Create large data (> 1MB)
|
||||
large_data = os.urandom(2_000_000) # 2MB of random data
|
||||
|
||||
# Send with file server URL
|
||||
env = smartsend(
|
||||
"/data/large",
|
||||
[("large_file", large_data, "binary")],
|
||||
nats_url="nats://localhost:4222",
|
||||
fileserver_url="http://localhost:8080",
|
||||
size_threshold=1_000_000
|
||||
)
|
||||
|
||||
# The envelope will contain the download URL
|
||||
print(f"File uploaded to: {env.payloads[0].data}")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const { smartsend } = require('./src/NATSBridge');
|
||||
|
||||
// Create large data (> 1MB)
|
||||
const largeData = new ArrayBuffer(2_000_000);
|
||||
const view = new Uint8Array(largeData);
|
||||
view.fill(42); // Fill with some data
|
||||
|
||||
await smartsend("/data/large", [
|
||||
{ dataname: "large_file", data: largeData, type: "binary" }
|
||||
], {
|
||||
fileserverUrl: "http://localhost:8080",
|
||||
sizeThreshold: 1_000_000
|
||||
});
|
||||
```
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
@@ -380,59 +492,86 @@ using NATSBridge
|
||||
# Create large data (> 1MB)
|
||||
large_data = rand(UInt8, 2_000_000)
|
||||
|
||||
env = smartsend(
|
||||
env, env_json_str = smartsend(
|
||||
"/data/large",
|
||||
[("large_file", large_data, "binary")],
|
||||
broker_url="nats://localhost:4222",
|
||||
fileserver_url="http://localhost:8080"
|
||||
)
|
||||
|
||||
# The envelope will contain the download URL
|
||||
println("File uploaded to: $(env.payloads[1].data)")
|
||||
```
|
||||
|
||||
### Example 5: Mixed Content (Chat with Text + Image)
|
||||
|
||||
NATSBridge supports sending multiple payloads with different types in a single message:
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
# Read image file
|
||||
with open("avatar.png", "rb") as f:
|
||||
image_data = f.read()
|
||||
|
||||
# Send mixed content
|
||||
data = [
|
||||
("message_text", "Hello with image!", "text"),
|
||||
("user_avatar", image_data, "image")
|
||||
]
|
||||
|
||||
env = smartsend("/chat/mixed", data, nats_url="nats://localhost:4222")
|
||||
# Note: For link transport, data field contains the URL string
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const { smartsend } = require('./src/NATSBridge');
|
||||
const NATSBridge = require('./src/natbridge.js');
|
||||
|
||||
const fs = require('fs');
|
||||
// Create large data (> 1MB)
|
||||
const large_data = Buffer.alloc(2_000_000);
|
||||
for (let i = 0; i < large_data.length; i++) {
|
||||
large_data[i] = Math.floor(Math.random() * 256);
|
||||
}
|
||||
|
||||
await smartsend("/chat/mixed", [
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/data/large",
|
||||
[["large_file", large_data, "binary"]],
|
||||
{
|
||||
dataname: "message_text",
|
||||
data: "Hello with image!",
|
||||
type: "text"
|
||||
},
|
||||
{
|
||||
dataname: "user_avatar",
|
||||
data: fs.readFileSync("avatar.png"),
|
||||
type: "image"
|
||||
broker_url: "nats://localhost:4222",
|
||||
fileserver_url: "http://localhost:8080"
|
||||
}
|
||||
]);
|
||||
);
|
||||
|
||||
console.log("File uploaded to:", env.payloads[0].data);
|
||||
// Note: For link transport, data field contains the URL string
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import smartsend
|
||||
|
||||
# Create large data (> 1MB)
|
||||
import os
|
||||
large_data = os.urandom(2_000_000)
|
||||
|
||||
env, env_json_str = await smartsend(
|
||||
"/data/large",
|
||||
[("large_file", large_data, "binary")],
|
||||
broker_url="nats://localhost:4222",
|
||||
fileserver_url="http://localhost:8080"
|
||||
)
|
||||
|
||||
print(f"File uploaded to: {env['payloads'][0]['data']}")
|
||||
# Note: For link transport, data field contains the URL string
|
||||
```
|
||||
|
||||
#### MicroPython
|
||||
|
||||
MicroPython enforces a hard limit of 50KB per payload:
|
||||
|
||||
```python
|
||||
from natbridge_mpy import NATSBridge
|
||||
|
||||
bridge = NATSBridge()
|
||||
|
||||
# MicroPython has a hard limit of 50KB per payload
|
||||
# Use streaming or chunking for larger data
|
||||
small_data = bytes(1000) # 1KB
|
||||
|
||||
data = [("small_file", small_data, "binary")]
|
||||
env, env_json_str = bridge.smartsend(
|
||||
"/data/small",
|
||||
data,
|
||||
size_threshold=100000 # Enforced max: 50000 bytes
|
||||
)
|
||||
```
|
||||
|
||||
### Example 5: Mixed Content (Chat with Text + Image)
|
||||
|
||||
NATSBridge supports sending multiple payloads with different types in a single message:
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
@@ -445,31 +584,54 @@ data = [
|
||||
("user_avatar", image_data, "image")
|
||||
]
|
||||
|
||||
smartsend("/chat/mixed", data)
|
||||
env, env_json_str = smartsend("/chat/mixed", data, broker_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const NATSBridge = require('./src/natbridge.js');
|
||||
const fs = require('fs');
|
||||
|
||||
const image_data = fs.readFileSync('avatar.png');
|
||||
|
||||
const data = [
|
||||
["message_text", "Hello with image!", "text"],
|
||||
["user_avatar", image_data, "image"]
|
||||
];
|
||||
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/chat/mixed",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222" }
|
||||
);
|
||||
```
|
||||
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from natbridge import smartsend
|
||||
|
||||
with open("avatar.png", "rb") as f:
|
||||
image_data = f.read()
|
||||
|
||||
data = [
|
||||
("message_text", "Hello with image!", "text"),
|
||||
("user_avatar", image_data, "image")
|
||||
]
|
||||
|
||||
env, env_json_str = await smartsend(
|
||||
"/chat/mixed",
|
||||
data,
|
||||
broker_url="nats://localhost:4222"
|
||||
)
|
||||
# env: Dict with all metadata and payloads
|
||||
```
|
||||
|
||||
### Example 6: Table Data (Arrow IPC)
|
||||
|
||||
For tabular data, NATSBridge uses Apache Arrow IPC format:
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
import pandas as pd
|
||||
|
||||
# Create DataFrame
|
||||
df = pd.DataFrame({
|
||||
"id": [1, 2, 3],
|
||||
"name": ["Alice", "Bob", "Charlie"],
|
||||
"score": [95, 88, 92]
|
||||
})
|
||||
|
||||
# Send as table type
|
||||
data = [("students", df, "table")]
|
||||
env = smartsend("/data/students", data, nats_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
@@ -484,89 +646,53 @@ df = DataFrame(
|
||||
)
|
||||
|
||||
data = [("students", df, "table")]
|
||||
smartsend("/data/students", data)
|
||||
env, env_json_str = smartsend("/data/students", data, broker_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cross-Platform Communication
|
||||
|
||||
NATSBridge enables seamless communication between different platforms:
|
||||
|
||||
### Julia ↔ JavaScript
|
||||
|
||||
#### Julia Sender
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Send dictionary from Julia to JavaScript
|
||||
config = Dict("step_size" => 0.01, "iterations" => 1000)
|
||||
data = [("config", config, "dictionary")]
|
||||
smartsend("/analysis/config", data, nats_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
#### JavaScript Receiver
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const { smartreceive } = require('./src/NATSBridge');
|
||||
const NATSBridge = require('./src/natbridge.js');
|
||||
|
||||
// Receive dictionary from Julia
|
||||
const envelope = await smartreceive(msg);
|
||||
for (const payload of envelope.payloads) {
|
||||
if (payload.type === "dictionary") {
|
||||
console.log("Received config:", payload.data);
|
||||
// payload.data = { step_size: 0.01, iterations: 1000 }
|
||||
}
|
||||
}
|
||||
// Create table data (array of objects)
|
||||
const table_data = [
|
||||
{ id: 1, name: "Alice", score: 95 },
|
||||
{ id: 2, name: "Bob", score: 88 },
|
||||
{ id: 3, name: "Charlie", score: 92 }
|
||||
];
|
||||
|
||||
const data = [["students", table_data, "table"]];
|
||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||
"/data/students",
|
||||
data,
|
||||
{ broker_url: "nats://localhost:4222" }
|
||||
);
|
||||
```
|
||||
|
||||
### JavaScript ↔ Python
|
||||
|
||||
#### JavaScript Sender
|
||||
|
||||
```javascript
|
||||
const { smartsend } = require('./src/NATSBridge');
|
||||
|
||||
await smartsend("/data/transfer", [
|
||||
{ dataname: "message", data: "Hello from JS!", type: "text" }
|
||||
]);
|
||||
```
|
||||
|
||||
#### Python Receiver
|
||||
#### Python
|
||||
|
||||
```python
|
||||
from nats_bridge import smartreceive
|
||||
from natbridge import smartsend
|
||||
import pandas as pd
|
||||
|
||||
envelope = smartreceive(msg)
|
||||
for dataname, data, type in envelope["payloads"]:
|
||||
if type == "text":
|
||||
print(f"Received from JS: {data}")
|
||||
# Create DataFrame
|
||||
df = pd.DataFrame({
|
||||
'id': [1, 2, 3],
|
||||
'name': ['Alice', 'Bob', 'Charlie'],
|
||||
'score': [95, 88, 92]
|
||||
})
|
||||
|
||||
data = [("students", df, "table")]
|
||||
env, env_json_str = await smartsend(
|
||||
"/data/students",
|
||||
data,
|
||||
broker_url="nats://localhost:4222"
|
||||
)
|
||||
```
|
||||
|
||||
### Python ↔ Julia
|
||||
#### MicroPython
|
||||
|
||||
#### Python Sender
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
data = [("message", "Hello from Python!", "text")]
|
||||
smartsend("/chat/python", data)
|
||||
```
|
||||
|
||||
#### Julia Receiver
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
envelope = smartreceive(msg, fileserverDownloadHandler)
|
||||
for (dataname, data, type) in envelope["payloads"]
|
||||
if type == "text"
|
||||
println("Received from Python: $data")
|
||||
end
|
||||
end
|
||||
```
|
||||
MicroPython does not support table type due to memory constraints. Use dictionary or binary instead.
|
||||
|
||||
---
|
||||
|
||||
@@ -574,7 +700,7 @@ end
|
||||
|
||||
1. **Explore the test directory** for more examples
|
||||
2. **Check the documentation** for advanced configuration options
|
||||
3. **Join the community** to share your use cases
|
||||
3. **Read the walkthrough** for building real-world applications
|
||||
|
||||
---
|
||||
|
||||
@@ -595,7 +721,8 @@ end
|
||||
### Serialization Errors
|
||||
|
||||
- Verify data type matches the specified type
|
||||
- Check that binary data is in the correct format (bytes/Vector{UInt8})
|
||||
- Check that binary data is in the correct format
|
||||
- MicroPython: Ensure payload size < 50KB
|
||||
|
||||
---
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
28
package.json
28
package.json
@@ -1,28 +0,0 @@
|
||||
{
|
||||
"name": "natsbridge",
|
||||
"version": "1.0.0",
|
||||
"description": "Bi-Directional Data Bridge for JavaScript using NATS",
|
||||
"main": "src/NATSBridge.js",
|
||||
"scripts": {
|
||||
"test": "echo \"Error: no test specified\" && exit 1",
|
||||
"lint": "eslint src/*.js test/*.js"
|
||||
},
|
||||
"keywords": [
|
||||
"nats",
|
||||
"message-broker",
|
||||
"bridge",
|
||||
"arrow",
|
||||
"serialization"
|
||||
],
|
||||
"author": "",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"nats": "^2.9.0",
|
||||
"apache-arrow": "^14.0.0",
|
||||
"uuid": "^9.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"eslint": "^8.0.0",
|
||||
"jest": "^29.0.0"
|
||||
}
|
||||
}
|
||||
14
plik_fileserver/docker-compose.yml
Normal file
14
plik_fileserver/docker-compose.yml
Normal file
@@ -0,0 +1,14 @@
|
||||
services:
|
||||
plik:
|
||||
image: rootgg/plik:latest
|
||||
container_name: plik-server
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
# # Mount the config file (created below)
|
||||
# - ./plikd.cfg:/home/plik/server/plikd.cfg
|
||||
# Mount local folder for uploads and database
|
||||
- ./plik-data:/data
|
||||
# Set user to match your host UID to avoid permission issues
|
||||
user: "1000:1000"
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,709 +0,0 @@
|
||||
/**
|
||||
* NATSBridge.js - Bi-Directional Data Bridge for JavaScript
|
||||
* Implements smartsend and smartreceive for NATS communication
|
||||
*
|
||||
* This module provides functionality for sending and receiving data across network boundaries
|
||||
* using NATS as the message bus, with support for both direct payload transport and
|
||||
* URL-based transport for larger payloads.
|
||||
*
|
||||
* File Server Handler Architecture:
|
||||
* The system uses handler functions to abstract file server operations, allowing support
|
||||
* for different file server implementations (e.g., Plik, AWS S3, custom HTTP server).
|
||||
*
|
||||
* Handler Function Signatures:
|
||||
*
|
||||
* ```javascript
|
||||
* // Upload handler - uploads data to file server and returns URL
|
||||
* // The handler is passed to smartsend as fileserverUploadHandler parameter
|
||||
* // It receives: (fileserver_url, dataname, data)
|
||||
* // Returns: { status, uploadid, fileid, url }
|
||||
* async function fileserverUploadHandler(fileserver_url, dataname, data) { ... }
|
||||
*
|
||||
* // Download handler - fetches data from file server URL with exponential backoff
|
||||
* // The handler is passed to smartreceive as fileserverDownloadHandler parameter
|
||||
* // It receives: (url, max_retries, base_delay, max_delay, correlation_id)
|
||||
* // Returns: ArrayBuffer (the downloaded data)
|
||||
* async function fileserverDownloadHandler(url, max_retries, base_delay, max_delay, correlation_id) { ... }
|
||||
* ```
|
||||
*
|
||||
* Multi-Payload Support (Standard API):
|
||||
* The system uses a standardized list-of-tuples format for all payload operations.
|
||||
* Even when sending a single payload, the user must wrap it in a list.
|
||||
*
|
||||
* API Standard:
|
||||
* ```javascript
|
||||
* // Input format for smartsend (always a list of tuples with type info)
|
||||
* [{ dataname, data, type }, ...]
|
||||
*
|
||||
* // Output format for smartreceive (always returns a list of tuples)
|
||||
* [{ dataname, data, type }, ...]
|
||||
* ```
|
||||
*
|
||||
* Supported types: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||
*/
|
||||
|
||||
// ---------------------------------------------- 100 --------------------------------------------- #
|
||||
|
||||
// Constants
|
||||
const DEFAULT_SIZE_THRESHOLD = 1_000_000; // 1MB - threshold for switching from direct to link transport
|
||||
const DEFAULT_NATS_URL = "nats://localhost:4222"; // Default NATS server URL
|
||||
const DEFAULT_FILESERVER_URL = "http://localhost:8080"; // Default HTTP file server URL for link transport
|
||||
|
||||
// Helper: Generate UUID v4
|
||||
function uuid4() {
|
||||
// Simple UUID v4 generator
|
||||
return 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, function(c) {
|
||||
var r = Math.random() * 16 | 0, v = c == 'x' ? r : (r & 0x3 | 0x8);
|
||||
return v.toString(16);
|
||||
});
|
||||
}
|
||||
|
||||
// Helper: Log with correlation ID and timestamp
|
||||
function log_trace(correlation_id, message) {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`[${timestamp}] [Correlation: ${correlation_id}] ${message}`);
|
||||
}
|
||||
|
||||
// Helper: Get size of data in bytes
|
||||
function getDataSize(data) {
|
||||
if (typeof data === 'string') {
|
||||
return new TextEncoder().encode(data).length;
|
||||
} else if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
||||
return data.byteLength;
|
||||
} else if (typeof data === 'object' && data !== null) {
|
||||
// For objects, serialize to JSON and measure
|
||||
return new TextEncoder().encode(JSON.stringify(data)).length;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
// Helper: Convert ArrayBuffer to Base64 string
|
||||
function arrayBufferToBase64(buffer) {
|
||||
const bytes = new Uint8Array(buffer);
|
||||
let binary = '';
|
||||
for (let i = 0; i < bytes.length; i++) {
|
||||
binary += String.fromCharCode(bytes[i]);
|
||||
}
|
||||
return btoa(binary);
|
||||
}
|
||||
|
||||
// Helper: Convert Base64 string to ArrayBuffer
|
||||
function base64ToArrayBuffer(base64) {
|
||||
const binaryString = atob(base64);
|
||||
const len = binaryString.length;
|
||||
const bytes = new Uint8Array(len);
|
||||
for (let i = 0; i < len; i++) {
|
||||
bytes[i] = binaryString.charCodeAt(i);
|
||||
}
|
||||
return bytes.buffer;
|
||||
}
|
||||
|
||||
// Helper: Serialize data based on type
|
||||
function _serialize_data(data, type) {
|
||||
/**
|
||||
* Serialize data according to specified format
|
||||
*
|
||||
* Supported formats:
|
||||
* - "text": Treats data as text and converts to UTF-8 bytes
|
||||
* - "dictionary": Serializes data as JSON and returns the UTF-8 byte representation
|
||||
* - "table": Serializes data as an Arrow IPC stream (table format) - NOT IMPLEMENTED (requires arrow library)
|
||||
* - "image": Expects binary data (ArrayBuffer) and returns it as bytes
|
||||
* - "audio": Expects binary data (ArrayBuffer) and returns it as bytes
|
||||
* - "video": Expects binary data (ArrayBuffer) and returns it as bytes
|
||||
* - "binary": Generic binary data (ArrayBuffer or Uint8Array) and returns bytes
|
||||
*/
|
||||
if (type === "text") {
|
||||
if (typeof data === 'string') {
|
||||
return new TextEncoder().encode(data).buffer;
|
||||
} else {
|
||||
throw new Error("Text data must be a String");
|
||||
}
|
||||
} else if (type === "dictionary") {
|
||||
// JSON data - serialize directly
|
||||
const jsonStr = JSON.stringify(data);
|
||||
return new TextEncoder().encode(jsonStr).buffer;
|
||||
} else if (type === "table") {
|
||||
// Table data - convert to Arrow IPC stream (NOT IMPLEMENTED in pure JavaScript)
|
||||
// This would require the apache-arrow library
|
||||
throw new Error("Table serialization requires apache-arrow library");
|
||||
} else if (type === "image") {
|
||||
if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
||||
return data instanceof ArrayBuffer ? data : data.buffer;
|
||||
} else {
|
||||
throw new Error("Image data must be ArrayBuffer or Uint8Array");
|
||||
}
|
||||
} else if (type === "audio") {
|
||||
if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
||||
return data instanceof ArrayBuffer ? data : data.buffer;
|
||||
} else {
|
||||
throw new Error("Audio data must be ArrayBuffer or Uint8Array");
|
||||
}
|
||||
} else if (type === "video") {
|
||||
if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
||||
return data instanceof ArrayBuffer ? data : data.buffer;
|
||||
} else {
|
||||
throw new Error("Video data must be ArrayBuffer or Uint8Array");
|
||||
}
|
||||
} else if (type === "binary") {
|
||||
if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
||||
return data instanceof ArrayBuffer ? data : data.buffer;
|
||||
} else {
|
||||
throw new Error("Binary data must be ArrayBuffer or Uint8Array");
|
||||
}
|
||||
} else {
|
||||
throw new Error(`Unknown type: ${type}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Helper: Deserialize bytes based on type
|
||||
function _deserialize_data(data, type, correlation_id) {
|
||||
/**
|
||||
* Deserialize bytes to data based on type
|
||||
*
|
||||
* Supported formats:
|
||||
* - "text": Converts bytes to string
|
||||
* - "dictionary": Parses JSON string
|
||||
* - "table": Parses Arrow IPC stream - NOT IMPLEMENTED (requires apache-arrow library)
|
||||
* - "image": Returns binary data
|
||||
* - "audio": Returns binary data
|
||||
* - "video": Returns binary data
|
||||
* - "binary": Returns binary data
|
||||
*/
|
||||
if (type === "text") {
|
||||
const decoder = new TextDecoder();
|
||||
return decoder.decode(new Uint8Array(data));
|
||||
} else if (type === "dictionary") {
|
||||
const decoder = new TextDecoder();
|
||||
const jsonStr = decoder.decode(new Uint8Array(data));
|
||||
return JSON.parse(jsonStr);
|
||||
} else if (type === "table") {
|
||||
// Table data - deserialize Arrow IPC stream (NOT IMPLEMENTED in pure JavaScript)
|
||||
throw new Error("Table deserialization requires apache-arrow library");
|
||||
} else if (type === "image") {
|
||||
return data;
|
||||
} else if (type === "audio") {
|
||||
return data;
|
||||
} else if (type === "video") {
|
||||
return data;
|
||||
} else if (type === "binary") {
|
||||
return data;
|
||||
} else {
|
||||
throw new Error(`Unknown type: ${type}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Helper: Upload data to file server
|
||||
async function _upload_to_fileserver(fileserver_url, dataname, data, correlation_id) {
|
||||
/**
|
||||
* Upload data to HTTP file server (plik-like API)
|
||||
*
|
||||
* This function implements the plik one-shot upload mode:
|
||||
* 1. Creates a one-shot upload session by sending POST request with {"OneShot": true}
|
||||
* 2. Uploads the file data as multipart form data
|
||||
* 3. Returns identifiers and download URL for the uploaded file
|
||||
*/
|
||||
log_trace(correlation_id, `Uploading ${dataname} to fileserver: ${fileserver_url}`);
|
||||
|
||||
// Step 1: Get upload ID and token
|
||||
const url_getUploadID = `${fileserver_url}/upload`;
|
||||
const headers = {
|
||||
"Content-Type": "application/json"
|
||||
};
|
||||
const body = JSON.stringify({ OneShot: true });
|
||||
|
||||
let response = await fetch(url_getUploadID, {
|
||||
method: "POST",
|
||||
headers: headers,
|
||||
body: body
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to get upload ID: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const responseJson = await response.json();
|
||||
const uploadid = responseJson.id;
|
||||
const uploadtoken = responseJson.uploadToken;
|
||||
|
||||
// Step 2: Upload file data
|
||||
const url_upload = `${fileserver_url}/file/${uploadid}`;
|
||||
|
||||
// Create multipart form data
|
||||
const formData = new FormData();
|
||||
// Create a Blob from the ArrayBuffer
|
||||
const blob = new Blob([data], { type: "application/octet-stream" });
|
||||
formData.append("file", blob, dataname);
|
||||
|
||||
response = await fetch(url_upload, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"X-UploadToken": uploadtoken
|
||||
},
|
||||
body: formData
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to upload file: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const fileResponseJson = await response.json();
|
||||
const fileid = fileResponseJson.id;
|
||||
|
||||
// Build the download URL
|
||||
const url = `${fileserver_url}/file/${uploadid}/${fileid}/${encodeURIComponent(dataname)}`;
|
||||
|
||||
log_trace(correlation_id, `Uploaded to URL: ${url}`);
|
||||
|
||||
return {
|
||||
status: response.status,
|
||||
uploadid: uploadid,
|
||||
fileid: fileid,
|
||||
url: url
|
||||
};
|
||||
}
|
||||
|
||||
// Helper: Fetch data from URL with exponential backoff
|
||||
async function _fetch_with_backoff(url, max_retries, base_delay, max_delay, correlation_id) {
|
||||
/**
|
||||
* Fetch data from URL with retry logic using exponential backoff
|
||||
*/
|
||||
let delay = base_delay;
|
||||
|
||||
for (let attempt = 1; attempt <= max_retries; attempt++) {
|
||||
try {
|
||||
const response = await fetch(url);
|
||||
|
||||
if (response.status === 200) {
|
||||
log_trace(correlation_id, `Successfully fetched data from ${url} on attempt ${attempt}`);
|
||||
const arrayBuffer = await response.arrayBuffer();
|
||||
return arrayBuffer;
|
||||
} else {
|
||||
throw new Error(`Failed to fetch: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
} catch (e) {
|
||||
log_trace(correlation_id, `Attempt ${attempt} failed: ${e.message}`);
|
||||
|
||||
if (attempt < max_retries) {
|
||||
// Sleep with exponential backoff
|
||||
await new Promise(resolve => setTimeout(resolve, delay));
|
||||
delay = Math.min(delay * 2, max_delay);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error(`Failed to fetch data after ${max_retries} attempts`);
|
||||
}
|
||||
|
||||
// Helper: Get payload bytes from data
|
||||
function _get_payload_bytes(data) {
|
||||
if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
||||
return data instanceof ArrayBuffer ? new Uint8Array(data) : data;
|
||||
} else if (typeof data === 'string') {
|
||||
return new TextEncoder().encode(data);
|
||||
} else {
|
||||
// For objects, serialize to JSON
|
||||
return new TextEncoder().encode(JSON.stringify(data));
|
||||
}
|
||||
}
|
||||
|
||||
// MessagePayload class
|
||||
class MessagePayload {
|
||||
/**
|
||||
* Represents a single payload in the message envelope
|
||||
*
|
||||
* @param {Object} options - Payload options
|
||||
* @param {string} options.id - ID of this payload (e.g., "uuid4")
|
||||
* @param {string} options.dataname - Name of this payload (e.g., "login_image")
|
||||
* @param {string} options.type - Payload type: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||
* @param {string} options.transport - "direct" or "link"
|
||||
* @param {string} options.encoding - "none", "json", "base64", "arrow-ipc"
|
||||
* @param {number} options.size - Data size in bytes
|
||||
* @param {string|ArrayBuffer} options.data - Payload data (direct) or URL (link)
|
||||
* @param {Object} options.metadata - Metadata for this payload
|
||||
*/
|
||||
constructor(options) {
|
||||
this.id = options.id || uuid4();
|
||||
this.dataname = options.dataname;
|
||||
this.type = options.type;
|
||||
this.transport = options.transport;
|
||||
this.encoding = options.encoding;
|
||||
this.size = options.size;
|
||||
this.data = options.data;
|
||||
this.metadata = options.metadata || {};
|
||||
}
|
||||
|
||||
// Convert to JSON object
|
||||
toJSON() {
|
||||
const obj = {
|
||||
id: this.id,
|
||||
dataname: this.dataname,
|
||||
type: this.type,
|
||||
transport: this.transport,
|
||||
encoding: this.encoding,
|
||||
size: this.size
|
||||
};
|
||||
|
||||
// Include data based on transport type
|
||||
if (this.transport === "direct" && this.data !== null) {
|
||||
if (this.encoding === "base64" || this.encoding === "json") {
|
||||
obj.data = this.data;
|
||||
} else {
|
||||
// For other encodings, use base64
|
||||
const payloadBytes = _get_payload_bytes(this.data);
|
||||
obj.data = arrayBufferToBase64(payloadBytes);
|
||||
}
|
||||
} else if (this.transport === "link" && this.data !== null) {
|
||||
// For link transport, data is a URL string
|
||||
obj.data = this.data;
|
||||
}
|
||||
|
||||
if (Object.keys(this.metadata).length > 0) {
|
||||
obj.metadata = this.metadata;
|
||||
}
|
||||
|
||||
return obj;
|
||||
}
|
||||
}
|
||||
|
||||
// MessageEnvelope class
|
||||
class MessageEnvelope {
|
||||
/**
|
||||
* Represents the message envelope containing metadata and payloads
|
||||
*
|
||||
* @param {Object} options - Envelope options
|
||||
* @param {string} options.sendTo - Topic/subject the sender sends to
|
||||
* @param {Array<MessagePayload>} options.payloads - Array of payloads
|
||||
* @param {string} options.correlationId - Unique identifier to track messages
|
||||
* @param {string} options.msgId - This message id
|
||||
* @param {string} options.timestamp - Message published timestamp
|
||||
* @param {string} options.msgPurpose - Purpose of this message
|
||||
* @param {string} options.senderName - Name of the sender
|
||||
* @param {string} options.senderId - UUID of the sender
|
||||
* @param {string} options.receiverName - Name of the receiver
|
||||
* @param {string} options.receiverId - UUID of the receiver
|
||||
* @param {string} options.replyTo - Topic to reply to
|
||||
* @param {string} options.replyToMsgId - Message id this message is replying to
|
||||
* @param {string} options.brokerURL - NATS server address
|
||||
* @param {Object} options.metadata - Metadata for the envelope
|
||||
*/
|
||||
constructor(options) {
|
||||
this.correlationId = options.correlationId || uuid4();
|
||||
this.msgId = options.msgId || uuid4();
|
||||
this.timestamp = options.timestamp || new Date().toISOString();
|
||||
this.sendTo = options.sendTo;
|
||||
this.msgPurpose = options.msgPurpose || "";
|
||||
this.senderName = options.senderName || "";
|
||||
this.senderId = options.senderId || uuid4();
|
||||
this.receiverName = options.receiverName || "";
|
||||
this.receiverId = options.receiverId || "";
|
||||
this.replyTo = options.replyTo || "";
|
||||
this.replyToMsgId = options.replyToMsgId || "";
|
||||
this.brokerURL = options.brokerURL || DEFAULT_NATS_URL;
|
||||
this.metadata = options.metadata || {};
|
||||
this.payloads = options.payloads || [];
|
||||
}
|
||||
|
||||
// Convert to JSON string
|
||||
toJSON() {
|
||||
const obj = {
|
||||
correlationId: this.correlationId,
|
||||
msgId: this.msgId,
|
||||
timestamp: this.timestamp,
|
||||
sendTo: this.sendTo,
|
||||
msgPurpose: this.msgPurpose,
|
||||
senderName: this.senderName,
|
||||
senderId: this.senderId,
|
||||
receiverName: this.receiverName,
|
||||
receiverId: this.receiverId,
|
||||
replyTo: this.replyTo,
|
||||
replyToMsgId: this.replyToMsgId,
|
||||
brokerURL: this.brokerURL
|
||||
};
|
||||
|
||||
if (Object.keys(this.metadata).length > 0) {
|
||||
obj.metadata = this.metadata;
|
||||
}
|
||||
|
||||
if (this.payloads.length > 0) {
|
||||
obj.payloads = this.payloads.map(p => p.toJSON());
|
||||
}
|
||||
|
||||
return obj;
|
||||
}
|
||||
|
||||
// Convert to JSON string
|
||||
toString() {
|
||||
return JSON.stringify(this.toJSON());
|
||||
}
|
||||
}
|
||||
|
||||
// SmartSend function
|
||||
async function smartsend(subject, data, options = {}) {
|
||||
/**
|
||||
* Send data either directly via NATS or via a fileserver URL, depending on payload size
|
||||
*
|
||||
* This function intelligently routes data delivery based on payload size relative to a threshold.
|
||||
* If the serialized payload is smaller than `size_threshold`, it encodes the data as Base64 and publishes directly over NATS.
|
||||
* Otherwise, it uploads the data to a fileserver and publishes only the download URL over NATS.
|
||||
*
|
||||
* @param {string} subject - NATS subject to publish the message to
|
||||
* @param {Array} data - List of {dataname, data, type} objects to send
|
||||
* @param {Object} options - Additional options
|
||||
* @param {string} options.natsUrl - URL of the NATS server (default: "nats://localhost:4222")
|
||||
* @param {string} options.fileserverUrl - Base URL of the file server (default: "http://localhost:8080")
|
||||
* @param {Function} options.fileserverUploadHandler - Function to handle fileserver uploads
|
||||
* @param {number} options.sizeThreshold - Threshold in bytes separating direct vs link transport (default: 1MB)
|
||||
* @param {string} options.correlationId - Optional correlation ID for tracing
|
||||
* @param {string} options.msgPurpose - Purpose of the message (default: "chat")
|
||||
* @param {string} options.senderName - Name of the sender (default: "NATSBridge")
|
||||
* @param {string} options.receiverName - Name of the receiver (default: "")
|
||||
* @param {string} options.receiverId - UUID of the receiver (default: "")
|
||||
* @param {string} options.replyTo - Topic to reply to (default: "")
|
||||
* @param {string} options.replyToMsgId - Message ID this message is replying to (default: "")
|
||||
*
|
||||
* @returns {Promise<MessageEnvelope>} - The envelope for tracking
|
||||
*/
|
||||
const {
|
||||
natsUrl = DEFAULT_NATS_URL,
|
||||
fileserverUrl = DEFAULT_FILESERVER_URL,
|
||||
fileserverUploadHandler = _upload_to_fileserver,
|
||||
sizeThreshold = DEFAULT_SIZE_THRESHOLD,
|
||||
correlationId = uuid4(),
|
||||
msgPurpose = "chat",
|
||||
senderName = "NATSBridge",
|
||||
receiverName = "",
|
||||
receiverId = "",
|
||||
replyTo = "",
|
||||
replyToMsgId = ""
|
||||
} = options;
|
||||
|
||||
log_trace(correlationId, `Starting smartsend for subject: ${subject}`);
|
||||
|
||||
// Generate message metadata
|
||||
const msgId = uuid4();
|
||||
|
||||
// Process each payload in the list
|
||||
const payloads = [];
|
||||
|
||||
for (const payload of data) {
|
||||
const dataname = payload.dataname;
|
||||
const payloadData = payload.data;
|
||||
const payloadType = payload.type;
|
||||
|
||||
// Serialize data based on type
|
||||
const payloadBytes = _serialize_data(payloadData, payloadType);
|
||||
const payloadSize = payloadBytes.byteLength;
|
||||
|
||||
log_trace(correlationId, `Serialized payload '${dataname}' (type: ${payloadType}) size: ${payloadSize} bytes`);
|
||||
|
||||
// Decision: Direct vs Link
|
||||
if (payloadSize < sizeThreshold) {
|
||||
// Direct path - Base64 encode and send via NATS
|
||||
const payloadB64 = arrayBufferToBase64(payloadBytes);
|
||||
log_trace(correlationId, `Using direct transport for ${payloadSize} bytes`);
|
||||
|
||||
// Create MessagePayload for direct transport
|
||||
const payloadObj = new MessagePayload({
|
||||
dataname: dataname,
|
||||
type: payloadType,
|
||||
transport: "direct",
|
||||
encoding: "base64",
|
||||
size: payloadSize,
|
||||
data: payloadB64,
|
||||
metadata: { payload_bytes: payloadSize }
|
||||
});
|
||||
payloads.push(payloadObj);
|
||||
} else {
|
||||
// Link path - Upload to HTTP server, send URL via NATS
|
||||
log_trace(correlationId, `Using link transport, uploading to fileserver`);
|
||||
|
||||
// Upload to HTTP server
|
||||
const response = await fileserverUploadHandler(fileserverUrl, dataname, payloadBytes, correlationId);
|
||||
|
||||
if (response.status !== 200) {
|
||||
throw new Error(`Failed to upload data to fileserver: ${response.status}`);
|
||||
}
|
||||
|
||||
const url = response.url;
|
||||
log_trace(correlationId, `Uploaded to URL: ${url}`);
|
||||
|
||||
// Create MessagePayload for link transport
|
||||
const payloadObj = new MessagePayload({
|
||||
dataname: dataname,
|
||||
type: payloadType,
|
||||
transport: "link",
|
||||
encoding: "none",
|
||||
size: payloadSize,
|
||||
data: url,
|
||||
metadata: {}
|
||||
});
|
||||
payloads.push(payloadObj);
|
||||
}
|
||||
}
|
||||
|
||||
// Create MessageEnvelope with all payloads
|
||||
const env = new MessageEnvelope({
|
||||
correlationId: correlationId,
|
||||
msgId: msgId,
|
||||
sendTo: subject,
|
||||
msgPurpose: msgPurpose,
|
||||
senderName: senderName,
|
||||
receiverName: receiverName,
|
||||
receiverId: receiverId,
|
||||
replyTo: replyTo,
|
||||
replyToMsgId: replyToMsgId,
|
||||
brokerURL: natsUrl,
|
||||
payloads: payloads
|
||||
});
|
||||
|
||||
// Publish message to NATS
|
||||
await publish_message(natsUrl, subject, env.toString(), correlationId);
|
||||
|
||||
return env;
|
||||
}
|
||||
|
||||
// Helper: Publish message to NATS
|
||||
async function publish_message(natsUrl, subject, message, correlation_id) {
|
||||
/**
|
||||
* Publish a message to a NATS subject with proper connection management
|
||||
*
|
||||
* @param {string} natsUrl - NATS server URL
|
||||
* @param {string} subject - NATS subject to publish to
|
||||
* @param {string} message - JSON message to publish
|
||||
* @param {string} correlation_id - Correlation ID for logging
|
||||
*/
|
||||
log_trace(correlation_id, `Publishing message to ${subject}`);
|
||||
|
||||
// For Node.js, we would use nats.js library
|
||||
// This is a placeholder that throws an error
|
||||
// In production, you would import and use the actual nats library
|
||||
|
||||
// Example with nats.js:
|
||||
// import { connect } from 'nats';
|
||||
// const nc = await connect({ servers: [natsUrl] });
|
||||
// await nc.publish(subject, message);
|
||||
// nc.close();
|
||||
|
||||
// For now, just log the message
|
||||
console.log(`[NATS PUBLISH] Subject: ${subject}, Message: ${message.substring(0, 100)}...`);
|
||||
}
|
||||
|
||||
// SmartReceive function
|
||||
async function smartreceive(msg, options = {}) {
|
||||
/**
|
||||
* Receive and process messages from NATS
|
||||
*
|
||||
* This function processes incoming NATS messages, handling both direct transport
|
||||
* (base64 decoded payloads) and link transport (URL-based payloads).
|
||||
*
|
||||
* @param {Object} msg - NATS message object with payload property
|
||||
* @param {Object} options - Additional options
|
||||
* @param {Function} options.fileserverDownloadHandler - Function to handle downloading data from file server URLs
|
||||
* @param {number} options.maxRetries - Maximum retry attempts for fetching URL (default: 5)
|
||||
* @param {number} options.baseDelay - Initial delay for exponential backoff in ms (default: 100)
|
||||
* @param {number} options.maxDelay - Maximum delay for exponential backoff in ms (default: 5000)
|
||||
*
|
||||
* @returns {Promise<Object>} - Envelope dictionary with metadata and payloads field containing list of {dataname, data, type} objects
|
||||
*/
|
||||
const {
|
||||
fileserverDownloadHandler = _fetch_with_backoff,
|
||||
maxRetries = 5,
|
||||
baseDelay = 100,
|
||||
maxDelay = 5000
|
||||
} = options;
|
||||
|
||||
// Parse the JSON envelope
|
||||
const jsonStr = typeof msg.payload === 'string' ? msg.payload : new TextDecoder().decode(msg.payload);
|
||||
const json_data = JSON.parse(jsonStr);
|
||||
|
||||
log_trace(json_data.correlationId, `Processing received message`);
|
||||
|
||||
// Process all payloads in the envelope
|
||||
const payloads_list = [];
|
||||
|
||||
// Get number of payloads
|
||||
const num_payloads = json_data.payloads ? json_data.payloads.length : 0;
|
||||
|
||||
for (let i = 0; i < num_payloads; i++) {
|
||||
const payload = json_data.payloads[i];
|
||||
const transport = payload.transport;
|
||||
const dataname = payload.dataname;
|
||||
|
||||
if (transport === "direct") {
|
||||
// Direct transport - payload is in the message
|
||||
log_trace(json_data.correlationId, `Direct transport - decoding payload '${dataname}'`);
|
||||
|
||||
// Extract base64 payload from the payload
|
||||
const payload_b64 = payload.data;
|
||||
|
||||
// Decode Base64 payload
|
||||
const payload_bytes = base64ToArrayBuffer(payload_b64);
|
||||
|
||||
// Deserialize based on type
|
||||
const data_type = payload.type;
|
||||
const data = _deserialize_data(payload_bytes, data_type, json_data.correlationId);
|
||||
|
||||
payloads_list.push({ dataname, data, type: data_type });
|
||||
} else if (transport === "link") {
|
||||
// Link transport - payload is at URL
|
||||
const url = payload.data;
|
||||
log_trace(json_data.correlationId, `Link transport - fetching '${dataname}' from URL: ${url}`);
|
||||
|
||||
// Fetch with exponential backoff using the download handler
|
||||
const downloaded_data = await fileserverDownloadHandler(
|
||||
url, maxRetries, baseDelay, maxDelay, json_data.correlationId
|
||||
);
|
||||
|
||||
// Deserialize based on type
|
||||
const data_type = payload.type;
|
||||
const data = _deserialize_data(downloaded_data, data_type, json_data.correlationId);
|
||||
|
||||
payloads_list.push({ dataname, data, type: data_type });
|
||||
} else {
|
||||
throw new Error(`Unknown transport type for payload '${dataname}': ${transport}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Replace payloads array with the processed list of {dataname, data, type} tuples
|
||||
json_data.payloads = payloads_list;
|
||||
|
||||
return json_data;
|
||||
}
|
||||
|
||||
// Export for Node.js
|
||||
if (typeof module !== 'undefined' && module.exports) {
|
||||
module.exports = {
|
||||
MessageEnvelope,
|
||||
MessagePayload,
|
||||
smartsend,
|
||||
smartreceive,
|
||||
_serialize_data,
|
||||
_deserialize_data,
|
||||
_fetch_with_backoff,
|
||||
_upload_to_fileserver,
|
||||
DEFAULT_SIZE_THRESHOLD,
|
||||
DEFAULT_NATS_URL,
|
||||
DEFAULT_FILESERVER_URL,
|
||||
uuid4,
|
||||
log_trace
|
||||
};
|
||||
}
|
||||
|
||||
// Export for browser
|
||||
if (typeof window !== 'undefined') {
|
||||
window.NATSBridge = {
|
||||
MessageEnvelope,
|
||||
MessagePayload,
|
||||
smartsend,
|
||||
smartreceive,
|
||||
_serialize_data,
|
||||
_deserialize_data,
|
||||
_fetch_with_backoff,
|
||||
_upload_to_fileserver,
|
||||
DEFAULT_SIZE_THRESHOLD,
|
||||
DEFAULT_NATS_URL,
|
||||
DEFAULT_FILESERVER_URL,
|
||||
uuid4,
|
||||
log_trace
|
||||
};
|
||||
}
|
||||
295
src/README.md
295
src/README.md
@@ -1,295 +0,0 @@
|
||||
# NATSBridge
|
||||
|
||||
A high-performance, bi-directional data bridge for **Julia**, **JavaScript**, and **Python/Micropython** using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||
|
||||
## Overview
|
||||
|
||||
NATSBridge enables seamless communication between Julia, JavaScript, and Python/Micropython applications through NATS, with automatic transport selection based on payload size:
|
||||
|
||||
- **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded)
|
||||
- **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL
|
||||
|
||||
## Features
|
||||
|
||||
- ✅ Bi-directional NATS communication across Julia ↔ JavaScript ↔ Python/Micropython
|
||||
- ✅ Multi-payload support (mixed content in single message)
|
||||
- ✅ Automatic transport selection based on payload size
|
||||
- ✅ File server integration for large payloads
|
||||
- ✅ Exponential backoff for URL fetching
|
||||
- ✅ Correlation ID tracking
|
||||
- ✅ Reply-to support for request-response pattern
|
||||
|
||||
## Supported Payload Types
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `text` | Plain text strings |
|
||||
| `dictionary` | JSON-serializable dictionaries |
|
||||
| `table` | Tabular data (Arrow IPC format) |
|
||||
| `image` | Image data (PNG, JPG bytes) |
|
||||
| `audio` | Audio data (WAV, MP3 bytes) |
|
||||
| `video` | Video data (MP4, AVI bytes) |
|
||||
| `binary` | Generic binary data |
|
||||
|
||||
## Implementation Guides
|
||||
|
||||
### [Julia Implementation](../tutorial_julia.md)
|
||||
|
||||
See the [Julia tutorial](../tutorial_julia.md) for getting started with Julia.
|
||||
|
||||
### [JavaScript Implementation](#javascript-implementation)
|
||||
|
||||
See [`NATSBridge.js`](NATSBridge.js) for the JavaScript implementation.
|
||||
|
||||
### [Python/Micropython Implementation](#pythonmicropython-implementation)
|
||||
|
||||
See [`nats_bridge.py`](nats_bridge.py) for the Python/Micropython implementation.
|
||||
|
||||
## Installation
|
||||
|
||||
### Julia
|
||||
|
||||
```julia
|
||||
using Pkg
|
||||
Pkg.add("NATS")
|
||||
Pkg.add("Arrow")
|
||||
Pkg.add("JSON3")
|
||||
Pkg.add("HTTP")
|
||||
Pkg.add("UUIDs")
|
||||
Pkg.add("Dates")
|
||||
```
|
||||
|
||||
### JavaScript
|
||||
|
||||
```bash
|
||||
npm install nats.js apache-arrow uuid base64-url
|
||||
```
|
||||
|
||||
### Python/Micropython
|
||||
|
||||
1. Copy `nats_bridge.py` to your device
|
||||
2. Ensure you have the following dependencies:
|
||||
- `urequests` for HTTP requests (Micropython)
|
||||
- `requests` for HTTP requests (Python)
|
||||
- `base64` for base64 encoding
|
||||
- `json` for JSON handling
|
||||
- `socket` for networking (Micropython)
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Text Message
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend, smartreceive
|
||||
|
||||
# Sender
|
||||
data = [("message", "Hello World", "text")]
|
||||
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
||||
|
||||
# Receiver
|
||||
payloads = smartreceive(msg)
|
||||
for dataname, data, type in payloads:
|
||||
print("Received {}: {}".format(dataname, data))
|
||||
```
|
||||
|
||||
#### Julia
|
||||
|
||||
```julia
|
||||
using NATSBridge
|
||||
|
||||
# Sender
|
||||
data = [("message", "Hello World", "text")]
|
||||
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
||||
|
||||
# Receiver
|
||||
envelope = smartreceive(msg, fileserverDownloadHandler)
|
||||
# envelope["payloads"] = [("message", "Hello World", "text"), ...]
|
||||
```
|
||||
|
||||
#### JavaScript
|
||||
|
||||
```javascript
|
||||
const { smartsend, smartreceive } = require('./src/NATSBridge');
|
||||
|
||||
// Sender
|
||||
await smartsend("/chat/room1", [
|
||||
{ dataname: "message", data: "Hello World", type: "text" }
|
||||
], { natsUrl: "nats://localhost:4222" });
|
||||
|
||||
// Receiver
|
||||
const envelope = await smartreceive(msg);
|
||||
// envelope.payloads = [{ dataname: "message", data: "Hello World", type: "text" }, ...]
|
||||
```
|
||||
|
||||
### Sending JSON Configuration
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
config = {
|
||||
"wifi_ssid": "MyNetwork",
|
||||
"wifi_password": "password123",
|
||||
"update_interval": 60
|
||||
}
|
||||
|
||||
data = [("config", config, "dictionary")]
|
||||
env = smartsend("/device/config", data, nats_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
### Mixed Content (Chat with Text + Image)
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
image_data = b"\x89PNG..." # PNG bytes
|
||||
|
||||
data = [
|
||||
("message_text", "Hello with image!", "text"),
|
||||
("user_avatar", image_data, "binary")
|
||||
]
|
||||
|
||||
env = smartsend("/chat/mixed", data, nats_url="nats://localhost:4222")
|
||||
```
|
||||
|
||||
### Request-Response Pattern
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
# Send command with reply-to
|
||||
data = [("command", {"action": "read_sensor"}, "dictionary")]
|
||||
env = smartsend(
|
||||
"/device/command",
|
||||
data,
|
||||
nats_url="nats://localhost:4222",
|
||||
reply_to="/device/response",
|
||||
reply_to_msg_id="cmd-001"
|
||||
)
|
||||
```
|
||||
|
||||
### Large Payloads (File Server)
|
||||
|
||||
#### Python/Micropython
|
||||
|
||||
```python
|
||||
from nats_bridge import smartsend
|
||||
|
||||
# Large data (> 1MB)
|
||||
large_data = b"A" * 2000000 # 2MB
|
||||
|
||||
env = smartsend(
|
||||
"/data/large",
|
||||
[("large_file", large_data, "binary")],
|
||||
nats_url="nats://localhost:4222",
|
||||
fileserver_url="http://localhost:8080",
|
||||
size_threshold=1000000 # 1MB threshold
|
||||
)
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `smartsend(subject, data, ...)`
|
||||
|
||||
Send data via NATS with automatic transport selection.
|
||||
|
||||
**Arguments:**
|
||||
- `subject` (str): NATS subject to publish to
|
||||
- `data` (list): List of `(dataname, data, type)` tuples
|
||||
- `nats_url` (str): NATS server URL (default: `nats://localhost:4222`)
|
||||
- `fileserver_url` (str): HTTP file server URL (default: `http://localhost:8080`)
|
||||
- `size_threshold` (int): Threshold in bytes (default: 1,000,000)
|
||||
- `correlation_id` (str): Optional correlation ID for tracing
|
||||
- `msg_purpose` (str): Message purpose (default: `"chat"`)
|
||||
- `sender_name` (str): Sender name (default: `"NATSBridge"`)
|
||||
- `receiver_name` (str): Receiver name (default: `""`)
|
||||
- `receiver_id` (str): Receiver ID (default: `""`)
|
||||
- `reply_to` (str): Reply topic (default: `""`)
|
||||
- `reply_to_msg_id` (str): Reply message ID (default: `""`)
|
||||
|
||||
**Returns:** `MessageEnvelope` object
|
||||
|
||||
### `smartreceive(msg, ...)`
|
||||
|
||||
Receive and process NATS messages.
|
||||
|
||||
**Arguments:**
|
||||
- `msg`: NATS message (dict or JSON string)
|
||||
- `fileserver_download_handler` (function): Function to fetch data from URLs
|
||||
- `max_retries` (int): Maximum retry attempts (default: 5)
|
||||
- `base_delay` (int): Initial delay in ms (default: 100)
|
||||
- `max_delay` (int): Maximum delay in ms (default: 5000)
|
||||
|
||||
**Returns:** List of `(dataname, data, type)` tuples
|
||||
|
||||
### `MessageEnvelope`
|
||||
|
||||
Represents a complete NATS message envelope.
|
||||
|
||||
**Attributes:**
|
||||
- `correlation_id`: Unique identifier for tracing
|
||||
- `msg_id`: Unique message identifier
|
||||
- `timestamp`: Message publication timestamp
|
||||
- `send_to`: NATS subject
|
||||
- `msg_purpose`: Message purpose
|
||||
- `sender_name`: Sender name
|
||||
- `sender_id`: Sender UUID
|
||||
- `receiver_name`: Receiver name
|
||||
- `receiver_id`: Receiver UUID
|
||||
- `reply_to`: Reply topic
|
||||
- `reply_to_msg_id`: Reply message ID
|
||||
- `broker_url`: NATS broker URL
|
||||
- `metadata`: Message-level metadata
|
||||
- `payloads`: List of MessagePayload objects
|
||||
|
||||
### `MessagePayload`
|
||||
|
||||
Represents a single payload within a message envelope.
|
||||
|
||||
**Attributes:**
|
||||
- `id`: Unique payload identifier
|
||||
- `dataname`: Name of the payload
|
||||
- `type`: Payload type ("text", "dictionary", etc.)
|
||||
- `transport`: Transport method ("direct" or "link")
|
||||
- `encoding`: Encoding method ("none", "base64", etc.)
|
||||
- `size`: Payload size in bytes
|
||||
- `data`: Payload data (bytes for direct, URL for link)
|
||||
- `metadata`: Payload-level metadata
|
||||
|
||||
## Examples
|
||||
|
||||
See [`examples/micropython_example.py`](../examples/micropython_example.py) for more detailed examples.
|
||||
|
||||
## Testing
|
||||
|
||||
Run the test suite:
|
||||
|
||||
```bash
|
||||
# Python/Micropython
|
||||
python test/test_micropython_basic.py
|
||||
|
||||
# JavaScript
|
||||
node test/test_js_to_js_text_sender.js
|
||||
node test/test_js_to_js_text_receiver.js
|
||||
|
||||
# Julia
|
||||
julia test/test_julia_to_julia_text_sender.jl
|
||||
julia test/test_julia_to_julia_text_receiver.jl
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- **Julia**: NATS server (nats.io), HTTP file server (optional)
|
||||
- **JavaScript**: NATS server (nats.io), HTTP file server (optional)
|
||||
- **Python/Micropython**: NATS server (nats.io), HTTP file server (optional)
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
674
src/natbridge.js
Normal file
674
src/natbridge.js
Normal file
@@ -0,0 +1,674 @@
|
||||
/**
|
||||
* NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||
* JavaScript/Node.js Implementation
|
||||
*
|
||||
* This module provides functionality for sending and receiving data across network boundaries
|
||||
* using NATS as the message bus, with support for both direct payload transport and
|
||||
* URL-based transport for larger payloads.
|
||||
*
|
||||
* @module NATSBridge
|
||||
*/
|
||||
|
||||
const nats = require('nats');
|
||||
const { v4: uuidv4 } = require('uuid');
|
||||
const fetch = require('node-fetch');
|
||||
const arrow = require('apache-arrow');
|
||||
|
||||
// ---------------------------------------------- Constants ---------------------------------------------- //
|
||||
|
||||
/**
|
||||
* Default size threshold for switching from direct to link transport (1MB)
|
||||
*/
|
||||
const DEFAULT_SIZE_THRESHOLD = 1_000_000;
|
||||
|
||||
/**
|
||||
* Default NATS server URL
|
||||
*/
|
||||
const DEFAULT_BROKER_URL = 'nats://localhost:4222';
|
||||
|
||||
/**
|
||||
* Default HTTP file server URL for link transport
|
||||
*/
|
||||
const DEFAULT_FILESERVER_URL = 'http://localhost:8080';
|
||||
|
||||
// ---------------------------------------------- Utility Functions ---------------------------------------------- //
|
||||
|
||||
/**
|
||||
* Convert Buffer to Base64 string
|
||||
* @param {Buffer} buffer - Buffer to encode
|
||||
* @returns {string} Base64 encoded string
|
||||
*/
|
||||
function bufferToBase64(buffer) {
|
||||
return buffer.toString('base64');
|
||||
}
|
||||
|
||||
/**
|
||||
* Log a trace message with correlation ID and timestamp
|
||||
* @param {string} correlationId - Correlation ID for tracing
|
||||
* @param {string} message - Message content to log
|
||||
*/
|
||||
function logTrace(correlationId, message) {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`[${timestamp}] [Correlation: ${correlationId}] ${message}`);
|
||||
}
|
||||
|
||||
// ---------------------------------------------- Serialization Functions ---------------------------------------------- //
|
||||
|
||||
/**
|
||||
* Serialize data according to specified format
|
||||
* @param {any} data - Data to serialize
|
||||
* @param {string} payloadType - Target format: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||
* @returns {Buffer} Binary representation of the serialized data
|
||||
*/
|
||||
async function serializeData(data, payloadType) {
|
||||
if (payloadType === 'text') {
|
||||
if (typeof data === 'string') {
|
||||
return Buffer.from(data, 'utf8');
|
||||
} else {
|
||||
throw new Error('Text data must be a string');
|
||||
}
|
||||
} else if (payloadType === 'dictionary') {
|
||||
const jsonStr = JSON.stringify(data);
|
||||
return Buffer.from(jsonStr, 'utf8');
|
||||
} else if (payloadType === 'table') {
|
||||
// Convert array of objects to Arrow IPC format
|
||||
if (!Array.isArray(data) || data.length === 0) {
|
||||
throw new Error('Table data must be a non-empty array of objects');
|
||||
}
|
||||
|
||||
return serializeArrowTable(data);
|
||||
} else if (payloadType === 'image') {
|
||||
if (data instanceof Uint8Array || Buffer.isBuffer(data)) {
|
||||
return Buffer.from(data);
|
||||
} else {
|
||||
throw new Error('Image data must be Uint8Array or Buffer');
|
||||
}
|
||||
} else if (payloadType === 'audio') {
|
||||
if (data instanceof Uint8Array || Buffer.isBuffer(data)) {
|
||||
return Buffer.from(data);
|
||||
} else {
|
||||
throw new Error('Audio data must be Uint8Array or Buffer');
|
||||
}
|
||||
} else if (payloadType === 'video') {
|
||||
if (data instanceof Uint8Array || Buffer.isBuffer(data)) {
|
||||
return Buffer.from(data);
|
||||
} else {
|
||||
throw new Error('Video data must be Uint8Array or Buffer');
|
||||
}
|
||||
} else if (payloadType === 'binary') {
|
||||
if (data instanceof Uint8Array || Buffer.isBuffer(data)) {
|
||||
return Buffer.from(data);
|
||||
} else {
|
||||
throw new Error('Binary data must be Uint8Array or Buffer');
|
||||
}
|
||||
} else {
|
||||
throw new Error(`Unknown payload_type: ${payloadType}`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper function to properly serialize table data to Arrow IPC
|
||||
* @param {Array<Object>} data - Array of objects representing table rows
|
||||
* @returns {Buffer} Arrow IPC formatted buffer
|
||||
*/
|
||||
function serializeArrowTable(data) {
|
||||
if (!Array.isArray(data) || data.length === 0) {
|
||||
throw new Error('Table data must be a non-empty array of objects');
|
||||
}
|
||||
|
||||
// Build schema from first row
|
||||
const fields = Object.keys(data[0]).map(key => {
|
||||
const value = data[0][key];
|
||||
let arrowType;
|
||||
if (typeof value === 'number') {
|
||||
arrowType = Number.isInteger(value) ? arrow.Int64 : arrow.Float64;
|
||||
} else if (typeof value === 'boolean') {
|
||||
arrowType = arrow.Boolean;
|
||||
} else if (value instanceof Date) {
|
||||
arrowType = arrow.Date;
|
||||
} else {
|
||||
arrowType = arrow.Utf8;
|
||||
}
|
||||
return new arrow.Field(key, arrowType, true);
|
||||
});
|
||||
|
||||
const schema = new arrow.Schema(fields);
|
||||
const batches = [];
|
||||
|
||||
// Create record batches
|
||||
for (const row of data) {
|
||||
const batch = arrow.recordBatch.fromObjects([row], schema);
|
||||
batches.push(batch);
|
||||
}
|
||||
|
||||
// Write to buffer using IPC format
|
||||
const buffers = arrow.ipc.recordBatchesToMessage(batches, schema).buffers;
|
||||
const combined = new Uint8Array(buffers.reduce((acc, b) => acc + b.byteLength, 0));
|
||||
let offset = 0;
|
||||
for (const buf of buffers) {
|
||||
combined.set(new Uint8Array(buf), offset);
|
||||
offset += buf.byteLength;
|
||||
}
|
||||
|
||||
return Buffer.from(combined);
|
||||
}
|
||||
|
||||
/**
|
||||
* Deserialize bytes to data based on type
|
||||
* @param {Buffer|Uint8Array} data - Serialized data as bytes
|
||||
* @param {string} payloadType - Data type
|
||||
* @param {string} correlationId - Correlation ID for logging
|
||||
* @returns {any} Deserialized data
|
||||
*/
|
||||
async function deserializeData(data, payloadType, correlationId) {
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data);
|
||||
|
||||
if (payloadType === 'text') {
|
||||
return buffer.toString('utf8');
|
||||
} else if (payloadType === 'dictionary') {
|
||||
const jsonStr = buffer.toString('utf8');
|
||||
return JSON.parse(jsonStr);
|
||||
} else if (payloadType === 'table') {
|
||||
const table = arrow.tableFromRawBytes(buffer);
|
||||
return table;
|
||||
} else if (payloadType === 'image') {
|
||||
return buffer;
|
||||
} else if (payloadType === 'audio') {
|
||||
return buffer;
|
||||
} else if (payloadType === 'video') {
|
||||
return buffer;
|
||||
} else if (payloadType === 'binary') {
|
||||
return buffer;
|
||||
} else {
|
||||
throw new Error(`Unknown payload_type: ${payloadType}`);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------- File Server Handlers ---------------------------------------------- //
|
||||
|
||||
/**
|
||||
* Upload data to plik server in one-shot mode
|
||||
* @param {string} fileServerUrl - Base URL of the plik server
|
||||
* @param {string} dataname - Name of the file being uploaded
|
||||
* @param {Buffer|Uint8Array} data - Raw byte data of the file content
|
||||
* @returns {Promise<{status: number, uploadid: string, fileid: string, url: string}>}
|
||||
*/
|
||||
async function plikOneshotUpload(fileServerUrl, dataname, data) {
|
||||
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data);
|
||||
|
||||
// Get upload id
|
||||
const urlGetUploadID = `${fileServerUrl}/upload`;
|
||||
const headers = { 'Content-Type': 'application/json' };
|
||||
const body = JSON.stringify({ OneShot: true });
|
||||
|
||||
const httpResponse = await fetch(urlGetUploadID, {
|
||||
method: 'POST',
|
||||
headers,
|
||||
body
|
||||
});
|
||||
|
||||
const responseJson = await httpResponse.json();
|
||||
const uploadid = responseJson.id;
|
||||
const uploadtoken = responseJson.uploadToken;
|
||||
|
||||
// Upload file
|
||||
const urlUpload = `${fileServerUrl}/file/${uploadid}`;
|
||||
const form = new FormData();
|
||||
const blob = new Blob([buffer], { type: 'application/octet-stream' });
|
||||
form.append('file', blob, dataname);
|
||||
|
||||
const uploadHeaders = {
|
||||
'X-UploadToken': uploadtoken
|
||||
};
|
||||
|
||||
const uploadResponse = await fetch(urlUpload, {
|
||||
method: 'POST',
|
||||
headers: uploadHeaders,
|
||||
body: form
|
||||
});
|
||||
|
||||
const uploadJson = await uploadResponse.json();
|
||||
const fileid = uploadJson.id;
|
||||
|
||||
const url = `${fileServerUrl}/file/${uploadid}/${fileid}/${dataname}`;
|
||||
|
||||
return {
|
||||
status: uploadResponse.status,
|
||||
uploadid,
|
||||
fileid,
|
||||
url
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch data from URL with exponential backoff
|
||||
* @param {string} url - URL to fetch from
|
||||
* @param {number} maxRetries - Maximum number of retry attempts
|
||||
* @param {number} baseDelay - Initial delay in milliseconds
|
||||
* @param {number} maxDelay - Maximum delay in milliseconds
|
||||
* @param {string} correlationId - Correlation ID for logging
|
||||
* @returns {Promise<Uint8Array>} Fetched data as bytes
|
||||
*/
|
||||
async function fetchWithBackoff(url, maxRetries, baseDelay, maxDelay, correlationId) {
|
||||
let delay = baseDelay;
|
||||
|
||||
for (let attempt = 1; attempt <= maxRetries; attempt++) {
|
||||
try {
|
||||
const response = await fetch(url);
|
||||
|
||||
if (response.status === 200) {
|
||||
logTrace(correlationId, `Successfully fetched data from ${url} on attempt ${attempt}`);
|
||||
const arrayBuffer = await response.arrayBuffer();
|
||||
return new Uint8Array(arrayBuffer);
|
||||
} else {
|
||||
throw new Error(`Failed to fetch: ${response.status}`);
|
||||
}
|
||||
} catch (e) {
|
||||
logTrace(correlationId, `Attempt ${attempt} failed: ${e.constructor.name}`);
|
||||
|
||||
if (attempt < maxRetries) {
|
||||
await new Promise(resolve => setTimeout(resolve, delay));
|
||||
delay = Math.min(delay * 2, maxDelay);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error(`Failed to fetch data after ${maxRetries} attempts`);
|
||||
}
|
||||
|
||||
// ---------------------------------------------- NATS Client ---------------------------------------------- //
|
||||
|
||||
/**
|
||||
* NATS client wrapper for connection management
|
||||
*/
|
||||
class NATSClient {
|
||||
/**
|
||||
* Create a new NATS client
|
||||
* @param {string} url - NATS server URL
|
||||
*/
|
||||
constructor(url) {
|
||||
this.url = url;
|
||||
this.connection = null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Connect to NATS server
|
||||
* @returns {Promise<NATS.Connection>}
|
||||
*/
|
||||
async connect() {
|
||||
this.connection = await nats.connect({ servers: this.url });
|
||||
return this.connection;
|
||||
}
|
||||
|
||||
/**
|
||||
* Publish message to NATS subject
|
||||
* @param {string} subject - NATS subject to publish to
|
||||
* @param {string} message - Message to publish
|
||||
* @param {string} correlationId - Correlation ID for logging
|
||||
*/
|
||||
async publish(subject, message, correlationId) {
|
||||
if (!this.connection) {
|
||||
await this.connect();
|
||||
}
|
||||
await this.connection.publish(subject, message);
|
||||
logTrace(correlationId, `Message published to ${subject}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Close the NATS connection
|
||||
*/
|
||||
async close() {
|
||||
if (this.connection) {
|
||||
this.connection.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------- Core Functions ---------------------------------------------- //
|
||||
|
||||
/**
|
||||
* Publish message to NATS
|
||||
* @param {string|NATSClient|NATS.Connection} brokerUrlOrClient - NATS URL, client, or connection
|
||||
* @param {string} subject - NATS subject to publish to
|
||||
* @param {string} message - JSON message to publish
|
||||
* @param {string} correlationId - Correlation ID for tracing
|
||||
*/
|
||||
async function publishMessage(brokerUrlOrClient, subject, message, correlationId) {
|
||||
let conn;
|
||||
|
||||
if (brokerUrlOrClient instanceof NATSClient) {
|
||||
conn = brokerUrlOrClient;
|
||||
} else if (brokerUrlOrClient instanceof nats.Connection) {
|
||||
// Create a wrapper for direct connection
|
||||
conn = {
|
||||
async publish(subj, msg) {
|
||||
await brokerUrlOrClient.publish(subj, msg);
|
||||
},
|
||||
async close() {
|
||||
await brokerUrlOrClient.close();
|
||||
}
|
||||
};
|
||||
} else {
|
||||
// String URL - create new client
|
||||
const client = new NATSClient(brokerUrlOrClient);
|
||||
conn = client;
|
||||
}
|
||||
|
||||
await conn.publish(subject, message, correlationId);
|
||||
|
||||
if (conn instanceof NATSClient) {
|
||||
await conn.close();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build message envelope from payloads and metadata
|
||||
* @param {string} subject - NATS subject
|
||||
* @param {Array} payloads - Array of payload objects
|
||||
* @param {Object} options - Envelope metadata options
|
||||
* @returns {Object} Envelope object
|
||||
*/
|
||||
function buildEnvelope(subject, payloads, options) {
|
||||
return {
|
||||
correlation_id: options.correlation_id,
|
||||
msg_id: options.msg_id,
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: subject,
|
||||
msg_purpose: options.msg_purpose,
|
||||
sender_name: options.sender_name,
|
||||
sender_id: options.sender_id,
|
||||
receiver_name: options.receiver_name,
|
||||
receiver_id: options.receiver_id,
|
||||
reply_to: options.reply_to,
|
||||
reply_to_msg_id: options.reply_to_msg_id,
|
||||
broker_url: options.broker_url,
|
||||
metadata: options.metadata || {},
|
||||
payloads: payloads
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Build payload object from serialized data
|
||||
* @param {string} dataname - Name of the payload
|
||||
* @param {string} payloadType - Type of the payload
|
||||
* @param {Buffer} payloadBytes - Serialized payload bytes
|
||||
* @param {string} transport - Transport type ("direct" or "link")
|
||||
* @param {string} data - Data (base64 for direct, URL for link)
|
||||
* @returns {Object} Payload object
|
||||
*/
|
||||
function buildPayload(dataname, payloadType, payloadBytes, transport, data) {
|
||||
return {
|
||||
id: uuidv4(),
|
||||
dataname,
|
||||
payload_type: payloadType,
|
||||
transport,
|
||||
encoding: transport === 'direct' ? 'base64' : 'none',
|
||||
size: payloadBytes.byteLength,
|
||||
data,
|
||||
metadata: transport === 'direct' ? { payload_bytes: payloadBytes.byteLength } : {}
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Send data via NATS with automatic transport selection
|
||||
*
|
||||
* This function intelligently routes data delivery based on payload size.
|
||||
* If the serialized payload is smaller than size_threshold, it encodes the data as Base64
|
||||
* and publishes directly over NATS. Otherwise, it uploads the data to a fileserver
|
||||
* and publishes only the download URL over NATS.
|
||||
*
|
||||
* @param {string} subject - NATS subject to publish the message to
|
||||
* @param {Array} data - List of [dataname, data, type] tuples to send
|
||||
* @param {Object} options - Optional configuration
|
||||
* @param {string} [options.broker_url=DEFAULT_BROKER_URL] - URL of the NATS server
|
||||
* @param {string} [options.fileserver_url=DEFAULT_FILESERVER_URL] - URL of the HTTP file server
|
||||
* @param {Function} [options.fileserver_upload_handler=plikOneshotUpload] - Function to handle fileserver uploads
|
||||
* @param {number} [options.size_threshold=DEFAULT_SIZE_THRESHOLD] - Threshold separating direct vs link transport
|
||||
* @param {string} [options.correlation_id=uuidv4()] - Correlation ID for tracing
|
||||
* @param {string} [options.msg_purpose="chat"] - Purpose of the message
|
||||
* @param {string} [options.sender_name="NATSBridge"] - Name of the sender
|
||||
* @param {string} [options.receiver_name=""] - Name of the receiver (empty means broadcast)
|
||||
* @param {string} [options.receiver_id=""] - UUID of the receiver (empty means broadcast)
|
||||
* @param {string} [options.reply_to=""] - Topic to reply to
|
||||
* @param {string} [options.reply_to_msg_id=""] - Message ID this message is replying to
|
||||
* @param {boolean} [options.is_publish=true] - Whether to automatically publish the message
|
||||
* @param {NATSClient|NATS.Connection} [options.nats_connection=null] - Pre-existing NATS connection
|
||||
* @param {string} [options.msg_id=uuidv4()] - Message ID
|
||||
* @param {string} [options.sender_id=uuidv4()] - Sender ID
|
||||
* @returns {Promise<[Object, string]>} Tuple of [env, env_json_str]
|
||||
*
|
||||
* @example
|
||||
* // Send a single payload
|
||||
* const [env, envJsonStr] = await smartsend(
|
||||
* "/test",
|
||||
* [["dataname1", data1, "dictionary"]],
|
||||
* { broker_url: "nats://localhost:4222" }
|
||||
* );
|
||||
*
|
||||
* // Send multiple payloads
|
||||
* const [env, envJsonStr] = await smartsend(
|
||||
* "/test",
|
||||
* [
|
||||
* ["dataname1", data1, "dictionary"],
|
||||
* ["dataname2", data2, "table"]
|
||||
* ],
|
||||
* { broker_url: "nats://localhost:4222" }
|
||||
* );
|
||||
*
|
||||
* // Send with pre-existing connection
|
||||
* const client = await NATSBridge.NATSClient.connect("nats://localhost:4222");
|
||||
* const [env, envJsonStr] = await smartsend(
|
||||
* "/test",
|
||||
* [["data", myData, "text"]],
|
||||
* { nats_connection: client }
|
||||
* );
|
||||
*/
|
||||
async function smartsend(subject, data, options = {}) {
|
||||
const {
|
||||
broker_url = DEFAULT_BROKER_URL,
|
||||
fileserver_url = DEFAULT_FILESERVER_URL,
|
||||
fileserver_upload_handler = plikOneshotUpload,
|
||||
size_threshold = DEFAULT_SIZE_THRESHOLD,
|
||||
correlation_id = uuidv4(),
|
||||
msg_purpose = 'chat',
|
||||
sender_name = 'NATSBridge',
|
||||
receiver_name = '',
|
||||
receiver_id = '',
|
||||
reply_to = '',
|
||||
reply_to_msg_id = '',
|
||||
is_publish = true,
|
||||
nats_connection = null,
|
||||
msg_id = uuidv4(),
|
||||
sender_id = uuidv4()
|
||||
} = options;
|
||||
|
||||
logTrace(correlation_id, `Starting smartsend for subject: ${subject}`);
|
||||
|
||||
// Process payloads
|
||||
const payloads = [];
|
||||
for (const [dataname, payloadData, payloadType] of data) {
|
||||
const payloadBytes = await serializeData(payloadData, payloadType);
|
||||
const payloadSize = payloadBytes.byteLength;
|
||||
|
||||
logTrace(correlation_id, `Serialized payload '${dataname}' (type: ${payloadType}) size: ${payloadSize} bytes`);
|
||||
|
||||
if (payloadSize < size_threshold) {
|
||||
// Direct path
|
||||
const payloadB64 = bufferToBase64(payloadBytes);
|
||||
logTrace(correlation_id, `Using direct transport for ${payloadSize} bytes`);
|
||||
|
||||
const payload = buildPayload(dataname, payloadType, payloadBytes, 'direct', payloadB64);
|
||||
payloads.push(payload);
|
||||
} else {
|
||||
// Link path
|
||||
logTrace(correlation_id, `Using link transport, uploading to fileserver`);
|
||||
|
||||
const response = await fileserver_upload_handler(fileserver_url, dataname, payloadBytes);
|
||||
|
||||
if (response.status !== 200) {
|
||||
throw new Error(`Failed to upload data to fileserver: ${response.status}`);
|
||||
}
|
||||
|
||||
logTrace(correlation_id, `Uploaded to URL: ${response.url}`);
|
||||
|
||||
const payload = buildPayload(dataname, payloadType, payloadBytes, 'link', response.url);
|
||||
payloads.push(payload);
|
||||
}
|
||||
}
|
||||
|
||||
// Build envelope
|
||||
const env = buildEnvelope(subject, payloads, {
|
||||
correlation_id,
|
||||
msg_id,
|
||||
msg_purpose,
|
||||
sender_name,
|
||||
sender_id,
|
||||
receiver_name,
|
||||
receiver_id,
|
||||
reply_to,
|
||||
reply_to_msg_id,
|
||||
broker_url
|
||||
});
|
||||
|
||||
const env_json_str = JSON.stringify(env);
|
||||
|
||||
if (is_publish) {
|
||||
if (nats_connection) {
|
||||
await publishMessage(nats_connection, subject, env_json_str, correlation_id);
|
||||
} else {
|
||||
await publishMessage(broker_url, subject, env_json_str, correlation_id);
|
||||
}
|
||||
}
|
||||
|
||||
return [env, env_json_str];
|
||||
}
|
||||
|
||||
/**
|
||||
* Receive and process NATS message
|
||||
*
|
||||
* This function processes incoming NATS messages, handling both direct transport
|
||||
* (base64 decoded payloads) and link transport (URL-based payloads).
|
||||
* It deserializes the data based on the transport type and returns the result.
|
||||
*
|
||||
* @param {Object} msg - NATS message object with payload property
|
||||
* @param {Object} options - Optional configuration
|
||||
* @param {Function} [options.fileserver_download_handler=fetchWithBackoff] - Function to handle fileserver downloads
|
||||
* @param {number} [options.max_retries=5] - Maximum retry attempts for fetching URL
|
||||
* @param {number} [options.base_delay=100] - Initial delay for exponential backoff in ms
|
||||
* @param {number} [options.max_delay=5000] - Maximum delay for exponential backoff in ms
|
||||
* @returns {Promise<Object>} Envelope object with processed payloads
|
||||
*
|
||||
* @example
|
||||
* // Receive and process message
|
||||
* const env = await smartreceive(msg, {
|
||||
* fileserver_download_handler: fetchWithBackoff,
|
||||
* max_retries: 5,
|
||||
* base_delay: 100,
|
||||
* max_delay: 5000
|
||||
* });
|
||||
* // env.payloads is an Array of [dataname, data, type] arrays
|
||||
* for (const [dataname, data, type] of env.payloads) {
|
||||
* console.log(`${dataname}: ${data} (type: ${type})`);
|
||||
* }
|
||||
*/
|
||||
async function smartreceive(msg, options = {}) {
|
||||
const {
|
||||
fileserver_download_handler = fetchWithBackoff,
|
||||
max_retries = 5,
|
||||
base_delay = 100,
|
||||
max_delay = 5000
|
||||
} = options;
|
||||
|
||||
// Parse the JSON envelope
|
||||
const payload = typeof msg.payload === 'string' ? msg.payload : Buffer.from(msg.payload).toString('utf8');
|
||||
const envJsonObj = JSON.parse(payload);
|
||||
logTrace(envJsonObj.correlation_id, 'Processing received message');
|
||||
|
||||
// Process all payloads in the envelope
|
||||
const payloadsList = [];
|
||||
const numPayloads = envJsonObj.payloads.length;
|
||||
|
||||
for (let i = 0; i < numPayloads; i++) {
|
||||
const payloadObj = envJsonObj.payloads[i];
|
||||
const transport = payloadObj.transport;
|
||||
const dataname = payloadObj.dataname;
|
||||
|
||||
if (transport === 'direct') {
|
||||
logTrace(envJsonObj.correlation_id, `Direct transport - decoding payload '${dataname}'`);
|
||||
|
||||
// Extract base64 payload from the payload
|
||||
const payloadB64 = payloadObj.data;
|
||||
|
||||
// Decode Base64 payload
|
||||
const payloadBytes = Buffer.from(payloadB64, 'base64');
|
||||
|
||||
// Deserialize based on type
|
||||
const dataType = payloadObj.payload_type;
|
||||
const data = await deserializeData(payloadBytes, dataType, envJsonObj.correlation_id);
|
||||
|
||||
payloadsList.push([dataname, data, dataType]);
|
||||
} else if (transport === 'link') {
|
||||
// Extract download URL from the payload
|
||||
const url = payloadObj.data;
|
||||
logTrace(envJsonObj.correlation_id, `Link transport - fetching '${dataname}' from URL: ${url}`);
|
||||
|
||||
// Fetch with exponential backoff using the download handler
|
||||
const downloadedData = await fileserver_download_handler(
|
||||
url,
|
||||
max_retries,
|
||||
base_delay,
|
||||
max_delay,
|
||||
envJsonObj.correlation_id
|
||||
);
|
||||
|
||||
// Deserialize based on type
|
||||
const dataType = payloadObj.payload_type;
|
||||
const data = await deserializeData(downloadedData, dataType, envJsonObj.correlation_id);
|
||||
|
||||
payloadsList.push([dataname, data, dataType]);
|
||||
} else {
|
||||
throw new Error(`Unknown transport type for payload '${dataname}': ${transport}`);
|
||||
}
|
||||
}
|
||||
|
||||
envJsonObj.payloads = payloadsList;
|
||||
return envJsonObj;
|
||||
}
|
||||
|
||||
// ---------------------------------------------- Module Exports ---------------------------------------------- //
|
||||
|
||||
const NATSBridge = {
|
||||
/**
|
||||
* NATS client class for connection management
|
||||
*/
|
||||
NATSClient,
|
||||
|
||||
/**
|
||||
* Send data via NATS with automatic transport selection
|
||||
*/
|
||||
smartsend,
|
||||
|
||||
/**
|
||||
* Receive and process NATS message
|
||||
*/
|
||||
smartreceive,
|
||||
|
||||
/**
|
||||
* Upload data to plik server in one-shot mode
|
||||
*/
|
||||
plikOneshotUpload,
|
||||
|
||||
/**
|
||||
* Fetch data from URL with exponential backoff
|
||||
*/
|
||||
fetchWithBackoff,
|
||||
|
||||
/**
|
||||
* Default constants
|
||||
*/
|
||||
DEFAULT_SIZE_THRESHOLD,
|
||||
DEFAULT_BROKER_URL,
|
||||
DEFAULT_FILESERVER_URL
|
||||
};
|
||||
|
||||
module.exports = NATSBridge;
|
||||
815
src/natbridge.py
Normal file
815
src/natbridge.py
Normal file
@@ -0,0 +1,815 @@
|
||||
"""
|
||||
NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||
Python Desktop Implementation
|
||||
|
||||
This module provides functionality for sending and receiving data across network boundaries
|
||||
using NATS as the message bus, with support for both direct payload transport and
|
||||
URL-based transport for larger payloads.
|
||||
|
||||
@package natbridge
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from typing import Any, Callable, Dict, List, Tuple, Union
|
||||
import aiohttp
|
||||
|
||||
try:
|
||||
import pyarrow as arrow
|
||||
import pyarrow.ipc as ipc
|
||||
ARROW_AVAILABLE = True
|
||||
except ImportError:
|
||||
ARROW_AVAILABLE = False
|
||||
|
||||
try:
|
||||
import nats
|
||||
from nats.aio.client import Client as NATSClient
|
||||
NATS_AVAILABLE = True
|
||||
except ImportError:
|
||||
NATS_AVAILABLE = False
|
||||
|
||||
# ---------------------------------------------- Constants ---------------------------------------------- #
|
||||
|
||||
"""
|
||||
Default size threshold for switching from direct to link transport (1MB)
|
||||
"""
|
||||
DEFAULT_SIZE_THRESHOLD = 1_000_000
|
||||
|
||||
"""
|
||||
Default NATS server URL
|
||||
"""
|
||||
DEFAULT_BROKER_URL = "nats://localhost:4222"
|
||||
|
||||
"""
|
||||
Default HTTP file server URL for link transport
|
||||
"""
|
||||
DEFAULT_FILESERVER_URL = "http://localhost:8080"
|
||||
|
||||
|
||||
# ---------------------------------------------- Utility Functions ---------------------------------------------- #
|
||||
|
||||
def log_trace(correlation_id: str, message: str) -> None:
|
||||
"""
|
||||
Log a trace message with correlation ID and timestamp.
|
||||
|
||||
Args:
|
||||
correlation_id: Correlation ID for tracing
|
||||
message: Message content to log
|
||||
"""
|
||||
timestamp = datetime.utcnow().isoformat() + 'Z'
|
||||
print(f"[{timestamp}] [Correlation: {correlation_id}] {message}")
|
||||
|
||||
|
||||
# ---------------------------------------------- Serialization Functions ---------------------------------------------- #
|
||||
|
||||
def _serialize_data(data: Any, payload_type: str) -> bytes:
|
||||
"""
|
||||
Serialize data according to specified format.
|
||||
|
||||
Args:
|
||||
data: Data to serialize (string for "text", JSON-serializable for "dictionary",
|
||||
table-like for "table", binary for "image", "audio", "video", "binary")
|
||||
payload_type: Target format: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||
|
||||
Returns:
|
||||
Binary representation of the serialized data
|
||||
|
||||
Raises:
|
||||
Error: If payload_type is not one of the supported types
|
||||
Error: If payload_type is "image", "audio", or "video" but data is not bytes
|
||||
Error: If payload_type is "table" but data is not a pandas DataFrame or pyarrow Table
|
||||
"""
|
||||
if payload_type == 'text':
|
||||
if isinstance(data, str):
|
||||
return data.encode('utf-8')
|
||||
else:
|
||||
raise ValueError('Text data must be a string')
|
||||
elif payload_type == 'dictionary':
|
||||
json_str = json.dumps(data)
|
||||
return json_str.encode('utf-8')
|
||||
elif payload_type == 'table':
|
||||
if not ARROW_AVAILABLE:
|
||||
raise RuntimeError('pyarrow not available for table serialization')
|
||||
|
||||
import io
|
||||
buf = io.BytesIO()
|
||||
|
||||
import pandas as pd
|
||||
if isinstance(data, pd.DataFrame):
|
||||
table = arrow.Table.from_pandas(data)
|
||||
sink = ipc.new_file(buf, table.schema)
|
||||
ipc.write_table(table, sink)
|
||||
sink.close()
|
||||
return buf.getvalue()
|
||||
elif isinstance(data, arrow.Table):
|
||||
sink = ipc.new_file(buf, data.schema)
|
||||
ipc.write_table(data, sink)
|
||||
sink.close()
|
||||
return buf.getvalue()
|
||||
else:
|
||||
raise ValueError('Table data must be a pandas DataFrame or pyarrow Table')
|
||||
elif payload_type == 'image':
|
||||
if isinstance(data, (bytes, bytearray)):
|
||||
return bytes(data)
|
||||
else:
|
||||
raise ValueError('Image data must be bytes')
|
||||
elif payload_type == 'audio':
|
||||
if isinstance(data, (bytes, bytearray)):
|
||||
return bytes(data)
|
||||
else:
|
||||
raise ValueError('Audio data must be bytes')
|
||||
elif payload_type == 'video':
|
||||
if isinstance(data, (bytes, bytearray)):
|
||||
return bytes(data)
|
||||
else:
|
||||
raise ValueError('Video data must be bytes')
|
||||
elif payload_type == 'binary':
|
||||
if isinstance(data, (bytes, bytearray)):
|
||||
return bytes(data)
|
||||
else:
|
||||
raise ValueError('Binary data must be bytes')
|
||||
else:
|
||||
raise ValueError(f'Unknown payload_type: {payload_type}')
|
||||
|
||||
|
||||
def _deserialize_data(data: bytes, payload_type: str, correlation_id: str) -> Any:
|
||||
"""
|
||||
Deserialize bytes to data based on type.
|
||||
|
||||
Args:
|
||||
data: Serialized data as bytes
|
||||
payload_type: Data type ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
||||
correlation_id: Correlation ID for logging
|
||||
|
||||
Returns:
|
||||
Deserialized data (String for "text", DataFrame for "table", JSON data for "dictionary",
|
||||
bytes for "image", "audio", "video", "binary")
|
||||
|
||||
Raises:
|
||||
Error: If payload_type is not one of the supported types
|
||||
"""
|
||||
if payload_type == 'text':
|
||||
return data.decode('utf-8')
|
||||
elif payload_type == 'dictionary':
|
||||
json_str = data.decode('utf-8')
|
||||
return json.loads(json_str)
|
||||
elif payload_type == 'table':
|
||||
if not ARROW_AVAILABLE:
|
||||
raise RuntimeError('pyarrow not available for table deserialization')
|
||||
|
||||
import io
|
||||
buf = io.BytesIO(data)
|
||||
reader = ipc.open_file(buf)
|
||||
return reader.read_all().to_pandas()
|
||||
elif payload_type == 'image':
|
||||
return data
|
||||
elif payload_type == 'audio':
|
||||
return data
|
||||
elif payload_type == 'video':
|
||||
return data
|
||||
elif payload_type == 'binary':
|
||||
return data
|
||||
else:
|
||||
raise ValueError(f'Unknown payload_type: {payload_type}')
|
||||
|
||||
|
||||
# ---------------------------------------------- File Server Handlers ---------------------------------------------- #
|
||||
|
||||
async def plik_oneshot_upload(
|
||||
file_server_url: str,
|
||||
dataname: str,
|
||||
data: bytes
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Upload data to plik server in one-shot mode.
|
||||
|
||||
This function uploads a raw byte array to a plik server in one-shot mode (no upload session).
|
||||
It first creates a one-shot upload session by sending a POST request with {"OneShot": true},
|
||||
retrieves an upload ID and token, then uploads the file data as multipart form data using the token.
|
||||
|
||||
Args:
|
||||
file_server_url: Base URL of the plik server (e.g., "http://localhost:8080")
|
||||
dataname: Name of the file being uploaded
|
||||
data: Raw byte data of the file content
|
||||
|
||||
Returns:
|
||||
Dict with keys:
|
||||
- "status": HTTP server response status
|
||||
- "uploadid": ID of the one-shot upload session
|
||||
- "fileid": ID of the uploaded file within the session
|
||||
- "url": Full URL to download the uploaded file
|
||||
|
||||
Example:
|
||||
>>> fileserver_url = "http://localhost:8080"
|
||||
>>> dataname = "test.txt"
|
||||
>>> data = b"hello world"
|
||||
>>> result = await plik_oneshot_upload(file_server_url, dataname, data)
|
||||
>>> result["status"], result["uploadid"], result["fileid"], result["url"]
|
||||
"""
|
||||
async with aiohttp.ClientSession() as session:
|
||||
# Get upload id
|
||||
url_getUploadID = f"{file_server_url}/upload"
|
||||
headers = {'Content-Type': 'application/json'}
|
||||
body = json.dumps({"OneShot": True})
|
||||
|
||||
async with session.post(url_getUploadID, headers=headers, data=body) as response:
|
||||
response_json = await response.json()
|
||||
uploadid = response_json['id']
|
||||
uploadtoken = response_json['uploadToken']
|
||||
|
||||
# Upload file
|
||||
url_upload = f"{file_server_url}/file/{uploadid}"
|
||||
headers = {'X-UploadToken': uploadtoken}
|
||||
|
||||
form = aiohttp.FormData()
|
||||
form.add_field('file', data, filename=dataname, content_type='application/octet-stream')
|
||||
|
||||
async with session.post(url_upload, headers=headers, data=form) as upload_response:
|
||||
upload_json = await upload_response.json()
|
||||
fileid = upload_json['id']
|
||||
|
||||
url = f"{file_server_url}/file/{uploadid}/{fileid}/{dataname}"
|
||||
|
||||
return {
|
||||
'status': upload_response.status,
|
||||
'uploadid': uploadid,
|
||||
'fileid': fileid,
|
||||
'url': url
|
||||
}
|
||||
|
||||
|
||||
async def fetch_with_backoff(
|
||||
url: str,
|
||||
max_retries: int,
|
||||
base_delay: int,
|
||||
max_delay: int,
|
||||
correlation_id: str
|
||||
) -> bytes:
|
||||
"""
|
||||
Fetch data from URL with exponential backoff.
|
||||
|
||||
This internal function retrieves data from a URL with retry logic using
|
||||
exponential backoff to handle transient failures.
|
||||
|
||||
Args:
|
||||
url: URL to fetch from
|
||||
max_retries: Maximum number of retry attempts
|
||||
base_delay: Initial delay in milliseconds
|
||||
max_delay: Maximum delay in milliseconds
|
||||
correlation_id: Correlation ID for logging
|
||||
|
||||
Returns:
|
||||
Fetched data as bytes
|
||||
|
||||
Raises:
|
||||
Error: If all retry attempts fail
|
||||
|
||||
Example:
|
||||
>>> data = await fetch_with_backoff("http://example.com/file.zip", 5, 100, 5000, "correlation123")
|
||||
"""
|
||||
delay = base_delay
|
||||
|
||||
for attempt in range(1, max_retries + 1):
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url) as response:
|
||||
if response.status == 200:
|
||||
log_trace(correlation_id, f"Successfully fetched data from {url} on attempt {attempt}")
|
||||
return await response.read()
|
||||
else:
|
||||
raise Exception(f"Failed to fetch: {response.status}")
|
||||
except Exception as e:
|
||||
log_trace(correlation_id, f"Attempt {attempt} failed: {type(e).__name__}")
|
||||
|
||||
if attempt < max_retries:
|
||||
await asyncio.sleep(delay / 1000.0)
|
||||
delay = min(delay * 2, max_delay)
|
||||
|
||||
raise Exception(f"Failed to fetch data after {max_retries} attempts")
|
||||
|
||||
|
||||
# ---------------------------------------------- NATS Client ---------------------------------------------- #
|
||||
|
||||
class NATSClient:
|
||||
"""NATS client wrapper for connection management."""
|
||||
|
||||
def __init__(self, url: str = DEFAULT_BROKER_URL):
|
||||
"""
|
||||
Create a new NATS client.
|
||||
|
||||
Args:
|
||||
url: NATS server URL
|
||||
"""
|
||||
self.url = url
|
||||
self._client: NATSClient = None
|
||||
|
||||
async def connect(self) -> NATSClient:
|
||||
"""
|
||||
Connect to NATS server.
|
||||
|
||||
Returns:
|
||||
NATS client instance
|
||||
"""
|
||||
if NATS_AVAILABLE:
|
||||
self._client = nats.connect(self.url)
|
||||
await self._client
|
||||
else:
|
||||
raise Error('nats-py not available')
|
||||
return self._client
|
||||
|
||||
async def publish(self, subject: str, message: str, correlation_id: str = "") -> None:
|
||||
"""
|
||||
Publish message to NATS subject.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
message: Message to publish
|
||||
correlation_id: Correlation ID for logging
|
||||
"""
|
||||
if self._client:
|
||||
await self._client.publish(subject, message)
|
||||
if correlation_id:
|
||||
log_trace(correlation_id, f"Message published to {subject}")
|
||||
|
||||
async def close(self) -> None:
|
||||
"""Close the NATS connection."""
|
||||
if self._client:
|
||||
await self._client.drain()
|
||||
await self._client.close()
|
||||
|
||||
|
||||
# ---------------------------------------------- Core Functions ---------------------------------------------- #
|
||||
|
||||
def _build_envelope(
|
||||
subject: str,
|
||||
payloads: List[Dict[str, Any]],
|
||||
options: Dict[str, Any]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Build message envelope from payloads and metadata.
|
||||
|
||||
Args:
|
||||
subject: NATS subject
|
||||
payloads: Array of payload objects
|
||||
options: Envelope metadata options
|
||||
|
||||
Returns:
|
||||
Envelope object
|
||||
"""
|
||||
return {
|
||||
'correlation_id': options['correlation_id'],
|
||||
'msg_id': options['msg_id'],
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'send_to': subject,
|
||||
'msg_purpose': options['msg_purpose'],
|
||||
'sender_name': options['sender_name'],
|
||||
'sender_id': options['sender_id'],
|
||||
'receiver_name': options['receiver_name'],
|
||||
'receiver_id': options['receiver_id'],
|
||||
'reply_to': options['reply_to'],
|
||||
'reply_to_msg_id': options['reply_to_msg_id'],
|
||||
'broker_url': options['broker_url'],
|
||||
'metadata': options.get('metadata', {}),
|
||||
'payloads': payloads
|
||||
}
|
||||
|
||||
|
||||
def _build_payload(
|
||||
dataname: str,
|
||||
payload_type: str,
|
||||
payload_bytes: bytes,
|
||||
transport: str,
|
||||
data: Union[str, bytes]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Build payload object from serialized data.
|
||||
|
||||
Args:
|
||||
dataname: Name of the payload
|
||||
payload_type: Type of the payload
|
||||
payload_bytes: Serialized payload bytes
|
||||
transport: Transport type ("direct" or "link")
|
||||
data: Data (base64 for direct, URL for link)
|
||||
|
||||
Returns:
|
||||
Payload object
|
||||
"""
|
||||
return {
|
||||
'id': str(uuid.uuid4()),
|
||||
'dataname': dataname,
|
||||
'payload_type': payload_type,
|
||||
'transport': transport,
|
||||
'encoding': 'base64' if transport == 'direct' else 'none',
|
||||
'size': len(payload_bytes),
|
||||
'data': data,
|
||||
'metadata': {'payload_bytes': len(payload_bytes)} if transport == 'direct' else {}
|
||||
}
|
||||
|
||||
|
||||
async def publish_message(
|
||||
broker_url_or_client: Union[str, NATSClient, Any],
|
||||
subject: str,
|
||||
message: str,
|
||||
correlation_id: str
|
||||
) -> None:
|
||||
"""
|
||||
Publish message to NATS.
|
||||
|
||||
Args:
|
||||
broker_url_or_client: NATS URL, client, or connection
|
||||
subject: NATS subject to publish to
|
||||
message: JSON message to publish
|
||||
correlation_id: Correlation ID for tracing
|
||||
"""
|
||||
if isinstance(broker_url_or_client, NATSClient):
|
||||
client = broker_url_or_client
|
||||
elif NATS_AVAILABLE and hasattr(broker_url_or_client, 'publish'):
|
||||
# Direct NATS client connection
|
||||
await broker_url_or_client.publish(subject, message)
|
||||
log_trace(correlation_id, f"Message published to {subject}")
|
||||
return
|
||||
else:
|
||||
# String URL - create new client
|
||||
client = NATSClient(broker_url_or_client)
|
||||
await client.connect()
|
||||
|
||||
await client.publish(subject, message, correlation_id)
|
||||
|
||||
if isinstance(broker_url_or_client, NATSClient):
|
||||
await broker_url_or_client.close()
|
||||
elif not (NATS_AVAILABLE and hasattr(broker_url_or_client, 'publish')):
|
||||
await client.close()
|
||||
|
||||
|
||||
async def smartsend(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
broker_url: str = DEFAULT_BROKER_URL,
|
||||
fileserver_url: str = DEFAULT_FILESERVER_URL,
|
||||
fileserver_upload_handler: Callable = plik_oneshot_upload,
|
||||
size_threshold: int = DEFAULT_SIZE_THRESHOLD,
|
||||
correlation_id: str = None,
|
||||
msg_purpose: str = "chat",
|
||||
sender_name: str = "NATSBridge",
|
||||
receiver_name: str = "",
|
||||
receiver_id: str = "",
|
||||
reply_to: str = "",
|
||||
reply_to_msg_id: str = "",
|
||||
is_publish: bool = True,
|
||||
nats_connection: Any = None,
|
||||
msg_id: str = None,
|
||||
sender_id: str = None
|
||||
) -> Tuple[Dict, str]:
|
||||
"""
|
||||
Send data via NATS with automatic transport selection.
|
||||
|
||||
This function intelligently routes data delivery based on payload size.
|
||||
If the serialized payload is smaller than size_threshold, it encodes the data as Base64
|
||||
and publishes directly over NATS. Otherwise, it uploads the data to a fileserver
|
||||
and publishes only the download URL over NATS.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish the message to
|
||||
data: List of (dataname, data, type) tuples to send
|
||||
- dataname: Name of the payload
|
||||
- data: The actual data to send
|
||||
- type: Payload type: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||
broker_url: URL of the NATS server
|
||||
fileserver_url: URL of the HTTP file server for large payloads
|
||||
fileserver_upload_handler: Function to handle fileserver uploads (must return Dict with "status",
|
||||
"uploadid", "fileid", "url" keys)
|
||||
size_threshold: Threshold in bytes separating direct vs link transport
|
||||
correlation_id: Correlation ID for tracing (auto-generated UUID if not provided)
|
||||
msg_purpose: Purpose of the message: "ACK", "NACK", "updateStatus", "shutdown", "chat", etc.
|
||||
sender_name: Name of the sender
|
||||
receiver_name: Name of the receiver (empty string means broadcast)
|
||||
receiver_id: UUID of the receiver (empty string means broadcast)
|
||||
reply_to: Topic to reply to (empty string if no reply expected)
|
||||
reply_to_msg_id: Message ID this message is replying to
|
||||
is_publish: Whether to automatically publish the message to NATS
|
||||
nats_connection: Pre-existing NATS connection (if provided, uses this connection instead of
|
||||
creating a new one; saves connection establishment overhead)
|
||||
msg_id: Message ID (auto-generated UUID if not provided)
|
||||
sender_id: Sender ID (auto-generated UUID if not provided)
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str) where:
|
||||
- env: Dict containing all metadata and payloads
|
||||
- env_json_str: JSON string for publishing to NATS
|
||||
|
||||
Example:
|
||||
>>> # Send a single payload (still wrapped in a list)
|
||||
>>> data = {"key": "value"}
|
||||
>>> env, env_json_str = await smartsend(
|
||||
... "my.subject",
|
||||
... [("dataname1", data, "dictionary")],
|
||||
... broker_url="nats://localhost:4222"
|
||||
... )
|
||||
>>>
|
||||
>>> # Send multiple payloads with different types
|
||||
>>> data1 = {"key1": "value1"}
|
||||
>>> data2 = [1, 2, 3, 4, 5]
|
||||
>>> env, env_json_str = await smartsend(
|
||||
... "my.subject",
|
||||
... [("dataname1", data1, "dictionary"), ("dataname2", data2, "table")]
|
||||
... )
|
||||
>>>
|
||||
>>> # Send a large array using fileserver upload
|
||||
>>> data = list(range(10_000_000)) # ~80 MB
|
||||
>>> env, env_json_str = await smartsend(
|
||||
... "large.data",
|
||||
... [("large_table", data, "table")]
|
||||
... )
|
||||
>>>
|
||||
>>> # Mixed content (e.g., chat with text and image)
|
||||
>>> env, env_json_str = await smartsend(
|
||||
... "chat.subject",
|
||||
... [
|
||||
... ("message_text", "Hello!", "text"),
|
||||
... ("user_image", image_data, "image"),
|
||||
... ("audio_clip", audio_data, "audio")
|
||||
... ]
|
||||
... )
|
||||
>>>
|
||||
>>> # Publish the JSON string directly using NATS request-reply pattern
|
||||
>>> # reply = await nats.request(broker_url, subject, env_json_str, reply_to=reply_to_topic)
|
||||
"""
|
||||
if correlation_id is None:
|
||||
correlation_id = str(uuid.uuid4())
|
||||
if msg_id is None:
|
||||
msg_id = str(uuid.uuid4())
|
||||
if sender_id is None:
|
||||
sender_id = str(uuid.uuid4())
|
||||
|
||||
log_trace(correlation_id, f"Starting smartsend for subject: {subject}")
|
||||
|
||||
# Process payloads
|
||||
payloads = []
|
||||
for dataname, payload_data, payload_type in data:
|
||||
payload_bytes = _serialize_data(payload_data, payload_type)
|
||||
payload_size = len(payload_bytes)
|
||||
|
||||
log_trace(correlation_id, f"Serialized payload '{dataname}' (type: {payload_type}) size: {payload_size} bytes")
|
||||
|
||||
if payload_size < size_threshold:
|
||||
# Direct path
|
||||
payload_b64 = base64.b64encode(payload_bytes).decode('utf-8')
|
||||
log_trace(correlation_id, f"Using direct transport for {payload_size} bytes")
|
||||
|
||||
payload = _build_payload(dataname, payload_type, payload_bytes, 'direct', payload_b64)
|
||||
payloads.append(payload)
|
||||
else:
|
||||
# Link path
|
||||
log_trace(correlation_id, "Using link transport, uploading to fileserver")
|
||||
|
||||
response = await fileserver_upload_handler(fileserver_url, dataname, payload_bytes)
|
||||
|
||||
if response['status'] != 200:
|
||||
raise Exception(f"Failed to upload data to fileserver: {response['status']}")
|
||||
|
||||
log_trace(correlation_id, f"Uploaded to URL: {response['url']}")
|
||||
|
||||
payload = _build_payload(dataname, payload_type, payload_bytes, 'link', response['url'])
|
||||
payloads.append(payload)
|
||||
|
||||
# Build envelope
|
||||
env = _build_envelope(subject, payloads, {
|
||||
'correlation_id': correlation_id,
|
||||
'msg_id': msg_id,
|
||||
'msg_purpose': msg_purpose,
|
||||
'sender_name': sender_name,
|
||||
'sender_id': sender_id,
|
||||
'receiver_name': receiver_name,
|
||||
'receiver_id': receiver_id,
|
||||
'reply_to': reply_to,
|
||||
'reply_to_msg_id': reply_to_msg_id,
|
||||
'broker_url': broker_url
|
||||
})
|
||||
|
||||
env_json_str = json.dumps(env)
|
||||
|
||||
if is_publish:
|
||||
if nats_connection:
|
||||
await publish_message(nats_connection, subject, env_json_str, correlation_id)
|
||||
else:
|
||||
await publish_message(broker_url, subject, env_json_str, correlation_id)
|
||||
|
||||
return env, env_json_str
|
||||
|
||||
|
||||
async def smartreceive(
|
||||
msg: Any,
|
||||
fileserver_download_handler: Callable = fetch_with_backoff,
|
||||
max_retries: int = 5,
|
||||
base_delay: int = 100,
|
||||
max_delay: int = 5000
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Receive and process NATS messages.
|
||||
|
||||
This function processes incoming NATS messages, handling both direct transport
|
||||
(base64 decoded payloads) and link transport (URL-based payloads).
|
||||
It deserializes the data based on the transport type and returns the result.
|
||||
|
||||
Args:
|
||||
msg: NATS message to process
|
||||
fileserver_download_handler: Function to handle downloading data from file server URLs
|
||||
max_retries: Maximum retry attempts for fetching URL
|
||||
base_delay: Initial delay for exponential backoff in ms
|
||||
max_delay: Maximum delay for exponential backoff in ms
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads field containing List[Tuple[str, Any, str]]
|
||||
|
||||
Example:
|
||||
>>> # Receive and process message
|
||||
>>> env = await smartreceive(msg, fileserver_download_handler=fetch_with_backoff)
|
||||
>>> # env is a Dict with "payloads" key containing List[Tuple[str, Any, str]]
|
||||
>>> # Access payloads: for dataname, data, type_ in env["payloads"]
|
||||
>>> for dataname, data, type_ in env["payloads"]:
|
||||
>>> print(f"{dataname}: {data} (type: {type_})")
|
||||
"""
|
||||
# Parse the JSON envelope
|
||||
if isinstance(msg, dict):
|
||||
# Already parsed
|
||||
env_json_obj = msg
|
||||
elif hasattr(msg, 'payload'):
|
||||
# NATS message object
|
||||
payload = msg.payload if isinstance(msg.payload, str) else msg.payload.decode('utf-8')
|
||||
env_json_obj = json.loads(payload)
|
||||
else:
|
||||
# Assume it's already a JSON string or dict
|
||||
env_json_obj = json.loads(msg) if isinstance(msg, str) else msg
|
||||
|
||||
log_trace(env_json_obj['correlation_id'], "Processing received message")
|
||||
|
||||
# Process all payloads in the envelope
|
||||
payloads_list = []
|
||||
num_payloads = len(env_json_obj['payloads'])
|
||||
|
||||
for i in range(num_payloads):
|
||||
payload_obj = env_json_obj['payloads'][i]
|
||||
transport = payload_obj['transport']
|
||||
dataname = payload_obj['dataname']
|
||||
|
||||
if transport == 'direct':
|
||||
log_trace(env_json_obj['correlation_id'], f"Direct transport - decoding payload '{dataname}'")
|
||||
|
||||
# Extract base64 payload from the payload
|
||||
payload_b64 = payload_obj['data']
|
||||
|
||||
# Decode Base64 payload
|
||||
payload_bytes = base64.b64decode(payload_b64)
|
||||
|
||||
# Deserialize based on type
|
||||
data_type = payload_obj['payload_type']
|
||||
data = _deserialize_data(payload_bytes, data_type, env_json_obj['correlation_id'])
|
||||
|
||||
payloads_list.append((dataname, data, data_type))
|
||||
elif transport == 'link':
|
||||
# Extract download URL from the payload
|
||||
url = payload_obj['data']
|
||||
log_trace(env_json_obj['correlation_id'], f"Link transport - fetching '{dataname}' from URL: {url}")
|
||||
|
||||
# Fetch with exponential backoff using the download handler
|
||||
downloaded_data = await fileserver_download_handler(
|
||||
url,
|
||||
max_retries,
|
||||
base_delay,
|
||||
max_delay,
|
||||
env_json_obj['correlation_id']
|
||||
)
|
||||
|
||||
# Deserialize based on type
|
||||
data_type = payload_obj['payload_type']
|
||||
data = _deserialize_data(downloaded_data, data_type, env_json_obj['correlation_id'])
|
||||
|
||||
payloads_list.append((dataname, data, data_type))
|
||||
else:
|
||||
raise Exception(f"Unknown transport type for payload '{dataname}': {transport}")
|
||||
|
||||
env_json_obj['payloads'] = payloads_list
|
||||
return env_json_obj
|
||||
|
||||
|
||||
# ---------------------------------------------- Module Exports ---------------------------------------------- #
|
||||
|
||||
class NATSBridge:
|
||||
"""
|
||||
Cross-platform NATS bridge implementation.
|
||||
|
||||
This class provides a convenient interface for NATSBridge functionality,
|
||||
encapsulating the main functions and providing a class-based API.
|
||||
"""
|
||||
|
||||
DEFAULT_SIZE_THRESHOLD = DEFAULT_SIZE_THRESHOLD
|
||||
DEFAULT_BROKER_URL = DEFAULT_BROKER_URL
|
||||
DEFAULT_FILESERVER_URL = DEFAULT_FILESERVER_URL
|
||||
|
||||
def __init__(self, broker_url: str = None, fileserver_url: str = None):
|
||||
"""
|
||||
Initialize NATSBridge.
|
||||
|
||||
Args:
|
||||
broker_url: NATS server URL (defaults to DEFAULT_BROKER_URL)
|
||||
fileserver_url: HTTP file server URL (defaults to DEFAULT_FILESERVER_URL)
|
||||
"""
|
||||
self.broker_url = broker_url or self.DEFAULT_BROKER_URL
|
||||
self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL
|
||||
|
||||
async def smartsend(
|
||||
self,
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
**kwargs
|
||||
) -> Tuple[Dict, str]:
|
||||
"""
|
||||
Send data via NATS.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
data: List of (dataname, data, type) tuples
|
||||
**kwargs: Additional options passed to smartsend
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str)
|
||||
"""
|
||||
kwargs['broker_url'] = kwargs.get('broker_url', self.broker_url)
|
||||
kwargs['fileserver_url'] = kwargs.get('fileserver_url', self.fileserver_url)
|
||||
return await smartsend(subject, data, **kwargs)
|
||||
|
||||
async def smartreceive(
|
||||
self,
|
||||
msg: Any,
|
||||
**kwargs
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Receive and process NATS message.
|
||||
|
||||
Args:
|
||||
msg: NATS message to process
|
||||
**kwargs: Additional options passed to smartreceive
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads
|
||||
"""
|
||||
return await smartreceive(msg, **kwargs)
|
||||
|
||||
|
||||
# Convenience functions for module-level usage
|
||||
def send(
|
||||
subject: str,
|
||||
data: List[Tuple[str, Any, str]],
|
||||
**kwargs
|
||||
) -> Tuple[Dict, str]:
|
||||
"""
|
||||
Convenience function for sending data.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
data: List of (dataname, data, type) tuples
|
||||
**kwargs: Additional options
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str)
|
||||
"""
|
||||
return asyncio.run(smartsend(subject, data, **kwargs))
|
||||
|
||||
|
||||
def receive(
|
||||
msg: Any,
|
||||
**kwargs
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Convenience function for receiving messages.
|
||||
|
||||
Args:
|
||||
msg: NATS message to process
|
||||
**kwargs: Additional options
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads
|
||||
"""
|
||||
return asyncio.run(smartreceive(msg, **kwargs))
|
||||
|
||||
|
||||
__all__ = [
|
||||
'smartsend',
|
||||
'smartreceive',
|
||||
'plik_oneshot_upload',
|
||||
'fetch_with_backoff',
|
||||
'NATSBridge',
|
||||
'send',
|
||||
'receive',
|
||||
'DEFAULT_SIZE_THRESHOLD',
|
||||
'DEFAULT_BROKER_URL',
|
||||
'DEFAULT_FILESERVER_URL',
|
||||
'NATSClient',
|
||||
'_serialize_data',
|
||||
'_deserialize_data',
|
||||
'log_trace',
|
||||
'publish_message'
|
||||
]
|
||||
673
src/natbridge_mpy.py
Normal file
673
src/natbridge_mpy.py
Normal file
@@ -0,0 +1,673 @@
|
||||
"""
|
||||
NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||
MicroPython Implementation
|
||||
|
||||
This module provides functionality for sending and receiving data across network boundaries
|
||||
using NATS as the message bus, with support for both direct payload transport and
|
||||
URL-based transport for larger payloads.
|
||||
|
||||
Note: MicroPython has significant constraints compared to desktop implementations:
|
||||
- Limited memory (~256KB - 1MB)
|
||||
- No Arrow IPC support (memory constraints)
|
||||
- Synchronous API (no async/await)
|
||||
- Lower size threshold for direct transport
|
||||
"""
|
||||
|
||||
import network
|
||||
import time
|
||||
import json
|
||||
import base64
|
||||
import uos
|
||||
import struct
|
||||
import random
|
||||
|
||||
# ---------------------------------------------- Constants ---------------------------------------------- #
|
||||
|
||||
"""
|
||||
Default size threshold for switching from direct to link transport (100KB for MicroPython)
|
||||
"""
|
||||
DEFAULT_SIZE_THRESHOLD = 100000
|
||||
|
||||
"""
|
||||
Default NATS server URL
|
||||
"""
|
||||
DEFAULT_BROKER_URL = "nats://localhost:4222"
|
||||
|
||||
"""
|
||||
Default HTTP file server URL for link transport
|
||||
"""
|
||||
DEFAULT_FILESERVER_URL = "http://localhost:8080"
|
||||
|
||||
"""
|
||||
Hard limit for payload size in MicroPython (50KB)
|
||||
"""
|
||||
MAX_PAYLOAD_SIZE = 50000
|
||||
|
||||
|
||||
# ---------------------------------------------- Utility Functions ---------------------------------------------- #
|
||||
|
||||
def log_trace(correlation_id, message):
|
||||
"""
|
||||
Log a trace message with correlation ID and timestamp.
|
||||
|
||||
Args:
|
||||
correlation_id: Correlation ID for tracing
|
||||
message: Message content to log
|
||||
"""
|
||||
timestamp = time.strftime('%Y-%m-%dT%H:%M:%SZ', time.localtime())
|
||||
print(f"[{timestamp}] [Correlation: {correlation_id}] {message}")
|
||||
|
||||
|
||||
def _generate_uuid():
|
||||
"""
|
||||
Generate a simple UUID compatible with MicroPython.
|
||||
|
||||
Returns:
|
||||
UUID string
|
||||
"""
|
||||
# Generate a simple UUID-like string
|
||||
# Format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
|
||||
hex_chars = '0123456789abcdef'
|
||||
uuid_str = ''.join([random.choice(hex_chars) for _ in range(32)])
|
||||
# Insert hyphens at proper positions
|
||||
return f"{uuid_str[:8]}-{uuid_str[8:12]}-{uuid_str[12:16]}-{uuid_str[16:20]}-{uuid_str[20:]}"
|
||||
|
||||
|
||||
# ---------------------------------------------- Serialization Functions ---------------------------------------------- #
|
||||
|
||||
def _serialize_data(data, payload_type):
|
||||
"""
|
||||
Serialize data according to specified format.
|
||||
|
||||
Args:
|
||||
data: Data to serialize (string for "text", dict for "dictionary",
|
||||
bytes for "image", "audio", "video", "binary")
|
||||
payload_type: Target format: "text", "dictionary", "image", "audio", "video", "binary"
|
||||
|
||||
Returns:
|
||||
Binary representation of the serialized data
|
||||
|
||||
Note:
|
||||
MicroPython does not support "table" type due to memory constraints.
|
||||
|
||||
Raises:
|
||||
ValueError: If payload_type is not one of the supported types
|
||||
"""
|
||||
if payload_type == 'text':
|
||||
if isinstance(data, str):
|
||||
return data.encode('utf-8')
|
||||
else:
|
||||
raise ValueError('Text data must be a string')
|
||||
elif payload_type == 'dictionary':
|
||||
json_str = json.dumps(data)
|
||||
return json_str.encode('utf-8')
|
||||
elif payload_type in ('image', 'audio', 'video', 'binary'):
|
||||
if isinstance(data, (bytes, bytearray, memoryview)):
|
||||
return bytes(data)
|
||||
else:
|
||||
raise ValueError(f'{payload_type} data must be bytes')
|
||||
else:
|
||||
raise ValueError(f'Unknown payload_type: {payload_type}')
|
||||
|
||||
|
||||
def _deserialize_data(data, payload_type):
|
||||
"""
|
||||
Deserialize bytes to data based on type.
|
||||
|
||||
Args:
|
||||
data: Serialized data as bytes
|
||||
payload_type: Data type ("text", "dictionary", "image", "audio", "video", "binary")
|
||||
|
||||
Returns:
|
||||
Deserialized data (String for "text", dict for "dictionary", bytes for others)
|
||||
|
||||
Note:
|
||||
MicroPython does not support "table" type due to memory constraints.
|
||||
|
||||
Raises:
|
||||
ValueError: If payload_type is not one of the supported types
|
||||
"""
|
||||
if payload_type == 'text':
|
||||
return data.decode('utf-8')
|
||||
elif payload_type == 'dictionary':
|
||||
json_str = data.decode('utf-8')
|
||||
return json.loads(json_str)
|
||||
elif payload_type in ('image', 'audio', 'video', 'binary'):
|
||||
return data
|
||||
else:
|
||||
raise ValueError(f'Unknown payload_type: {payload_type}')
|
||||
|
||||
|
||||
# ---------------------------------------------- File Server Handlers ---------------------------------------------- #
|
||||
|
||||
def _sync_fileserver_upload(file_server_url, dataname, data):
|
||||
"""
|
||||
Synchronous file upload to HTTP server.
|
||||
|
||||
Note:
|
||||
This is a simplified implementation for MicroPython.
|
||||
In practice, would use network.HTTP or similar.
|
||||
Currently raises NotImplementedError as file upload is not fully supported.
|
||||
|
||||
Args:
|
||||
file_server_url: Base URL of the file server
|
||||
dataname: Name of the file being uploaded
|
||||
data: Raw byte data of the file content
|
||||
|
||||
Returns:
|
||||
Dict with keys: 'status', 'url'
|
||||
|
||||
Raises:
|
||||
NotImplementedError: File upload is not implemented in MicroPython
|
||||
"""
|
||||
raise NotImplementedError("File upload not fully implemented in MicroPython. "
|
||||
"Use direct transport only for memory-constrained devices.")
|
||||
|
||||
|
||||
def _sync_fileserver_download(url, max_retries, base_delay, max_delay, correlation_id):
|
||||
"""
|
||||
Synchronous file download with exponential backoff.
|
||||
|
||||
Note:
|
||||
This is a simplified implementation for MicroPython.
|
||||
In practice, would use network.HTTP or similar.
|
||||
Currently raises NotImplementedError as file download is not fully supported.
|
||||
|
||||
Args:
|
||||
url: URL to download from
|
||||
max_retries: Maximum retry attempts
|
||||
base_delay: Initial delay in ms
|
||||
max_delay: Maximum delay in ms
|
||||
correlation_id: Correlation ID for logging
|
||||
|
||||
Returns:
|
||||
Downloaded bytes
|
||||
|
||||
Raises:
|
||||
NotImplementedError: File download is not implemented in MicroPython
|
||||
"""
|
||||
raise NotImplementedError("File download not fully implemented in MicroPython. "
|
||||
"Use direct transport only for memory-constrained devices.")
|
||||
|
||||
|
||||
# ---------------------------------------------- NATS Client ---------------------------------------------- #
|
||||
|
||||
class NATSClient:
|
||||
"""
|
||||
NATS client wrapper for MicroPython.
|
||||
|
||||
Note:
|
||||
This is a simplified implementation for MicroPython.
|
||||
Full NATS client implementation would require additional network stack support.
|
||||
"""
|
||||
|
||||
def __init__(self, url=DEFAULT_BROKER_URL):
|
||||
"""
|
||||
Initialize NATS client.
|
||||
|
||||
Args:
|
||||
url: NATS server URL
|
||||
"""
|
||||
self.url = url
|
||||
self._connected = False
|
||||
|
||||
def connect(self):
|
||||
"""
|
||||
Connect to NATS server.
|
||||
|
||||
Note:
|
||||
This is a placeholder implementation.
|
||||
Actual NATS client would require network stack support.
|
||||
|
||||
Returns:
|
||||
True if connected, False otherwise
|
||||
"""
|
||||
# Placeholder - actual implementation would connect to NATS server
|
||||
self._connected = True
|
||||
return self._connected
|
||||
|
||||
def publish(self, subject, message):
|
||||
"""
|
||||
Publish message to NATS subject.
|
||||
|
||||
Note:
|
||||
This is a placeholder implementation.
|
||||
Actual NATS client would require network stack support.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
message: Message to publish
|
||||
"""
|
||||
if not self._connected:
|
||||
raise RuntimeError("Not connected to NATS server")
|
||||
# Placeholder - actual implementation would publish to NATS
|
||||
print(f"[NATS] Publish to {subject}: {message[:50]}...")
|
||||
|
||||
def close(self):
|
||||
"""Close the NATS connection."""
|
||||
self._connected = False
|
||||
|
||||
|
||||
# ---------------------------------------------- Core Functions ---------------------------------------------- #
|
||||
|
||||
def _build_envelope(subject, payloads, options):
|
||||
"""
|
||||
Build message envelope from payloads and metadata.
|
||||
|
||||
Args:
|
||||
subject: NATS subject
|
||||
payloads: Array of payload objects
|
||||
options: Envelope metadata options
|
||||
|
||||
Returns:
|
||||
Envelope dict
|
||||
"""
|
||||
return {
|
||||
'correlation_id': options['correlation_id'],
|
||||
'msg_id': options['msg_id'],
|
||||
'timestamp': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.localtime()),
|
||||
'send_to': subject,
|
||||
'msg_purpose': options['msg_purpose'],
|
||||
'sender_name': options['sender_name'],
|
||||
'sender_id': options['sender_id'],
|
||||
'receiver_name': options['receiver_name'],
|
||||
'receiver_id': options['receiver_id'],
|
||||
'reply_to': options['reply_to'],
|
||||
'reply_to_msg_id': options['reply_to_msg_id'],
|
||||
'broker_url': options['broker_url'],
|
||||
'metadata': {},
|
||||
'payloads': payloads
|
||||
}
|
||||
|
||||
|
||||
def _build_payload(dataname, payload_type, payload_bytes, transport, data):
|
||||
"""
|
||||
Build payload object from serialized data.
|
||||
|
||||
Args:
|
||||
dataname: Name of the payload
|
||||
payload_type: Type of the payload
|
||||
payload_bytes: Serialized payload bytes
|
||||
transport: Transport type ("direct" or "link")
|
||||
data: Data (base64 for direct, URL for link)
|
||||
|
||||
Returns:
|
||||
Payload dict
|
||||
"""
|
||||
return {
|
||||
'id': _generate_uuid(),
|
||||
'dataname': dataname,
|
||||
'payload_type': payload_type,
|
||||
'transport': transport,
|
||||
'encoding': 'base64' if transport == 'direct' else 'none',
|
||||
'size': len(payload_bytes),
|
||||
'data': data,
|
||||
'metadata': {'payload_bytes': len(payload_bytes)} if transport == 'direct' else {}
|
||||
}
|
||||
|
||||
|
||||
def _publish(subject, message, correlation_id):
|
||||
"""
|
||||
Publish message to NATS.
|
||||
|
||||
Note:
|
||||
This is a simplified implementation for MicroPython.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
message: JSON message to publish
|
||||
correlation_id: Correlation ID for logging
|
||||
"""
|
||||
log_trace(correlation_id, f"Publishing to {subject}")
|
||||
# Placeholder - actual implementation would use NATSClient
|
||||
# client = NATSClient()
|
||||
# client.connect()
|
||||
# client.publish(subject, message)
|
||||
# client.close()
|
||||
|
||||
|
||||
def smartsend(subject, data, **kwargs):
|
||||
"""
|
||||
Send data via NATS with automatic transport selection.
|
||||
|
||||
This function intelligently routes data delivery based on payload size.
|
||||
If the serialized payload is smaller than size_threshold, it encodes the data as Base64
|
||||
and publishes directly over NATS. Otherwise, it uploads the data to a fileserver
|
||||
and publishes only the download URL over NATS.
|
||||
|
||||
Note:
|
||||
MicroPython has memory constraints, so the default size_threshold is lower (100KB).
|
||||
Table type is not supported due to memory constraints.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish the message to
|
||||
data: List of (dataname, data, type) tuples to send
|
||||
- dataname: Name of the payload
|
||||
- data: The actual data to send
|
||||
- type: Payload type: "text", "dictionary", "image", "audio", "video", "binary"
|
||||
broker_url: NATS server URL (default: DEFAULT_BROKER_URL)
|
||||
fileserver_url: HTTP file server URL (default: DEFAULT_FILESERVER_URL)
|
||||
fileserver_upload_handler: Function to handle fileserver uploads (default: _sync_fileserver_upload)
|
||||
size_threshold: Threshold in bytes separating direct vs link transport (default: 100000)
|
||||
correlation_id: Correlation ID for tracing (auto-generated if not provided)
|
||||
msg_purpose: Purpose of the message (default: "chat")
|
||||
sender_name: Name of the sender (default: "NATSBridge")
|
||||
receiver_name: Name of the receiver (empty means broadcast)
|
||||
receiver_id: UUID of the receiver (empty means broadcast)
|
||||
reply_to: Topic to reply to (empty if no reply expected)
|
||||
reply_to_msg_id: Message ID this message is replying to
|
||||
is_publish: Whether to automatically publish the message (default: True)
|
||||
msg_id: Message ID (auto-generated if not provided)
|
||||
sender_id: Sender ID (auto-generated if not provided)
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str) where:
|
||||
- env: Dict containing all metadata and payloads
|
||||
- env_json_str: JSON string for publishing to NATS
|
||||
|
||||
Example:
|
||||
>>> # Send text payload
|
||||
>>> env, env_json_str = NATSBridge.smartsend(
|
||||
... "/chat",
|
||||
... [("message", "Hello!", "text")],
|
||||
... broker_url="nats://localhost:4222"
|
||||
... )
|
||||
>>>
|
||||
>>> # Send dictionary payload
|
||||
>>> env, env_json_str = NATSBridge.smartsend(
|
||||
... "/config",
|
||||
... [("config", {"key": "value"}, "dictionary")],
|
||||
... broker_url="nats://localhost:4222"
|
||||
... )
|
||||
>>>
|
||||
>>> # Send binary payload (image, audio, video)
|
||||
>>> env, env_json_str = NATSBridge.smartsend(
|
||||
... "/media",
|
||||
... [("image", image_bytes, "image")],
|
||||
... broker_url="nats://localhost:4222"
|
||||
... )
|
||||
"""
|
||||
# Extract options with defaults
|
||||
correlation_id = kwargs.get('correlation_id', _generate_uuid())
|
||||
msg_id = kwargs.get('msg_id', _generate_uuid())
|
||||
sender_id = kwargs.get('sender_id', _generate_uuid())
|
||||
broker_url = kwargs.get('broker_url', DEFAULT_BROKER_URL)
|
||||
fileserver_url = kwargs.get('fileserver_url', DEFAULT_FILESERVER_URL)
|
||||
size_threshold = kwargs.get('size_threshold', DEFAULT_SIZE_THRESHOLD)
|
||||
msg_purpose = kwargs.get('msg_purpose', 'chat')
|
||||
sender_name = kwargs.get('sender_name', 'NATSBridge')
|
||||
receiver_name = kwargs.get('receiver_name', '')
|
||||
receiver_id = kwargs.get('receiver_id', '')
|
||||
reply_to = kwargs.get('reply_to', '')
|
||||
reply_to_msg_id = kwargs.get('reply_to_msg_id', '')
|
||||
is_publish = kwargs.get('is_publish', True)
|
||||
fileserver_upload_handler = kwargs.get('fileserver_upload_handler', _sync_fileserver_upload)
|
||||
|
||||
log_trace(correlation_id, f"Starting smartsend for subject: {subject}")
|
||||
|
||||
# Process payloads
|
||||
payloads = []
|
||||
for dataname, payload_data, payload_type in data:
|
||||
payload_bytes = _serialize_data(payload_data, payload_type)
|
||||
payload_size = len(payload_bytes)
|
||||
|
||||
# Check against hard limit for MicroPython
|
||||
if payload_size > MAX_PAYLOAD_SIZE:
|
||||
raise MemoryError(f"Payload '{dataname}' exceeds max size {MAX_PAYLOAD_SIZE} bytes")
|
||||
|
||||
log_trace(correlation_id, f"Serialized payload '{dataname}' (type: {payload_type}) size: {payload_size} bytes")
|
||||
|
||||
if payload_size < size_threshold:
|
||||
# Direct path
|
||||
payload_b64 = base64.b64encode(payload_bytes).decode('ascii')
|
||||
log_trace(correlation_id, f"Using direct transport for {payload_size} bytes")
|
||||
|
||||
payload = _build_payload(dataname, payload_type, payload_bytes, 'direct', payload_b64)
|
||||
payloads.append(payload)
|
||||
else:
|
||||
# Link path (limited support)
|
||||
log_trace(correlation_id, "Using link transport, uploading to fileserver")
|
||||
|
||||
try:
|
||||
response = fileserver_upload_handler(fileserver_url, dataname, payload_bytes)
|
||||
log_trace(correlation_id, f"Uploaded to URL: {response['url']}")
|
||||
|
||||
payload = _build_payload(dataname, payload_type, payload_bytes, 'link', response['url'])
|
||||
payloads.append(payload)
|
||||
except NotImplementedError:
|
||||
# Fall back to direct transport if file upload not available
|
||||
log_trace(correlation_id, "File upload not available, using direct transport")
|
||||
payload_b64 = base64.b64encode(payload_bytes).decode('ascii')
|
||||
payload = _build_payload(dataname, payload_type, payload_bytes, 'direct', payload_b64)
|
||||
payloads.append(payload)
|
||||
|
||||
# Build envelope
|
||||
env = _build_envelope(subject, payloads, {
|
||||
'correlation_id': correlation_id,
|
||||
'msg_id': msg_id,
|
||||
'msg_purpose': msg_purpose,
|
||||
'sender_name': sender_name,
|
||||
'sender_id': sender_id,
|
||||
'receiver_name': receiver_name,
|
||||
'receiver_id': receiver_id,
|
||||
'reply_to': reply_to,
|
||||
'reply_to_msg_id': reply_to_msg_id,
|
||||
'broker_url': broker_url
|
||||
})
|
||||
|
||||
env_json_str = json.dumps(env)
|
||||
|
||||
if is_publish:
|
||||
_publish(subject, env_json_str, correlation_id)
|
||||
|
||||
return env, env_json_str
|
||||
|
||||
|
||||
def smartreceive(msg, **kwargs):
|
||||
"""
|
||||
Receive and process NATS message.
|
||||
|
||||
This function processes incoming NATS messages, handling both direct transport
|
||||
(base64 decoded payloads) and link transport (URL-based payloads).
|
||||
It deserializes the data based on the transport type and returns the result.
|
||||
|
||||
Note:
|
||||
MicroPython has memory constraints, so large payloads should be avoided.
|
||||
Table type is not supported due to memory constraints.
|
||||
|
||||
Args:
|
||||
msg: NATS message to process (can be string, dict, or object with 'payload' attribute)
|
||||
fileserver_download_handler: Function to handle downloading data from file server URLs
|
||||
max_retries: Maximum retry attempts (default: 3)
|
||||
base_delay: Initial delay in ms (default: 100)
|
||||
max_delay: Maximum delay in ms (default: 1000)
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads field containing List[Tuple[str, Any, str]]
|
||||
|
||||
Example:
|
||||
>>> # Receive and process message
|
||||
>>> env = NATSBridge.smartreceive(msg, fileserver_download_handler=_sync_fileserver_download)
|
||||
>>> # env is a Dict with "payloads" key containing List[Tuple[str, Any, str]]
|
||||
>>> for dataname, data, type_ in env["payloads"]:
|
||||
... print(f"{dataname}: {data} (type: {type_})")
|
||||
"""
|
||||
# Parse the JSON envelope
|
||||
if isinstance(msg, dict):
|
||||
# Already parsed
|
||||
env_json_obj = msg
|
||||
elif hasattr(msg, 'payload'):
|
||||
# Object with payload attribute
|
||||
payload = msg.payload if isinstance(msg.payload, str) else msg.payload.decode('utf-8')
|
||||
env_json_obj = json.loads(payload)
|
||||
else:
|
||||
# Assume it's already a JSON string or dict
|
||||
env_json_obj = json.loads(msg) if isinstance(msg, str) else msg
|
||||
|
||||
correlation_id = env_json_obj['correlation_id']
|
||||
log_trace(correlation_id, "Processing received message")
|
||||
|
||||
# Process all payloads in the envelope
|
||||
payloads_list = []
|
||||
num_payloads = len(env_json_obj['payloads'])
|
||||
|
||||
for i in range(num_payloads):
|
||||
payload_obj = env_json_obj['payloads'][i]
|
||||
transport = payload_obj['transport']
|
||||
dataname = payload_obj['dataname']
|
||||
|
||||
if transport == 'direct':
|
||||
log_trace(correlation_id, f"Direct transport - decoding payload '{dataname}'")
|
||||
|
||||
# Extract base64 payload from the payload
|
||||
payload_b64 = payload_obj['data']
|
||||
|
||||
# Decode Base64 payload
|
||||
payload_bytes = base64.b64decode(payload_b64)
|
||||
|
||||
# Deserialize based on type
|
||||
data_type = payload_obj['payload_type']
|
||||
data = _deserialize_data(payload_bytes, data_type)
|
||||
|
||||
payloads_list.append((dataname, data, data_type))
|
||||
elif transport == 'link':
|
||||
# Extract download URL from the payload
|
||||
url = payload_obj['data']
|
||||
log_trace(correlation_id, f"Link transport - fetching '{dataname}' from URL: {url}")
|
||||
|
||||
# Fetch with exponential backoff using the download handler
|
||||
fileserver_download_handler = kwargs.get('fileserver_download_handler', _sync_fileserver_download)
|
||||
max_retries = kwargs.get('max_retries', 3)
|
||||
base_delay = kwargs.get('base_delay', 100)
|
||||
max_delay = kwargs.get('max_delay', 1000)
|
||||
|
||||
downloaded_data = fileserver_download_handler(
|
||||
url,
|
||||
max_retries,
|
||||
base_delay,
|
||||
max_delay,
|
||||
correlation_id
|
||||
)
|
||||
|
||||
# Deserialize based on type
|
||||
data_type = payload_obj['payload_type']
|
||||
data = _deserialize_data(downloaded_data, data_type)
|
||||
|
||||
payloads_list.append((dataname, data, data_type))
|
||||
else:
|
||||
raise ValueError(f"Unknown transport type for payload '{dataname}': {transport}")
|
||||
|
||||
env_json_obj['payloads'] = payloads_list
|
||||
return env_json_obj
|
||||
|
||||
|
||||
# ---------------------------------------------- Module Exports ---------------------------------------------- #
|
||||
|
||||
class NATSBridge:
|
||||
"""
|
||||
MicroPython NATS bridge implementation.
|
||||
|
||||
This class provides a convenient interface for NATSBridge functionality,
|
||||
encapsulating the main functions and providing a class-based API.
|
||||
|
||||
Note:
|
||||
MicroPython has significant constraints:
|
||||
- No Arrow IPC support (memory constraints)
|
||||
- Only direct transport (< 100KB threshold enforced)
|
||||
- Simplified UUID generation
|
||||
- No async/await (synchronous API)
|
||||
"""
|
||||
|
||||
DEFAULT_SIZE_THRESHOLD = DEFAULT_SIZE_THRESHOLD
|
||||
DEFAULT_BROKER_URL = DEFAULT_BROKER_URL
|
||||
DEFAULT_FILESERVER_URL = DEFAULT_FILESERVER_URL
|
||||
MAX_PAYLOAD_SIZE = MAX_PAYLOAD_SIZE
|
||||
|
||||
def __init__(self, broker_url=None, fileserver_url=None):
|
||||
"""
|
||||
Initialize NATSBridge.
|
||||
|
||||
Args:
|
||||
broker_url: NATS server URL (defaults to DEFAULT_BROKER_URL)
|
||||
fileserver_url: HTTP file server URL (defaults to DEFAULT_FILESERVER_URL)
|
||||
"""
|
||||
self.broker_url = broker_url or self.DEFAULT_BROKER_URL
|
||||
self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL
|
||||
|
||||
def smartsend(self, subject, data, **kwargs):
|
||||
"""
|
||||
Send data via NATS.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
data: List of (dataname, data, type) tuples
|
||||
**kwargs: Additional options passed to smartsend
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str)
|
||||
"""
|
||||
kwargs['broker_url'] = kwargs.get('broker_url', self.broker_url)
|
||||
kwargs['fileserver_url'] = kwargs.get('fileserver_url', self.fileserver_url)
|
||||
return smartsend(subject, data, **kwargs)
|
||||
|
||||
def smartreceive(self, msg, **kwargs):
|
||||
"""
|
||||
Receive and process NATS message.
|
||||
|
||||
Args:
|
||||
msg: NATS message to process
|
||||
**kwargs: Additional options passed to smartreceive
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads
|
||||
"""
|
||||
return smartreceive(msg, **kwargs)
|
||||
|
||||
|
||||
# Convenience functions for module-level usage
|
||||
def send(subject, data, **kwargs):
|
||||
"""
|
||||
Convenience function for sending data.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
data: List of (dataname, data, type) tuples
|
||||
**kwargs: Additional options
|
||||
|
||||
Returns:
|
||||
Tuple of (env, env_json_str)
|
||||
"""
|
||||
return smartsend(subject, data, **kwargs)
|
||||
|
||||
|
||||
def receive(msg, **kwargs):
|
||||
"""
|
||||
Convenience function for receiving messages.
|
||||
|
||||
Args:
|
||||
msg: NATS message to process
|
||||
**kwargs: Additional options
|
||||
|
||||
Returns:
|
||||
Dict with envelope metadata and payloads
|
||||
"""
|
||||
return smartreceive(msg, **kwargs)
|
||||
|
||||
|
||||
__all__ = [
|
||||
'smartsend',
|
||||
'smartreceive',
|
||||
'NATSBridge',
|
||||
'send',
|
||||
'receive',
|
||||
'DEFAULT_SIZE_THRESHOLD',
|
||||
'DEFAULT_BROKER_URL',
|
||||
'DEFAULT_FILESERVER_URL',
|
||||
'MAX_PAYLOAD_SIZE',
|
||||
'NATSClient',
|
||||
'_serialize_data',
|
||||
'_deserialize_data',
|
||||
'log_trace',
|
||||
'_sync_fileserver_upload',
|
||||
'_sync_fileserver_download'
|
||||
]
|
||||
@@ -1,667 +0,0 @@
|
||||
"""
|
||||
Micropython NATS Bridge - Bi-Directional Data Bridge for Micropython
|
||||
|
||||
This module provides functionality for sending and receiving data over NATS
|
||||
using the Claim-Check pattern for large payloads.
|
||||
|
||||
Supported types: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||
"""
|
||||
|
||||
import json
|
||||
import random
|
||||
import time
|
||||
import usocket
|
||||
import uselect
|
||||
import ustruct
|
||||
import uuid
|
||||
|
||||
try:
|
||||
import ussl
|
||||
HAS_SSL = True
|
||||
except ImportError:
|
||||
HAS_SSL = False
|
||||
|
||||
# Constants
|
||||
DEFAULT_SIZE_THRESHOLD = 1000000 # 1MB - threshold for switching from direct to link transport
|
||||
DEFAULT_NATS_URL = "nats://localhost:4222"
|
||||
DEFAULT_FILESERVER_URL = "http://localhost:8080"
|
||||
|
||||
# ============================================= 100 ============================================== #
|
||||
|
||||
|
||||
class MessagePayload:
|
||||
"""Internal message payload structure representing a single payload within a NATS message envelope."""
|
||||
|
||||
def __init__(self, data, msg_type, id="", dataname="", transport="direct",
|
||||
encoding="none", size=0, metadata=None):
|
||||
"""
|
||||
Initialize a MessagePayload.
|
||||
|
||||
Args:
|
||||
data: Payload data (bytes for direct, URL string for link)
|
||||
msg_type: Payload type ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
||||
id: Unique identifier for this payload (auto-generated if empty)
|
||||
dataname: Name of the payload (auto-generated UUID if empty)
|
||||
transport: Transport method ("direct" or "link")
|
||||
encoding: Encoding method ("none", "json", "base64", "arrow-ipc")
|
||||
size: Size of the payload in bytes
|
||||
metadata: Optional metadata dictionary
|
||||
"""
|
||||
self.id = id if id else self._generate_uuid()
|
||||
self.dataname = dataname if dataname else self._generate_uuid()
|
||||
self.type = msg_type
|
||||
self.transport = transport
|
||||
self.encoding = encoding
|
||||
self.size = size
|
||||
self.data = data
|
||||
self.metadata = metadata if metadata else {}
|
||||
|
||||
def _generate_uuid(self):
|
||||
"""Generate a UUID string."""
|
||||
return str(uuid.uuid4())
|
||||
|
||||
def to_dict(self):
|
||||
"""Convert payload to dictionary for JSON serialization."""
|
||||
payload_dict = {
|
||||
"id": self.id,
|
||||
"dataname": self.dataname,
|
||||
"type": self.type,
|
||||
"transport": self.transport,
|
||||
"encoding": self.encoding,
|
||||
"size": self.size,
|
||||
}
|
||||
|
||||
# Include data based on transport type
|
||||
if self.transport == "direct" and self.data is not None:
|
||||
if self.encoding == "base64" or self.encoding == "json":
|
||||
payload_dict["data"] = self.data
|
||||
else:
|
||||
# For other encodings, use base64
|
||||
payload_dict["data"] = self._to_base64(self.data)
|
||||
elif self.transport == "link" and self.data is not None:
|
||||
# For link transport, data is a URL string
|
||||
payload_dict["data"] = self.data
|
||||
|
||||
if self.metadata:
|
||||
payload_dict["metadata"] = self.metadata
|
||||
|
||||
return payload_dict
|
||||
|
||||
def _to_base64(self, data):
|
||||
"""Convert bytes to base64 string."""
|
||||
if isinstance(data, bytes):
|
||||
# Simple base64 encoding without library
|
||||
import ubinascii
|
||||
return ubinascii.b2a_base64(data).decode('utf-8').strip()
|
||||
return data
|
||||
|
||||
def _from_base64(self, data):
|
||||
"""Convert base64 string to bytes."""
|
||||
import ubinascii
|
||||
return ubinascii.a2b_base64(data)
|
||||
|
||||
|
||||
class MessageEnvelope:
|
||||
"""Internal message envelope structure containing multiple payloads with metadata."""
|
||||
|
||||
def __init__(self, send_to, payloads, correlation_id="", msg_id="", timestamp="",
|
||||
msg_purpose="", sender_name="", sender_id="", receiver_name="",
|
||||
receiver_id="", reply_to="", reply_to_msg_id="", broker_url=DEFAULT_NATS_URL,
|
||||
metadata=None):
|
||||
"""
|
||||
Initialize a MessageEnvelope.
|
||||
|
||||
Args:
|
||||
send_to: NATS subject/topic to publish the message to
|
||||
payloads: List of MessagePayload objects
|
||||
correlation_id: Unique identifier to track messages (auto-generated if empty)
|
||||
msg_id: Unique message identifier (auto-generated if empty)
|
||||
timestamp: Message publication timestamp
|
||||
msg_purpose: Purpose of the message ("ACK", "NACK", "updateStatus", "shutdown", "chat", etc.)
|
||||
sender_name: Name of the sender
|
||||
sender_id: UUID of the sender
|
||||
receiver_name: Name of the receiver (empty means broadcast)
|
||||
receiver_id: UUID of the receiver (empty means broadcast)
|
||||
reply_to: Topic where receiver should reply
|
||||
reply_to_msg_id: Message ID this message is replying to
|
||||
broker_url: NATS broker URL
|
||||
metadata: Optional message-level metadata
|
||||
"""
|
||||
self.correlation_id = correlation_id if correlation_id else self._generate_uuid()
|
||||
self.msg_id = msg_id if msg_id else self._generate_uuid()
|
||||
self.timestamp = timestamp if timestamp else self._get_timestamp()
|
||||
self.send_to = send_to
|
||||
self.msg_purpose = msg_purpose
|
||||
self.sender_name = sender_name
|
||||
self.sender_id = sender_id if sender_id else self._generate_uuid()
|
||||
self.receiver_name = receiver_name
|
||||
self.receiver_id = receiver_id if receiver_id else self._generate_uuid()
|
||||
self.reply_to = reply_to
|
||||
self.reply_to_msg_id = reply_to_msg_id
|
||||
self.broker_url = broker_url
|
||||
self.metadata = metadata if metadata else {}
|
||||
self.payloads = payloads
|
||||
|
||||
def _generate_uuid(self):
|
||||
"""Generate a UUID string."""
|
||||
return str(uuid.uuid4())
|
||||
|
||||
def _get_timestamp(self):
|
||||
"""Get current timestamp in ISO format."""
|
||||
# Simplified timestamp - Micropython may not have full datetime
|
||||
return "2026-02-21T" + time.strftime("%H:%M:%S", time.localtime())
|
||||
|
||||
def to_json(self):
|
||||
"""Convert envelope to JSON string."""
|
||||
obj = {
|
||||
"correlationId": self.correlation_id,
|
||||
"msgId": self.msg_id,
|
||||
"timestamp": self.timestamp,
|
||||
"sendTo": self.send_to,
|
||||
"msgPurpose": self.msg_purpose,
|
||||
"senderName": self.sender_name,
|
||||
"senderId": self.sender_id,
|
||||
"receiverName": self.receiver_name,
|
||||
"receiverId": self.receiver_id,
|
||||
"replyTo": self.reply_to,
|
||||
"replyToMsgId": self.reply_to_msg_id,
|
||||
"brokerURL": self.broker_url
|
||||
}
|
||||
|
||||
# Include metadata if not empty
|
||||
if self.metadata:
|
||||
obj["metadata"] = self.metadata
|
||||
|
||||
# Convert payloads to JSON array
|
||||
if self.payloads:
|
||||
payloads_json = []
|
||||
for payload in self.payloads:
|
||||
payloads_json.append(payload.to_dict())
|
||||
obj["payloads"] = payloads_json
|
||||
|
||||
return json.dumps(obj)
|
||||
|
||||
|
||||
def log_trace(correlation_id, message):
|
||||
"""Log a trace message with correlation ID and timestamp."""
|
||||
timestamp = time.strftime("%Y-%m-%dT%H:%M:%S", time.localtime())
|
||||
print("[{}] [Correlation: {}] {}".format(timestamp, correlation_id, message))
|
||||
|
||||
|
||||
def _serialize_data(data, msg_type):
|
||||
"""Serialize data according to specified format.
|
||||
|
||||
Args:
|
||||
data: Data to serialize
|
||||
msg_type: Target format ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
||||
|
||||
Returns:
|
||||
bytes: Binary representation of the serialized data
|
||||
"""
|
||||
if msg_type == "text":
|
||||
if isinstance(data, str):
|
||||
return data.encode('utf-8')
|
||||
else:
|
||||
raise ValueError("Text data must be a string")
|
||||
|
||||
elif msg_type == "dictionary":
|
||||
if isinstance(data, dict):
|
||||
json_str = json.dumps(data)
|
||||
return json_str.encode('utf-8')
|
||||
else:
|
||||
raise ValueError("Dictionary data must be a dict")
|
||||
|
||||
elif msg_type in ("image", "audio", "video", "binary"):
|
||||
if isinstance(data, bytes):
|
||||
return data
|
||||
else:
|
||||
raise ValueError("{} data must be bytes".format(msg_type.capitalize()))
|
||||
|
||||
else:
|
||||
raise ValueError("Unknown type: {}".format(msg_type))
|
||||
|
||||
|
||||
def _deserialize_data(data_bytes, msg_type, correlation_id):
|
||||
"""Deserialize bytes to data based on type.
|
||||
|
||||
Args:
|
||||
data_bytes: Serialized data as bytes
|
||||
msg_type: Data type ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
||||
correlation_id: Correlation ID for logging
|
||||
|
||||
Returns:
|
||||
Deserialized data
|
||||
"""
|
||||
if msg_type == "text":
|
||||
return data_bytes.decode('utf-8')
|
||||
|
||||
elif msg_type == "dictionary":
|
||||
json_str = data_bytes.decode('utf-8')
|
||||
return json.loads(json_str)
|
||||
|
||||
elif msg_type in ("image", "audio", "video", "binary"):
|
||||
return data_bytes
|
||||
|
||||
else:
|
||||
raise ValueError("Unknown type: {}".format(msg_type))
|
||||
|
||||
|
||||
class NATSConnection:
|
||||
"""Simple NATS connection for Micropython."""
|
||||
|
||||
def __init__(self, url=DEFAULT_NATS_URL):
|
||||
"""Initialize NATS connection.
|
||||
|
||||
Args:
|
||||
url: NATS server URL (e.g., "nats://localhost:4222")
|
||||
"""
|
||||
self.url = url
|
||||
self.host = "localhost"
|
||||
self.port = 4222
|
||||
self.conn = None
|
||||
self._parse_url(url)
|
||||
|
||||
def _parse_url(self, url):
|
||||
"""Parse NATS URL to extract host and port."""
|
||||
if url.startswith("nats://"):
|
||||
url = url[7:]
|
||||
elif url.startswith("tls://"):
|
||||
url = url[6:]
|
||||
|
||||
if ":" in url:
|
||||
self.host, port_str = url.split(":")
|
||||
self.port = int(port_str)
|
||||
else:
|
||||
self.host = url
|
||||
|
||||
def connect(self):
|
||||
"""Connect to NATS server."""
|
||||
addr = usocket.getaddrinfo(self.host, self.port)[0][-1]
|
||||
self.conn = usocket.socket()
|
||||
self.conn.connect(addr)
|
||||
log_trace("", "Connected to NATS server at {}:{}".format(self.host, self.port))
|
||||
|
||||
def publish(self, subject, message):
|
||||
"""Publish a message to a NATS subject.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish to
|
||||
message: Message to publish (should be bytes or string)
|
||||
"""
|
||||
if isinstance(message, str):
|
||||
message = message.encode('utf-8')
|
||||
|
||||
# Simple NATS protocol implementation
|
||||
msg = "PUB {} {}\r\n".format(subject, len(message))
|
||||
msg = msg.encode('utf-8') + message + b"\r\n"
|
||||
self.conn.send(msg)
|
||||
log_trace("", "Message published to {}".format(subject))
|
||||
|
||||
def subscribe(self, subject, callback):
|
||||
"""Subscribe to a NATS subject.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to subscribe to
|
||||
callback: Callback function to handle incoming messages
|
||||
"""
|
||||
log_trace("", "Subscribed to {}".format(subject))
|
||||
# Simplified subscription - in a real implementation, you'd handle SUB/PUB messages
|
||||
# For Micropython, we'll use a simple polling approach
|
||||
self.subscribed_subject = subject
|
||||
self.subscription_callback = callback
|
||||
|
||||
def wait_message(self, timeout=1000):
|
||||
"""Wait for incoming message.
|
||||
|
||||
Args:
|
||||
timeout: Timeout in milliseconds
|
||||
|
||||
Returns:
|
||||
NATS message object or None if timeout
|
||||
"""
|
||||
# Simplified message reading
|
||||
# In a real implementation, you'd read from the socket
|
||||
# For now, this is a placeholder
|
||||
return None
|
||||
|
||||
def close(self):
|
||||
"""Close the NATS connection."""
|
||||
if self.conn:
|
||||
self.conn.close()
|
||||
self.conn = None
|
||||
log_trace("", "NATS connection closed")
|
||||
|
||||
|
||||
def _fetch_with_backoff(url, max_retries=5, base_delay=100, max_delay=5000, correlation_id=""):
|
||||
"""Fetch data from URL with exponential backoff.
|
||||
|
||||
Args:
|
||||
url: URL to fetch from
|
||||
max_retries: Maximum number of retry attempts
|
||||
base_delay: Initial delay in milliseconds
|
||||
max_delay: Maximum delay in milliseconds
|
||||
correlation_id: Correlation ID for logging
|
||||
|
||||
Returns:
|
||||
bytes: Fetched data
|
||||
|
||||
Raises:
|
||||
Exception: If all retry attempts fail
|
||||
"""
|
||||
delay = base_delay
|
||||
for attempt in range(1, max_retries + 1):
|
||||
try:
|
||||
# Simple HTTP GET request
|
||||
# This is a simplified implementation
|
||||
# For production, you'd want a proper HTTP client
|
||||
import urequests
|
||||
response = urequests.get(url)
|
||||
if response.status_code == 200:
|
||||
log_trace(correlation_id, "Successfully fetched data from {} on attempt {}".format(url, attempt))
|
||||
return response.content
|
||||
else:
|
||||
raise Exception("Failed to fetch: {}".format(response.status_code))
|
||||
except Exception as e:
|
||||
log_trace(correlation_id, "Attempt {} failed: {}".format(attempt, str(e)))
|
||||
if attempt < max_retries:
|
||||
time.sleep(delay / 1000.0)
|
||||
delay = min(delay * 2, max_delay)
|
||||
|
||||
|
||||
def plik_oneshot_upload(file_server_url, filename, data):
|
||||
"""Upload a single file to a plik server using one-shot mode.
|
||||
|
||||
Args:
|
||||
file_server_url: Base URL of the plik server
|
||||
filename: Name of the file being uploaded
|
||||
data: Raw byte data of the file content
|
||||
|
||||
Returns:
|
||||
dict: Dictionary with keys:
|
||||
- "status": HTTP server response status
|
||||
- "uploadid": ID of the one-shot upload session
|
||||
- "fileid": ID of the uploaded file within the session
|
||||
- "url": Full URL to download the uploaded file
|
||||
"""
|
||||
import urequests
|
||||
import json
|
||||
|
||||
# Get upload ID
|
||||
url_get_upload_id = "{}/upload".format(file_server_url)
|
||||
headers = {"Content-Type": "application/json"}
|
||||
body = json.dumps({"OneShot": True})
|
||||
|
||||
response = urequests.post(url_get_upload_id, headers=headers, data=body)
|
||||
response_json = json.loads(response.content)
|
||||
|
||||
uploadid = response_json.get("id")
|
||||
uploadtoken = response_json.get("uploadToken")
|
||||
|
||||
# Upload file
|
||||
url_upload = "{}/file/{}".format(file_server_url, uploadid)
|
||||
headers = {"X-UploadToken": uploadtoken}
|
||||
|
||||
# For Micropython, we need to construct the multipart form data manually
|
||||
# This is a simplified approach
|
||||
boundary = "----WebKitFormBoundary{}".format(uuid.uuid4().hex[:16])
|
||||
|
||||
# Create multipart body
|
||||
part1 = "--{}\r\n".format(boundary)
|
||||
part1 += "Content-Disposition: form-data; name=\"file\"; filename=\"{}\"\r\n".format(filename)
|
||||
part1 += "Content-Type: application/octet-stream\r\n\r\n"
|
||||
part1_bytes = part1.encode('utf-8')
|
||||
|
||||
part2 = "\r\n--{}--".format(boundary)
|
||||
part2_bytes = part2.encode('utf-8')
|
||||
|
||||
# Combine all parts
|
||||
full_body = part1_bytes + data + part2_bytes
|
||||
|
||||
# Set content type with boundary
|
||||
content_type = "multipart/form-data; boundary={}".format(boundary)
|
||||
|
||||
response = urequests.post(url_upload, headers={"Content-Type": content_type}, data=full_body)
|
||||
response_json = json.loads(response.content)
|
||||
|
||||
fileid = response_json.get("id")
|
||||
url = "{}/file/{}/{}".format(file_server_url, uploadid, filename)
|
||||
|
||||
return {
|
||||
"status": response.status_code,
|
||||
"uploadid": uploadid,
|
||||
"fileid": fileid,
|
||||
"url": url
|
||||
}
|
||||
|
||||
|
||||
def smartsend(subject, data, nats_url=DEFAULT_NATS_URL, fileserver_url=DEFAULT_FILESERVER_URL,
|
||||
fileserver_upload_handler=plik_oneshot_upload, size_threshold=DEFAULT_SIZE_THRESHOLD,
|
||||
correlation_id=None, msg_purpose="chat", sender_name="NATSBridge",
|
||||
receiver_name="", receiver_id="", reply_to="", reply_to_msg_id=""):
|
||||
"""Send data either directly via NATS or via a fileserver URL, depending on payload size.
|
||||
|
||||
This function intelligently routes data delivery based on payload size relative to a threshold.
|
||||
If the serialized payload is smaller than `size_threshold`, it encodes the data as Base64 and
|
||||
publishes directly over NATS. Otherwise, it uploads the data to a fileserver and publishes
|
||||
only the download URL over NATS.
|
||||
|
||||
Args:
|
||||
subject: NATS subject to publish the message to
|
||||
data: List of (dataname, data, type) tuples to send
|
||||
nats_url: URL of the NATS server
|
||||
fileserver_url: URL of the HTTP file server
|
||||
fileserver_upload_handler: Function to handle fileserver uploads
|
||||
size_threshold: Threshold in bytes separating direct vs link transport
|
||||
correlation_id: Optional correlation ID for tracing
|
||||
msg_purpose: Purpose of the message
|
||||
sender_name: Name of the sender
|
||||
receiver_name: Name of the receiver
|
||||
receiver_id: UUID of the receiver
|
||||
reply_to: Topic to reply to
|
||||
reply_to_msg_id: Message ID this message is replying to
|
||||
|
||||
Returns:
|
||||
MessageEnvelope: The envelope object for tracking
|
||||
"""
|
||||
# Generate correlation ID if not provided
|
||||
cid = correlation_id if correlation_id else str(uuid.uuid4())
|
||||
|
||||
log_trace(cid, "Starting smartsend for subject: {}".format(subject))
|
||||
|
||||
# Generate message metadata
|
||||
msg_id = str(uuid.uuid4())
|
||||
|
||||
# Process each payload in the list
|
||||
payloads = []
|
||||
|
||||
for dataname, payload_data, payload_type in data:
|
||||
# Serialize data based on type
|
||||
payload_bytes = _serialize_data(payload_data, payload_type)
|
||||
|
||||
payload_size = len(payload_bytes)
|
||||
log_trace(cid, "Serialized payload '{}' (type: {}) size: {} bytes".format(
|
||||
dataname, payload_type, payload_size))
|
||||
|
||||
# Decision: Direct vs Link
|
||||
if payload_size < size_threshold:
|
||||
# Direct path - Base64 encode and send via NATS
|
||||
payload_b64 = _serialize_data(payload_bytes, "binary") # Already bytes
|
||||
# Convert to base64 string for JSON
|
||||
import ubinascii
|
||||
payload_b64_str = ubinascii.b2a_base64(payload_bytes).decode('utf-8').strip()
|
||||
|
||||
log_trace(cid, "Using direct transport for {} bytes".format(payload_size))
|
||||
|
||||
# Create MessagePayload for direct transport
|
||||
payload = MessagePayload(
|
||||
payload_b64_str,
|
||||
payload_type,
|
||||
id=str(uuid.uuid4()),
|
||||
dataname=dataname,
|
||||
transport="direct",
|
||||
encoding="base64",
|
||||
size=payload_size,
|
||||
metadata={"payload_bytes": payload_size}
|
||||
)
|
||||
payloads.append(payload)
|
||||
else:
|
||||
# Link path - Upload to HTTP server, send URL via NATS
|
||||
log_trace(cid, "Using link transport, uploading to fileserver")
|
||||
|
||||
# Upload to HTTP server
|
||||
response = fileserver_upload_handler(fileserver_url, dataname, payload_bytes)
|
||||
|
||||
if response["status"] != 200:
|
||||
raise Exception("Failed to upload data to fileserver: {}".format(response["status"]))
|
||||
|
||||
url = response["url"]
|
||||
log_trace(cid, "Uploaded to URL: {}".format(url))
|
||||
|
||||
# Create MessagePayload for link transport
|
||||
payload = MessagePayload(
|
||||
url,
|
||||
payload_type,
|
||||
id=str(uuid.uuid4()),
|
||||
dataname=dataname,
|
||||
transport="link",
|
||||
encoding="none",
|
||||
size=payload_size,
|
||||
metadata={}
|
||||
)
|
||||
payloads.append(payload)
|
||||
|
||||
# Create MessageEnvelope with all payloads
|
||||
env = MessageEnvelope(
|
||||
subject,
|
||||
payloads,
|
||||
correlation_id=cid,
|
||||
msg_id=msg_id,
|
||||
msg_purpose=msg_purpose,
|
||||
sender_name=sender_name,
|
||||
sender_id=str(uuid.uuid4()),
|
||||
receiver_name=receiver_name,
|
||||
receiver_id=receiver_id,
|
||||
reply_to=reply_to,
|
||||
reply_to_msg_id=reply_to_msg_id,
|
||||
broker_url=nats_url,
|
||||
metadata={}
|
||||
)
|
||||
|
||||
msg_json = env.to_json()
|
||||
|
||||
# Publish to NATS
|
||||
nats_conn = NATSConnection(nats_url)
|
||||
nats_conn.connect()
|
||||
nats_conn.publish(subject, msg_json)
|
||||
nats_conn.close()
|
||||
|
||||
return env
|
||||
|
||||
|
||||
def smartreceive(msg, fileserver_download_handler=_fetch_with_backoff, max_retries=5,
|
||||
base_delay=100, max_delay=5000):
|
||||
"""Receive and process messages from NATS.
|
||||
|
||||
This function processes incoming NATS messages, handling both direct transport
|
||||
(base64 decoded payloads) and link transport (URL-based payloads).
|
||||
|
||||
Args:
|
||||
msg: NATS message to process (dict with payload data)
|
||||
fileserver_download_handler: Function to handle downloading data from file server URLs
|
||||
max_retries: Maximum retry attempts for fetching URL
|
||||
base_delay: Initial delay for exponential backoff in ms
|
||||
max_delay: Maximum delay for exponential backoff in ms
|
||||
|
||||
Returns:
|
||||
dict: Envelope dictionary with metadata and 'payloads' field containing list of (dataname, data, type) tuples
|
||||
"""
|
||||
# Parse the JSON envelope
|
||||
json_data = msg if isinstance(msg, dict) else json.loads(msg)
|
||||
log_trace(json_data.get("correlationId", ""), "Processing received message")
|
||||
|
||||
# Process all payloads in the envelope
|
||||
payloads_list = []
|
||||
|
||||
# Get number of payloads
|
||||
num_payloads = len(json_data.get("payloads", []))
|
||||
|
||||
for i in range(num_payloads):
|
||||
payload = json_data["payloads"][i]
|
||||
transport = payload.get("transport", "")
|
||||
dataname = payload.get("dataname", "")
|
||||
|
||||
if transport == "direct":
|
||||
log_trace(json_data.get("correlationId", ""),
|
||||
"Direct transport - decoding payload '{}'".format(dataname))
|
||||
|
||||
# Extract base64 payload from the payload
|
||||
payload_b64 = payload.get("data", "")
|
||||
|
||||
# Decode Base64 payload
|
||||
import ubinascii
|
||||
payload_bytes = ubinascii.a2b_base64(payload_b64.encode('utf-8'))
|
||||
|
||||
# Deserialize based on type
|
||||
data_type = payload.get("type", "")
|
||||
data = _deserialize_data(payload_bytes, data_type, json_data.get("correlationId", ""))
|
||||
|
||||
payloads_list.append((dataname, data, data_type))
|
||||
|
||||
elif transport == "link":
|
||||
# Extract download URL from the payload
|
||||
url = payload.get("data", "")
|
||||
log_trace(json_data.get("correlationId", ""),
|
||||
"Link transport - fetching '{}' from URL: {}".format(dataname, url))
|
||||
|
||||
# Fetch with exponential backoff
|
||||
downloaded_data = fileserver_download_handler(
|
||||
url, max_retries, base_delay, max_delay, json_data.get("correlationId", "")
|
||||
)
|
||||
|
||||
# Deserialize based on type
|
||||
data_type = payload.get("type", "")
|
||||
data = _deserialize_data(downloaded_data, data_type, json_data.get("correlationId", ""))
|
||||
|
||||
payloads_list.append((dataname, data, data_type))
|
||||
|
||||
else:
|
||||
raise ValueError("Unknown transport type for payload '{}': {}".format(dataname, transport))
|
||||
|
||||
# Replace payloads field with the processed list of (dataname, data, type) tuples
|
||||
json_data["payloads"] = payloads_list
|
||||
|
||||
return json_data
|
||||
|
||||
|
||||
# Utility functions
|
||||
def generate_uuid():
|
||||
"""Generate a UUID string."""
|
||||
return str(uuid.uuid4())
|
||||
|
||||
|
||||
def get_timestamp():
|
||||
"""Get current timestamp in ISO format."""
|
||||
return time.strftime("%Y-%m-%dT%H:%M:%S", time.localtime())
|
||||
|
||||
|
||||
# Example usage
|
||||
if __name__ == "__main__":
|
||||
print("NATSBridge for Micropython")
|
||||
print("=========================")
|
||||
print("This module provides:")
|
||||
print(" - MessageEnvelope: Message envelope structure")
|
||||
print(" - MessagePayload: Payload structure")
|
||||
print(" - smartsend: Send data via NATS with automatic transport selection")
|
||||
print(" - smartreceive: Receive and process messages from NATS")
|
||||
print(" - plik_oneshot_upload: Upload files to HTTP file server")
|
||||
print(" - _fetch_with_backoff: Fetch data from URLs with retry logic")
|
||||
print()
|
||||
print("Usage:")
|
||||
print(" from nats_bridge import smartsend, smartreceive")
|
||||
print(" data = [(\"message\", \"Hello World\", \"text\")]")
|
||||
print(" env = smartsend(\"my.subject\", data)")
|
||||
print()
|
||||
print(" # On receiver:")
|
||||
print(" payloads = smartreceive(msg)")
|
||||
print(" for dataname, data, type in payloads:")
|
||||
print(" print(f\"Received {dataname} of type {type}: {data}\")")
|
||||
215
test/test_js_binary_receiver.js
Normal file
215
test/test_js_binary_receiver.js
Normal file
@@ -0,0 +1,215 @@
|
||||
/**
|
||||
* JavaScript Binary Receiver Test
|
||||
* Tests the smartreceive function with binary/image/audio/video payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natbridge.js');
|
||||
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Binary Receiver Test ===\n');
|
||||
|
||||
// Create mock NATS message with binary payloads
|
||||
const binaryData = Buffer.from([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]); // PNG header
|
||||
const audioData = Buffer.from([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]); // FLAC header
|
||||
const videoData = Buffer.from([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]); // MP4 header
|
||||
const genericBinary = Buffer.from([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE]);
|
||||
|
||||
const testData = {
|
||||
correlation_id: 'js-binary-receiver-' + Date.now(),
|
||||
msg_id: 'msg-' + Date.now(),
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: '/test/binary',
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-binary-test',
|
||||
sender_id: 'sender-' + Date.now(),
|
||||
receiver_name: 'js-receiver',
|
||||
receiver_id: 'receiver-' + Date.now(),
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
broker_url: TEST_BROKER_URL,
|
||||
metadata: {},
|
||||
payloads: [
|
||||
{
|
||||
id: 'payload-1',
|
||||
dataname: 'image',
|
||||
payload_type: 'image',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: binaryData.length,
|
||||
data: binaryData.toString('base64'),
|
||||
metadata: { payload_bytes: binaryData.length }
|
||||
},
|
||||
{
|
||||
id: 'payload-2',
|
||||
dataname: 'audio',
|
||||
payload_type: 'audio',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: audioData.length,
|
||||
data: audioData.toString('base64'),
|
||||
metadata: { payload_bytes: audioData.length }
|
||||
},
|
||||
{
|
||||
id: 'payload-3',
|
||||
dataname: 'video',
|
||||
payload_type: 'video',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: videoData.length,
|
||||
data: videoData.toString('base64'),
|
||||
metadata: { payload_bytes: videoData.length }
|
||||
},
|
||||
{
|
||||
id: 'payload-4',
|
||||
dataname: 'binary',
|
||||
payload_type: 'binary',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: genericBinary.length,
|
||||
data: genericBinary.toString('base64'),
|
||||
metadata: { payload_bytes: genericBinary.length }
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const mockMsg = {
|
||||
payload: JSON.stringify(testData)
|
||||
};
|
||||
|
||||
console.log('Mock Message Created:');
|
||||
console.log(` Correlation ID: ${testData.correlation_id}`);
|
||||
console.log(` Payloads: ${testData.payloads.length}`);
|
||||
console.log(` Payload types: ${testData.payloads.map(p => p.payload_type).join(', ')}\n`);
|
||||
|
||||
try {
|
||||
// Receive and process the message
|
||||
console.log('Receiving and processing message...');
|
||||
const env = await NATSBridge.smartreceive(
|
||||
mockMsg,
|
||||
{
|
||||
max_retries: 3,
|
||||
base_delay: 100,
|
||||
max_delay: 1000
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Received Envelope ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate received data
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (!env.correlation_id) {
|
||||
console.log('❌ correlation_id is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ correlation_id present');
|
||||
}
|
||||
|
||||
if (env.payloads.length !== 4) {
|
||||
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
// Expected data
|
||||
const expectedData = [
|
||||
['image', binaryData, 'image'],
|
||||
['audio', audioData, 'audio'],
|
||||
['video', videoData, 'video'],
|
||||
['binary', genericBinary, 'binary']
|
||||
];
|
||||
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
const expected = expectedData[i];
|
||||
|
||||
if (payload[0] !== expected[0]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected dataname '${expected[0]}', got '${payload[0]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct dataname`);
|
||||
}
|
||||
|
||||
if (payload[2] !== expected[2]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected type '${expected[2]}', got '${payload[2]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct type`);
|
||||
}
|
||||
|
||||
// Verify binary data integrity
|
||||
const receivedData = payload[1];
|
||||
if (!(receivedData instanceof Buffer || receivedData instanceof Uint8Array)) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected Buffer/Uint8Array, got ${typeof receivedData}`);
|
||||
passed = false;
|
||||
} else if (Buffer.isBuffer(receivedData)) {
|
||||
if (receivedData.length !== expected[1].length) {
|
||||
console.log(`❌ Payload ${i + 1}: Length mismatch`);
|
||||
passed = false;
|
||||
} else {
|
||||
let dataMatch = true;
|
||||
for (let j = 0; j < expected[1].length; j++) {
|
||||
if (receivedData[j] !== expected[1][j]) {
|
||||
dataMatch = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (dataMatch) {
|
||||
console.log(`✅ Payload ${i + 1}: Data correctly deserialized`);
|
||||
} else {
|
||||
console.log(`❌ Payload ${i + 1}: Data mismatch`);
|
||||
passed = false;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Uint8Array comparison
|
||||
const receivedBuffer = Buffer.from(receivedData);
|
||||
if (receivedBuffer.length !== expected[1].length) {
|
||||
console.log(`❌ Payload ${i + 1}: Length mismatch`);
|
||||
passed = false;
|
||||
} else {
|
||||
let dataMatch = true;
|
||||
for (let j = 0; j < expected[1].length; j++) {
|
||||
if (receivedBuffer[j] !== expected[1][j]) {
|
||||
dataMatch = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (dataMatch) {
|
||||
console.log(`✅ Payload ${i + 1}: Data correctly deserialized`);
|
||||
} else {
|
||||
console.log(`❌ Payload ${i + 1}: Data mismatch`);
|
||||
passed = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
173
test/test_js_binary_sender.js
Normal file
173
test/test_js_binary_sender.js
Normal file
@@ -0,0 +1,173 @@
|
||||
/**
|
||||
* JavaScript Binary Sender Test
|
||||
* Tests the smartsend function with binary/image/audio/video payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natbridge.js');
|
||||
|
||||
const TEST_SUBJECT = '/test/binary';
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Binary Sender Test ===\n');
|
||||
|
||||
const correlationId = NATSBridge.uuidv4();
|
||||
console.log(`Correlation ID: ${correlationId}`);
|
||||
console.log(`Subject: ${TEST_SUBJECT}`);
|
||||
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
|
||||
|
||||
// Test data - binary data for different types
|
||||
const binaryData = Buffer.from([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]); // PNG header
|
||||
const audioData = Buffer.from([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]); // FLAC header
|
||||
const videoData = Buffer.from([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]); // MP4 header
|
||||
const genericBinary = Buffer.from([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE]);
|
||||
|
||||
const testData = [
|
||||
['image', binaryData, 'image'],
|
||||
['audio', audioData, 'audio'],
|
||||
['video', videoData, 'video'],
|
||||
['binary', genericBinary, 'binary']
|
||||
];
|
||||
|
||||
try {
|
||||
// Send the message
|
||||
console.log('Sending binary payloads...');
|
||||
const [env, envJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
testData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: correlationId,
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-binary-test',
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Envelope Created ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Purpose: ${env.msg_purpose}`);
|
||||
console.log(`Sender: ${env.sender_name}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate envelope structure
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (env.payloads.length !== 4) {
|
||||
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
// Test each payload
|
||||
const expectedDatanames = ['image', 'audio', 'video', 'binary'];
|
||||
const expectedTypes = ['image', 'audio', 'video', 'binary'];
|
||||
const expectedData = [binaryData, audioData, videoData, genericBinary];
|
||||
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
|
||||
if (payload.dataname !== expectedDatanames[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected dataname '${expectedDatanames[i]}', got '${payload.dataname}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct dataname`);
|
||||
}
|
||||
|
||||
if (payload.payload_type !== expectedTypes[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected type '${expectedTypes[i]}', got '${payload.payload_type}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct type`);
|
||||
}
|
||||
|
||||
if (payload.transport !== 'direct') {
|
||||
console.log(`❌ Payload ${i + 1}: Expected transport 'direct', got '${payload.transport}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct transport`);
|
||||
}
|
||||
|
||||
if (payload.encoding !== 'base64') {
|
||||
console.log(`❌ Payload ${i + 1}: Expected encoding 'base64', got '${payload.encoding}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct encoding`);
|
||||
}
|
||||
|
||||
// Decode and verify the data
|
||||
const decodedData = Buffer.from(payload.data, 'base64');
|
||||
const originalData = expectedData[i];
|
||||
|
||||
if (decodedData.length !== originalData.length) {
|
||||
console.log(`❌ Payload ${i + 1}: Length mismatch (${decodedData.length} vs ${originalData.length})`);
|
||||
passed = false;
|
||||
} else {
|
||||
let dataMatch = true;
|
||||
for (let j = 0; j < originalData.length; j++) {
|
||||
if (decodedData[j] !== originalData[j]) {
|
||||
dataMatch = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (dataMatch) {
|
||||
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
|
||||
} else {
|
||||
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
|
||||
passed = false;
|
||||
}
|
||||
}
|
||||
|
||||
console.log(` Size: ${payload.size} bytes\n`);
|
||||
}
|
||||
|
||||
// Test with larger binary data (simulating file upload scenario)
|
||||
console.log('=== Large Binary Data Test ===');
|
||||
const largeData = Buffer.alloc(10000, 0xFF); // 10KB of binary data
|
||||
const largeTestData = [
|
||||
['large_binary', largeData, 'binary']
|
||||
];
|
||||
|
||||
const [largeEnv, _] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
largeTestData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: 'large-' + correlationId,
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
if (largeEnv.payloads.length === 1 && largeEnv.payloads[0].size === 10000) {
|
||||
console.log('✅ Large binary data handled correctly');
|
||||
} else {
|
||||
console.log('❌ Large binary data handling failed');
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
220
test/test_js_dictionary_receiver.js
Normal file
220
test/test_js_dictionary_receiver.js
Normal file
@@ -0,0 +1,220 @@
|
||||
/**
|
||||
* JavaScript Dictionary Receiver Test
|
||||
* Tests the smartreceive function with dictionary payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natbridge.js');
|
||||
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Dictionary Receiver Test ===\n');
|
||||
|
||||
// Create a mock NATS message with dictionary payloads
|
||||
const simpleDict = { key1: 'value1', key2: 'value2' };
|
||||
const nestedDict = { outer: { inner: 'value', number: 42 } };
|
||||
const arrayDict = { items: [1, 2, 3, 'four', 'five'] };
|
||||
const mixedDict = { string: 'text', number: 123, boolean: true, null_val: null };
|
||||
|
||||
const testData = {
|
||||
correlation_id: 'test-receiver-dict-' + Date.now(),
|
||||
msg_id: 'msg-' + Date.now(),
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: '/test/dictionary',
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-dict-test',
|
||||
sender_id: 'sender-' + Date.now(),
|
||||
receiver_name: 'js-receiver',
|
||||
receiver_id: 'receiver-' + Date.now(),
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
broker_url: TEST_BROKER_URL,
|
||||
metadata: {},
|
||||
payloads: [
|
||||
{
|
||||
id: 'payload-1',
|
||||
dataname: 'simple_dict',
|
||||
payload_type: 'dictionary',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: Buffer.from(JSON.stringify(simpleDict)).length,
|
||||
data: Buffer.from(JSON.stringify(simpleDict)).toString('base64'),
|
||||
metadata: { payload_bytes: Buffer.from(JSON.stringify(simpleDict)).length }
|
||||
},
|
||||
{
|
||||
id: 'payload-2',
|
||||
dataname: 'nested_dict',
|
||||
payload_type: 'dictionary',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: Buffer.from(JSON.stringify(nestedDict)).length,
|
||||
data: Buffer.from(JSON.stringify(nestedDict)).toString('base64'),
|
||||
metadata: { payload_bytes: Buffer.from(JSON.stringify(nestedDict)).length }
|
||||
},
|
||||
{
|
||||
id: 'payload-3',
|
||||
dataname: 'array_dict',
|
||||
payload_type: 'dictionary',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: Buffer.from(JSON.stringify(arrayDict)).length,
|
||||
data: Buffer.from(JSON.stringify(arrayDict)).toString('base64'),
|
||||
metadata: { payload_bytes: Buffer.from(JSON.stringify(arrayDict)).length }
|
||||
},
|
||||
{
|
||||
id: 'payload-4',
|
||||
dataname: 'mixed_dict',
|
||||
payload_type: 'dictionary',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: Buffer.from(JSON.stringify(mixedDict)).length,
|
||||
data: Buffer.from(JSON.stringify(mixedDict)).toString('base64'),
|
||||
metadata: { payload_bytes: Buffer.from(JSON.stringify(mixedDict)).length }
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const mockMsg = {
|
||||
payload: JSON.stringify(testData)
|
||||
};
|
||||
|
||||
console.log('Mock Message Created:');
|
||||
console.log(` Correlation ID: ${testData.correlation_id}`);
|
||||
console.log(` Payloads: ${testData.payloads.length}`);
|
||||
console.log(` Payload types: ${testData.payloads.map(p => p.payload_type).join(', ')}\n`);
|
||||
|
||||
try {
|
||||
// Receive and process the message
|
||||
console.log('Receiving and processing message...');
|
||||
const env = await NATSBridge.smartreceive(
|
||||
mockMsg,
|
||||
{
|
||||
max_retries: 3,
|
||||
base_delay: 100,
|
||||
max_delay: 1000
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Received Envelope ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate received data
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (!env.correlation_id) {
|
||||
console.log('❌ correlation_id is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ correlation_id present');
|
||||
}
|
||||
|
||||
if (env.payloads.length !== 4) {
|
||||
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
// Expected data
|
||||
const expectedData = [
|
||||
['simple_dict', simpleDict, 'dictionary'],
|
||||
['nested_dict', nestedDict, 'dictionary'],
|
||||
['array_dict', arrayDict, 'dictionary'],
|
||||
['mixed_dict', mixedDict, 'dictionary']
|
||||
];
|
||||
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
const expected = expectedData[i];
|
||||
|
||||
if (payload[0] !== expected[0]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected dataname '${expected[0]}', got '${payload[0]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct dataname`);
|
||||
}
|
||||
|
||||
if (payload[2] !== expected[2]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected type '${expected[2]}', got '${payload[2]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct type`);
|
||||
}
|
||||
|
||||
const dataMatch = JSON.stringify(payload[1]) === JSON.stringify(expected[1]);
|
||||
if (!dataMatch) {
|
||||
console.log(`❌ Payload ${i + 1}: Data mismatch`);
|
||||
console.log(` Expected: ${JSON.stringify(expected[1])}`);
|
||||
console.log(` Got: ${JSON.stringify(payload[1])}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Data correctly deserialized`);
|
||||
}
|
||||
}
|
||||
|
||||
// Test round-trip with receive
|
||||
console.log('\n=== Round-trip Test ===');
|
||||
const roundTripData = {
|
||||
correlation_id: 'roundtrip-' + Date.now(),
|
||||
msg_id: 'msg-' + Date.now(),
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: '/test/dictionary',
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-test',
|
||||
sender_id: 'sender-' + Date.now(),
|
||||
receiver_name: 'js-receiver',
|
||||
receiver_id: 'receiver-' + Date.now(),
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
broker_url: TEST_BROKER_URL,
|
||||
metadata: {},
|
||||
payloads: [
|
||||
{
|
||||
id: 'payload-rt',
|
||||
dataname: 'roundtrip',
|
||||
payload_type: 'dictionary',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: Buffer.from(JSON.stringify({ test: 'data', nested: { a: 1, b: 2 } })).length,
|
||||
data: Buffer.from(JSON.stringify({ test: 'data', nested: { a: 1, b: 2 } })).toString('base64'),
|
||||
metadata: { payload_bytes: Buffer.from(JSON.stringify({ test: 'data', nested: { a: 1, b: 2 } })).length }
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const mockRtMsg = { payload: JSON.stringify(roundTripData) };
|
||||
const rtEnv = await NATSBridge.smartreceive(mockRtMsg);
|
||||
|
||||
if (rtEnv.payloads.length === 1 &&
|
||||
rtEnv.payloads[0][0] === 'roundtrip' &&
|
||||
rtEnv.payloads[0][2] === 'dictionary') {
|
||||
console.log('✅ Round-trip test successful');
|
||||
} else {
|
||||
console.log('❌ Round-trip test failed');
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
178
test/test_js_dictionary_sender.js
Normal file
178
test/test_js_dictionary_sender.js
Normal file
@@ -0,0 +1,178 @@
|
||||
/**
|
||||
* JavaScript Dictionary Sender Test
|
||||
* Tests the smartsend function with dictionary payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natbridge.js');
|
||||
|
||||
const TEST_SUBJECT = '/test/dictionary';
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Dictionary Sender Test ===\n');
|
||||
|
||||
const correlationId = NATSBridge.uuidv4();
|
||||
console.log(`Correlation ID: ${correlationId}`);
|
||||
console.log(`Subject: ${TEST_SUBJECT}`);
|
||||
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
|
||||
|
||||
// Test data - various dictionary structures
|
||||
const testData = [
|
||||
['simple_dict', { key1: 'value1', key2: 'value2' }, 'dictionary'],
|
||||
['nested_dict', { outer: { inner: 'value', number: 42 } }, 'dictionary'],
|
||||
['array_dict', { items: [1, 2, 3, 'four', 'five'] }, 'dictionary'],
|
||||
['mixed_dict', { string: 'text', number: 123, boolean: true, null_val: null }, 'dictionary']
|
||||
];
|
||||
|
||||
try {
|
||||
// Send the message
|
||||
console.log('Sending dictionary payloads...');
|
||||
const [env, envJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
testData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: correlationId,
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-dict-test',
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Envelope Created ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Purpose: ${env.msg_purpose}`);
|
||||
console.log(`Sender: ${env.sender_name}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate envelope structure
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (env.payloads.length !== 4) {
|
||||
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
// Test each payload
|
||||
const expectedDatanames = ['simple_dict', 'nested_dict', 'array_dict', 'mixed_dict'];
|
||||
const expectedTypes = ['dictionary', 'dictionary', 'dictionary', 'dictionary'];
|
||||
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
|
||||
if (payload.dataname !== expectedDatanames[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected dataname '${expectedDatanames[i]}', got '${payload.dataname}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct dataname`);
|
||||
}
|
||||
|
||||
if (payload.payload_type !== expectedTypes[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected type '${expectedTypes[i]}', got '${payload.payload_type}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct type`);
|
||||
}
|
||||
|
||||
if (payload.transport !== 'direct') {
|
||||
console.log(`❌ Payload ${i + 1}: Expected transport 'direct', got '${payload.transport}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct transport`);
|
||||
}
|
||||
|
||||
if (payload.encoding !== 'base64') {
|
||||
console.log(`❌ Payload ${i + 1}: Expected encoding 'base64', got '${payload.encoding}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct encoding`);
|
||||
}
|
||||
|
||||
// Decode and verify the data
|
||||
const decodedData = JSON.parse(Buffer.from(payload.data, 'base64').toString('utf8'));
|
||||
const originalData = testData[i][1];
|
||||
|
||||
const originalJson = JSON.stringify(originalData);
|
||||
const decodedJson = JSON.stringify(decodedData);
|
||||
|
||||
if (originalJson !== decodedJson) {
|
||||
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
|
||||
console.log(` Expected: ${originalJson}`);
|
||||
console.log(` Got: ${decodedJson}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
|
||||
}
|
||||
|
||||
console.log(` Size: ${payload.size} bytes\n`);
|
||||
}
|
||||
|
||||
// Test round-trip serialization
|
||||
console.log('=== Round-trip Serialization Test ===');
|
||||
const roundTripTestData = [
|
||||
['roundtrip', { test: 'data', numbers: [1, 2, 3], nested: { a: 1, b: 2 } }, 'dictionary']
|
||||
];
|
||||
|
||||
const [rtEnv, rtEnvJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
roundTripTestData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: 'roundtrip-' + correlationId,
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
const rtPayload = rtEnv.payloads[0];
|
||||
const rtDecoded = JSON.parse(Buffer.from(rtPayload.data, 'base64').toString('utf8'));
|
||||
|
||||
if (JSON.stringify(rtDecoded) === JSON.stringify(roundTripTestData[0][1])) {
|
||||
console.log('✅ Round-trip serialization successful');
|
||||
} else {
|
||||
console.log('❌ Round-trip serialization failed');
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Test JSON string output
|
||||
console.log('\n=== JSON String Output Test ===');
|
||||
try {
|
||||
const parsed = JSON.parse(envJsonStr);
|
||||
if (parsed.correlation_id === env.correlation_id &&
|
||||
parsed.payloads.length === env.payloads.length) {
|
||||
console.log('✅ JSON string is valid and matches envelope');
|
||||
} else {
|
||||
console.log('❌ JSON string does not match envelope');
|
||||
passed = false;
|
||||
}
|
||||
} catch (e) {
|
||||
console.log('❌ JSON string is invalid:', e.message);
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
204
test/test_js_mix_payloads_sender.js
Normal file
204
test/test_js_mix_payloads_sender.js
Normal file
@@ -0,0 +1,204 @@
|
||||
/**
|
||||
* JavaScript Mix Payloads Sender Test
|
||||
* Tests the smartsend function with mixed payload types
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natbridge.js');
|
||||
|
||||
const TEST_SUBJECT = '/test/mix';
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Mix Payloads Sender Test ===\n');
|
||||
|
||||
const correlationId = NATSBridge.uuidv4();
|
||||
console.log(`Correlation ID: ${correlationId}`);
|
||||
console.log(`Subject: ${TEST_SUBJECT}`);
|
||||
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
|
||||
|
||||
// Test data - mixed payload types
|
||||
const textData = 'Hello, NATSBridge!';
|
||||
const dictData = { key1: 'value1', key2: 42, nested: { a: 1, b: 2 } };
|
||||
const binaryData = Buffer.from([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]); // PNG header
|
||||
|
||||
// Table data
|
||||
const tableData = [
|
||||
{ id: 1, name: 'Alice', age: 30 },
|
||||
{ id: 2, name: 'Bob', age: 25 },
|
||||
{ id: 3, name: 'Charlie', age: 35 }
|
||||
];
|
||||
|
||||
const testData = [
|
||||
['message', textData, 'text'],
|
||||
['config', dictData, 'dictionary'],
|
||||
['image', binaryData, 'image'],
|
||||
['users', tableData, 'table']
|
||||
];
|
||||
|
||||
try {
|
||||
// Send the message
|
||||
console.log('Sending mixed payloads...');
|
||||
const [env, envJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
testData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: correlationId,
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-mix-test',
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Envelope Created ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Purpose: ${env.msg_purpose}`);
|
||||
console.log(`Sender: ${env.sender_name}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate envelope structure
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (env.payloads.length !== 4) {
|
||||
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
// Test each payload
|
||||
const expectedDatanames = ['message', 'config', 'image', 'users'];
|
||||
const expectedTypes = ['text', 'dictionary', 'image', 'table'];
|
||||
const expectedData = [textData, dictData, binaryData, tableData];
|
||||
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
|
||||
if (payload.dataname !== expectedDatanames[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected dataname '${expectedDatanames[i]}', got '${payload.dataname}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct dataname`);
|
||||
}
|
||||
|
||||
if (payload.payload_type !== expectedTypes[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Expected type '${expectedTypes[i]}', got '${payload.payload_type}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct type`);
|
||||
}
|
||||
|
||||
if (payload.transport !== 'direct') {
|
||||
console.log(`❌ Payload ${i + 1}: Expected transport 'direct', got '${payload.transport}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct transport`);
|
||||
}
|
||||
|
||||
if (payload.encoding !== 'base64') {
|
||||
console.log(`❌ Payload ${i + 1}: Expected encoding 'base64', got '${payload.encoding}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Correct encoding`);
|
||||
}
|
||||
|
||||
// Verify data integrity based on type
|
||||
if (expectedTypes[i] === 'text') {
|
||||
const decodedData = Buffer.from(payload.data, 'base64').toString('utf8');
|
||||
if (decodedData !== expectedData[i]) {
|
||||
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
|
||||
}
|
||||
} else if (expectedTypes[i] === 'dictionary') {
|
||||
const decodedData = JSON.parse(Buffer.from(payload.data, 'base64').toString('utf8'));
|
||||
if (JSON.stringify(decodedData) !== JSON.stringify(expectedData[i])) {
|
||||
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
|
||||
}
|
||||
} else if (expectedTypes[i] === 'image') {
|
||||
const decodedData = Buffer.from(payload.data, 'base64');
|
||||
if (decodedData.length !== expectedData[i].length) {
|
||||
console.log(`❌ Payload ${i + 1}: Length mismatch`);
|
||||
passed = false;
|
||||
} else {
|
||||
let dataMatch = true;
|
||||
for (let j = 0; j < expectedData[i].length; j++) {
|
||||
if (decodedData[j] !== expectedData[i][j]) {
|
||||
dataMatch = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (dataMatch) {
|
||||
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
|
||||
} else {
|
||||
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
|
||||
passed = false;
|
||||
}
|
||||
}
|
||||
} else if (expectedTypes[i] === 'table') {
|
||||
const decodedData = Buffer.from(payload.data, 'base64');
|
||||
if (decodedData.length > 0) {
|
||||
console.log(`✅ Payload ${i + 1}: Arrow IPC data present (${decodedData.length} bytes)`);
|
||||
} else {
|
||||
console.log(`❌ Payload ${i + 1}: Arrow IPC data is empty`);
|
||||
passed = false;
|
||||
}
|
||||
}
|
||||
|
||||
console.log(` Size: ${payload.size} bytes\n`);
|
||||
}
|
||||
|
||||
// Test with chat-like payload (text + image + audio)
|
||||
console.log('=== Chat-like Payload Test ===');
|
||||
const chatData = [
|
||||
['text', 'Hello!', 'text'],
|
||||
['image', Buffer.from([0xFF, 0xD8, 0xFF, 0xE0]), 'image'],
|
||||
['audio', Buffer.from([0x46, 0x4C, 0x41, 0x43]), 'audio']
|
||||
];
|
||||
|
||||
const [chatEnv, _] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
chatData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: 'chat-' + correlationId,
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
if (chatEnv.payloads.length === 3) {
|
||||
console.log('✅ Chat-like payloads handled correctly');
|
||||
} else {
|
||||
console.log('❌ Chat-like payloads handling failed');
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
172
test/test_js_table_receiver.js
Normal file
172
test/test_js_table_receiver.js
Normal file
@@ -0,0 +1,172 @@
|
||||
/**
|
||||
* JavaScript Table Receiver Test
|
||||
* Tests the smartreceive function with table (Arrow IPC) payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natbridge.js');
|
||||
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Table Receiver Test ===\n');
|
||||
|
||||
// Create a mock NATS message with table payload
|
||||
const tableData = [
|
||||
{ id: 1, name: 'Alice', age: 30, active: true },
|
||||
{ id: 2, name: 'Bob', age: 25, active: false },
|
||||
{ id: 3, name: 'Charlie', age: 35, active: true }
|
||||
];
|
||||
|
||||
// Convert to Arrow IPC format
|
||||
const arrow = require('apache-arrow');
|
||||
const fields = [
|
||||
new arrow.Field('id', arrow.Int64, true),
|
||||
new arrow.Field('name', arrow.Utf8, true),
|
||||
new arrow.Field('age', arrow.Int64, true),
|
||||
new arrow.Field('active', arrow.Boolean, true)
|
||||
];
|
||||
const schema = new arrow.Schema(fields);
|
||||
const batches = [];
|
||||
for (const row of tableData) {
|
||||
const batch = arrow.recordBatch.fromObjects([row], schema);
|
||||
batches.push(batch);
|
||||
}
|
||||
const buffers = arrow.ipc.recordBatchesToMessage(batches, schema).buffers;
|
||||
const combined = new Uint8Array(buffers.reduce((acc, b) => acc + b.byteLength, 0));
|
||||
let offset = 0;
|
||||
for (const buf of buffers) {
|
||||
combined.set(new Uint8Array(buf), offset);
|
||||
offset += buf.byteLength;
|
||||
}
|
||||
const arrowBuffer = Buffer.from(combined);
|
||||
|
||||
const testData = {
|
||||
correlation_id: 'js-table-receiver-' + Date.now(),
|
||||
msg_id: 'msg-' + Date.now(),
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: '/test/table',
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-table-test',
|
||||
sender_id: 'sender-' + Date.now(),
|
||||
receiver_name: 'js-receiver',
|
||||
receiver_id: 'receiver-' + Date.now(),
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
broker_url: TEST_BROKER_URL,
|
||||
metadata: {},
|
||||
payloads: [
|
||||
{
|
||||
id: 'payload-1',
|
||||
dataname: 'users_table',
|
||||
payload_type: 'table',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: arrowBuffer.length,
|
||||
data: arrowBuffer.toString('base64'),
|
||||
metadata: { payload_bytes: arrowBuffer.length }
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const mockMsg = {
|
||||
payload: JSON.stringify(testData)
|
||||
};
|
||||
|
||||
console.log('Mock Message Created:');
|
||||
console.log(` Correlation ID: ${testData.correlation_id}`);
|
||||
console.log(` Payloads: ${testData.payloads.length}`);
|
||||
console.log(` Payload type: ${testData.payloads[0].payload_type}`);
|
||||
console.log(` Transport: ${testData.payloads[0].transport}\n`);
|
||||
|
||||
try {
|
||||
// Receive and process the message
|
||||
console.log('Receiving and processing message...');
|
||||
const env = await NATSBridge.smartreceive(
|
||||
mockMsg,
|
||||
{
|
||||
max_retries: 3,
|
||||
base_delay: 100,
|
||||
max_delay: 1000
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Received Envelope ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate received data
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (!env.correlation_id) {
|
||||
console.log('❌ correlation_id is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ correlation_id present');
|
||||
}
|
||||
|
||||
if (env.payloads.length !== 1) {
|
||||
console.log(`❌ Expected 1 payload, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
const payload = env.payloads[0];
|
||||
if (payload[0] !== 'users_table') {
|
||||
console.log(`❌ Expected dataname 'users_table', got '${payload[0]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct dataname');
|
||||
}
|
||||
|
||||
if (payload[2] !== 'table') {
|
||||
console.log(`❌ Expected type 'table', got '${payload[2]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct type');
|
||||
}
|
||||
|
||||
// Verify table data is a Buffer (Arrow IPC format)
|
||||
if (payload[1] instanceof Buffer || payload[1] instanceof Uint8Array) {
|
||||
console.log('✅ Table data is Arrow IPC buffer');
|
||||
console.log(` Buffer size: ${payload[1].length} bytes`);
|
||||
} else {
|
||||
console.log(`❌ Expected Buffer/Uint8Array, got ${typeof payload[1]}`);
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Test round-trip with Arrow deserialization
|
||||
console.log('\n=== Arrow Deserialization Test ===');
|
||||
try {
|
||||
const table = arrow.tableFromRawBytes(payload[1]);
|
||||
console.log(`✅ Arrow table deserialized successfully`);
|
||||
console.log(` Schema: ${table.schema.fields.map(f => f.name).join(', ')}`);
|
||||
console.log(` Num rows: ${table.numRows}`);
|
||||
} catch (e) {
|
||||
console.log('❌ Arrow deserialization failed:', e.message);
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
179
test/test_js_table_sender.js
Normal file
179
test/test_js_table_sender.js
Normal file
@@ -0,0 +1,179 @@
|
||||
/**
|
||||
* JavaScript Table Sender Test
|
||||
* Tests the smartsend function with table (Arrow IPC) payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natbridge.js');
|
||||
|
||||
const TEST_SUBJECT = '/test/table';
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Table Sender Test ===\n');
|
||||
|
||||
const correlationId = NATSBridge.uuidv4();
|
||||
console.log(`Correlation ID: ${correlationId}`);
|
||||
console.log(`Subject: ${TEST_SUBJECT}`);
|
||||
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
|
||||
|
||||
// Test data - table data as array of objects
|
||||
const tableData = [
|
||||
{ id: 1, name: 'Alice', age: 30, active: true },
|
||||
{ id: 2, name: 'Bob', age: 25, active: false },
|
||||
{ id: 3, name: 'Charlie', age: 35, active: true },
|
||||
{ id: 4, name: 'Diana', age: 28, active: true },
|
||||
{ id: 5, name: 'Eve', age: 32, active: false }
|
||||
];
|
||||
|
||||
const testData = [
|
||||
['users_table', tableData, 'table']
|
||||
];
|
||||
|
||||
try {
|
||||
// Send the message
|
||||
console.log('Sending table payload...');
|
||||
const [env, envJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
testData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: correlationId,
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-table-test',
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Envelope Created ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Purpose: ${env.msg_purpose}`);
|
||||
console.log(`Sender: ${env.sender_name}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate envelope structure
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (env.payloads.length !== 1) {
|
||||
console.log(`❌ Expected 1 payload, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
const payload = env.payloads[0];
|
||||
if (payload.dataname !== 'users_table') {
|
||||
console.log(`❌ Expected dataname 'users_table', got '${payload.dataname}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct dataname');
|
||||
}
|
||||
|
||||
if (payload.payload_type !== 'table') {
|
||||
console.log(`❌ Expected payload_type 'table', got '${payload.payload_type}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct payload_type');
|
||||
}
|
||||
|
||||
if (payload.transport !== 'direct') {
|
||||
console.log(`❌ Expected transport 'direct', got '${payload.transport}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct transport');
|
||||
}
|
||||
|
||||
if (payload.encoding !== 'base64') {
|
||||
console.log(`❌ Expected encoding 'base64', got '${payload.encoding}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct encoding');
|
||||
}
|
||||
|
||||
// Verify Arrow IPC data can be decoded
|
||||
console.log('\n=== Arrow IPC Verification ===');
|
||||
const decodedData = Buffer.from(payload.data, 'base64');
|
||||
console.log(`Arrow IPC buffer size: ${decodedData.length} bytes`);
|
||||
|
||||
if (decodedData.length > 0) {
|
||||
console.log('✅ Arrow IPC data present');
|
||||
} else {
|
||||
console.log('❌ Arrow IPC data is empty');
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Test with larger table
|
||||
console.log('\n=== Larger Table Test ===');
|
||||
const largeTableData = [];
|
||||
for (let i = 1; i <= 100; i++) {
|
||||
largeTableData.push({
|
||||
id: i,
|
||||
name: `User${i}`,
|
||||
age: Math.floor(Math.random() * 100),
|
||||
active: Math.random() > 0.5,
|
||||
score: Math.random() * 100
|
||||
});
|
||||
}
|
||||
|
||||
const largeTestData = [
|
||||
['large_table', largeTableData, 'table']
|
||||
];
|
||||
|
||||
const [largeEnv, _] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
largeTestData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: 'large-' + correlationId,
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
if (largeEnv.payloads.length === 1) {
|
||||
console.log('✅ Large table handled correctly');
|
||||
console.log(` Size: ${largeEnv.payloads[0].size} bytes`);
|
||||
} else {
|
||||
console.log('❌ Large table handling failed');
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Test JSON string output
|
||||
console.log('\n=== JSON String Output Test ===');
|
||||
try {
|
||||
const parsed = JSON.parse(envJsonStr);
|
||||
if (parsed.correlation_id === env.correlation_id &&
|
||||
parsed.payloads.length === env.payloads.length) {
|
||||
console.log('✅ JSON string is valid and matches envelope');
|
||||
} else {
|
||||
console.log('❌ JSON string does not match envelope');
|
||||
passed = false;
|
||||
}
|
||||
} catch (e) {
|
||||
console.log('❌ JSON string is invalid:', e.message);
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
206
test/test_js_text_receiver.js
Normal file
206
test/test_js_text_receiver.js
Normal file
@@ -0,0 +1,206 @@
|
||||
/**
|
||||
* JavaScript Text Receiver Test
|
||||
* Tests the smartreceive function with text payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natbridge.js');
|
||||
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Text Receiver Test ===\n');
|
||||
|
||||
// Create a mock NATS message with text payload
|
||||
const testData = {
|
||||
correlation_id: 'test-receiver-' + Date.now(),
|
||||
msg_id: 'msg-' + Date.now(),
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: '/test/text',
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-text-test',
|
||||
sender_id: 'sender-' + Date.now(),
|
||||
receiver_name: 'js-receiver',
|
||||
receiver_id: 'receiver-' + Date.now(),
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
broker_url: TEST_BROKER_URL,
|
||||
metadata: {},
|
||||
payloads: [
|
||||
{
|
||||
id: 'payload-' + Date.now(),
|
||||
dataname: 'message',
|
||||
payload_type: 'text',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: 38,
|
||||
data: Buffer.from('Hello, NATSBridge! This is a test message.').toString('base64'),
|
||||
metadata: { payload_bytes: 38 }
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const mockMsg = {
|
||||
payload: JSON.stringify(testData)
|
||||
};
|
||||
|
||||
console.log('Mock Message Created:');
|
||||
console.log(` Correlation ID: ${testData.correlation_id}`);
|
||||
console.log(` Payloads: ${testData.payloads.length}`);
|
||||
console.log(` Payload dataname: ${testData.payloads[0].dataname}`);
|
||||
console.log(` Payload type: ${testData.payloads[0].payload_type}`);
|
||||
console.log(` Transport: ${testData.payloads[0].transport}\n`);
|
||||
|
||||
try {
|
||||
// Receive and process the message
|
||||
console.log('Receiving and processing message...');
|
||||
const env = await NATSBridge.smartreceive(
|
||||
mockMsg,
|
||||
{
|
||||
max_retries: 3,
|
||||
base_delay: 100,
|
||||
max_delay: 1000
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Received Envelope ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate received data
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (!env.correlation_id) {
|
||||
console.log('❌ correlation_id is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ correlation_id present');
|
||||
}
|
||||
|
||||
if (env.payloads.length !== 1) {
|
||||
console.log(`❌ Expected 1 payload, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
const payload = env.payloads[0];
|
||||
if (payload[0] !== 'message') {
|
||||
console.log(`❌ Expected dataname 'message', got '${payload[0]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct dataname');
|
||||
}
|
||||
|
||||
if (payload[2] !== 'text') {
|
||||
console.log(`❌ Expected type 'text', got '${payload[2]}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct type');
|
||||
}
|
||||
|
||||
if (payload[1] !== 'Hello, NATSBridge! This is a test message.') {
|
||||
console.log(`❌ Data mismatch`);
|
||||
console.log(` Expected: Hello, NATSBridge! This is a test message.`);
|
||||
console.log(` Got: ${payload[1]}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Data correctly deserialized');
|
||||
}
|
||||
|
||||
// Test with multiple text payloads
|
||||
console.log('\n=== Multiple Text Payloads Test ===');
|
||||
const multiTestData = {
|
||||
correlation_id: 'multi-receiver-' + Date.now(),
|
||||
msg_id: 'msg-' + Date.now(),
|
||||
timestamp: new Date().toISOString(),
|
||||
send_to: '/test/text',
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-text-test',
|
||||
sender_id: 'sender-' + Date.now(),
|
||||
receiver_name: 'js-receiver',
|
||||
receiver_id: 'receiver-' + Date.now(),
|
||||
reply_to: '',
|
||||
reply_to_msg_id: '',
|
||||
broker_url: TEST_BROKER_URL,
|
||||
metadata: {},
|
||||
payloads: [
|
||||
{
|
||||
id: 'payload-1',
|
||||
dataname: 'msg1',
|
||||
payload_type: 'text',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: 16,
|
||||
data: Buffer.from('First message').toString('base64'),
|
||||
metadata: { payload_bytes: 16 }
|
||||
},
|
||||
{
|
||||
id: 'payload-2',
|
||||
dataname: 'msg2',
|
||||
payload_type: 'text',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: 16,
|
||||
data: Buffer.from('Second message').toString('base64'),
|
||||
metadata: { payload_bytes: 16 }
|
||||
},
|
||||
{
|
||||
id: 'payload-3',
|
||||
dataname: 'msg3',
|
||||
payload_type: 'text',
|
||||
transport: 'direct',
|
||||
encoding: 'base64',
|
||||
size: 16,
|
||||
data: Buffer.from('Third message').toString('base64'),
|
||||
metadata: { payload_bytes: 16 }
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
const mockMultiMsg = {
|
||||
payload: JSON.stringify(multiTestData)
|
||||
};
|
||||
|
||||
const multiEnv = await NATSBridge.smartreceive(mockMultiMsg);
|
||||
|
||||
if (multiEnv.payloads.length === 3) {
|
||||
console.log('✅ Multiple payloads handled correctly');
|
||||
|
||||
// Verify each payload
|
||||
const expectedMessages = ['First message', 'Second message', 'Third message'];
|
||||
for (let i = 0; i < 3; i++) {
|
||||
if (multiEnv.payloads[i][1] === expectedMessages[i]) {
|
||||
console.log(`✅ Payload ${i + 1} correctly deserialized`);
|
||||
} else {
|
||||
console.log(`❌ Payload ${i + 1} mismatch`);
|
||||
passed = false;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
console.log(`❌ Expected 3 payloads, got ${multiEnv.payloads.length}`);
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
169
test/test_js_text_sender.js
Normal file
169
test/test_js_text_sender.js
Normal file
@@ -0,0 +1,169 @@
|
||||
/**
|
||||
* JavaScript Text Sender Test
|
||||
* Tests the smartsend function with text payloads
|
||||
*/
|
||||
|
||||
const NATSBridge = require('../src/natbridge.js');
|
||||
|
||||
const TEST_SUBJECT = '/test/text';
|
||||
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
|
||||
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
|
||||
|
||||
async function runTest() {
|
||||
console.log('=== JavaScript Text Sender Test ===\n');
|
||||
|
||||
const correlationId = NATSBridge.uuidv4();
|
||||
console.log(`Correlation ID: ${correlationId}`);
|
||||
console.log(`Subject: ${TEST_SUBJECT}`);
|
||||
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
|
||||
|
||||
// Test data
|
||||
const textData = 'Hello, NATSBridge! This is a test message.';
|
||||
const testData = [
|
||||
['message', textData, 'text']
|
||||
];
|
||||
|
||||
try {
|
||||
// Send the message
|
||||
console.log('Sending text payload...');
|
||||
const [env, envJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
testData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: correlationId,
|
||||
msg_purpose: 'test',
|
||||
sender_name: 'js-text-test',
|
||||
is_publish: false // Don't actually publish for this test
|
||||
}
|
||||
);
|
||||
|
||||
console.log('\n=== Envelope Created ===');
|
||||
console.log(`Correlation ID: ${env.correlation_id}`);
|
||||
console.log(`Message ID: ${env.msg_id}`);
|
||||
console.log(`Timestamp: ${env.timestamp}`);
|
||||
console.log(`Subject: ${env.send_to}`);
|
||||
console.log(`Purpose: ${env.msg_purpose}`);
|
||||
console.log(`Sender: ${env.sender_name}`);
|
||||
console.log(`Payloads: ${env.payloads.length}\n`);
|
||||
|
||||
// Validate envelope structure
|
||||
console.log('=== Validation ===');
|
||||
let passed = true;
|
||||
|
||||
if (!env.correlation_id) {
|
||||
console.log('❌ correlation_id is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ correlation_id present');
|
||||
}
|
||||
|
||||
if (!env.msg_id) {
|
||||
console.log('❌ msg_id is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ msg_id present');
|
||||
}
|
||||
|
||||
if (!env.timestamp) {
|
||||
console.log('❌ timestamp is missing');
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ timestamp present');
|
||||
}
|
||||
|
||||
if (env.payloads.length !== 1) {
|
||||
console.log(`❌ Expected 1 payload, got ${env.payloads.length}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct number of payloads');
|
||||
}
|
||||
|
||||
const payload = env.payloads[0];
|
||||
if (payload.dataname !== 'message') {
|
||||
console.log(`❌ Expected dataname 'message', got '${payload.dataname}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct dataname');
|
||||
}
|
||||
|
||||
if (payload.payload_type !== 'text') {
|
||||
console.log(`❌ Expected payload_type 'text', got '${payload.payload_type}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct payload_type');
|
||||
}
|
||||
|
||||
if (payload.transport !== 'direct') {
|
||||
console.log(`❌ Expected transport 'direct', got '${payload.transport}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct transport');
|
||||
}
|
||||
|
||||
if (payload.encoding !== 'base64') {
|
||||
console.log(`❌ Expected encoding 'base64', got '${payload.encoding}'`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Correct encoding');
|
||||
}
|
||||
|
||||
// Decode and verify the data
|
||||
const decodedData = Buffer.from(payload.data, 'base64').toString('utf8');
|
||||
if (decodedData !== textData) {
|
||||
console.log(`❌ Decoded data mismatch`);
|
||||
console.log(` Expected: ${textData}`);
|
||||
console.log(` Got: ${decodedData}`);
|
||||
passed = false;
|
||||
} else {
|
||||
console.log('✅ Data integrity verified');
|
||||
}
|
||||
|
||||
console.log(`\nPayload size: ${payload.size} bytes`);
|
||||
console.log(`Base64 data length: ${payload.data.length} chars`);
|
||||
|
||||
// Test with multiple text payloads
|
||||
console.log('\n=== Multiple Text Payloads Test ===');
|
||||
const multiTestData = [
|
||||
['msg1', 'First message', 'text'],
|
||||
['msg2', 'Second message', 'text'],
|
||||
['msg3', 'Third message', 'text']
|
||||
];
|
||||
|
||||
const [multiEnv, multiEnvJsonStr] = await NATSBridge.smartsend(
|
||||
TEST_SUBJECT,
|
||||
multiTestData,
|
||||
{
|
||||
broker_url: TEST_BROKER_URL,
|
||||
fileserver_url: TEST_FILESERVER_URL,
|
||||
correlation_id: 'multi-test-' + correlationId,
|
||||
is_publish: false
|
||||
}
|
||||
);
|
||||
|
||||
if (multiEnv.payloads.length === 3) {
|
||||
console.log('✅ Multiple payloads handled correctly');
|
||||
} else {
|
||||
console.log(`❌ Expected 3 payloads, got ${multiEnv.payloads.length}`);
|
||||
passed = false;
|
||||
}
|
||||
|
||||
// Final result
|
||||
console.log('\n=== Test Result ===');
|
||||
if (passed) {
|
||||
console.log('✅ ALL TESTS PASSED');
|
||||
process.exit(0);
|
||||
} else {
|
||||
console.log('❌ SOME TESTS FAILED');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('❌ Test failed with error:', error.message);
|
||||
console.error(error.stack);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runTest();
|
||||
@@ -1,80 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Test script for Dictionary transport testing
|
||||
// Tests receiving 1 large and 1 small Dictionaries via direct and link transport
|
||||
// Uses NATSBridge.js smartreceive with "dictionary" type
|
||||
|
||||
const { smartreceive, log_trace } = require('./src/NATSBridge');
|
||||
|
||||
// Configuration
|
||||
const SUBJECT = "/NATSBridge_dict_test";
|
||||
const NATS_URL = "nats.yiem.cc";
|
||||
|
||||
// Helper: Log with correlation ID
|
||||
function log_trace(message) {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`[${timestamp}] ${message}`);
|
||||
}
|
||||
|
||||
// Receiver: Listen for messages and verify Dictionary handling
|
||||
async function test_dict_receive() {
|
||||
// Connect to NATS
|
||||
const { connect } = require('nats');
|
||||
const nc = await connect({ servers: [NATS_URL] });
|
||||
|
||||
// Subscribe to the subject
|
||||
const sub = nc.subscribe(SUBJECT);
|
||||
|
||||
for await (const msg of sub) {
|
||||
log_trace(`Received message on ${msg.subject}`);
|
||||
|
||||
// Use NATSBridge.smartreceive to handle the data
|
||||
const result = await smartreceive(
|
||||
msg,
|
||||
{
|
||||
maxRetries: 5,
|
||||
baseDelay: 100,
|
||||
maxDelay: 5000
|
||||
}
|
||||
);
|
||||
|
||||
// Result is an envelope dictionary with payloads field
|
||||
// Access payloads with result.payloads
|
||||
for (const { dataname, data, type } of result.payloads) {
|
||||
if (typeof data === 'object' && data !== null && !Array.isArray(data)) {
|
||||
log_trace(`Received Dictionary '${dataname}' of type ${type}`);
|
||||
|
||||
// Display dictionary contents
|
||||
console.log(" Contents:");
|
||||
for (const [key, value] of Object.entries(data)) {
|
||||
console.log(` ${key} => ${value}`);
|
||||
}
|
||||
|
||||
// Save to JSON file
|
||||
const fs = require('fs');
|
||||
const output_path = `./received_${dataname}.json`;
|
||||
const json_str = JSON.stringify(data, null, 2);
|
||||
fs.writeFileSync(output_path, json_str);
|
||||
log_trace(`Saved Dictionary to ${output_path}`);
|
||||
} else {
|
||||
log_trace(`Received unexpected data type for '${dataname}': ${typeof data}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Keep listening for 10 seconds
|
||||
setTimeout(() => {
|
||||
nc.close();
|
||||
process.exit(0);
|
||||
}, 120000);
|
||||
}
|
||||
|
||||
// Run the test
|
||||
console.log("Starting Dictionary transport test...");
|
||||
console.log("Note: This receiver will wait for messages from the sender.");
|
||||
console.log("Run test_js_to_js_dict_sender.js first to send test data.");
|
||||
|
||||
// Run receiver
|
||||
console.log("testing smartreceive");
|
||||
test_dict_receive();
|
||||
|
||||
console.log("Test completed.");
|
||||
@@ -1,164 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Test script for Dictionary transport testing
|
||||
// Tests sending 1 large and 1 small Dictionaries via direct and link transport
|
||||
// Uses NATSBridge.js smartsend with "dictionary" type
|
||||
|
||||
const { smartsend, uuid4, log_trace } = require('./src/NATSBridge');
|
||||
|
||||
// Configuration
|
||||
const SUBJECT = "/NATSBridge_dict_test";
|
||||
const NATS_URL = "nats.yiem.cc";
|
||||
const FILESERVER_URL = "http://192.168.88.104:8080";
|
||||
|
||||
// Create correlation ID for tracing
|
||||
const correlation_id = uuid4();
|
||||
|
||||
// Helper: Log with correlation ID
|
||||
function log_trace(message) {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`[${timestamp}] [Correlation: ${correlation_id}] ${message}`);
|
||||
}
|
||||
|
||||
// File upload handler for plik server
|
||||
async function plik_upload_handler(fileserver_url, dataname, data, correlation_id) {
|
||||
// Get upload ID
|
||||
const url_getUploadID = `${fileserver_url}/upload`;
|
||||
const headers = {
|
||||
"Content-Type": "application/json"
|
||||
};
|
||||
const body = JSON.stringify({ OneShot: true });
|
||||
|
||||
let response = await fetch(url_getUploadID, {
|
||||
method: "POST",
|
||||
headers: headers,
|
||||
body: body
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to get upload ID: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const responseJson = await response.json();
|
||||
const uploadid = responseJson.id;
|
||||
const uploadtoken = responseJson.uploadToken;
|
||||
|
||||
// Upload file
|
||||
const formData = new FormData();
|
||||
const blob = new Blob([data], { type: "application/octet-stream" });
|
||||
formData.append("file", blob, dataname);
|
||||
|
||||
response = await fetch(`${fileserver_url}/file/${uploadid}`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"X-UploadToken": uploadtoken
|
||||
},
|
||||
body: formData
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to upload file: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const fileResponseJson = await response.json();
|
||||
const fileid = fileResponseJson.id;
|
||||
|
||||
const url = `${fileserver_url}/file/${uploadid}/${fileid}/${encodeURIComponent(dataname)}`;
|
||||
|
||||
return {
|
||||
status: response.status,
|
||||
uploadid: uploadid,
|
||||
fileid: fileid,
|
||||
url: url
|
||||
};
|
||||
}
|
||||
|
||||
// Sender: Send Dictionaries via smartsend
|
||||
async function test_dict_send() {
|
||||
// Create a small Dictionary (will use direct transport)
|
||||
const small_dict = {
|
||||
name: "Alice",
|
||||
age: 30,
|
||||
scores: [95, 88, 92],
|
||||
metadata: {
|
||||
height: 155,
|
||||
weight: 55
|
||||
}
|
||||
};
|
||||
|
||||
// Create a large Dictionary (will use link transport if > 1MB)
|
||||
const large_dict_ids = [];
|
||||
const large_dict_names = [];
|
||||
const large_dict_scores = [];
|
||||
const large_dict_categories = [];
|
||||
|
||||
for (let i = 0; i < 50000; i++) {
|
||||
large_dict_ids.push(i + 1);
|
||||
large_dict_names.push(`User_${i}`);
|
||||
large_dict_scores.push(Math.floor(Math.random() * 100) + 1);
|
||||
large_dict_categories.push(`Category_${Math.floor(Math.random() * 10) + 1}`);
|
||||
}
|
||||
|
||||
const large_dict = {
|
||||
ids: large_dict_ids,
|
||||
names: large_dict_names,
|
||||
scores: large_dict_scores,
|
||||
categories: large_dict_categories,
|
||||
metadata: {
|
||||
source: "test_generator",
|
||||
timestamp: new Date().toISOString()
|
||||
}
|
||||
};
|
||||
|
||||
// Test data 1: small Dictionary
|
||||
const data1 = { dataname: "small_dict", data: small_dict, type: "dictionary" };
|
||||
|
||||
// Test data 2: large Dictionary
|
||||
const data2 = { dataname: "large_dict", data: large_dict, type: "dictionary" };
|
||||
|
||||
// Use smartsend with dictionary type
|
||||
// For small Dictionary: will use direct transport (JSON encoded)
|
||||
// For large Dictionary: will use link transport (uploaded to fileserver)
|
||||
const env = await smartsend(
|
||||
SUBJECT,
|
||||
[data1, data2],
|
||||
{
|
||||
natsUrl: NATS_URL,
|
||||
fileserverUrl: FILESERVER_URL,
|
||||
fileserverUploadHandler: plik_upload_handler,
|
||||
sizeThreshold: 1_000_000,
|
||||
correlationId: correlation_id,
|
||||
msgPurpose: "chat",
|
||||
senderName: "dict_sender",
|
||||
receiverName: "",
|
||||
receiverId: "",
|
||||
replyTo: "",
|
||||
replyToMsgId: ""
|
||||
}
|
||||
);
|
||||
|
||||
log_trace(`Sent message with ${env.payloads.length} payloads`);
|
||||
|
||||
// Log transport type for each payload
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
log_trace(`Payload ${i + 1} ('${payload.dataname}'):`);
|
||||
log_trace(` Transport: ${payload.transport}`);
|
||||
log_trace(` Type: ${payload.type}`);
|
||||
log_trace(` Size: ${payload.size} bytes`);
|
||||
log_trace(` Encoding: ${payload.encoding}`);
|
||||
|
||||
if (payload.transport === "link") {
|
||||
log_trace(` URL: ${payload.data}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Run the test
|
||||
console.log("Starting Dictionary transport test...");
|
||||
console.log(`Correlation ID: ${correlation_id}`);
|
||||
|
||||
// Run sender
|
||||
console.log("start smartsend for dictionaries");
|
||||
test_dict_send();
|
||||
|
||||
console.log("Test completed.");
|
||||
@@ -1,71 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Test script for large payload testing using binary transport
|
||||
// Tests receiving a large file (> 1MB) via smartsend with binary type
|
||||
|
||||
const { smartreceive, log_trace } = require('./src/NATSBridge');
|
||||
|
||||
// Configuration
|
||||
const SUBJECT = "/NATSBridge_test";
|
||||
const NATS_URL = "nats.yiem.cc";
|
||||
|
||||
// Helper: Log with correlation ID
|
||||
function log_trace(message) {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`[${timestamp}] ${message}`);
|
||||
}
|
||||
|
||||
// Receiver: Listen for messages and verify large payload handling
|
||||
async function test_large_binary_receive() {
|
||||
// Connect to NATS
|
||||
const { connect } = require('nats');
|
||||
const nc = await connect({ servers: [NATS_URL] });
|
||||
|
||||
// Subscribe to the subject
|
||||
const sub = nc.subscribe(SUBJECT);
|
||||
|
||||
for await (const msg of sub) {
|
||||
log_trace(`Received message on ${msg.subject}`);
|
||||
|
||||
// Use NATSBridge.smartreceive to handle the data
|
||||
const result = await smartreceive(
|
||||
msg,
|
||||
{
|
||||
maxRetries: 5,
|
||||
baseDelay: 100,
|
||||
maxDelay: 5000
|
||||
}
|
||||
);
|
||||
|
||||
// Result is an envelope dictionary with payloads field
|
||||
// Access payloads with result.payloads
|
||||
for (const { dataname, data, type } of result.payloads) {
|
||||
if (data instanceof Uint8Array || Array.isArray(data)) {
|
||||
const file_size = data.length;
|
||||
log_trace(`Received ${file_size} bytes of binary data for '${dataname}' of type ${type}`);
|
||||
|
||||
// Save received data to a test file
|
||||
const fs = require('fs');
|
||||
const output_path = `./new_${dataname}`;
|
||||
fs.writeFileSync(output_path, Buffer.from(data));
|
||||
log_trace(`Saved received data to ${output_path}`);
|
||||
} else {
|
||||
log_trace(`Received unexpected data type for '${dataname}': ${typeof data}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Keep listening for 10 seconds
|
||||
setTimeout(() => {
|
||||
nc.close();
|
||||
process.exit(0);
|
||||
}, 120000);
|
||||
}
|
||||
|
||||
// Run the test
|
||||
console.log("Starting large binary payload test...");
|
||||
|
||||
// Run receiver
|
||||
console.log("testing smartreceive");
|
||||
test_large_binary_receive();
|
||||
|
||||
console.log("Test completed.");
|
||||
@@ -1,143 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Test script for large payload testing using binary transport
|
||||
// Tests sending a large file (> 1MB) via smartsend with binary type
|
||||
|
||||
const { smartsend, uuid4, log_trace } = require('./src/NATSBridge');
|
||||
|
||||
// Configuration
|
||||
const SUBJECT = "/NATSBridge_test";
|
||||
const NATS_URL = "nats.yiem.cc";
|
||||
const FILESERVER_URL = "http://192.168.88.104:8080";
|
||||
|
||||
// Create correlation ID for tracing
|
||||
const correlation_id = uuid4();
|
||||
|
||||
// Helper: Log with correlation ID
|
||||
function log_trace(message) {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`[${timestamp}] [Correlation: ${correlation_id}] ${message}`);
|
||||
}
|
||||
|
||||
// File upload handler for plik server
|
||||
async function plik_upload_handler(fileserver_url, dataname, data, correlation_id) {
|
||||
log_trace(correlation_id, `Uploading ${dataname} to fileserver: ${fileserver_url}`);
|
||||
|
||||
// Step 1: Get upload ID and token
|
||||
const url_getUploadID = `${fileserver_url}/upload`;
|
||||
const headers = {
|
||||
"Content-Type": "application/json"
|
||||
};
|
||||
const body = JSON.stringify({ OneShot: true });
|
||||
|
||||
let response = await fetch(url_getUploadID, {
|
||||
method: "POST",
|
||||
headers: headers,
|
||||
body: body
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to get upload ID: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const responseJson = await response.json();
|
||||
const uploadid = responseJson.id;
|
||||
const uploadtoken = responseJson.uploadToken;
|
||||
|
||||
// Step 2: Upload file data
|
||||
const url_upload = `${fileserver_url}/file/${uploadid}`;
|
||||
|
||||
// Create multipart form data
|
||||
const formData = new FormData();
|
||||
const blob = new Blob([data], { type: "application/octet-stream" });
|
||||
formData.append("file", blob, dataname);
|
||||
|
||||
response = await fetch(url_upload, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"X-UploadToken": uploadtoken
|
||||
},
|
||||
body: formData
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to upload file: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const fileResponseJson = await response.json();
|
||||
const fileid = fileResponseJson.id;
|
||||
|
||||
// Build the download URL
|
||||
const url = `${fileserver_url}/file/${uploadid}/${fileid}/${encodeURIComponent(dataname)}`;
|
||||
|
||||
log_trace(correlation_id, `Uploaded to URL: ${url}`);
|
||||
|
||||
return {
|
||||
status: response.status,
|
||||
uploadid: uploadid,
|
||||
fileid: fileid,
|
||||
url: url
|
||||
};
|
||||
}
|
||||
|
||||
// Sender: Send large binary file via smartsend
|
||||
async function test_large_binary_send() {
|
||||
// Read the large file as binary data
|
||||
const fs = require('fs');
|
||||
|
||||
// Test data 1
|
||||
const file_path1 = './testFile_large.zip';
|
||||
const file_data1 = fs.readFileSync(file_path1);
|
||||
const filename1 = 'testFile_large.zip';
|
||||
const data1 = { dataname: filename1, data: file_data1, type: "binary" };
|
||||
|
||||
// Test data 2
|
||||
const file_path2 = './testFile_small.zip';
|
||||
const file_data2 = fs.readFileSync(file_path2);
|
||||
const filename2 = 'testFile_small.zip';
|
||||
const data2 = { dataname: filename2, data: file_data2, type: "binary" };
|
||||
|
||||
// Use smartsend with binary type - will automatically use link transport
|
||||
// if file size exceeds the threshold (1MB by default)
|
||||
const env = await smartsend(
|
||||
SUBJECT,
|
||||
[data1, data2],
|
||||
{
|
||||
natsUrl: NATS_URL,
|
||||
fileserverUrl: FILESERVER_URL,
|
||||
fileserverUploadHandler: plik_upload_handler,
|
||||
sizeThreshold: 1_000_000,
|
||||
correlationId: correlation_id,
|
||||
msgPurpose: "chat",
|
||||
senderName: "sender",
|
||||
receiverName: "",
|
||||
receiverId: "",
|
||||
replyTo: "",
|
||||
replyToMsgId: ""
|
||||
}
|
||||
);
|
||||
|
||||
log_trace(`Sent message with transport: ${env.payloads[0].transport}`);
|
||||
log_trace(`Envelope type: ${env.payloads[0].type}`);
|
||||
|
||||
// Check if link transport was used
|
||||
if (env.payloads[0].transport === "link") {
|
||||
log_trace("Using link transport - file uploaded to HTTP server");
|
||||
log_trace(`URL: ${env.payloads[0].data}`);
|
||||
} else {
|
||||
log_trace("Using direct transport - payload sent via NATS");
|
||||
}
|
||||
}
|
||||
|
||||
// Run the test
|
||||
console.log("Starting large binary payload test...");
|
||||
console.log(`Correlation ID: ${correlation_id}`);
|
||||
|
||||
// Run sender first
|
||||
console.log("start smartsend");
|
||||
test_large_binary_send();
|
||||
|
||||
// Run receiver
|
||||
// console.log("testing smartreceive");
|
||||
// test_large_binary_receive();
|
||||
|
||||
console.log("Test completed.");
|
||||
@@ -1,276 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Test script for mixed-content message testing
|
||||
// Tests sending a mix of text, json, table, image, audio, video, and binary data
|
||||
// from JavaScript serviceA to JavaScript serviceB using NATSBridge.js smartsend
|
||||
//
|
||||
// This test demonstrates that any combination and any number of mixed content
|
||||
// can be sent and received correctly.
|
||||
|
||||
const { smartsend, uuid4, log_trace, _serialize_data } = require('./src/NATSBridge');
|
||||
|
||||
// Configuration
|
||||
const SUBJECT = "/NATSBridge_mix_test";
|
||||
const NATS_URL = "nats.yiem.cc";
|
||||
const FILESERVER_URL = "http://192.168.88.104:8080";
|
||||
|
||||
// Create correlation ID for tracing
|
||||
const correlation_id = uuid4();
|
||||
|
||||
// Helper: Log with correlation ID
|
||||
function log_trace(message) {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`[${timestamp}] [Correlation: ${correlation_id}] ${message}`);
|
||||
}
|
||||
|
||||
// File upload handler for plik server
|
||||
async function plik_upload_handler(fileserver_url, dataname, data, correlation_id) {
|
||||
log_trace(correlation_id, `Uploading ${dataname} to fileserver: ${fileserver_url}`);
|
||||
|
||||
// Step 1: Get upload ID and token
|
||||
const url_getUploadID = `${fileserver_url}/upload`;
|
||||
const headers = {
|
||||
"Content-Type": "application/json"
|
||||
};
|
||||
const body = JSON.stringify({ OneShot: true });
|
||||
|
||||
let response = await fetch(url_getUploadID, {
|
||||
method: "POST",
|
||||
headers: headers,
|
||||
body: body
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to get upload ID: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const responseJson = await response.json();
|
||||
const uploadid = responseJson.id;
|
||||
const uploadtoken = responseJson.uploadToken;
|
||||
|
||||
// Step 2: Upload file data
|
||||
const url_upload = `${fileserver_url}/file/${uploadid}`;
|
||||
|
||||
// Create multipart form data
|
||||
const formData = new FormData();
|
||||
const blob = new Blob([data], { type: "application/octet-stream" });
|
||||
formData.append("file", blob, dataname);
|
||||
|
||||
response = await fetch(url_upload, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"X-UploadToken": uploadtoken
|
||||
},
|
||||
body: formData
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to upload file: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const fileResponseJson = await response.json();
|
||||
const fileid = fileResponseJson.id;
|
||||
|
||||
// Build the download URL
|
||||
const url = `${fileserver_url}/file/${uploadid}/${fileid}/${encodeURIComponent(dataname)}`;
|
||||
|
||||
log_trace(correlation_id, `Uploaded to URL: ${url}`);
|
||||
|
||||
return {
|
||||
status: response.status,
|
||||
uploadid: uploadid,
|
||||
fileid: fileid,
|
||||
url: url
|
||||
};
|
||||
}
|
||||
|
||||
// Helper: Create sample data for each type
|
||||
function create_sample_data() {
|
||||
// Text data (small - direct transport)
|
||||
const text_data = "Hello! This is a test chat message. 🎉\nHow are you doing today? 😊";
|
||||
|
||||
// Dictionary/JSON data (medium - could be direct or link)
|
||||
const dict_data = {
|
||||
type: "chat",
|
||||
sender: "serviceA",
|
||||
receiver: "serviceB",
|
||||
metadata: {
|
||||
timestamp: new Date().toISOString(),
|
||||
priority: "high",
|
||||
tags: ["urgent", "chat", "test"]
|
||||
},
|
||||
content: {
|
||||
text: "This is a JSON-formatted chat message with nested structure.",
|
||||
format: "markdown",
|
||||
mentions: ["user1", "user2"]
|
||||
}
|
||||
};
|
||||
|
||||
// Table data (small - direct transport) - NOT IMPLEMENTED (requires apache-arrow)
|
||||
// const table_data_small = {...};
|
||||
|
||||
// Table data (large - link transport) - NOT IMPLEMENTED (requires apache-arrow)
|
||||
// const table_data_large = {...};
|
||||
|
||||
// Image data (small binary - direct transport)
|
||||
// Create a simple 10x10 pixel PNG-like data
|
||||
const image_width = 10;
|
||||
const image_height = 10;
|
||||
let image_data = new Uint8Array(128); // PNG header + pixel data
|
||||
// PNG header
|
||||
image_data[0] = 0x89;
|
||||
image_data[1] = 0x50;
|
||||
image_data[2] = 0x4E;
|
||||
image_data[3] = 0x47;
|
||||
image_data[4] = 0x0D;
|
||||
image_data[5] = 0x0A;
|
||||
image_data[6] = 0x1A;
|
||||
image_data[7] = 0x0A;
|
||||
// Simple RGB data (10*10*3 = 300 bytes)
|
||||
for (let i = 0; i < 300; i++) {
|
||||
image_data[i + 8] = 0xFF; // Red pixel
|
||||
}
|
||||
|
||||
// Image data (large - link transport)
|
||||
const large_image_width = 500;
|
||||
const large_image_height = 1000;
|
||||
const large_image_data = new Uint8Array(large_image_width * large_image_height * 3 + 8);
|
||||
// PNG header
|
||||
large_image_data[0] = 0x89;
|
||||
large_image_data[1] = 0x50;
|
||||
large_image_data[2] = 0x4E;
|
||||
large_image_data[3] = 0x47;
|
||||
large_image_data[4] = 0x0D;
|
||||
large_image_data[5] = 0x0A;
|
||||
large_image_data[6] = 0x1A;
|
||||
large_image_data[7] = 0x0A;
|
||||
// Random RGB data
|
||||
for (let i = 0; i < large_image_width * large_image_height * 3; i++) {
|
||||
large_image_data[i + 8] = Math.floor(Math.random() * 255);
|
||||
}
|
||||
|
||||
// Audio data (small binary - direct transport)
|
||||
const audio_data = new Uint8Array(100);
|
||||
for (let i = 0; i < 100; i++) {
|
||||
audio_data[i] = Math.floor(Math.random() * 255);
|
||||
}
|
||||
|
||||
// Audio data (large - link transport)
|
||||
const large_audio_data = new Uint8Array(1_500_000);
|
||||
for (let i = 0; i < 1_500_000; i++) {
|
||||
large_audio_data[i] = Math.floor(Math.random() * 255);
|
||||
}
|
||||
|
||||
// Video data (small binary - direct transport)
|
||||
const video_data = new Uint8Array(150);
|
||||
for (let i = 0; i < 150; i++) {
|
||||
video_data[i] = Math.floor(Math.random() * 255);
|
||||
}
|
||||
|
||||
// Video data (large - link transport)
|
||||
const large_video_data = new Uint8Array(1_500_000);
|
||||
for (let i = 0; i < 1_500_000; i++) {
|
||||
large_video_data[i] = Math.floor(Math.random() * 255);
|
||||
}
|
||||
|
||||
// Binary data (small - direct transport)
|
||||
const binary_data = new Uint8Array(200);
|
||||
for (let i = 0; i < 200; i++) {
|
||||
binary_data[i] = Math.floor(Math.random() * 255);
|
||||
}
|
||||
|
||||
// Binary data (large - link transport)
|
||||
const large_binary_data = new Uint8Array(1_500_000);
|
||||
for (let i = 0; i < 1_500_000; i++) {
|
||||
large_binary_data[i] = Math.floor(Math.random() * 255);
|
||||
}
|
||||
|
||||
return {
|
||||
text_data,
|
||||
dict_data,
|
||||
// table_data_small,
|
||||
// table_data_large,
|
||||
image_data,
|
||||
large_image_data,
|
||||
audio_data,
|
||||
large_audio_data,
|
||||
video_data,
|
||||
large_video_data,
|
||||
binary_data,
|
||||
large_binary_data
|
||||
};
|
||||
}
|
||||
|
||||
// Sender: Send mixed content via smartsend
|
||||
async function test_mix_send() {
|
||||
// Create sample data
|
||||
const { text_data, dict_data, image_data, large_image_data, audio_data, large_audio_data, video_data, large_video_data, binary_data, large_binary_data } = create_sample_data();
|
||||
|
||||
// Create payloads list - mixed content with both small and large data
|
||||
// Small data uses direct transport, large data uses link transport
|
||||
const payloads = [
|
||||
// Small data (direct transport) - text, dictionary
|
||||
{ dataname: "chat_text", data: text_data, type: "text" },
|
||||
{ dataname: "chat_json", data: dict_data, type: "dictionary" },
|
||||
// { dataname: "chat_table_small", data: table_data_small, type: "table" },
|
||||
|
||||
// Large data (link transport) - large image, large audio, large video, large binary
|
||||
// { dataname: "chat_table_large", data: table_data_large, type: "table" },
|
||||
{ dataname: "user_image_large", data: large_image_data, type: "image" },
|
||||
{ dataname: "audio_clip_large", data: large_audio_data, type: "audio" },
|
||||
{ dataname: "video_clip_large", data: large_video_data, type: "video" },
|
||||
{ dataname: "binary_file_large", data: large_binary_data, type: "binary" }
|
||||
];
|
||||
|
||||
// Use smartsend with mixed content
|
||||
const env = await smartsend(
|
||||
SUBJECT,
|
||||
payloads,
|
||||
{
|
||||
natsUrl: NATS_URL,
|
||||
fileserverUrl: FILESERVER_URL,
|
||||
fileserverUploadHandler: plik_upload_handler,
|
||||
sizeThreshold: 1_000_000,
|
||||
correlationId: correlation_id,
|
||||
msgPurpose: "chat",
|
||||
senderName: "mix_sender",
|
||||
receiverName: "",
|
||||
receiverId: "",
|
||||
replyTo: "",
|
||||
replyToMsgId: ""
|
||||
}
|
||||
);
|
||||
|
||||
log_trace(`Sent message with ${env.payloads.length} payloads`);
|
||||
|
||||
// Log transport type for each payload
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
log_trace(`Payload ${i + 1} ('${payload.dataname}'):`);
|
||||
log_trace(` Transport: ${payload.transport}`);
|
||||
log_trace(` Type: ${payload.type}`);
|
||||
log_trace(` Size: ${payload.size} bytes`);
|
||||
log_trace(` Encoding: ${payload.encoding}`);
|
||||
|
||||
if (payload.transport === "link") {
|
||||
log_trace(` URL: ${payload.data}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Summary
|
||||
console.log("\n--- Transport Summary ---");
|
||||
const direct_count = env.payloads.filter(p => p.transport === "direct").length;
|
||||
const link_count = env.payloads.filter(p => p.transport === "link").length;
|
||||
log_trace(`Direct transport: ${direct_count} payloads`);
|
||||
log_trace(`Link transport: ${link_count} payloads`);
|
||||
}
|
||||
|
||||
// Run the test
|
||||
console.log("Starting mixed-content transport test...");
|
||||
console.log(`Correlation ID: ${correlation_id}`);
|
||||
|
||||
// Run sender
|
||||
console.log("start smartsend for mixed content");
|
||||
test_mix_send();
|
||||
|
||||
console.log("\nTest completed.");
|
||||
console.log("Note: Run test_js_to_js_mix_receiver.js to receive the messages.");
|
||||
@@ -1,173 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Test script for mixed-content message testing
|
||||
// Tests receiving a mix of text, json, table, image, audio, video, and binary data
|
||||
// from JavaScript serviceA to JavaScript serviceB using NATSBridge.js smartreceive
|
||||
//
|
||||
// This test demonstrates that any combination and any number of mixed content
|
||||
// can be sent and received correctly.
|
||||
|
||||
const { smartreceive, log_trace } = require('./src/NATSBridge');
|
||||
|
||||
// Configuration
|
||||
const SUBJECT = "/NATSBridge_mix_test";
|
||||
const NATS_URL = "nats.yiem.cc";
|
||||
|
||||
// Helper: Log with correlation ID
|
||||
function log_trace(message) {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`[${timestamp}] ${message}`);
|
||||
}
|
||||
|
||||
// Receiver: Listen for messages and verify mixed content handling
|
||||
async function test_mix_receive() {
|
||||
// Connect to NATS
|
||||
const { connect } = require('nats');
|
||||
const nc = await connect({ servers: [NATS_URL] });
|
||||
|
||||
// Subscribe to the subject
|
||||
const sub = nc.subscribe(SUBJECT);
|
||||
|
||||
for await (const msg of sub) {
|
||||
log_trace(`Received message on ${msg.subject}`);
|
||||
|
||||
// Use NATSBridge.smartreceive to handle the data
|
||||
const result = await smartreceive(
|
||||
msg,
|
||||
{
|
||||
maxRetries: 5,
|
||||
baseDelay: 100,
|
||||
maxDelay: 5000
|
||||
}
|
||||
);
|
||||
|
||||
log_trace(`Received ${result.payloads.length} payloads`);
|
||||
|
||||
// Result is an envelope dictionary with payloads field
|
||||
// Access payloads with result.payloads
|
||||
for (const { dataname, data, type } of result.payloads) {
|
||||
log_trace(`\n=== Payload: ${dataname} (type: ${type}) ===`);
|
||||
|
||||
// Handle different data types
|
||||
if (type === "text") {
|
||||
// Text data - should be a String
|
||||
if (typeof data === 'string') {
|
||||
log_trace(` Type: String`);
|
||||
log_trace(` Length: ${data.length} characters`);
|
||||
|
||||
// Display first 200 characters
|
||||
if (data.length > 200) {
|
||||
log_trace(` First 200 chars: ${data.substring(0, 200)}...`);
|
||||
} else {
|
||||
log_trace(` Content: ${data}`);
|
||||
}
|
||||
|
||||
// Save to file
|
||||
const fs = require('fs');
|
||||
const output_path = `./received_${dataname}.txt`;
|
||||
fs.writeFileSync(output_path, data);
|
||||
log_trace(` Saved to: ${output_path}`);
|
||||
} else {
|
||||
log_trace(` ERROR: Expected String, got ${typeof data}`);
|
||||
}
|
||||
|
||||
} else if (type === "dictionary") {
|
||||
// Dictionary data - should be an object
|
||||
if (typeof data === 'object' && data !== null && !Array.isArray(data)) {
|
||||
log_trace(` Type: Object`);
|
||||
log_trace(` Keys: ${Object.keys(data).join(', ')}`);
|
||||
|
||||
// Display nested content
|
||||
for (const [key, value] of Object.entries(data)) {
|
||||
log_trace(` ${key} => ${value}`);
|
||||
}
|
||||
|
||||
// Save to JSON file
|
||||
const fs = require('fs');
|
||||
const output_path = `./received_${dataname}.json`;
|
||||
const json_str = JSON.stringify(data, null, 2);
|
||||
fs.writeFileSync(output_path, json_str);
|
||||
log_trace(` Saved to: ${output_path}`);
|
||||
} else {
|
||||
log_trace(` ERROR: Expected Object, got ${typeof data}`);
|
||||
}
|
||||
|
||||
} else if (type === "table") {
|
||||
// Table data - should be an array of objects (requires apache-arrow)
|
||||
log_trace(` Type: Array (requires apache-arrow for full deserialization)`);
|
||||
if (Array.isArray(data)) {
|
||||
log_trace(` Length: ${data.length} items`);
|
||||
log_trace(` First item: ${JSON.stringify(data[0])}`);
|
||||
} else {
|
||||
log_trace(` ERROR: Expected Array, got ${typeof data}`);
|
||||
}
|
||||
|
||||
} else if (type === "image" || type === "audio" || type === "video" || type === "binary") {
|
||||
// Binary data - should be Uint8Array
|
||||
if (data instanceof Uint8Array || Array.isArray(data)) {
|
||||
log_trace(` Type: Uint8Array (binary)`);
|
||||
log_trace(` Size: ${data.length} bytes`);
|
||||
|
||||
// Save to file
|
||||
const fs = require('fs');
|
||||
const output_path = `./received_${dataname}.bin`;
|
||||
fs.writeFileSync(output_path, Buffer.from(data));
|
||||
log_trace(` Saved to: ${output_path}`);
|
||||
} else {
|
||||
log_trace(` ERROR: Expected Uint8Array, got ${typeof data}`);
|
||||
}
|
||||
|
||||
} else {
|
||||
log_trace(` ERROR: Unknown data type '${type}'`);
|
||||
}
|
||||
}
|
||||
|
||||
// Summary
|
||||
console.log("\n=== Verification Summary ===");
|
||||
const text_count = result.payloads.filter(x => x.type === "text").length;
|
||||
const dict_count = result.payloads.filter(x => x.type === "dictionary").length;
|
||||
const table_count = result.payloads.filter(x => x.type === "table").length;
|
||||
const image_count = result.payloads.filter(x => x.type === "image").length;
|
||||
const audio_count = result.payloads.filter(x => x.type === "audio").length;
|
||||
const video_count = result.payloads.filter(x => x.type === "video").length;
|
||||
const binary_count = result.payloads.filter(x => x.type === "binary").length;
|
||||
|
||||
log_trace(`Text payloads: ${text_count}`);
|
||||
log_trace(`Dictionary payloads: ${dict_count}`);
|
||||
log_trace(`Table payloads: ${table_count}`);
|
||||
log_trace(`Image payloads: ${image_count}`);
|
||||
log_trace(`Audio payloads: ${audio_count}`);
|
||||
log_trace(`Video payloads: ${video_count}`);
|
||||
log_trace(`Binary payloads: ${binary_count}`);
|
||||
|
||||
// Print transport type info for each payload if available
|
||||
console.log("\n=== Payload Details ===");
|
||||
for (const { dataname, data, type } of result.payloads) {
|
||||
if (["image", "audio", "video", "binary"].includes(type)) {
|
||||
log_trace(`${dataname}: ${data.length} bytes (binary)`);
|
||||
} else if (type === "table") {
|
||||
log_trace(`${dataname}: ${data.length} items (Array)`);
|
||||
} else if (type === "dictionary") {
|
||||
log_trace(`${dataname}: ${JSON.stringify(data).length} bytes (Object)`);
|
||||
} else if (type === "text") {
|
||||
log_trace(`${dataname}: ${data.length} characters (String)`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Keep listening for 2 minutes
|
||||
setTimeout(() => {
|
||||
nc.close();
|
||||
process.exit(0);
|
||||
}, 120000);
|
||||
}
|
||||
|
||||
// Run the test
|
||||
console.log("Starting mixed-content transport test...");
|
||||
console.log("Note: This receiver will wait for messages from the sender.");
|
||||
console.log("Run test_js_to_js_mix_sender.js first to send test data.");
|
||||
|
||||
// Run receiver
|
||||
console.log("\ntesting smartreceive for mixed content");
|
||||
test_mix_receive();
|
||||
|
||||
console.log("\nTest completed.");
|
||||
@@ -1,87 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Test script for Table transport testing
|
||||
// Tests receiving 1 large and 1 small Tables via direct and link transport
|
||||
// Uses NATSBridge.js smartreceive with "table" type
|
||||
//
|
||||
// Note: This test requires the apache-arrow library to deserialize table data.
|
||||
// The JavaScript implementation uses apache-arrow for Arrow IPC deserialization.
|
||||
|
||||
const { smartreceive, log_trace } = require('./src/NATSBridge');
|
||||
|
||||
// Configuration
|
||||
const SUBJECT = "/NATSBridge_table_test";
|
||||
const NATS_URL = "nats.yiem.cc";
|
||||
|
||||
// Helper: Log with correlation ID
|
||||
function log_trace(message) {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`[${timestamp}] ${message}`);
|
||||
}
|
||||
|
||||
// Receiver: Listen for messages and verify Table handling
|
||||
async function test_table_receive() {
|
||||
// Connect to NATS
|
||||
const { connect } = require('nats');
|
||||
const nc = await connect({ servers: [NATS_URL] });
|
||||
|
||||
// Subscribe to the subject
|
||||
const sub = nc.subscribe(SUBJECT);
|
||||
|
||||
for await (const msg of sub) {
|
||||
log_trace(`Received message on ${msg.subject}`);
|
||||
|
||||
// Use NATSBridge.smartreceive to handle the data
|
||||
const result = await smartreceive(
|
||||
msg,
|
||||
{
|
||||
maxRetries: 5,
|
||||
baseDelay: 100,
|
||||
maxDelay: 5000
|
||||
}
|
||||
);
|
||||
|
||||
// Result is an envelope dictionary with payloads field
|
||||
// Access payloads with result.payloads
|
||||
for (const { dataname, data, type } of result.payloads) {
|
||||
if (Array.isArray(data)) {
|
||||
log_trace(`Received Table '${dataname}' of type ${type}`);
|
||||
|
||||
// Display table contents
|
||||
console.log(` Dimensions: ${data.length} rows x ${data.length > 0 ? Object.keys(data[0]).length : 0} columns`);
|
||||
console.log(` Columns: ${data.length > 0 ? Object.keys(data[0]).join(', ') : ''}`);
|
||||
|
||||
// Display first few rows
|
||||
console.log(` First 5 rows:`);
|
||||
for (let i = 0; i < Math.min(5, data.length); i++) {
|
||||
console.log(` Row ${i}: ${JSON.stringify(data[i])}`);
|
||||
}
|
||||
|
||||
// Save to JSON file
|
||||
const fs = require('fs');
|
||||
const output_path = `./received_${dataname}.json`;
|
||||
const json_str = JSON.stringify(data, null, 2);
|
||||
fs.writeFileSync(output_path, json_str);
|
||||
log_trace(`Saved Table to ${output_path}`);
|
||||
} else {
|
||||
log_trace(`Received unexpected data type for '${dataname}': ${typeof data}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Keep listening for 10 seconds
|
||||
setTimeout(() => {
|
||||
nc.close();
|
||||
process.exit(0);
|
||||
}, 120000);
|
||||
}
|
||||
|
||||
// Run the test
|
||||
console.log("Starting Table transport test...");
|
||||
console.log("Note: This receiver will wait for messages from the sender.");
|
||||
console.log("Run test_js_to_js_table_sender.js first to send test data.");
|
||||
|
||||
// Run receiver
|
||||
console.log("testing smartreceive");
|
||||
test_table_receive();
|
||||
|
||||
console.log("Test completed.");
|
||||
@@ -1,164 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Test script for Table transport testing
|
||||
// Tests sending 1 large and 1 small Tables via direct and link transport
|
||||
// Uses NATSBridge.js smartsend with "table" type
|
||||
//
|
||||
// Note: This test requires the apache-arrow library to serialize/deserialize table data.
|
||||
// The JavaScript implementation uses apache-arrow for Arrow IPC serialization.
|
||||
|
||||
const { smartsend, uuid4, log_trace } = require('./src/NATSBridge');
|
||||
|
||||
// Configuration
|
||||
const SUBJECT = "/NATSBridge_table_test";
|
||||
const NATS_URL = "nats.yiem.cc";
|
||||
const FILESERVER_URL = "http://192.168.88.104:8080";
|
||||
|
||||
// Create correlation ID for tracing
|
||||
const correlation_id = uuid4();
|
||||
|
||||
// Helper: Log with correlation ID
|
||||
function log_trace(message) {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`[${timestamp}] [Correlation: ${correlation_id}] ${message}`);
|
||||
}
|
||||
|
||||
// File upload handler for plik server
|
||||
async function plik_upload_handler(fileserver_url, dataname, data, correlation_id) {
|
||||
log_trace(correlation_id, `Uploading ${dataname} to fileserver: ${fileserver_url}`);
|
||||
|
||||
// Step 1: Get upload ID and token
|
||||
const url_getUploadID = `${fileserver_url}/upload`;
|
||||
const headers = {
|
||||
"Content-Type": "application/json"
|
||||
};
|
||||
const body = JSON.stringify({ OneShot: true });
|
||||
|
||||
let response = await fetch(url_getUploadID, {
|
||||
method: "POST",
|
||||
headers: headers,
|
||||
body: body
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to get upload ID: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const responseJson = await response.json();
|
||||
const uploadid = responseJson.id;
|
||||
const uploadtoken = responseJson.uploadToken;
|
||||
|
||||
// Step 2: Upload file data
|
||||
const url_upload = `${fileserver_url}/file/${uploadid}`;
|
||||
|
||||
// Create multipart form data
|
||||
const formData = new FormData();
|
||||
const blob = new Blob([data], { type: "application/octet-stream" });
|
||||
formData.append("file", blob, dataname);
|
||||
|
||||
response = await fetch(url_upload, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"X-UploadToken": uploadtoken
|
||||
},
|
||||
body: formData
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to upload file: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const fileResponseJson = await response.json();
|
||||
const fileid = fileResponseJson.id;
|
||||
|
||||
// Build the download URL
|
||||
const url = `${fileserver_url}/file/${uploadid}/${fileid}/${encodeURIComponent(dataname)}`;
|
||||
|
||||
log_trace(correlation_id, `Uploaded to URL: ${url}`);
|
||||
|
||||
return {
|
||||
status: response.status,
|
||||
uploadid: uploadid,
|
||||
fileid: fileid,
|
||||
url: url
|
||||
};
|
||||
}
|
||||
|
||||
// Sender: Send Tables via smartsend
|
||||
async function test_table_send() {
|
||||
// Note: This test requires apache-arrow library to create Arrow IPC data.
|
||||
// For now, we'll use a simple array of objects as table data.
|
||||
// In production, you would use the apache-arrow library to create Arrow IPC data.
|
||||
|
||||
// Create a small Table (will use direct transport)
|
||||
const small_table = [
|
||||
{ id: 1, name: "Alice", score: 95 },
|
||||
{ id: 2, name: "Bob", score: 88 },
|
||||
{ id: 3, name: "Charlie", score: 92 }
|
||||
];
|
||||
|
||||
// Create a large Table (will use link transport if > 1MB)
|
||||
// Generate a larger dataset (~2MB to ensure link transport)
|
||||
const large_table = [];
|
||||
for (let i = 0; i < 50000; i++) {
|
||||
large_table.push({
|
||||
id: i,
|
||||
message: `msg_${i}`,
|
||||
sender: `sender_${i}`,
|
||||
timestamp: new Date().toISOString(),
|
||||
priority: Math.floor(Math.random() * 3) + 1
|
||||
});
|
||||
}
|
||||
|
||||
// Test data 1: small Table
|
||||
const data1 = { dataname: "small_table", data: small_table, type: "table" };
|
||||
|
||||
// Test data 2: large Table
|
||||
const data2 = { dataname: "large_table", data: large_table, type: "table" };
|
||||
|
||||
// Use smartsend with table type
|
||||
// For small Table: will use direct transport (Arrow IPC encoded)
|
||||
// For large Table: will use link transport (uploaded to fileserver)
|
||||
const env = await smartsend(
|
||||
SUBJECT,
|
||||
[data1, data2],
|
||||
{
|
||||
natsUrl: NATS_URL,
|
||||
fileserverUrl: FILESERVER_URL,
|
||||
fileserverUploadHandler: plik_upload_handler,
|
||||
sizeThreshold: 1_000_000,
|
||||
correlationId: correlation_id,
|
||||
msgPurpose: "chat",
|
||||
senderName: "table_sender",
|
||||
receiverName: "",
|
||||
receiverId: "",
|
||||
replyTo: "",
|
||||
replyToMsgId: ""
|
||||
}
|
||||
);
|
||||
|
||||
log_trace(`Sent message with ${env.payloads.length} payloads`);
|
||||
|
||||
// Log transport type for each payload
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
log_trace(`Payload ${i + 1} ('${payload.dataname}'):`);
|
||||
log_trace(` Transport: ${payload.transport}`);
|
||||
log_trace(` Type: ${payload.type}`);
|
||||
log_trace(` Size: ${payload.size} bytes`);
|
||||
log_trace(` Encoding: ${payload.encoding}`);
|
||||
|
||||
if (payload.transport === "link") {
|
||||
log_trace(` URL: ${payload.data}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Run the test
|
||||
console.log("Starting Table transport test...");
|
||||
console.log(`Correlation ID: ${correlation_id}`);
|
||||
|
||||
// Run sender
|
||||
console.log("start smartsend for tables");
|
||||
test_table_send();
|
||||
|
||||
console.log("Test completed.");
|
||||
@@ -1,81 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Test script for text transport testing
|
||||
// Tests receiving 1 large and 1 small text from JavaScript serviceA to JavaScript serviceB
|
||||
// Uses NATSBridge.js smartreceive with "text" type
|
||||
|
||||
const { smartreceive, log_trace } = require('./src/NATSBridge');
|
||||
|
||||
// Configuration
|
||||
const SUBJECT = "/NATSBridge_text_test";
|
||||
const NATS_URL = "nats.yiem.cc";
|
||||
|
||||
// Helper: Log with correlation ID
|
||||
function log_trace(message) {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`[${timestamp}] ${message}`);
|
||||
}
|
||||
|
||||
// Receiver: Listen for messages and verify text handling
|
||||
async function test_text_receive() {
|
||||
// Connect to NATS
|
||||
const { connect } = require('nats');
|
||||
const nc = await connect({ servers: [NATS_URL] });
|
||||
|
||||
// Subscribe to the subject
|
||||
const sub = nc.subscribe(SUBJECT);
|
||||
|
||||
for await (const msg of sub) {
|
||||
log_trace(`Received message on ${msg.subject}`);
|
||||
|
||||
// Use NATSBridge.smartreceive to handle the data
|
||||
const result = await smartreceive(
|
||||
msg,
|
||||
{
|
||||
maxRetries: 5,
|
||||
baseDelay: 100,
|
||||
maxDelay: 5000
|
||||
}
|
||||
);
|
||||
|
||||
// Result is an envelope dictionary with payloads field
|
||||
// Access payloads with result.payloads
|
||||
for (const { dataname, data, type } of result.payloads) {
|
||||
if (typeof data === 'string') {
|
||||
log_trace(`Received text '${dataname}' of type ${type}`);
|
||||
log_trace(` Length: ${data.length} characters`);
|
||||
|
||||
// Display first 100 characters
|
||||
if (data.length > 100) {
|
||||
log_trace(` First 100 characters: ${data.substring(0, 100)}...`);
|
||||
} else {
|
||||
log_trace(` Content: ${data}`);
|
||||
}
|
||||
|
||||
// Save to file
|
||||
const fs = require('fs');
|
||||
const output_path = `./received_${dataname}.txt`;
|
||||
fs.writeFileSync(output_path, data);
|
||||
log_trace(`Saved text to ${output_path}`);
|
||||
} else {
|
||||
log_trace(`Received unexpected data type for '${dataname}': ${typeof data}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Keep listening for 10 seconds
|
||||
setTimeout(() => {
|
||||
nc.close();
|
||||
process.exit(0);
|
||||
}, 120000);
|
||||
}
|
||||
|
||||
// Run the test
|
||||
console.log("Starting text transport test...");
|
||||
console.log("Note: This receiver will wait for messages from the sender.");
|
||||
console.log("Run test_js_to_js_text_sender.js first to send test data.");
|
||||
|
||||
// Run receiver
|
||||
console.log("testing smartreceive for text");
|
||||
test_text_receive();
|
||||
|
||||
console.log("Test completed.");
|
||||
@@ -1,140 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Test script for text transport testing
|
||||
// Tests sending 1 large and 1 small text from JavaScript serviceA to JavaScript serviceB
|
||||
// Uses NATSBridge.js smartsend with "text" type
|
||||
|
||||
const { smartsend, uuid4, log_trace } = require('./src/NATSBridge');
|
||||
|
||||
// Configuration
|
||||
const SUBJECT = "/NATSBridge_text_test";
|
||||
const NATS_URL = "nats.yiem.cc";
|
||||
const FILESERVER_URL = "http://192.168.88.104:8080";
|
||||
|
||||
// Create correlation ID for tracing
|
||||
const correlation_id = uuid4();
|
||||
|
||||
// Helper: Log with correlation ID
|
||||
function log_trace(message) {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`[${timestamp}] [Correlation: ${correlation_id}] ${message}`);
|
||||
}
|
||||
|
||||
// File upload handler for plik server
|
||||
async function plik_upload_handler(fileserver_url, dataname, data, correlation_id) {
|
||||
// Get upload ID
|
||||
const url_getUploadID = `${fileserver_url}/upload`;
|
||||
const headers = {
|
||||
"Content-Type": "application/json"
|
||||
};
|
||||
const body = JSON.stringify({ OneShot: true });
|
||||
|
||||
let response = await fetch(url_getUploadID, {
|
||||
method: "POST",
|
||||
headers: headers,
|
||||
body: body
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to get upload ID: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const responseJson = await response.json();
|
||||
const uploadid = responseJson.id;
|
||||
const uploadtoken = responseJson.uploadToken;
|
||||
|
||||
// Upload file
|
||||
const formData = new FormData();
|
||||
const blob = new Blob([data], { type: "application/octet-stream" });
|
||||
formData.append("file", blob, dataname);
|
||||
|
||||
response = await fetch(`${fileserver_url}/file/${uploadid}`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"X-UploadToken": uploadtoken
|
||||
},
|
||||
body: formData
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to upload file: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const fileResponseJson = await response.json();
|
||||
const fileid = fileResponseJson.id;
|
||||
|
||||
const url = `${fileserver_url}/file/${uploadid}/${fileid}/${encodeURIComponent(dataname)}`;
|
||||
|
||||
return {
|
||||
status: response.status,
|
||||
uploadid: uploadid,
|
||||
fileid: fileid,
|
||||
url: url
|
||||
};
|
||||
}
|
||||
|
||||
// Sender: Send text via smartsend
|
||||
async function test_text_send() {
|
||||
// Create a small text (will use direct transport)
|
||||
const small_text = "Hello, this is a small text message. Testing direct transport via NATS.";
|
||||
|
||||
// Create a large text (will use link transport if > 1MB)
|
||||
// Generate a larger text (~2MB to ensure link transport)
|
||||
const large_text_lines = [];
|
||||
for (let i = 0; i < 50000; i++) {
|
||||
large_text_lines.push(`Line ${i}: This is a sample text line with some content to pad the size. `);
|
||||
}
|
||||
const large_text = large_text_lines.join("");
|
||||
|
||||
// Test data 1: small text
|
||||
const data1 = { dataname: "small_text", data: small_text, type: "text" };
|
||||
|
||||
// Test data 2: large text
|
||||
const data2 = { dataname: "large_text", data: large_text, type: "text" };
|
||||
|
||||
// Use smartsend with text type
|
||||
// For small text: will use direct transport (Base64 encoded UTF-8)
|
||||
// For large text: will use link transport (uploaded to fileserver)
|
||||
const env = await smartsend(
|
||||
SUBJECT,
|
||||
[data1, data2],
|
||||
{
|
||||
natsUrl: NATS_URL,
|
||||
fileserverUrl: FILESERVER_URL,
|
||||
fileserverUploadHandler: plik_upload_handler,
|
||||
sizeThreshold: 1_000_000,
|
||||
correlationId: correlation_id,
|
||||
msgPurpose: "chat",
|
||||
senderName: "text_sender",
|
||||
receiverName: "",
|
||||
receiverId: "",
|
||||
replyTo: "",
|
||||
replyToMsgId: ""
|
||||
}
|
||||
);
|
||||
|
||||
log_trace(`Sent message with ${env.payloads.length} payloads`);
|
||||
|
||||
// Log transport type for each payload
|
||||
for (let i = 0; i < env.payloads.length; i++) {
|
||||
const payload = env.payloads[i];
|
||||
log_trace(`Payload ${i + 1} ('${payload.dataname}'):`);
|
||||
log_trace(` Transport: ${payload.transport}`);
|
||||
log_trace(` Type: ${payload.type}`);
|
||||
log_trace(` Size: ${payload.size} bytes`);
|
||||
log_trace(` Encoding: ${payload.encoding}`);
|
||||
|
||||
if (payload.transport === "link") {
|
||||
log_trace(` URL: ${payload.data}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Run the test
|
||||
console.log("Starting text transport test...");
|
||||
console.log(`Correlation ID: ${correlation_id}`);
|
||||
|
||||
// Run sender
|
||||
console.log("start smartsend for text");
|
||||
test_text_send();
|
||||
|
||||
console.log("Test completed.");
|
||||
@@ -92,12 +92,12 @@ function test_dict_send()
|
||||
# Use smartsend with dictionary type
|
||||
# For small Dictionary: will use direct transport (JSON encoded)
|
||||
# For large Dictionary: will use link transport (uploaded to fileserver)
|
||||
env = NATSBridge.smartsend(
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
SUBJECT,
|
||||
[data1, data2]; # List of (dataname, data, type) tuples
|
||||
nats_url = NATS_URL,
|
||||
broker_url = NATS_URL,
|
||||
fileserver_url = FILESERVER_URL,
|
||||
fileserverUploadHandler = plik_upload_handler,
|
||||
fileserver_upload_handler = plik_upload_handler,
|
||||
size_threshold = 1_000_000, # 1MB threshold
|
||||
correlation_id = correlation_id,
|
||||
msg_purpose = "chat",
|
||||
@@ -105,7 +105,8 @@ function test_dict_send()
|
||||
receiver_name = "",
|
||||
receiver_id = "",
|
||||
reply_to = "",
|
||||
reply_to_msg_id = ""
|
||||
reply_to_msg_id = "",
|
||||
is_publish = true # Publish the message to NATS
|
||||
)
|
||||
|
||||
log_trace("Sent message with $(length(env.payloads)) payloads")
|
||||
@@ -114,7 +115,7 @@ function test_dict_send()
|
||||
for (i, payload) in enumerate(env.payloads)
|
||||
log_trace("Payload $i ('$payload.dataname'):")
|
||||
log_trace(" Transport: $(payload.transport)")
|
||||
log_trace(" Type: $(payload.type)")
|
||||
log_trace(" Type: $(payload.payload_type)")
|
||||
log_trace(" Size: $(payload.size) bytes")
|
||||
log_trace(" Encoding: $(payload.encoding)")
|
||||
|
||||
@@ -79,12 +79,12 @@ function test_large_binary_send()
|
||||
# Use smartsend with binary type - will automatically use link transport
|
||||
# if file size exceeds the threshold (1MB by default)
|
||||
# API: smartsend(subject, [(dataname, data, type), ...]; keywords...)
|
||||
env = NATSBridge.smartsend(
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
SUBJECT,
|
||||
[data1, data2]; # List of (dataname, data, type) tuples
|
||||
nats_url = NATS_URL;
|
||||
broker_url = NATS_URL;
|
||||
fileserver_url = FILESERVER_URL,
|
||||
fileserverUploadHandler = plik_upload_handler,
|
||||
fileserver_upload_handler = plik_upload_handler,
|
||||
size_threshold = 1_000_000,
|
||||
correlation_id = correlation_id,
|
||||
msg_purpose = "chat",
|
||||
@@ -92,11 +92,12 @@ function test_large_binary_send()
|
||||
receiver_name = "",
|
||||
receiver_id = "",
|
||||
reply_to = "",
|
||||
reply_to_msg_id = ""
|
||||
reply_to_msg_id = "",
|
||||
is_publish = true # Publish the message to NATS
|
||||
)
|
||||
|
||||
log_trace("Sent message with transport: $(env.payloads[1].transport)")
|
||||
log_trace("Envelope type: $(env.payloads[1].type)")
|
||||
log_trace("Envelope type: $(env.payloads[1].payload_type)")
|
||||
|
||||
# Check if link transport was used
|
||||
if env.payloads[1].transport == "link"
|
||||
@@ -186,12 +186,12 @@ function test_mix_send()
|
||||
]
|
||||
|
||||
# Use smartsend with mixed content
|
||||
env = NATSBridge.smartsend(
|
||||
sendinfo = NATSBridge.smartsend(
|
||||
SUBJECT,
|
||||
payloads; # List of (dataname, data, type) tuples
|
||||
nats_url = NATS_URL,
|
||||
broker_url = NATS_URL,
|
||||
fileserver_url = FILESERVER_URL,
|
||||
fileserverUploadHandler = plik_upload_handler,
|
||||
fileserver_upload_handler = plik_upload_handler,
|
||||
size_threshold = 1_000_000, # 1MB threshold
|
||||
correlation_id = correlation_id,
|
||||
msg_purpose = "chat",
|
||||
@@ -199,16 +199,18 @@ function test_mix_send()
|
||||
receiver_name = "",
|
||||
receiver_id = "",
|
||||
reply_to = "",
|
||||
reply_to_msg_id = ""
|
||||
reply_to_msg_id = "",
|
||||
is_publish = true # Publish the message to NATS
|
||||
)
|
||||
|
||||
env, env_json_str = sendinfo
|
||||
log_trace("Sent message with $(length(env.payloads)) payloads")
|
||||
|
||||
# Log transport type for each payload
|
||||
for (i, payload) in enumerate(env.payloads)
|
||||
log_trace("Payload $i ('$payload.dataname'):")
|
||||
log_trace(" Transport: $(payload.transport)")
|
||||
log_trace(" Type: $(payload.type)")
|
||||
log_trace(" Type: $(payload.payload_type)")
|
||||
log_trace(" Size: $(payload.size) bytes")
|
||||
log_trace(" Encoding: $(payload.encoding)")
|
||||
|
||||
@@ -90,12 +90,12 @@ function test_table_send()
|
||||
# Use smartsend with table type
|
||||
# For small DataFrame: will use direct transport (Base64 encoded Arrow IPC)
|
||||
# For large DataFrame: will use link transport (uploaded to fileserver)
|
||||
env = NATSBridge.smartsend(
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
SUBJECT,
|
||||
[data1, data2]; # List of (dataname, data, type) tuples
|
||||
nats_url = NATS_URL,
|
||||
broker_url = NATS_URL,
|
||||
fileserver_url = FILESERVER_URL,
|
||||
fileserverUploadHandler = plik_upload_handler,
|
||||
fileserver_upload_handler = plik_upload_handler,
|
||||
size_threshold = 1_000_000, # 1MB threshold
|
||||
correlation_id = correlation_id,
|
||||
msg_purpose = "chat",
|
||||
@@ -103,7 +103,8 @@ function test_table_send()
|
||||
receiver_name = "",
|
||||
receiver_id = "",
|
||||
reply_to = "",
|
||||
reply_to_msg_id = ""
|
||||
reply_to_msg_id = "",
|
||||
is_publish = true # Publish the message to NATS
|
||||
)
|
||||
|
||||
log_trace("Sent message with $(length(env.payloads)) payloads")
|
||||
@@ -112,7 +113,7 @@ function test_table_send()
|
||||
for (i, payload) in enumerate(env.payloads)
|
||||
log_trace("Payload $i ('$payload.dataname'):")
|
||||
log_trace(" Transport: $(payload.transport)")
|
||||
log_trace(" Type: $(payload.type)")
|
||||
log_trace(" Type: $(payload.payload_type)")
|
||||
log_trace(" Size: $(payload.size) bytes")
|
||||
log_trace(" Encoding: $(payload.encoding)")
|
||||
|
||||
@@ -75,12 +75,12 @@ function test_text_send()
|
||||
# Use smartsend with text type
|
||||
# For small text: will use direct transport (Base64 encoded UTF-8)
|
||||
# For large text: will use link transport (uploaded to fileserver)
|
||||
env = NATSBridge.smartsend(
|
||||
env, env_json_str = NATSBridge.smartsend(
|
||||
SUBJECT,
|
||||
[data1, data2]; # List of (dataname, data, type) tuples
|
||||
nats_url = NATS_URL,
|
||||
broker_url = NATS_URL,
|
||||
fileserver_url = FILESERVER_URL,
|
||||
fileserverUploadHandler = plik_upload_handler,
|
||||
fileserver_upload_handler = plik_upload_handler,
|
||||
size_threshold = 1_000_000, # 1MB threshold
|
||||
correlation_id = correlation_id,
|
||||
msg_purpose = "chat",
|
||||
@@ -88,7 +88,8 @@ function test_text_send()
|
||||
receiver_name = "",
|
||||
receiver_id = "",
|
||||
reply_to = "",
|
||||
reply_to_msg_id = ""
|
||||
reply_to_msg_id = "",
|
||||
is_publish = true # Publish the message to NATS
|
||||
)
|
||||
|
||||
log_trace("Sent message with $(length(env.payloads)) payloads")
|
||||
@@ -97,7 +98,7 @@ function test_text_send()
|
||||
for (i, payload) in enumerate(env.payloads)
|
||||
log_trace("Payload $i ('$payload.dataname'):")
|
||||
log_trace(" Transport: $(payload.transport)")
|
||||
log_trace(" Type: $(payload.type)")
|
||||
log_trace(" Type: $(payload.payload_type)")
|
||||
log_trace(" Size: $(payload.size) bytes")
|
||||
log_trace(" Encoding: $(payload.encoding)")
|
||||
|
||||
@@ -1,207 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Basic functionality test for nats_bridge.py
|
||||
Tests the core classes and functions without NATS connection
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add src to path for import
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||
|
||||
from nats_bridge import (
|
||||
MessagePayload,
|
||||
MessageEnvelope,
|
||||
smartsend,
|
||||
smartreceive,
|
||||
log_trace,
|
||||
generate_uuid,
|
||||
get_timestamp,
|
||||
_serialize_data,
|
||||
_deserialize_data
|
||||
)
|
||||
import json
|
||||
|
||||
|
||||
def test_message_payload():
|
||||
"""Test MessagePayload class"""
|
||||
print("\n=== Testing MessagePayload ===")
|
||||
|
||||
# Test direct transport with text
|
||||
payload1 = MessagePayload(
|
||||
data="Hello World",
|
||||
msg_type="text",
|
||||
id="test-id-1",
|
||||
dataname="message",
|
||||
transport="direct",
|
||||
encoding="base64",
|
||||
size=11
|
||||
)
|
||||
|
||||
assert payload1.id == "test-id-1"
|
||||
assert payload1.dataname == "message"
|
||||
assert payload1.type == "text"
|
||||
assert payload1.transport == "direct"
|
||||
assert payload1.encoding == "base64"
|
||||
assert payload1.size == 11
|
||||
print(" [PASS] MessagePayload with text data")
|
||||
|
||||
# Test link transport with URL
|
||||
payload2 = MessagePayload(
|
||||
data="http://example.com/file.txt",
|
||||
msg_type="binary",
|
||||
id="test-id-2",
|
||||
dataname="file",
|
||||
transport="link",
|
||||
encoding="none",
|
||||
size=1000
|
||||
)
|
||||
|
||||
assert payload2.transport == "link"
|
||||
assert payload2.data == "http://example.com/file.txt"
|
||||
print(" [PASS] MessagePayload with link transport")
|
||||
|
||||
# Test to_dict method
|
||||
payload_dict = payload1.to_dict()
|
||||
assert "id" in payload_dict
|
||||
assert "dataname" in payload_dict
|
||||
assert "type" in payload_dict
|
||||
assert "transport" in payload_dict
|
||||
assert "data" in payload_dict
|
||||
print(" [PASS] MessagePayload.to_dict() method")
|
||||
|
||||
|
||||
def test_message_envelope():
|
||||
"""Test MessageEnvelope class"""
|
||||
print("\n=== Testing MessageEnvelope ===")
|
||||
|
||||
# Create payloads
|
||||
payload1 = MessagePayload("Hello", "text", id="p1", dataname="msg1")
|
||||
payload2 = MessagePayload("http://example.com/file", "binary", id="p2", dataname="file", transport="link")
|
||||
|
||||
# Create envelope
|
||||
env = MessageEnvelope(
|
||||
send_to="/test/subject",
|
||||
payloads=[payload1, payload2],
|
||||
correlation_id="test-correlation-id",
|
||||
msg_id="test-msg-id",
|
||||
msg_purpose="chat",
|
||||
sender_name="test_sender",
|
||||
receiver_name="test_receiver",
|
||||
reply_to="/test/reply"
|
||||
)
|
||||
|
||||
assert env.send_to == "/test/subject"
|
||||
assert env.correlation_id == "test-correlation-id"
|
||||
assert env.msg_id == "test-msg-id"
|
||||
assert env.msg_purpose == "chat"
|
||||
assert len(env.payloads) == 2
|
||||
print(" [PASS] MessageEnvelope creation")
|
||||
|
||||
# Test to_json method
|
||||
json_str = env.to_json()
|
||||
json_data = json.loads(json_str)
|
||||
assert json_data["sendTo"] == "/test/subject"
|
||||
assert json_data["correlationId"] == "test-correlation-id"
|
||||
assert json_data["msgPurpose"] == "chat"
|
||||
assert len(json_data["payloads"]) == 2
|
||||
print(" [PASS] MessageEnvelope.to_json() method")
|
||||
|
||||
|
||||
def test_serialize_data():
|
||||
"""Test _serialize_data function"""
|
||||
print("\n=== Testing _serialize_data ===")
|
||||
|
||||
# Test text serialization
|
||||
text_bytes = _serialize_data("Hello", "text")
|
||||
assert isinstance(text_bytes, bytes)
|
||||
assert text_bytes == b"Hello"
|
||||
print(" [PASS] Text serialization")
|
||||
|
||||
# Test dictionary serialization
|
||||
dict_data = {"key": "value", "number": 42}
|
||||
dict_bytes = _serialize_data(dict_data, "dictionary")
|
||||
assert isinstance(dict_bytes, bytes)
|
||||
parsed = json.loads(dict_bytes.decode('utf-8'))
|
||||
assert parsed["key"] == "value"
|
||||
print(" [PASS] Dictionary serialization")
|
||||
|
||||
# Test binary serialization
|
||||
binary_data = b"\x00\x01\x02"
|
||||
binary_bytes = _serialize_data(binary_data, "binary")
|
||||
assert binary_bytes == b"\x00\x01\x02"
|
||||
print(" [PASS] Binary serialization")
|
||||
|
||||
# Test image serialization
|
||||
image_data = bytes([1, 2, 3, 4, 5])
|
||||
image_bytes = _serialize_data(image_data, "image")
|
||||
assert image_bytes == image_data
|
||||
print(" [PASS] Image serialization")
|
||||
|
||||
|
||||
def test_deserialize_data():
|
||||
"""Test _deserialize_data function"""
|
||||
print("\n=== Testing _deserialize_data ===")
|
||||
|
||||
# Test text deserialization
|
||||
text_bytes = b"Hello"
|
||||
text_data = _deserialize_data(text_bytes, "text", "test-correlation-id")
|
||||
assert text_data == "Hello"
|
||||
print(" [PASS] Text deserialization")
|
||||
|
||||
# Test dictionary deserialization
|
||||
dict_bytes = b'{"key": "value"}'
|
||||
dict_data = _deserialize_data(dict_bytes, "dictionary", "test-correlation-id")
|
||||
assert dict_data == {"key": "value"}
|
||||
print(" [PASS] Dictionary deserialization")
|
||||
|
||||
# Test binary deserialization
|
||||
binary_data = b"\x00\x01\x02"
|
||||
binary_result = _deserialize_data(binary_data, "binary", "test-correlation-id")
|
||||
assert binary_result == b"\x00\x01\x02"
|
||||
print(" [PASS] Binary deserialization")
|
||||
|
||||
|
||||
def test_utilities():
|
||||
"""Test utility functions"""
|
||||
print("\n=== Testing Utility Functions ===")
|
||||
|
||||
# Test generate_uuid
|
||||
uuid1 = generate_uuid()
|
||||
uuid2 = generate_uuid()
|
||||
assert uuid1 != uuid2
|
||||
print(f" [PASS] generate_uuid() - generated: {uuid1}")
|
||||
|
||||
# Test get_timestamp
|
||||
timestamp = get_timestamp()
|
||||
assert "T" in timestamp
|
||||
print(f" [PASS] get_timestamp() - generated: {timestamp}")
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all tests"""
|
||||
print("=" * 60)
|
||||
print("NATSBridge Python/Micropython - Basic Functionality Tests")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
test_message_payload()
|
||||
test_message_envelope()
|
||||
test_serialize_data()
|
||||
test_deserialize_data()
|
||||
test_utilities()
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("ALL TESTS PASSED!")
|
||||
print("=" * 60)
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n[FAIL] Test failed with error: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,70 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for dictionary transport testing - Receiver
|
||||
Tests receiving dictionary messages via NATS using nats_bridge.py smartreceive
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add src to path for import
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||
|
||||
from nats_bridge import smartreceive, log_trace
|
||||
import nats
|
||||
import asyncio
|
||||
|
||||
# Configuration
|
||||
SUBJECT = "/NATSBridge_dict_test"
|
||||
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||
|
||||
|
||||
async def main():
|
||||
log_trace("", f"Starting dictionary transport receiver test...")
|
||||
log_trace("", f"Note: This receiver will wait for messages from the sender.")
|
||||
log_trace("", f"Run test_micropython_dict_sender.py first to send test data.")
|
||||
|
||||
# Connect to NATS
|
||||
nc = await nats.connect(NATS_URL)
|
||||
log_trace("", f"Connected to NATS at {NATS_URL}")
|
||||
|
||||
# Subscribe to the subject
|
||||
async def message_handler(msg):
|
||||
log_trace("", f"Received message on {msg.subject}")
|
||||
|
||||
# Use smartreceive to handle the data
|
||||
result = smartreceive(msg.data)
|
||||
|
||||
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
|
||||
for dataname, data, data_type in result["payloads"]:
|
||||
if isinstance(data, dict):
|
||||
log_trace(result.get("correlationId", ""), f"Received dictionary '{dataname}' of type {data_type}")
|
||||
log_trace(result.get("correlationId", ""), f" Keys: {list(data.keys())}")
|
||||
|
||||
# Display first few items for small dicts
|
||||
if isinstance(data, dict) and len(data) <= 10:
|
||||
log_trace(result.get("correlationId", ""), f" Content: {json.dumps(data, indent=2)}")
|
||||
else:
|
||||
# For large dicts, show summary
|
||||
log_trace(result.get("correlationId", ""), f" Summary: {json.dumps(data, default=str)[:200]}...")
|
||||
|
||||
# Save to file
|
||||
output_path = f"./received_{dataname}.json"
|
||||
with open(output_path, 'w') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
log_trace(result.get("correlationId", ""), f"Saved dictionary to {output_path}")
|
||||
else:
|
||||
log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}")
|
||||
|
||||
sid = await nc.subscribe(SUBJECT, cb=message_handler)
|
||||
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
|
||||
|
||||
# Keep listening for 120 seconds
|
||||
await asyncio.sleep(120)
|
||||
await nc.close()
|
||||
log_trace("", "Test completed.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -1,99 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for dictionary transport testing - Micropython
|
||||
Tests sending dictionary messages via NATS using nats_bridge.py smartsend
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add src to path for import
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||
|
||||
from nats_bridge import smartsend, log_trace
|
||||
import uuid
|
||||
|
||||
# Configuration
|
||||
SUBJECT = "/NATSBridge_dict_test"
|
||||
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||
FILESERVER_URL = "http://192.168.88.104:8080"
|
||||
SIZE_THRESHOLD = 1_000_000 # 1MB
|
||||
|
||||
# Create correlation ID for tracing
|
||||
correlation_id = str(uuid.uuid4())
|
||||
|
||||
|
||||
def main():
|
||||
# Create a small dictionary (will use direct transport)
|
||||
small_dict = {
|
||||
"name": "test",
|
||||
"value": 42,
|
||||
"enabled": True,
|
||||
"metadata": {
|
||||
"version": "1.0.0",
|
||||
"timestamp": "2026-02-22T12:00:00Z"
|
||||
}
|
||||
}
|
||||
|
||||
# Create a large dictionary (will use link transport if > 1MB)
|
||||
# Generate a larger dictionary (~2MB to ensure link transport)
|
||||
large_dict = {
|
||||
"id": str(uuid.uuid4()),
|
||||
"items": [
|
||||
{
|
||||
"index": i,
|
||||
"name": f"item_{i}",
|
||||
"value": i * 1.5,
|
||||
"data": "x" * 10000 # Large string per item
|
||||
}
|
||||
for i in range(200)
|
||||
],
|
||||
"metadata": {
|
||||
"count": 200,
|
||||
"created": "2026-02-22T12:00:00Z"
|
||||
}
|
||||
}
|
||||
|
||||
# Test data 1: small dictionary
|
||||
data1 = ("small_dict", small_dict, "dictionary")
|
||||
|
||||
# Test data 2: large dictionary
|
||||
data2 = ("large_dict", large_dict, "dictionary")
|
||||
|
||||
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
|
||||
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
||||
|
||||
# Use smartsend with dictionary type
|
||||
env = smartsend(
|
||||
SUBJECT,
|
||||
[data1, data2], # List of (dataname, data, type) tuples
|
||||
nats_url=NATS_URL,
|
||||
fileserver_url=FILESERVER_URL,
|
||||
size_threshold=SIZE_THRESHOLD,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose="chat",
|
||||
sender_name="dict_sender",
|
||||
receiver_name="",
|
||||
receiver_id="",
|
||||
reply_to="",
|
||||
reply_to_msg_id=""
|
||||
)
|
||||
|
||||
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
||||
|
||||
# Log transport type for each payload
|
||||
for i, payload in enumerate(env.payloads):
|
||||
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
|
||||
log_trace(correlation_id, f" Transport: {payload.transport}")
|
||||
log_trace(correlation_id, f" Type: {payload.type}")
|
||||
log_trace(correlation_id, f" Size: {payload.size} bytes")
|
||||
log_trace(correlation_id, f" Encoding: {payload.encoding}")
|
||||
|
||||
if payload.transport == "link":
|
||||
log_trace(correlation_id, f" URL: {payload.data}")
|
||||
|
||||
print(f"Test completed. Correlation ID: {correlation_id}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,65 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for file transport testing - Receiver
|
||||
Tests receiving binary files via NATS using nats_bridge.py smartreceive
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add src to path for import
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||
|
||||
from nats_bridge import smartreceive, log_trace
|
||||
import nats
|
||||
import asyncio
|
||||
|
||||
# Configuration
|
||||
SUBJECT = "/NATSBridge_file_test"
|
||||
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||
|
||||
|
||||
async def main():
|
||||
log_trace("", f"Starting file transport receiver test...")
|
||||
log_trace("", f"Note: This receiver will wait for messages from the sender.")
|
||||
log_trace("", f"Run test_micropython_file_sender.py first to send test data.")
|
||||
|
||||
# Connect to NATS
|
||||
nc = await nats.connect(NATS_URL)
|
||||
log_trace("", f"Connected to NATS at {NATS_URL}")
|
||||
|
||||
# Subscribe to the subject
|
||||
async def message_handler(msg):
|
||||
log_trace("", f"Received message on {msg.subject}")
|
||||
|
||||
# Use smartreceive to handle the data
|
||||
result = smartreceive(msg.data)
|
||||
|
||||
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
|
||||
for dataname, data, data_type in result["payloads"]:
|
||||
if isinstance(data, bytes):
|
||||
log_trace(result.get("correlationId", ""), f"Received binary '{dataname}' of type {data_type}")
|
||||
log_trace(result.get("correlationId", ""), f" Size: {len(data)} bytes")
|
||||
|
||||
# Display first 100 bytes as hex
|
||||
log_trace(result.get("correlationId", ""), f" First 100 bytes (hex): {data[:100].hex()}")
|
||||
|
||||
# Save to file
|
||||
output_path = f"./received_{dataname}.bin"
|
||||
with open(output_path, 'wb') as f:
|
||||
f.write(data)
|
||||
log_trace(result.get("correlationId", ""), f"Saved binary to {output_path}")
|
||||
else:
|
||||
log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}")
|
||||
|
||||
sid = await nc.subscribe(SUBJECT, cb=message_handler)
|
||||
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
|
||||
|
||||
# Keep listening for 120 seconds
|
||||
await asyncio.sleep(120)
|
||||
await nc.close()
|
||||
log_trace("", "Test completed.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -1,79 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for file transport testing - Micropython
|
||||
Tests sending binary files via NATS using nats_bridge.py smartsend
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add src to path for import
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||
|
||||
from nats_bridge import smartsend, log_trace
|
||||
import uuid
|
||||
|
||||
# Configuration
|
||||
SUBJECT = "/NATSBridge_file_test"
|
||||
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||
FILESERVER_URL = "http://192.168.88.104:8080"
|
||||
SIZE_THRESHOLD = 1_000_000 # 1MB
|
||||
|
||||
# Create correlation ID for tracing
|
||||
correlation_id = str(uuid.uuid4())
|
||||
|
||||
|
||||
def main():
|
||||
# Create small binary data (will use direct transport)
|
||||
small_binary = b"This is small binary data for testing direct transport."
|
||||
small_binary += b"\x00" * 100 # Add some null bytes
|
||||
|
||||
# Create large binary data (will use link transport if > 1MB)
|
||||
# Generate a larger binary (~2MB to ensure link transport)
|
||||
large_binary = bytes([
|
||||
(i * 7) % 256 for i in range(2_000_000)
|
||||
])
|
||||
|
||||
# Test data 1: small binary (direct transport)
|
||||
data1 = ("small_binary", small_binary, "binary")
|
||||
|
||||
# Test data 2: large binary (link transport)
|
||||
data2 = ("large_binary", large_binary, "binary")
|
||||
|
||||
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
|
||||
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
||||
|
||||
# Use smartsend with binary type
|
||||
env = smartsend(
|
||||
SUBJECT,
|
||||
[data1, data2], # List of (dataname, data, type) tuples
|
||||
nats_url=NATS_URL,
|
||||
fileserver_url=FILESERVER_URL,
|
||||
size_threshold=SIZE_THRESHOLD,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose="chat",
|
||||
sender_name="file_sender",
|
||||
receiver_name="",
|
||||
receiver_id="",
|
||||
reply_to="",
|
||||
reply_to_msg_id=""
|
||||
)
|
||||
|
||||
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
||||
|
||||
# Log transport type for each payload
|
||||
for i, payload in enumerate(env.payloads):
|
||||
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
|
||||
log_trace(correlation_id, f" Transport: {payload.transport}")
|
||||
log_trace(correlation_id, f" Type: {payload.type}")
|
||||
log_trace(correlation_id, f" Size: {payload.size} bytes")
|
||||
log_trace(correlation_id, f" Encoding: {payload.encoding}")
|
||||
|
||||
if payload.transport == "link":
|
||||
log_trace(correlation_id, f" URL: {payload.data}")
|
||||
|
||||
print(f"Test completed. Correlation ID: {correlation_id}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,97 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for mixed payload testing - Receiver
|
||||
Tests receiving mixed payload types via NATS using nats_bridge.py smartreceive
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add src to path for import
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||
|
||||
from nats_bridge import smartreceive, log_trace
|
||||
import nats
|
||||
import asyncio
|
||||
|
||||
# Configuration
|
||||
SUBJECT = "/NATSBridge_mixed_test"
|
||||
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||
|
||||
|
||||
async def main():
|
||||
log_trace("", f"Starting mixed payload receiver test...")
|
||||
log_trace("", f"Note: This receiver will wait for messages from the sender.")
|
||||
log_trace("", f"Run test_micropython_mixed_sender.py first to send test data.")
|
||||
|
||||
# Connect to NATS
|
||||
nc = await nats.connect(NATS_URL)
|
||||
log_trace("", f"Connected to NATS at {NATS_URL}")
|
||||
|
||||
# Subscribe to the subject
|
||||
async def message_handler(msg):
|
||||
log_trace("", f"Received message on {msg.subject}")
|
||||
|
||||
# Use smartreceive to handle the data
|
||||
result = smartreceive(msg.data)
|
||||
|
||||
log_trace(result.get("correlationId", ""), f"Received envelope with {len(result['payloads'])} payloads")
|
||||
|
||||
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
|
||||
for dataname, data, data_type in result["payloads"]:
|
||||
log_trace(result.get("correlationId", ""), f"\n--- Payload: {dataname} (type: {data_type}) ---")
|
||||
|
||||
if isinstance(data, str):
|
||||
log_trace(result.get("correlationId", ""), f" Type: text/string")
|
||||
log_trace(result.get("correlationId", ""), f" Length: {len(data)} characters")
|
||||
if len(data) <= 100:
|
||||
log_trace(result.get("correlationId", ""), f" Content: {data}")
|
||||
else:
|
||||
log_trace(result.get("correlationId", ""), f" First 100 chars: {data[:100]}...")
|
||||
# Save to file
|
||||
output_path = f"./received_{dataname}.txt"
|
||||
with open(output_path, 'w') as f:
|
||||
f.write(data)
|
||||
log_trace(result.get("correlationId", ""), f" Saved to: {output_path}")
|
||||
|
||||
elif isinstance(data, dict):
|
||||
log_trace(result.get("correlationId", ""), f" Type: dictionary")
|
||||
log_trace(result.get("correlationId", ""), f" Keys: {list(data.keys())}")
|
||||
log_trace(result.get("correlationId", ""), f" Content: {json.dumps(data, indent=2)}")
|
||||
# Save to file
|
||||
output_path = f"./received_{dataname}.json"
|
||||
with open(output_path, 'w') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
log_trace(result.get("correlationId", ""), f" Saved to: {output_path}")
|
||||
|
||||
elif isinstance(data, bytes):
|
||||
log_trace(result.get("correlationId", ""), f" Type: binary")
|
||||
log_trace(result.get("correlationId", ""), f" Size: {len(data)} bytes")
|
||||
log_trace(result.get("correlationId", ""), f" First 100 bytes (hex): {data[:100].hex()}")
|
||||
# Save to file
|
||||
output_path = f"./received_{dataname}.bin"
|
||||
with open(output_path, 'wb') as f:
|
||||
f.write(data)
|
||||
log_trace(result.get("correlationId", ""), f" Saved to: {output_path}")
|
||||
else:
|
||||
log_trace(result.get("correlationId", ""), f" Received unexpected data type: {type(data)}")
|
||||
|
||||
# Log envelope metadata
|
||||
log_trace(result.get("correlationId", ""), f"\n--- Envelope Metadata ---")
|
||||
log_trace(result.get("correlationId", ""), f" Correlation ID: {result.get('correlationId', 'N/A')}")
|
||||
log_trace(result.get("correlationId", ""), f" Message ID: {result.get('msgId', 'N/A')}")
|
||||
log_trace(result.get("correlationId", ""), f" Sender: {result.get('senderName', 'N/A')}")
|
||||
log_trace(result.get("correlationId", ""), f" Purpose: {result.get('msgPurpose', 'N/A')}")
|
||||
|
||||
sid = await nc.subscribe(SUBJECT, cb=message_handler)
|
||||
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
|
||||
|
||||
# Keep listening for 120 seconds
|
||||
await asyncio.sleep(120)
|
||||
await nc.close()
|
||||
log_trace("", "Test completed.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -1,93 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for mixed payload testing - Micropython
|
||||
Tests sending mixed payload types via NATS using nats_bridge.py smartsend
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add src to path for import
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||
|
||||
from nats_bridge import smartsend, log_trace
|
||||
import uuid
|
||||
|
||||
# Configuration
|
||||
SUBJECT = "/NATSBridge_mixed_test"
|
||||
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||
FILESERVER_URL = "http://192.168.88.104:8080"
|
||||
SIZE_THRESHOLD = 1_000_000 # 1MB
|
||||
|
||||
# Create correlation ID for tracing
|
||||
correlation_id = str(uuid.uuid4())
|
||||
|
||||
|
||||
def main():
|
||||
# Create payloads for mixed content test
|
||||
|
||||
# 1. Small text (direct transport)
|
||||
text_data = "Hello, this is a text message for testing mixed payloads!"
|
||||
|
||||
# 2. Small dictionary (direct transport)
|
||||
dict_data = {
|
||||
"status": "ok",
|
||||
"code": 200,
|
||||
"message": "Test successful",
|
||||
"items": [1, 2, 3]
|
||||
}
|
||||
|
||||
# 3. Small binary (direct transport)
|
||||
binary_data = b"\x00\x01\x02\x03\x04\x05" + b"\xff" * 100
|
||||
|
||||
# 4. Large text (link transport - will use fileserver)
|
||||
large_text = "\n".join([
|
||||
f"Line {i}: This is a large text payload for link transport testing. " * 50
|
||||
for i in range(100)
|
||||
])
|
||||
|
||||
# Test data list - mixed payload types
|
||||
data = [
|
||||
("message_text", text_data, "text"),
|
||||
("config_dict", dict_data, "dictionary"),
|
||||
("small_binary", binary_data, "binary"),
|
||||
("large_text", large_text, "text"),
|
||||
]
|
||||
|
||||
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
|
||||
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
||||
|
||||
# Use smartsend with mixed types
|
||||
env = smartsend(
|
||||
SUBJECT,
|
||||
data, # List of (dataname, data, type) tuples
|
||||
nats_url=NATS_URL,
|
||||
fileserver_url=FILESERVER_URL,
|
||||
size_threshold=SIZE_THRESHOLD,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose="chat",
|
||||
sender_name="mixed_sender",
|
||||
receiver_name="",
|
||||
receiver_id="",
|
||||
reply_to="",
|
||||
reply_to_msg_id=""
|
||||
)
|
||||
|
||||
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
||||
|
||||
# Log transport type for each payload
|
||||
for i, payload in enumerate(env.payloads):
|
||||
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
|
||||
log_trace(correlation_id, f" Transport: {payload.transport}")
|
||||
log_trace(correlation_id, f" Type: {payload.type}")
|
||||
log_trace(correlation_id, f" Size: {payload.size} bytes")
|
||||
log_trace(correlation_id, f" Encoding: {payload.encoding}")
|
||||
|
||||
if payload.transport == "link":
|
||||
log_trace(correlation_id, f" URL: {payload.data}")
|
||||
|
||||
print(f"Test completed. Correlation ID: {correlation_id}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,69 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for text transport testing - Receiver
|
||||
Tests receiving text messages via NATS using nats_bridge.py smartreceive
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add src to path for import
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||
|
||||
from nats_bridge import smartreceive, log_trace
|
||||
import nats
|
||||
import asyncio
|
||||
|
||||
# Configuration
|
||||
SUBJECT = "/NATSBridge_text_test"
|
||||
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||
|
||||
|
||||
async def main():
|
||||
log_trace("", f"Starting text transport receiver test...")
|
||||
log_trace("", f"Note: This receiver will wait for messages from the sender.")
|
||||
log_trace("", f"Run test_micropython_text_sender.py first to send test data.")
|
||||
|
||||
# Connect to NATS
|
||||
nc = await nats.connect(NATS_URL)
|
||||
log_trace("", f"Connected to NATS at {NATS_URL}")
|
||||
|
||||
# Subscribe to the subject
|
||||
async def message_handler(msg):
|
||||
log_trace("", f"Received message on {msg.subject}")
|
||||
|
||||
# Use smartreceive to handle the data
|
||||
result = smartreceive(msg.data)
|
||||
|
||||
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
|
||||
for dataname, data, data_type in result["payloads"]:
|
||||
if isinstance(data, str):
|
||||
log_trace(result.get("correlationId", ""), f"Received text '{dataname}' of type {data_type}")
|
||||
log_trace(result.get("correlationId", ""), f" Length: {len(data)} characters")
|
||||
|
||||
# Display first 100 characters
|
||||
if len(data) > 100:
|
||||
log_trace(result.get("correlationId", ""), f" First 100 characters: {data[:100]}...")
|
||||
else:
|
||||
log_trace(result.get("correlationId", ""), f" Content: {data}")
|
||||
|
||||
# Save to file
|
||||
output_path = f"./received_{dataname}.txt"
|
||||
with open(output_path, 'w') as f:
|
||||
f.write(data)
|
||||
log_trace(result.get("correlationId", ""), f"Saved text to {output_path}")
|
||||
else:
|
||||
log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}")
|
||||
|
||||
sid = await nc.subscribe(SUBJECT, cb=message_handler)
|
||||
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
|
||||
|
||||
# Keep listening for 120 seconds
|
||||
await asyncio.sleep(120)
|
||||
await nc.close()
|
||||
log_trace("", "Test completed.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -1,81 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for text transport testing - Micropython
|
||||
Tests sending text messages via NATS using nats_bridge.py smartsend
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add src to path for import
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||
|
||||
from nats_bridge import smartsend, log_trace
|
||||
import uuid
|
||||
|
||||
# Configuration
|
||||
SUBJECT = "/NATSBridge_text_test"
|
||||
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||
FILESERVER_URL = "http://192.168.88.104:8080"
|
||||
SIZE_THRESHOLD = 1_000_000 # 1MB
|
||||
|
||||
# Create correlation ID for tracing
|
||||
correlation_id = str(uuid.uuid4())
|
||||
|
||||
|
||||
def main():
|
||||
# Create a small text (will use direct transport)
|
||||
small_text = "Hello, this is a small text message. Testing direct transport via NATS."
|
||||
|
||||
# Create a large text (will use link transport if > 1MB)
|
||||
# Generate a larger text (~2MB to ensure link transport)
|
||||
large_text = "\n".join([
|
||||
f"Line {i}: This is a sample text line with some content to pad the size. " * 100
|
||||
for i in range(500)
|
||||
])
|
||||
|
||||
# Test data 1: small text
|
||||
data1 = ("small_text", small_text, "text")
|
||||
|
||||
# Test data 2: large text
|
||||
data2 = ("large_text", large_text, "text")
|
||||
|
||||
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
|
||||
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
||||
|
||||
# Use smartsend with text type
|
||||
# For small text: will use direct transport (Base64 encoded UTF-8)
|
||||
# For large text: will use link transport (uploaded to fileserver)
|
||||
env = smartsend(
|
||||
SUBJECT,
|
||||
[data1, data2], # List of (dataname, data, type) tuples
|
||||
nats_url=NATS_URL,
|
||||
fileserver_url=FILESERVER_URL,
|
||||
size_threshold=SIZE_THRESHOLD,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose="chat",
|
||||
sender_name="text_sender",
|
||||
receiver_name="",
|
||||
receiver_id="",
|
||||
reply_to="",
|
||||
reply_to_msg_id=""
|
||||
)
|
||||
|
||||
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
||||
|
||||
# Log transport type for each payload
|
||||
for i, payload in enumerate(env.payloads):
|
||||
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
|
||||
log_trace(correlation_id, f" Transport: {payload.transport}")
|
||||
log_trace(correlation_id, f" Type: {payload.type}")
|
||||
log_trace(correlation_id, f" Size: {payload.size} bytes")
|
||||
log_trace(correlation_id, f" Encoding: {payload.encoding}")
|
||||
|
||||
if payload.transport == "link":
|
||||
log_trace(correlation_id, f" URL: {payload.data}")
|
||||
|
||||
print(f"Test completed. Correlation ID: {correlation_id}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
185
test/test_mpy_binary_receiver.py
Normal file
185
test/test_mpy_binary_receiver.py
Normal file
@@ -0,0 +1,185 @@
|
||||
"""
|
||||
MicroPython Binary Receiver Test
|
||||
Tests the smartreceive function with binary/image/audio/video payloads
|
||||
|
||||
Note: This test is designed for both MicroPython and desktop Python
|
||||
for compatibility testing.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import base64
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge_mpy import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
def run_test():
|
||||
print('=== MicroPython Binary Receiver Test ===\n')
|
||||
|
||||
from natbridge_mpy import _generate_uuid
|
||||
|
||||
# Create mock NATS message with binary payloads
|
||||
image_data = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
|
||||
audio_data = bytes([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]) # FLAC header
|
||||
video_data = bytes([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]) # MP4 header
|
||||
generic_binary = bytes([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE])
|
||||
|
||||
test_data = {
|
||||
'correlation_id': 'mpy-binary-receiver-' + _generate_uuid(),
|
||||
'msg_id': _generate_uuid(),
|
||||
'timestamp': '2024-01-15T10:30:00Z',
|
||||
'send_to': '/test/binary',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'mpy-binary-test',
|
||||
'sender_id': _generate_uuid(),
|
||||
'receiver_name': 'mpy-receiver',
|
||||
'receiver_id': _generate_uuid(),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'image',
|
||||
'payload_type': 'image',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(image_data),
|
||||
'data': base64.b64encode(image_data).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(image_data)}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'audio',
|
||||
'payload_type': 'audio',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(audio_data),
|
||||
'data': base64.b64encode(audio_data).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(audio_data)}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'video',
|
||||
'payload_type': 'video',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(video_data),
|
||||
'data': base64.b64encode(video_data).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(video_data)}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'binary',
|
||||
'payload_type': 'binary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(generic_binary),
|
||||
'data': base64.b64encode(generic_binary).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(generic_binary)}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_msg = {
|
||||
'payload': json.dumps(test_data)
|
||||
}
|
||||
|
||||
print('Mock Message Created:')
|
||||
print(f' Correlation ID: {test_data["correlation_id"]}')
|
||||
print(f' Payloads: {len(test_data["payloads"])}')
|
||||
print(f' Payload types: {", ".join(p["payload_type"] for p in test_data["payloads"])}\n')
|
||||
|
||||
try:
|
||||
# Receive and process the message
|
||||
print('Receiving and processing message...')
|
||||
env = smartreceive(
|
||||
mock_msg,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
|
||||
print('\n=== Received Envelope ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate received data
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Expected data
|
||||
expected_data = [
|
||||
('image', image_data, 'image'),
|
||||
('audio', audio_data, 'audio'),
|
||||
('video', video_data, 'video'),
|
||||
('binary', generic_binary, 'binary')
|
||||
]
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
expected = expected_data[i]
|
||||
|
||||
if payload[0] != expected[0]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected[0]}', got '{payload[0]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload[2] != expected[2]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected[2]}', got '{payload[2]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
# Verify binary data integrity
|
||||
received_data = payload[1]
|
||||
if received_data != expected[1]:
|
||||
print(f'❌ Payload {i + 1}: Data mismatch')
|
||||
print(f' Expected: {expected[1]}')
|
||||
print(f' Got: {received_data}')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data correctly deserialized')
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_test()
|
||||
163
test/test_mpy_binary_sender.py
Normal file
163
test/test_mpy_binary_sender.py
Normal file
@@ -0,0 +1,163 @@
|
||||
"""
|
||||
MicroPython Binary Sender Test
|
||||
Tests the smartsend function with binary/image/audio/video payloads
|
||||
|
||||
Note: This test is designed for both MicroPython and desktop Python
|
||||
for compatibility testing.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import base64
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge_mpy import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL, DEFAULT_SIZE_THRESHOLD, MAX_PAYLOAD_SIZE
|
||||
|
||||
TEST_SUBJECT = '/test/binary'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
def run_test():
|
||||
print('=== MicroPython Binary Sender Test ===\n')
|
||||
|
||||
from natbridge_mpy import _generate_uuid
|
||||
correlation_id = 'mpy-binary-test-' + _generate_uuid()
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}')
|
||||
print(f'Default Size Threshold: {DEFAULT_SIZE_THRESHOLD} bytes')
|
||||
print(f'Max Payload Size: {MAX_PAYLOAD_SIZE} bytes\n')
|
||||
|
||||
# Test data - binary data for different types
|
||||
image_data = bytearray([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
|
||||
audio_data = bytearray([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]) # FLAC header
|
||||
video_data = bytearray([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]) # MP4 header
|
||||
generic_binary = bytearray([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE])
|
||||
|
||||
test_data = [
|
||||
('image', bytes(image_data), 'image'),
|
||||
('audio', bytes(audio_data), 'audio'),
|
||||
('video', bytes(video_data), 'video'),
|
||||
('binary', bytes(generic_binary), 'binary')
|
||||
]
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending binary payloads...')
|
||||
env, env_json_str = smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='mpy-binary-test',
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Test each payload
|
||||
expected_datanames = ['image', 'audio', 'video', 'binary']
|
||||
expected_types = ['image', 'audio', 'video', 'binary']
|
||||
expected_data = [bytes(image_data), bytes(audio_data), bytes(video_data), bytes(generic_binary)]
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
|
||||
if payload['dataname'] != expected_datanames[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload['payload_type'] != expected_types[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct encoding')
|
||||
|
||||
# Decode and verify the data
|
||||
decoded_data = base64.b64decode(payload['data'])
|
||||
original_data = expected_data[i]
|
||||
|
||||
if decoded_data != original_data:
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
|
||||
print(f' Size: {payload["size"]} bytes\n')
|
||||
|
||||
# Test with larger binary data
|
||||
print('=== Large Binary Data Test ===')
|
||||
large_data = bytes([0xFF] * 1000) # 1KB of binary data
|
||||
large_test_data = [
|
||||
('large_binary', large_data, 'binary')
|
||||
]
|
||||
|
||||
large_env, _ = smartsend(
|
||||
TEST_SUBJECT,
|
||||
large_test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='large-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if len(large_env['payloads']) == 1 and large_env['payloads'][0]['size'] == 1000:
|
||||
print('✅ Large binary data handled correctly')
|
||||
else:
|
||||
print('❌ Large binary data handling failed')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_test()
|
||||
224
test/test_mpy_dictionary_receiver.py
Normal file
224
test/test_mpy_dictionary_receiver.py
Normal file
@@ -0,0 +1,224 @@
|
||||
"""
|
||||
MicroPython Dictionary Receiver Test
|
||||
Tests the smartreceive function with dictionary payloads
|
||||
|
||||
Note: This test is designed for both MicroPython and desktop Python
|
||||
for compatibility testing.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge_mpy import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
def run_test():
|
||||
print('=== MicroPython Dictionary Receiver Test ===\n')
|
||||
|
||||
from natbridge_mpy import _generate_uuid
|
||||
|
||||
# Create a mock NATS message with dictionary payloads
|
||||
import base64
|
||||
|
||||
simple_dict = {'key1': 'value1', 'key2': 'value2'}
|
||||
nested_dict = {'outer': {'inner': 'value', 'number': 42}}
|
||||
array_dict = {'items': [1, 2, 3, 'four', 'five']}
|
||||
mixed_dict = {'string': 'text', 'number': 123, 'boolean': True, 'null_val': None}
|
||||
|
||||
test_data = {
|
||||
'correlation_id': 'mpy-receiver-dict-' + _generate_uuid(),
|
||||
'msg_id': _generate_uuid(),
|
||||
'timestamp': '2024-01-15T10:30:00Z',
|
||||
'send_to': '/test/dictionary',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'mpy-dict-test',
|
||||
'sender_id': _generate_uuid(),
|
||||
'receiver_name': 'mpy-receiver',
|
||||
'receiver_id': _generate_uuid(),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'simple_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(simple_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(simple_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(simple_dict).encode('utf8'))}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'nested_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(nested_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(nested_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(nested_dict).encode('utf8'))}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'array_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(array_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(array_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(array_dict).encode('utf8'))}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'mixed_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(mixed_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(mixed_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(mixed_dict).encode('utf8'))}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_msg = {
|
||||
'payload': json.dumps(test_data)
|
||||
}
|
||||
|
||||
print('Mock Message Created:')
|
||||
print(f' Correlation ID: {test_data["correlation_id"]}')
|
||||
print(f' Payloads: {len(test_data["payloads"])}')
|
||||
print(f' Payload types: {", ".join(p["payload_type"] for p in test_data["payloads"])}\n')
|
||||
|
||||
try:
|
||||
# Receive and process the message
|
||||
print('Receiving and processing message...')
|
||||
env = smartreceive(
|
||||
mock_msg,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
|
||||
print('\n=== Received Envelope ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate received data
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Expected data
|
||||
expected_data = [
|
||||
('simple_dict', simple_dict, 'dictionary'),
|
||||
('nested_dict', nested_dict, 'dictionary'),
|
||||
('array_dict', array_dict, 'dictionary'),
|
||||
('mixed_dict', mixed_dict, 'dictionary')
|
||||
]
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
expected = expected_data[i]
|
||||
|
||||
if payload[0] != expected[0]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected[0]}', got '{payload[0]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload[2] != expected[2]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected[2]}', got '{payload[2]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
data_match = json.dumps(payload[1], sort_keys=True) == json.dumps(expected[1], sort_keys=True)
|
||||
if not data_match:
|
||||
print(f'❌ Payload {i + 1}: Data mismatch')
|
||||
print(f' Expected: {json.dumps(expected[1], sort_keys=True)}')
|
||||
print(f' Got: {json.dumps(payload[1], sort_keys=True)}')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data correctly deserialized')
|
||||
|
||||
# Test round-trip with receive
|
||||
print('\n=== Round-trip Test ===')
|
||||
round_trip_data = {
|
||||
'correlation_id': 'roundtrip-' + _generate_uuid(),
|
||||
'msg_id': _generate_uuid(),
|
||||
'timestamp': '2024-01-15T10:30:00Z',
|
||||
'send_to': '/test/dictionary',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'mpy-test',
|
||||
'sender_id': _generate_uuid(),
|
||||
'receiver_name': 'mpy-receiver',
|
||||
'receiver_id': _generate_uuid(),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'roundtrip',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8'))}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_rt_msg = {'payload': json.dumps(round_trip_data)}
|
||||
rt_env = smartreceive(mock_rt_msg)
|
||||
|
||||
if rt_env['payloads'][0][0] == 'roundtrip' and rt_env['payloads'][0][2] == 'dictionary':
|
||||
print('✅ Round-trip test successful')
|
||||
else:
|
||||
print('❌ Round-trip test failed')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_test()
|
||||
177
test/test_mpy_dictionary_sender.py
Normal file
177
test/test_mpy_dictionary_sender.py
Normal file
@@ -0,0 +1,177 @@
|
||||
"""
|
||||
MicroPython Dictionary Sender Test
|
||||
Tests the smartsend function with dictionary payloads
|
||||
|
||||
Note: This test is designed for both MicroPython and desktop Python
|
||||
for compatibility testing.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge_mpy import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL, DEFAULT_SIZE_THRESHOLD, MAX_PAYLOAD_SIZE
|
||||
|
||||
TEST_SUBJECT = '/test/dictionary'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
def run_test():
|
||||
print('=== MicroPython Dictionary Sender Test ===\n')
|
||||
|
||||
from natbridge_mpy import _generate_uuid
|
||||
correlation_id = 'mpy-dict-test-' + _generate_uuid()
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}')
|
||||
print(f'Default Size Threshold: {DEFAULT_SIZE_THRESHOLD} bytes')
|
||||
print(f'Max Payload Size: {MAX_PAYLOAD_SIZE} bytes\n')
|
||||
|
||||
# Test data - various dictionary structures
|
||||
test_data = [
|
||||
('simple_dict', {'key1': 'value1', 'key2': 'value2'}, 'dictionary'),
|
||||
('nested_dict', {'outer': {'inner': 'value', 'number': 42}}, 'dictionary'),
|
||||
('array_dict', {'items': [1, 2, 3, 'four', 'five']}, 'dictionary'),
|
||||
('mixed_dict', {'string': 'text', 'number': 123, 'boolean': True, 'null_val': None}, 'dictionary')
|
||||
]
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending dictionary payloads...')
|
||||
env, env_json_str = smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='mpy-dict-test',
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Test each payload
|
||||
expected_datanames = ['simple_dict', 'nested_dict', 'array_dict', 'mixed_dict']
|
||||
expected_types = ['dictionary', 'dictionary', 'dictionary', 'dictionary']
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
|
||||
if payload['dataname'] != expected_datanames[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload['payload_type'] != expected_types[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct encoding')
|
||||
|
||||
# Decode and verify the data
|
||||
import base64
|
||||
decoded_data = json.loads(base64.b64decode(payload['data']).decode('ascii'))
|
||||
original_data = test_data[i][1]
|
||||
|
||||
# Normalize for comparison
|
||||
if json.dumps(decoded_data, sort_keys=True) != json.dumps(original_data, sort_keys=True):
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
print(f' Expected: {json.dumps(original_data)}')
|
||||
print(f' Got: {json.dumps(decoded_data)}')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
|
||||
print(f' Size: {payload["size"]} bytes\n')
|
||||
|
||||
# Test round-trip serialization
|
||||
print('=== Round-trip Serialization Test ===')
|
||||
round_trip_data = [
|
||||
('roundtrip', {'test': 'data', 'numbers': [1, 2, 3], 'nested': {'a': 1, 'b': 2}}, 'dictionary')
|
||||
]
|
||||
|
||||
rt_env, _ = smartsend(
|
||||
TEST_SUBJECT,
|
||||
round_trip_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='roundtrip-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
rt_payload = rt_env['payloads'][0]
|
||||
rt_decoded = json.loads(base64.b64decode(rt_payload['data']).decode('ascii'))
|
||||
|
||||
if json.dumps(rt_decoded, sort_keys=True) == json.dumps(round_trip_data[0][1], sort_keys=True):
|
||||
print('✅ Round-trip serialization successful')
|
||||
else:
|
||||
print('❌ Round-trip serialization failed')
|
||||
passed = False
|
||||
|
||||
# Test JSON string output
|
||||
print('\n=== JSON String Output Test ===')
|
||||
try:
|
||||
parsed = json.loads(env_json_str)
|
||||
if parsed['correlation_id'] == env['correlation_id'] and \
|
||||
len(parsed['payloads']) == len(env['payloads']):
|
||||
print('✅ JSON string is valid and matches envelope')
|
||||
else:
|
||||
print('❌ JSON string does not match envelope')
|
||||
passed = False
|
||||
except json.JSONDecodeError as e:
|
||||
print(f'❌ JSON string is invalid: {e}')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_test()
|
||||
209
test/test_mpy_text_receiver.py
Normal file
209
test/test_mpy_text_receiver.py
Normal file
@@ -0,0 +1,209 @@
|
||||
"""
|
||||
MicroPython Text Receiver Test
|
||||
Tests the smartreceive function with text payloads
|
||||
|
||||
Note: This test is designed for both MicroPython and desktop Python
|
||||
for compatibility testing.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge_mpy import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
def run_test():
|
||||
print('=== MicroPython Text Receiver Test ===\n')
|
||||
|
||||
from natbridge_mpy import _generate_uuid
|
||||
|
||||
# Create a mock NATS message with text payload
|
||||
test_text = 'Hello, NATSBridge! This is a test message.'
|
||||
import base64
|
||||
|
||||
test_data = {
|
||||
'correlation_id': 'mpy-receiver-test-' + _generate_uuid(),
|
||||
'msg_id': _generate_uuid(),
|
||||
'timestamp': '2024-01-15T10:30:00Z',
|
||||
'send_to': '/test/text',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'mpy-text-test',
|
||||
'sender_id': _generate_uuid(),
|
||||
'receiver_name': 'mpy-receiver',
|
||||
'receiver_id': _generate_uuid(),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'message',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(test_text.encode('utf8')),
|
||||
'data': base64.b64encode(test_text.encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(test_text.encode('utf8'))}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_msg = {
|
||||
'payload': json.dumps(test_data)
|
||||
}
|
||||
|
||||
print('Mock Message Created:')
|
||||
print(f' Correlation ID: {test_data["correlation_id"]}')
|
||||
print(f' Payloads: {len(test_data["payloads"])}')
|
||||
print(f' Payload dataname: {test_data["payloads"][0]["dataname"]}')
|
||||
print(f' Payload type: {test_data["payloads"][0]["payload_type"]}')
|
||||
print(f' Transport: {test_data["payloads"][0]["transport"]}\n')
|
||||
|
||||
try:
|
||||
# Receive and process the message
|
||||
print('Receiving and processing message...')
|
||||
env = smartreceive(
|
||||
mock_msg,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
|
||||
print('\n=== Received Envelope ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate received data
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if len(env['payloads']) != 1:
|
||||
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
payload = env['payloads'][0]
|
||||
if payload[0] != 'message':
|
||||
print(f"❌ Expected dataname 'message', got '{payload[0]}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct dataname')
|
||||
|
||||
if payload[2] != 'text':
|
||||
print(f"❌ Expected type 'text', got '{payload[2]}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct type')
|
||||
|
||||
if payload[1] != test_text:
|
||||
print('❌ Data mismatch')
|
||||
print(f' Expected: {test_text}')
|
||||
print(f' Got: {payload[1]}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Data correctly deserialized')
|
||||
|
||||
# Test with multiple text payloads
|
||||
print('\n=== Multiple Text Payloads Test ===')
|
||||
multi_test_data = {
|
||||
'correlation_id': 'multi-receiver-' + _generate_uuid(),
|
||||
'msg_id': _generate_uuid(),
|
||||
'timestamp': '2024-01-15T10:30:00Z',
|
||||
'send_to': '/test/text',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'mpy-text-test',
|
||||
'sender_id': _generate_uuid(),
|
||||
'receiver_name': 'mpy-receiver',
|
||||
'receiver_id': _generate_uuid(),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'msg1',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': 16,
|
||||
'data': base64.b64encode(b'First message').decode('ascii'),
|
||||
'metadata': {'payload_bytes': 16}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'msg2',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': 16,
|
||||
'data': base64.b64encode(b'Second message').decode('ascii'),
|
||||
'metadata': {'payload_bytes': 16}
|
||||
},
|
||||
{
|
||||
'id': _generate_uuid(),
|
||||
'dataname': 'msg3',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': 16,
|
||||
'data': base64.b64encode(b'Third message').decode('ascii'),
|
||||
'metadata': {'payload_bytes': 16}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_multi_msg = {'payload': json.dumps(multi_test_data)}
|
||||
multi_env = smartreceive(mock_multi_msg)
|
||||
|
||||
if len(multi_env['payloads']) == 3:
|
||||
print('✅ Multiple payloads handled correctly')
|
||||
|
||||
# Verify each payload
|
||||
expected_messages = ['First message', 'Second message', 'Third message']
|
||||
for i in range(3):
|
||||
if multi_env['payloads'][i][1] == expected_messages[i]:
|
||||
print(f'✅ Payload {i + 1} correctly deserialized')
|
||||
else:
|
||||
print(f'❌ Payload {i + 1} mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f'❌ Expected 3 payloads, got {len(multi_env["payloads"])}')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_test()
|
||||
205
test/test_mpy_text_sender.py
Normal file
205
test/test_mpy_text_sender.py
Normal file
@@ -0,0 +1,205 @@
|
||||
"""
|
||||
MicroPython Text Sender Test
|
||||
Tests the smartsend function with text payloads
|
||||
|
||||
Note: This test is designed for both MicroPython and desktop Python
|
||||
for compatibility testing.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge_mpy import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL, DEFAULT_SIZE_THRESHOLD, MAX_PAYLOAD_SIZE
|
||||
|
||||
TEST_SUBJECT = '/test/text'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
def run_test():
|
||||
print('=== MicroPython Text Sender Test ===\n')
|
||||
|
||||
from natbridge_mpy import _generate_uuid
|
||||
correlation_id = 'mpy-text-test-' + _generate_uuid()
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}')
|
||||
print(f'Default Size Threshold: {DEFAULT_SIZE_THRESHOLD} bytes')
|
||||
print(f'Max Payload Size: {MAX_PAYLOAD_SIZE} bytes\n')
|
||||
|
||||
# Test data
|
||||
text_data = 'Hello, NATSBridge! This is a test message.'
|
||||
test_data = [
|
||||
('message', text_data, 'text')
|
||||
]
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending text payload...')
|
||||
env, env_json_str = smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='mpy-text-test',
|
||||
is_publish=False # Don't actually publish for this test
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if not env.get('msg_id'):
|
||||
print('❌ msg_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ msg_id present')
|
||||
|
||||
if not env.get('timestamp'):
|
||||
print('❌ timestamp is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ timestamp present')
|
||||
|
||||
if len(env['payloads']) != 1:
|
||||
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
payload = env['payloads'][0]
|
||||
if payload['dataname'] != 'message':
|
||||
print(f"❌ Expected dataname 'message', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct dataname')
|
||||
|
||||
if payload['payload_type'] != 'text':
|
||||
print(f"❌ Expected payload_type 'text', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct payload_type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct encoding')
|
||||
|
||||
# Decode and verify the data
|
||||
import base64
|
||||
decoded_data = base64.b64decode(payload['data']).decode('ascii')
|
||||
if decoded_data != text_data:
|
||||
print('❌ Decoded data mismatch')
|
||||
print(f' Expected: {text_data}')
|
||||
print(f' Got: {decoded_data}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Data integrity verified')
|
||||
|
||||
print(f'\nPayload size: {payload["size"]} bytes')
|
||||
print(f'Base64 data length: {len(payload["data"])} chars')
|
||||
|
||||
# Test with multiple text payloads
|
||||
print('\n=== Multiple Text Payloads Test ===')
|
||||
multi_test_data = [
|
||||
('msg1', 'First message', 'text'),
|
||||
('msg2', 'Second message', 'text'),
|
||||
('msg3', 'Third message', 'text')
|
||||
]
|
||||
|
||||
multi_env, _ = smartsend(
|
||||
TEST_SUBJECT,
|
||||
multi_test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='multi-test-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if len(multi_env['payloads']) == 3:
|
||||
print('✅ Multiple payloads handled correctly')
|
||||
else:
|
||||
print(f'❌ Expected 3 payloads, got {len(multi_env["payloads"])}')
|
||||
passed = False
|
||||
|
||||
# Test size threshold enforcement
|
||||
print('\n=== Size Threshold Test ===')
|
||||
small_text = 'small'
|
||||
large_text = 'x' * (DEFAULT_SIZE_THRESHOLD - 100) # Just under threshold
|
||||
|
||||
small_env, _ = smartsend(
|
||||
TEST_SUBJECT,
|
||||
[('small', small_text, 'text')],
|
||||
broker_url=TEST_BROKER_URL,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if small_env['payloads'][0]['transport'] == 'direct':
|
||||
print('✅ Small payload uses direct transport')
|
||||
else:
|
||||
print('❌ Small payload should use direct transport')
|
||||
passed = False
|
||||
|
||||
# Test that large text (> MAX_PAYLOAD_SIZE) raises error
|
||||
print('\n=== Max Payload Size Test ===')
|
||||
try:
|
||||
too_large_text = 'x' * (MAX_PAYLOAD_SIZE + 1000)
|
||||
large_env, _ = smartsend(
|
||||
TEST_SUBJECT,
|
||||
[('large', too_large_text, 'text')],
|
||||
broker_url=TEST_BROKER_URL,
|
||||
is_publish=False
|
||||
)
|
||||
print('❌ Should have raised MemoryError for payload exceeding MAX_PAYLOAD_SIZE')
|
||||
passed = False
|
||||
except MemoryError as e:
|
||||
print(f'✅ Correctly raised MemoryError: {e}')
|
||||
except Exception as e:
|
||||
print(f'❌ Unexpected error: {type(e).__name__}: {e}')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_test()
|
||||
184
test/test_py_binary_receiver.py
Normal file
184
test/test_py_binary_receiver.py
Normal file
@@ -0,0 +1,184 @@
|
||||
"""
|
||||
Python Binary Receiver Test
|
||||
Tests the smartreceive function with binary/image/audio/video/table payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import base64
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Binary Receiver Test ===\n')
|
||||
|
||||
# Create mock NATS message with binary payloads
|
||||
image_data = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
|
||||
audio_data = bytes([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]) # FLAC header
|
||||
video_data = bytes([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]) # MP4 header
|
||||
generic_binary = bytes([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE])
|
||||
|
||||
test_data = {
|
||||
'correlation_id': 'py-binary-receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'timestamp': asyncio.get_event_loop().time().isoformat(),
|
||||
'send_to': '/test/binary',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'py-binary-test',
|
||||
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'receiver_name': 'py-receiver',
|
||||
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': 'payload-1',
|
||||
'dataname': 'image',
|
||||
'payload_type': 'image',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(image_data),
|
||||
'data': base64.b64encode(image_data).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(image_data)}
|
||||
},
|
||||
{
|
||||
'id': 'payload-2',
|
||||
'dataname': 'audio',
|
||||
'payload_type': 'audio',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(audio_data),
|
||||
'data': base64.b64encode(audio_data).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(audio_data)}
|
||||
},
|
||||
{
|
||||
'id': 'payload-3',
|
||||
'dataname': 'video',
|
||||
'payload_type': 'video',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(video_data),
|
||||
'data': base64.b64encode(video_data).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(video_data)}
|
||||
},
|
||||
{
|
||||
'id': 'payload-4',
|
||||
'dataname': 'binary',
|
||||
'payload_type': 'binary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(generic_binary),
|
||||
'data': base64.b64encode(generic_binary).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(generic_binary)}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_msg = {
|
||||
'payload': json.dumps(test_data)
|
||||
}
|
||||
|
||||
print('Mock Message Created:')
|
||||
print(f' Correlation ID: {test_data["correlation_id"]}')
|
||||
print(f' Payloads: {len(test_data["payloads"])}')
|
||||
print(f' Payload types: {", ".join(p["payload_type"] for p in test_data["payloads"])}\n')
|
||||
|
||||
try:
|
||||
# Receive and process the message
|
||||
print('Receiving and processing message...')
|
||||
env = await smartreceive(
|
||||
mock_msg,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
|
||||
print('\n=== Received Envelope ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate received data
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Expected data
|
||||
expected_data = [
|
||||
('image', image_data, 'image'),
|
||||
('audio', audio_data, 'audio'),
|
||||
('video', video_data, 'video'),
|
||||
('binary', generic_binary, 'binary')
|
||||
]
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
expected = expected_data[i]
|
||||
|
||||
if payload[0] != expected[0]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected[0]}', got '{payload[0]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload[2] != expected[2]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected[2]}', got '{payload[2]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
# Verify binary data integrity
|
||||
received_data = payload[1]
|
||||
if not isinstance(received_data, (bytes, bytearray)):
|
||||
print(f'❌ Payload {i + 1}: Expected bytes/bytearray, got {type(received_data)}')
|
||||
passed = False
|
||||
elif received_data != expected[1]:
|
||||
print(f'❌ Payload {i + 1}: Data mismatch')
|
||||
print(f' Expected: {expected[1]}')
|
||||
print(f' Got: {received_data}')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data correctly deserialized')
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
183
test/test_py_binary_sender.py
Normal file
183
test/test_py_binary_sender.py
Normal file
@@ -0,0 +1,183 @@
|
||||
"""
|
||||
Python Binary Sender Test
|
||||
Tests the smartsend function with binary/image/audio/video/table payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
import base64
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_SUBJECT = '/test/binary'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Binary Sender Test ===\n')
|
||||
|
||||
correlation_id = 'py-binary-test-' + str(asyncio.get_event_loop().time() * 1000000)
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}\n')
|
||||
|
||||
# Test data - binary data for different types
|
||||
image_data = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
|
||||
audio_data = bytes([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]) # FLAC header
|
||||
video_data = bytes([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]) # MP4 header
|
||||
generic_binary = bytes([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE])
|
||||
|
||||
# Test table data
|
||||
try:
|
||||
import pandas as pd
|
||||
table_data = pd.DataFrame({
|
||||
'id': [1, 2, 3, 4, 5],
|
||||
'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
|
||||
'value': [10.5, 20.3, 30.1, 40.9, 50.7]
|
||||
})
|
||||
table_available = True
|
||||
except ImportError:
|
||||
table_available = False
|
||||
table_data = None
|
||||
|
||||
test_data = [
|
||||
('image', image_data, 'image'),
|
||||
('audio', audio_data, 'audio'),
|
||||
('video', video_data, 'video'),
|
||||
('binary', generic_binary, 'binary')
|
||||
]
|
||||
|
||||
if table_available:
|
||||
test_data.append(('table', table_data, 'table'))
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending binary payloads...')
|
||||
env, env_json_str = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='py-binary-test',
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
expected_count = 5 if table_available else 4
|
||||
if len(env['payloads']) != expected_count:
|
||||
print(f'❌ Expected {expected_count} payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Test each payload
|
||||
expected_datanames = ['image', 'audio', 'video', 'binary']
|
||||
expected_types = ['image', 'audio', 'video', 'binary']
|
||||
expected_data = [image_data, audio_data, video_data, generic_binary]
|
||||
|
||||
if table_available:
|
||||
expected_datanames.append('table')
|
||||
expected_types.append('table')
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
|
||||
if payload['dataname'] != expected_datanames[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload['payload_type'] != expected_types[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct encoding')
|
||||
|
||||
# Decode and verify the data
|
||||
decoded_data = base64.b64decode(payload['data'])
|
||||
|
||||
if i < len(expected_data):
|
||||
original_data = expected_data[i]
|
||||
if decoded_data != original_data:
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
else:
|
||||
# Table payload - just verify it's present
|
||||
print(f'✅ Payload {i + 1}: Table data present (size: {payload["size"]} bytes)')
|
||||
|
||||
print(f' Size: {payload["size"]} bytes\n')
|
||||
|
||||
# Test with larger binary data
|
||||
print('=== Large Binary Data Test ===')
|
||||
large_data = bytes([0xFF] * 10000) # 10KB of binary data
|
||||
large_test_data = [
|
||||
('large_binary', large_data, 'binary')
|
||||
]
|
||||
|
||||
large_env, _ = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
large_test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='large-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if len(large_env['payloads']) == 1 and large_env['payloads'][0]['size'] == 10000:
|
||||
print('✅ Large binary data handled correctly')
|
||||
else:
|
||||
print('❌ Large binary data handling failed')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
220
test/test_py_dictionary_receiver.py
Normal file
220
test/test_py_dictionary_receiver.py
Normal file
@@ -0,0 +1,220 @@
|
||||
"""
|
||||
Python Dictionary Receiver Test
|
||||
Tests the smartreceive function with dictionary payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Dictionary Receiver Test ===\n')
|
||||
|
||||
# Create a mock NATS message with dictionary payloads
|
||||
import base64
|
||||
|
||||
simple_dict = {'key1': 'value1', 'key2': 'value2'}
|
||||
nested_dict = {'outer': {'inner': 'value', 'number': 42}}
|
||||
array_dict = {'items': [1, 2, 3, 'four', 'five']}
|
||||
mixed_dict = {'string': 'text', 'number': 123, 'boolean': True, 'null_val': None}
|
||||
|
||||
test_data = {
|
||||
'correlation_id': 'py-receiver-dict-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'timestamp': asyncio.get_event_loop().time().isoformat(),
|
||||
'send_to': '/test/dictionary',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'py-dict-test',
|
||||
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'receiver_name': 'py-receiver',
|
||||
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': 'payload-1',
|
||||
'dataname': 'simple_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(simple_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(simple_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(simple_dict).encode('utf8'))}
|
||||
},
|
||||
{
|
||||
'id': 'payload-2',
|
||||
'dataname': 'nested_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(nested_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(nested_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(nested_dict).encode('utf8'))}
|
||||
},
|
||||
{
|
||||
'id': 'payload-3',
|
||||
'dataname': 'array_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(array_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(array_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(array_dict).encode('utf8'))}
|
||||
},
|
||||
{
|
||||
'id': 'payload-4',
|
||||
'dataname': 'mixed_dict',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps(mixed_dict).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps(mixed_dict).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps(mixed_dict).encode('utf8'))}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_msg = {
|
||||
'payload': json.dumps(test_data)
|
||||
}
|
||||
|
||||
print('Mock Message Created:')
|
||||
print(f' Correlation ID: {test_data["correlation_id"]}')
|
||||
print(f' Payloads: {len(test_data["payloads"])}')
|
||||
print(f' Payload types: {", ".join(p["payload_type"] for p in test_data["payloads"])}\n')
|
||||
|
||||
try:
|
||||
# Receive and process the message
|
||||
print('Receiving and processing message...')
|
||||
env = await smartreceive(
|
||||
mock_msg,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
|
||||
print('\n=== Received Envelope ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate received data
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Expected data
|
||||
expected_data = [
|
||||
('simple_dict', simple_dict, 'dictionary'),
|
||||
('nested_dict', nested_dict, 'dictionary'),
|
||||
('array_dict', array_dict, 'dictionary'),
|
||||
('mixed_dict', mixed_dict, 'dictionary')
|
||||
]
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
expected = expected_data[i]
|
||||
|
||||
if payload[0] != expected[0]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected[0]}', got '{payload[0]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload[2] != expected[2]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected[2]}', got '{payload[2]}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
data_match = json.dumps(payload[1], sort_keys=True) == json.dumps(expected[1], sort_keys=True)
|
||||
if not data_match:
|
||||
print(f'❌ Payload {i + 1}: Data mismatch')
|
||||
print(f' Expected: {json.dumps(expected[1], sort_keys=True)}')
|
||||
print(f' Got: {json.dumps(payload[1], sort_keys=True)}')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data correctly deserialized')
|
||||
|
||||
# Test round-trip with receive
|
||||
print('\n=== Round-trip Test ===')
|
||||
round_trip_data = {
|
||||
'correlation_id': 'roundtrip-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'timestamp': asyncio.get_event_loop().time().isoformat(),
|
||||
'send_to': '/test/dictionary',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'py-test',
|
||||
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'receiver_name': 'py-receiver',
|
||||
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': 'payload-rt',
|
||||
'dataname': 'roundtrip',
|
||||
'payload_type': 'dictionary',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8')),
|
||||
'data': base64.b64encode(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8'))}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_rt_msg = {'payload': json.dumps(round_trip_data)}
|
||||
rt_env = await smartreceive(mock_rt_msg)
|
||||
|
||||
if rt_env['payloads'][0][0] == 'roundtrip' and rt_env['payloads'][0][2] == 'dictionary':
|
||||
print('✅ Round-trip test successful')
|
||||
else:
|
||||
print('❌ Round-trip test failed')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
172
test/test_py_dictionary_sender.py
Normal file
172
test/test_py_dictionary_sender.py
Normal file
@@ -0,0 +1,172 @@
|
||||
"""
|
||||
Python Dictionary Sender Test
|
||||
Tests the smartsend function with dictionary payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_SUBJECT = '/test/dictionary'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Dictionary Sender Test ===\n')
|
||||
|
||||
correlation_id = 'py-dict-test-' + str(asyncio.get_event_loop().time() * 1000000)
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}\n')
|
||||
|
||||
# Test data - various dictionary structures
|
||||
test_data = [
|
||||
('simple_dict', {'key1': 'value1', 'key2': 'value2'}, 'dictionary'),
|
||||
('nested_dict', {'outer': {'inner': 'value', 'number': 42}}, 'dictionary'),
|
||||
('array_dict', {'items': [1, 2, 3, 'four', 'five']}, 'dictionary'),
|
||||
('mixed_dict', {'string': 'text', 'number': 123, 'boolean': True, 'null_val': None}, 'dictionary')
|
||||
]
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending dictionary payloads...')
|
||||
env, env_json_str = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='py-dict-test',
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if len(env['payloads']) != 4:
|
||||
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Test each payload
|
||||
expected_datanames = ['simple_dict', 'nested_dict', 'array_dict', 'mixed_dict']
|
||||
expected_types = ['dictionary', 'dictionary', 'dictionary', 'dictionary']
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
|
||||
if payload['dataname'] != expected_datanames[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload['payload_type'] != expected_types[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct encoding')
|
||||
|
||||
# Decode and verify the data
|
||||
import base64
|
||||
decoded_data = json.loads(base64.b64decode(payload['data']).decode('utf8'))
|
||||
original_data = test_data[i][1]
|
||||
|
||||
# Normalize for comparison (None vs null, True vs true, etc.)
|
||||
if json.dumps(decoded_data, sort_keys=True) != json.dumps(original_data, sort_keys=True):
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
print(f' Expected: {json.dumps(original_data)}')
|
||||
print(f' Got: {json.dumps(decoded_data)}')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
|
||||
print(f' Size: {payload["size"]} bytes\n')
|
||||
|
||||
# Test round-trip serialization
|
||||
print('=== Round-trip Serialization Test ===')
|
||||
round_trip_data = [
|
||||
('roundtrip', {'test': 'data', 'numbers': [1, 2, 3], 'nested': {'a': 1, 'b': 2}}, 'dictionary')
|
||||
]
|
||||
|
||||
rt_env, _ = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
round_trip_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='roundtrip-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
rt_payload = rt_env['payloads'][0]
|
||||
rt_decoded = json.loads(base64.b64decode(rt_payload['data']).decode('utf8'))
|
||||
|
||||
if json.dumps(rt_decoded, sort_keys=True) == json.dumps(round_trip_data[0][1], sort_keys=True):
|
||||
print('✅ Round-trip serialization successful')
|
||||
else:
|
||||
print('❌ Round-trip serialization failed')
|
||||
passed = False
|
||||
|
||||
# Test JSON string output
|
||||
print('\n=== JSON String Output Test ===')
|
||||
try:
|
||||
parsed = json.loads(env_json_str)
|
||||
if parsed['correlation_id'] == env['correlation_id'] and \
|
||||
len(parsed['payloads']) == len(env['payloads']):
|
||||
print('✅ JSON string is valid and matches envelope')
|
||||
else:
|
||||
print('❌ JSON string does not match envelope')
|
||||
passed = False
|
||||
except json.JSONDecodeError as e:
|
||||
print(f'❌ JSON string is invalid: {e}')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
199
test/test_py_mix_payloads_sender.py
Normal file
199
test/test_py_mix_payloads_sender.py
Normal file
@@ -0,0 +1,199 @@
|
||||
"""
|
||||
Python Mix Payloads Sender Test
|
||||
Tests the smartsend function with mixed payload types
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
import base64
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_SUBJECT = '/test/mix'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Mix Payloads Sender Test ===\n')
|
||||
|
||||
correlation_id = 'py-mix-test-' + str(asyncio.get_event_loop().time() * 1000000)
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}\n')
|
||||
|
||||
# Test data - mixed payload types
|
||||
text_data = 'Hello, NATSBridge!'
|
||||
dict_data = {'key1': 'value1', 'key2': 42, 'nested': {'a': 1, 'b': 2}}
|
||||
image_data = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
|
||||
|
||||
# Table data
|
||||
try:
|
||||
import pandas as pd
|
||||
table_data = pd.DataFrame({
|
||||
'id': [1, 2, 3],
|
||||
'name': ['Alice', 'Bob', 'Charlie'],
|
||||
'age': [30, 25, 35]
|
||||
})
|
||||
table_available = True
|
||||
except ImportError:
|
||||
table_available = False
|
||||
table_data = None
|
||||
|
||||
test_data = [
|
||||
('message', text_data, 'text'),
|
||||
('config', dict_data, 'dictionary'),
|
||||
('image', image_data, 'image')
|
||||
]
|
||||
|
||||
if table_available:
|
||||
test_data.append(('users', table_data, 'table'))
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending mixed payloads...')
|
||||
env, env_json_str = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='py-mix-test',
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
expected_count = 4 if table_available else 3
|
||||
if len(env['payloads']) != expected_count:
|
||||
print(f'❌ Expected {expected_count} payloads, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
# Test each payload
|
||||
expected_datanames = ['message', 'config', 'image']
|
||||
expected_types = ['text', 'dictionary', 'image']
|
||||
expected_data = [text_data, dict_data, image_data]
|
||||
|
||||
if table_available:
|
||||
expected_datanames.append('users')
|
||||
expected_types.append('table')
|
||||
|
||||
for i in range(len(env['payloads'])):
|
||||
payload = env['payloads'][i]
|
||||
|
||||
if payload['dataname'] != expected_datanames[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct dataname')
|
||||
|
||||
if payload['payload_type'] != expected_types[i]:
|
||||
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Correct encoding')
|
||||
|
||||
# Verify data integrity based on type
|
||||
decoded_data = base64.b64decode(payload['data'])
|
||||
|
||||
if expected_types[i] == 'text':
|
||||
decoded_text = decoded_data.decode('utf8')
|
||||
if decoded_text != expected_data[i]:
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
elif expected_types[i] == 'dictionary':
|
||||
import json
|
||||
decoded_dict = json.loads(decoded_data.decode('utf8'))
|
||||
if json.dumps(decoded_dict, sort_keys=True) != json.dumps(expected_data[i], sort_keys=True):
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
elif expected_types[i] == 'image':
|
||||
if decoded_data != expected_data[i]:
|
||||
print(f'❌ Payload {i + 1}: Data integrity mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f'✅ Payload {i + 1}: Data integrity verified')
|
||||
elif expected_types[i] == 'table':
|
||||
if len(decoded_data) > 0:
|
||||
print(f'✅ Payload {i + 1}: Arrow IPC data present ({len(decoded_data)} bytes)')
|
||||
else:
|
||||
print(f'❌ Payload {i + 1}: Arrow IPC data is empty')
|
||||
passed = False
|
||||
|
||||
print(f' Size: {payload["size"]} bytes\n')
|
||||
|
||||
# Test with chat-like payload (text + image + audio)
|
||||
print('=== Chat-like Payload Test ===')
|
||||
chat_data = [
|
||||
('text', 'Hello!', 'text'),
|
||||
('image', bytes([0xFF, 0xD8, 0xFF, 0xE0]), 'image'),
|
||||
('audio', bytes([0x46, 0x4C, 0x41, 0x43]), 'audio')
|
||||
]
|
||||
|
||||
chat_env, _ = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
chat_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='chat-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if len(chat_env['payloads']) == 3:
|
||||
print('✅ Chat-like payloads handled correctly')
|
||||
else:
|
||||
print('❌ Chat-like payloads handling failed')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
167
test/test_py_table_sender.py
Normal file
167
test/test_py_table_sender.py
Normal file
@@ -0,0 +1,167 @@
|
||||
"""
|
||||
Python Table Sender Test
|
||||
Tests the smartsend function with table (Arrow IPC) payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_SUBJECT = '/test/table'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Table Sender Test ===\n')
|
||||
|
||||
correlation_id = 'py-table-test-' + str(asyncio.get_event_loop().time() * 1000000)
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}\n')
|
||||
|
||||
# Test data - pandas DataFrame
|
||||
try:
|
||||
import pandas as pd
|
||||
table_data = pd.DataFrame({
|
||||
'id': [1, 2, 3, 4, 5],
|
||||
'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
|
||||
'age': [30, 25, 35, 28, 32],
|
||||
'active': [True, False, True, True, False]
|
||||
})
|
||||
table_available = True
|
||||
except ImportError:
|
||||
print('❌ pandas not available - skipping table tests')
|
||||
sys.exit(0)
|
||||
|
||||
test_data = [
|
||||
('users_table', table_data, 'table')
|
||||
]
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending table payload...')
|
||||
env, env_json_str = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='py-table-test',
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if len(env['payloads']) != 1:
|
||||
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
payload = env['payloads'][0]
|
||||
if payload['dataname'] != 'users_table':
|
||||
print(f"❌ Expected dataname 'users_table', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct dataname')
|
||||
|
||||
if payload['payload_type'] != 'table':
|
||||
print(f"❌ Expected payload_type 'table', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct payload_type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct encoding')
|
||||
|
||||
print(f'\nPayload size: {payload["size"]} bytes')
|
||||
|
||||
# Test with larger table
|
||||
print('\n=== Larger Table Test ===')
|
||||
large_table_data = pd.DataFrame({
|
||||
'id': range(100),
|
||||
'name': [f'User{i}' for i in range(100)],
|
||||
'age': [20 + (i % 50) for i in range(100)],
|
||||
'active': [i % 2 == 0 for i in range(100)]
|
||||
})
|
||||
|
||||
large_test_data = [
|
||||
('large_table', large_table_data, 'table')
|
||||
]
|
||||
|
||||
large_env, _ = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
large_test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='large-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if len(large_env['payloads']) == 1:
|
||||
print('✅ Large table handled correctly')
|
||||
print(f' Size: {large_env["payloads"][0]["size"]} bytes')
|
||||
else:
|
||||
print('❌ Large table handling failed')
|
||||
passed = False
|
||||
|
||||
# Test JSON string output
|
||||
print('\n=== JSON String Output Test ===')
|
||||
import json
|
||||
try:
|
||||
parsed = json.loads(env_json_str)
|
||||
if parsed['correlation_id'] == env['correlation_id'] and \
|
||||
len(parsed['payloads']) == len(env['payloads']):
|
||||
print('✅ JSON string is valid and matches envelope')
|
||||
else:
|
||||
print('❌ JSON string does not match envelope')
|
||||
passed = False
|
||||
except json.JSONDecodeError as e:
|
||||
print(f'❌ JSON string is invalid: {e}')
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
205
test/test_py_text_receiver.py
Normal file
205
test/test_py_text_receiver.py
Normal file
@@ -0,0 +1,205 @@
|
||||
"""
|
||||
Python Text Receiver Test
|
||||
Tests the smartreceive function with text payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Text Receiver Test ===\n')
|
||||
|
||||
# Create a mock NATS message with text payload
|
||||
test_text = 'Hello, NATSBridge! This is a test message.'
|
||||
import base64
|
||||
|
||||
test_data = {
|
||||
'correlation_id': 'py-receiver-test-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'timestamp': asyncio.get_event_loop().time().isoformat(),
|
||||
'send_to': '/test/text',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'py-text-test',
|
||||
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'receiver_name': 'py-receiver',
|
||||
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': 'payload-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'dataname': 'message',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': len(test_text.encode('utf8')),
|
||||
'data': base64.b64encode(test_text.encode('utf8')).decode('ascii'),
|
||||
'metadata': {'payload_bytes': len(test_text.encode('utf8'))}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_msg = {
|
||||
'payload': json.dumps(test_data)
|
||||
}
|
||||
|
||||
print('Mock Message Created:')
|
||||
print(f' Correlation ID: {test_data["correlation_id"]}')
|
||||
print(f' Payloads: {len(test_data["payloads"])}')
|
||||
print(f' Payload dataname: {test_data["payloads"][0]["dataname"]}')
|
||||
print(f' Payload type: {test_data["payloads"][0]["payload_type"]}')
|
||||
print(f' Transport: {test_data["payloads"][0]["transport"]}\n')
|
||||
|
||||
try:
|
||||
# Receive and process the message
|
||||
print('Receiving and processing message...')
|
||||
env = await smartreceive(
|
||||
mock_msg,
|
||||
max_retries=3,
|
||||
base_delay=100,
|
||||
max_delay=1000
|
||||
)
|
||||
|
||||
print('\n=== Received Envelope ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate received data
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if len(env['payloads']) != 1:
|
||||
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
payload = env['payloads'][0]
|
||||
if payload[0] != 'message':
|
||||
print(f"❌ Expected dataname 'message', got '{payload[0]}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct dataname')
|
||||
|
||||
if payload[2] != 'text':
|
||||
print(f"❌ Expected type 'text', got '{payload[2]}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct type')
|
||||
|
||||
if payload[1] != test_text:
|
||||
print('❌ Data mismatch')
|
||||
print(f' Expected: {test_text}')
|
||||
print(f' Got: {payload[1]}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Data correctly deserialized')
|
||||
|
||||
# Test with multiple text payloads
|
||||
print('\n=== Multiple Text Payloads Test ===')
|
||||
multi_test_data = {
|
||||
'correlation_id': 'multi-receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'timestamp': asyncio.get_event_loop().time().isoformat(),
|
||||
'send_to': '/test/text',
|
||||
'msg_purpose': 'test',
|
||||
'sender_name': 'py-text-test',
|
||||
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'receiver_name': 'py-receiver',
|
||||
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
|
||||
'reply_to': '',
|
||||
'reply_to_msg_id': '',
|
||||
'broker_url': TEST_BROKER_URL,
|
||||
'metadata': {},
|
||||
'payloads': [
|
||||
{
|
||||
'id': 'payload-1',
|
||||
'dataname': 'msg1',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': 16,
|
||||
'data': base64.b64encode(b'First message').decode('ascii'),
|
||||
'metadata': {'payload_bytes': 16}
|
||||
},
|
||||
{
|
||||
'id': 'payload-2',
|
||||
'dataname': 'msg2',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': 16,
|
||||
'data': base64.b64encode(b'Second message').decode('ascii'),
|
||||
'metadata': {'payload_bytes': 16}
|
||||
},
|
||||
{
|
||||
'id': 'payload-3',
|
||||
'dataname': 'msg3',
|
||||
'payload_type': 'text',
|
||||
'transport': 'direct',
|
||||
'encoding': 'base64',
|
||||
'size': 16,
|
||||
'data': base64.b64encode(b'Third message').decode('ascii'),
|
||||
'metadata': {'payload_bytes': 16}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
mock_multi_msg = {'payload': json.dumps(multi_test_data)}
|
||||
multi_env = await smartreceive(mock_multi_msg)
|
||||
|
||||
if len(multi_env['payloads']) == 3:
|
||||
print('✅ Multiple payloads handled correctly')
|
||||
|
||||
# Verify each payload
|
||||
expected_messages = ['First message', 'Second message', 'Third message']
|
||||
for i in range(3):
|
||||
if multi_env['payloads'][i][1] == expected_messages[i]:
|
||||
print(f'✅ Payload {i + 1} correctly deserialized')
|
||||
else:
|
||||
print(f'❌ Payload {i + 1} mismatch')
|
||||
passed = False
|
||||
else:
|
||||
print(f"❌ Expected 3 payloads, got {len(multi_env['payloads'])}")
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
164
test/test_py_text_sender.py
Normal file
164
test/test_py_text_sender.py
Normal file
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
Python Text Sender Test
|
||||
Tests the smartsend function with text payloads
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from natbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
|
||||
|
||||
TEST_SUBJECT = '/test/text'
|
||||
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
|
||||
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
|
||||
|
||||
|
||||
async def run_test():
|
||||
print('=== Python Text Sender Test ===\n')
|
||||
|
||||
correlation_id = 'py-text-test-' + str(asyncio.get_event_loop().time() * 1000000)
|
||||
print(f'Correlation ID: {correlation_id}')
|
||||
print(f'Subject: {TEST_SUBJECT}')
|
||||
print(f'Broker URL: {TEST_BROKER_URL}\n')
|
||||
|
||||
# Test data
|
||||
text_data = 'Hello, NATSBridge! This is a test message.'
|
||||
test_data = [
|
||||
('message', text_data, 'text')
|
||||
]
|
||||
|
||||
try:
|
||||
# Send the message
|
||||
print('Sending text payload...')
|
||||
env, env_json_str = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id=correlation_id,
|
||||
msg_purpose='test',
|
||||
sender_name='py-text-test',
|
||||
is_publish=False # Don't actually publish for this test
|
||||
)
|
||||
|
||||
print('\n=== Envelope Created ===')
|
||||
print(f'Correlation ID: {env["correlation_id"]}')
|
||||
print(f'Message ID: {env["msg_id"]}')
|
||||
print(f'Timestamp: {env["timestamp"]}')
|
||||
print(f'Subject: {env["send_to"]}')
|
||||
print(f'Purpose: {env["msg_purpose"]}')
|
||||
print(f'Sender: {env["sender_name"]}')
|
||||
print(f'Payloads: {len(env["payloads"])}\n')
|
||||
|
||||
# Validate envelope structure
|
||||
print('=== Validation ===')
|
||||
passed = True
|
||||
|
||||
if not env.get('correlation_id'):
|
||||
print('❌ correlation_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ correlation_id present')
|
||||
|
||||
if not env.get('msg_id'):
|
||||
print('❌ msg_id is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ msg_id present')
|
||||
|
||||
if not env.get('timestamp'):
|
||||
print('❌ timestamp is missing')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ timestamp present')
|
||||
|
||||
if len(env['payloads']) != 1:
|
||||
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct number of payloads')
|
||||
|
||||
payload = env['payloads'][0]
|
||||
if payload['dataname'] != 'message':
|
||||
print(f"❌ Expected dataname 'message', got '{payload['dataname']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct dataname')
|
||||
|
||||
if payload['payload_type'] != 'text':
|
||||
print(f"❌ Expected payload_type 'text', got '{payload['payload_type']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct payload_type')
|
||||
|
||||
if payload['transport'] != 'direct':
|
||||
print(f"❌ Expected transport 'direct', got '{payload['transport']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct transport')
|
||||
|
||||
if payload['encoding'] != 'base64':
|
||||
print(f"❌ Expected encoding 'base64', got '{payload['encoding']}'")
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Correct encoding')
|
||||
|
||||
# Decode and verify the data
|
||||
import base64
|
||||
decoded_data = base64.b64decode(payload['data']).decode('utf8')
|
||||
if decoded_data != text_data:
|
||||
print('❌ Decoded data mismatch')
|
||||
print(f' Expected: {text_data}')
|
||||
print(f' Got: {decoded_data}')
|
||||
passed = False
|
||||
else:
|
||||
print('✅ Data integrity verified')
|
||||
|
||||
print(f"\nPayload size: {payload['size']} bytes")
|
||||
print(f'Base64 data length: {len(payload["data"])} chars')
|
||||
|
||||
# Test with multiple text payloads
|
||||
print('\n=== Multiple Text Payloads Test ===')
|
||||
multi_test_data = [
|
||||
('msg1', 'First message', 'text'),
|
||||
('msg2', 'Second message', 'text'),
|
||||
('msg3', 'Third message', 'text')
|
||||
]
|
||||
|
||||
multi_env, _ = await smartsend(
|
||||
TEST_SUBJECT,
|
||||
multi_test_data,
|
||||
broker_url=TEST_BROKER_URL,
|
||||
fileserver_url=TEST_FILESERVER_URL,
|
||||
correlation_id='multi-test-' + correlation_id,
|
||||
is_publish=False
|
||||
)
|
||||
|
||||
if len(multi_env['payloads']) == 3:
|
||||
print('✅ Multiple payloads handled correctly')
|
||||
else:
|
||||
print(f"❌ Expected 3 payloads, got {len(multi_env['payloads'])}")
|
||||
passed = False
|
||||
|
||||
# Final result
|
||||
print('\n=== Test Result ===')
|
||||
if passed:
|
||||
print('✅ ALL TESTS PASSED')
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('❌ SOME TESTS FAILED')
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f'❌ Test failed with error: {e}')
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(run_test())
|
||||
Binary file not shown.
Binary file not shown.
Reference in New Issue
Block a user