104 Commits

Author SHA1 Message Date
1b86a9252d update 2026-03-06 08:15:34 +07:00
e9fd148235 update 2026-03-06 07:43:26 +07:00
34ea1ed8ec update 2026-03-06 07:42:15 +07:00
aa92fb6d0d update 2026-03-06 07:27:07 +07:00
fbbea7b42b update 2026-03-06 07:19:03 +07:00
b2859710cd update 2026-03-06 07:18:08 +07:00
bc0ce7159c update 2026-03-06 07:14:40 +07:00
4614f99358 update 2026-03-05 20:17:36 +07:00
1ecc55f8aa update 2026-03-05 17:54:36 +07:00
ae0f24ccb2 update 2026-03-05 17:32:20 +07:00
060c68cd05 update 2026-03-05 11:00:46 +07:00
e85eba4cea update 2026-03-05 07:28:28 +07:00
206467e1fa update 2026-03-05 07:23:24 +07:00
a98394b9b9 update 2026-03-05 07:15:33 +07:00
c448811aa9 update 2026-03-05 06:35:48 +07:00
c3225a90c7 update 2026-03-04 20:50:12 +07:00
89acf780bf update 2026-03-04 20:42:15 +07:00
e5f4793370 fix output annotation 2026-03-04 11:58:19 +07:00
95fe697501 update diagram 2026-03-04 10:23:40 +07:00
ee2d2c7238 minor fix 2026-03-04 10:02:31 +07:00
ton
1dfa277279 Merge pull request 'split_smartsend' (#8) from split_smartsend into main
Reviewed-on: #8
2026-02-26 09:52:56 +00:00
78a8952383 update 2026-02-26 16:51:39 +07:00
fcc50847e4 update 2026-02-25 20:29:08 +07:00
f8d93991f5 update 2026-02-25 20:27:51 +07:00
bee9f783d9 update 2026-02-25 17:38:50 +07:00
3e1c8d563e update 2026-02-25 15:20:29 +07:00
1299febcdc update 2026-02-25 14:25:08 +07:00
be94c62760 update 2026-02-25 12:24:02 +07:00
6a862ef243 update 2026-02-25 12:09:00 +07:00
ae2de5fc62 update 2026-02-25 10:33:30 +07:00
df0bbc7327 update 2026-02-25 09:58:10 +07:00
d94761c866 update 2026-02-25 09:44:08 +07:00
f8235e1a59 update 2026-02-25 08:54:04 +07:00
647cadf497 update 2026-02-25 08:33:32 +07:00
8c793a81b6 update 2026-02-25 08:02:03 +07:00
6a42ba7e43 update 2026-02-25 07:29:42 +07:00
14b3790251 update 2026-02-25 06:23:24 +07:00
61d81bed62 update 2026-02-25 06:04:40 +07:00
1a10bc1a5f update 2026-02-25 05:32:59 +07:00
7f68d08134 update 2026-02-24 21:40:33 +07:00
ab20cd896f update 2026-02-24 21:18:19 +07:00
5a9e93d6e7 update 2026-02-24 20:38:45 +07:00
b51641dc7e update 2026-02-24 20:09:10 +07:00
45f1257896 update 2026-02-24 18:50:28 +07:00
3e2b8b1e3a update 2026-02-24 18:19:03 +07:00
90d81617ef update 2026-02-24 17:58:59 +07:00
64c62e616b update 2026-02-23 22:06:57 +07:00
2c340e37c7 update 2026-02-23 22:00:06 +07:00
7853e94d2e update 2026-02-23 21:54:50 +07:00
99bf57b154 update 2026-02-23 21:43:09 +07:00
0fa6eaf95b update 2026-02-23 21:37:50 +07:00
76f42be740 update 2026-02-23 21:32:22 +07:00
d99dc41be9 update 2026-02-23 21:09:36 +07:00
263508b8f7 update 2026-02-23 20:50:41 +07:00
0c2cca30ed update 2026-02-23 20:34:08 +07:00
46fdf668c6 update 2026-02-23 19:18:12 +07:00
f8a92a45a0 update README.md 2026-02-23 09:39:24 +07:00
cec70e6036 update 2026-02-23 08:11:03 +07:00
f9e08ba628 add Plik fileserver 2026-02-23 07:58:18 +07:00
c12a078149 update README.md 2026-02-23 07:55:10 +07:00
dedd803dc3 fix README.md 2026-02-23 07:24:54 +07:00
e8e927a491 move README.md 2026-02-23 07:17:31 +07:00
ton
d950bbac23 Merge pull request 'smartreceive_return_envelope' (#7) from smartreceive_return_envelope into main
Reviewed-on: #7
2026-02-23 00:11:09 +00:00
fc8da2ebf5 update 2026-02-23 07:08:17 +07:00
f6e50c405f update 2026-02-23 07:06:53 +07:00
ton
c06f508e8f Merge pull request 'smartreceive_return_envelope' (#6) from smartreceive_return_envelope into main
Reviewed-on: #6
2026-02-22 23:59:13 +00:00
97bf1e47f4 update 2026-02-23 06:58:16 +07:00
ef47fddd56 update 2026-02-23 06:28:41 +07:00
896dd84d2a update 2026-02-22 22:19:47 +07:00
def75d8f86 update 2026-02-22 21:55:18 +07:00
69f2173f75 update 2026-02-22 20:52:13 +07:00
075d355c58 update 2026-02-22 20:43:28 +07:00
ton
0de9725ba8 Merge pull request 'add Base64 in project.toml' (#5) from fix_precompile_issue into main
Reviewed-on: #5
2026-02-22 07:16:15 +00:00
6dcccc903f add Base64 in project.toml 2026-02-22 14:15:24 +07:00
ton
507b4951b4 Merge pull request 'add julia project file' (#4) from add_julia_project_file into main
Reviewed-on: #4
2026-02-22 07:02:07 +00:00
a064be0e5c update 2026-02-22 13:54:36 +07:00
ton
8a35f1d4dc Merge pull request 'add micropython support' (#3) from add_micropython into main
Reviewed-on: #3
2026-02-22 06:28:25 +00:00
9e5ee61785 update 2026-02-22 13:26:44 +07:00
ton
4b5b5d6ed8 Merge pull request 'add_mix_content_capability' (#2) from add_mix_content_capability into main
Reviewed-on: #2
2026-02-19 12:27:59 +00:00
3f45052193 update 2026-02-19 19:25:34 +07:00
7dc7ab67e4 update 2026-02-19 18:41:39 +07:00
e7c5e5f77f update 2026-02-19 16:39:31 +07:00
4e32a958ea update 2026-02-19 16:22:01 +07:00
a260def38d update 2026-02-19 15:58:22 +07:00
782a935d3d update 2026-02-19 14:30:01 +07:00
3fbdabc874 update 2026-02-19 12:29:04 +07:00
7386f8ed0b update 2026-02-19 12:27:15 +07:00
51e494c48b update 2026-02-19 11:23:15 +07:00
9ea9d55eee update 2026-02-19 07:33:35 +07:00
8c106464fd add test 2026-02-19 07:08:57 +07:00
7433c147c9 update 2026-02-18 20:55:18 +07:00
9c4a9ea1e5 update 2026-02-18 20:26:25 +07:00
82804c6803 update 2026-02-18 19:43:57 +07:00
483caab54c update 2026-02-18 19:40:41 +07:00
a9821b1ae6 update 2026-02-18 19:31:00 +07:00
0744642985 update 2026-02-17 21:19:08 +07:00
1d5c6f3348 update 2026-02-17 21:04:32 +07:00
ad87934abf update 2026-02-17 18:02:43 +07:00
6b49fa68c0 update 2026-02-15 21:02:22 +07:00
f0df169689 architecture.md rev1 2026-02-14 13:04:28 +07:00
d9fd7a61bb update 2026-02-13 21:38:20 +07:00
897f717da5 update 2026-02-13 21:26:30 +07:00
51e1a065ad update new struct 2026-02-13 21:23:38 +07:00
e7f50e899d add new struct 2026-02-13 20:16:22 +07:00
57 changed files with 15011 additions and 1882 deletions

View File

@@ -12,3 +12,54 @@ Role: Principal Systems Architect & Lead Software Engineer.Objective: Implement
Create a walkthrough for Julia service-A service sending a mix-content chat message to Julia service-B. the chat message must includes
I updated the following:
- NATSBridge.jl. Essentially I add NATS_connection keyword and new publish_message function to support the keyword.
Use them and ONLY them as ground truth.
Then update the following files accordingly:
- architecture.md
- implementation.md
All API should be semantically consistent and naming should be consistent across the board.
Task: Update NATSBridge.js to reflect recent changes in NATSBridge.jl and docs
Context: NATSBridge.jl and docs has been updated.
Requirements:
Source of Truth: Treat the updated NATSBridge.jl and docs as the definitive source.
API Consistency: Ensure the Main Package API (e.g., smartsend(), publish_message()) uses consistent naming across all three supported languages.
Ecosystem Variance: Low-level native functions (e.g., NATS.connect(), JSON.read()) should follow the conventions of the specific language ecosystem and do not require cross-language consistency.
I'm expanding this Julia package (NATSBridge) into a cross-platform project by adding a JavaScript and Python/MicroPython implementation. To ensure accuracy, the Julia src directory will serve as the ground truth, as the documentation may be outdated.
My goal is to maintain interface parity at the high-level API for a consistent user experience, while ensuring the low-level implementation adheres strictly to the idiomatic conventions of each respective language (e.g., multiple dispatch in Julia vs. asynchronous, prototype, or class-based patterns in JS and Python/MicroPython)
Now, help me do the following:
1) check architecture.md for any mistake.

View File

@@ -1,8 +1,8 @@
# This file is machine-generated - editing it directly is not advised
julia_version = "1.12.4"
julia_version = "1.12.5"
manifest_format = "2.0"
project_hash = "be1e3c2d8b7f4f0ee7375c94aaf704ce73ba57b9"
project_hash = "b632f853bcf5355f5c53ad3efa7a19f70444dc6c"
[[deps.AliasTables]]
deps = ["PtrArrays", "Random"]
@@ -436,6 +436,12 @@ git-tree-sha1 = "d9d9a189fb9155a460e6b5e8966bf6a66737abf8"
uuid = "55e73f9c-eeeb-467f-b4cc-a633fde63d2a"
version = "0.1.0"
[[deps.NATSBridge]]
deps = ["Arrow", "DataFrames", "Dates", "GeneralUtils", "HTTP", "JSON", "NATS", "PrettyPrinting", "Revise", "UUIDs"]
path = "."
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
version = "0.4.1"
[[deps.NanoDates]]
deps = ["Dates", "Parsers"]
git-tree-sha1 = "850a0557ae5934f6e67ac0dc5ca13d0328422d1f"

View File

@@ -1,194 +0,0 @@
### API
Plik server expose a REST-full API to manage uploads and get files :
Get and create upload :
- **POST** /upload
- Params (json object in request body) :
- oneshot (bool)
- stream (bool)
- removable (bool)
- ttl (int)
- login (string)
- password (string)
- files (see below)
- Return :
JSON formatted upload object.
Important fields :
- id (required to upload files)
- uploadToken (required to upload/remove files)
- files (see below)
For stream mode you need to know the file id before the upload starts as it will block.
File size and/or file type also need to be known before the upload starts as they have to be printed
in HTTP response headers.
To get the file ids pass a "files" json object with each file you are about to upload.
Fill the reference field with an arbitrary string to avoid matching file ids using the fileName field.
This is also used to notify of MISSING files when file upload is not yet finished or has failed.
```
"files" : [
{
"fileName": "file.txt",
"fileSize": 12345,
"fileType": "text/plain",
"reference": "0"
},...
]
```
- **GET** /upload/:uploadid:
- Get upload metadata (files list, upload date, ttl,...)
Upload file :
- **POST** /$mode/:uploadid:/:fileid:/:filename:
- Request body must be a multipart request with a part named "file" containing file data.
- **POST** /file/:uploadid:
- Same as above without passing file id, won't work for stream mode.
- **POST** /:
- Quick mode, automatically create an upload with default parameters and add the file to it.
Get file :
- **HEAD** /$mode/:uploadid:/:fileid:/:filename:
- Returns only HTTP headers. Useful to know Content-Type and Content-Length without downloading the file. Especially if upload has OneShot option enabled.
- **GET** /$mode/:uploadid:/:fileid:/:filename:
- Download file. Filename **MUST** match. A browser, might try to display the file if it's a jpeg for example. You may try to force download with ?dl=1 in url.
- **GET** /archive/:uploadid:/:filename:
- Download uploaded files in a zip archive. :filename: must end with .zip
Remove file :
- **DELETE** /$mode/:uploadid:/:fileid:/:filename:
- Delete file. Upload **MUST** have "removable" option enabled.
Show server details :
- **GET** /version
- Show plik server version, and some build information (build host, date, git revision,...)
- **GET** /config
- Show plik server configuration (ttl values, max file size, ...)
- **GET** /stats
- Get server statistics ( upload/file count, user count, total size used )
- Admin only
User authentication :
-
Plik can authenticate users using Google and/or OVH third-party API.
The /auth API is designed for the Plik web application nevertheless if you want to automatize it be sure to provide a valid
Referrer HTTP header and forward all session cookies.
Plik session cookies have the "secure" flag set, so they can only be transmitted over secure HTTPS connections.
To avoid CSRF attacks the value of the plik-xsrf cookie MUST be copied in the X-XSRFToken HTTP header of each
authenticated request.
Once authenticated a user can generate upload tokens. Those tokens can be used in the X-PlikToken HTTP header used to link
an upload to the user account. It can be put in the ~/.plikrc file of the Plik command line client.
- **Local** :
- You'll need to create users using the server command line
- **Google** :
- You'll need to create a new application in the [Google Developper Console](https://console.developers.google.com)
- You'll be handed a Google API ClientID and a Google API ClientSecret that you'll need to put in the plikd.cfg file
- Do not forget to whitelist valid origin and redirect url ( https://yourdomain/auth/google/callback ) for your domain
- **OVH** :
- You'll need to create a new application in the OVH API : https://eu.api.ovh.com/createApp/
- You'll be handed an OVH application key and an OVH application secret key that you'll need to put in the plikd.cfg file
- **GET** /auth/google/login
- Get Google user consent URL. User have to visit this URL to authenticate
- **GET** /auth/google/callback
- Callback of the user consent dialog
- The user will be redirected back to the web application with a Plik session cookie at the end of this call
- **GET** /auth/ovh/login
- Get OVH user consent URL. User have to visit this URL to authenticate
- The response will contain a temporary session cookie to forward the API endpoint and OVH consumer key to the callback
- **GET** /auth/ovh/callback
- Callback of the user consent dialog.
- The user will be redirected back to the web application with a Plik session cookie at the end of this call
- **POST** /auth/local/login
- Params :
- login : user login
- password : user password
- **GET** /auth/logout
- Invalidate Plik session cookies
- **GET** /me
- Return basic user info ( ID, name, email ) and tokens
- **DELETE** /me
- Remove user account.
- **GET** /me/token
- List user tokens
- This call use pagination
- **POST** /me/token
- Create a new upload token
- A comment can be passed in the json body
- **DELETE** /me/token/{token}
- Revoke an upload token
- **GET** /me/uploads
- List user uploads
- Params :
- token : filter by token
- This call use pagination
- **DELETE** /me/uploads
- Remove all uploads linked to a user account
- Params :
- token : filter by token
- **GET** /me/stats
- Get user statistics ( upload/file count, total size used )
- **GET** /users
- List all users
- This call use pagination
- Admin only
QRCode :
- **GET** /qrcode
- Generate a QRCode image from an url
- Params :
- url : The url you want to store in the QRCode
- size : The size of the generated image in pixels (default: 250, max: 1000)
$mode can be "file" or "stream" depending if stream mode is enabled. See FAQ for more details.
Examples :
```sh
Create an upload (in the json response, you'll have upload id and upload token)
$ curl -X POST http://127.0.0.1:8080/upload
Create a OneShot upload
$ curl -X POST -d '{ "OneShot" : true }' http://127.0.0.1:8080/upload
Upload a file to upload
$ curl -X POST --header "X-UploadToken: M9PJftiApG1Kqr81gN3Fq1HJItPENMhl" -F "file=@test.txt" http://127.0.0.1:8080/file/IsrIPIsDskFpN12E
Get headers
$ curl -I http://127.0.0.1:8080/file/IsrIPIsDskFpN12E/sFjIeokH23M35tN4/test.txt
HTTP/1.1 200 OK
Content-Disposition: filename=test.txt
Content-Length: 3486
Content-Type: text/plain; charset=utf-8
Date: Fri, 15 May 2015 09:16:20 GMT
```

View File

@@ -1,8 +1,21 @@
name = "NATSBridge"
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
version = "0.4.5"
authors = ["narawat <narawat@gmail.com>"]
[deps]
Arrow = "69666777-d1a9-59fb-9406-91d4454c9d45"
Base64 = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
GeneralUtils = "c6c72f09-b708-4ac8-ac7c-2084d70108fe"
HTTP = "cd3eb016-35fb-5094-929b-558a96fad6f3"
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
NATS = "55e73f9c-eeeb-467f-b4cc-a633fde63d2a"
PrettyPrinting = "54e16d92-306c-5ea0-a30b-337be88ac337"
Revise = "295af30f-e4ad-537b-8983-00126c2a3abe"
UUIDs = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"
[compat]
Base64 = "1.11.0"
JSON = "1.4.0"

939
README.md Normal file
View File

@@ -0,0 +1,939 @@
# NATSBridge - Cross-Platform Bi-Directional Data Bridge
A high-performance, bi-directional data bridge for **Julia, JavaScript, Python, and MicroPython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![NATS](https://img.shields.io/badge/NATS-Enabled-green.svg)](https://nats.io)
---
## Table of Contents
- [Overview](#overview)
- [Cross-Platform Support](#cross-platform-support)
- [Features](#features)
- [Architecture](#architecture)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [API Reference](#api-reference)
- [Payload Types](#payload-types)
- [Transport Strategies](#transport-strategies)
- [Cross-Platform Examples](#cross-platform-examples)
- [Testing](#testing)
- [License](#license)
---
## Overview
NATSBridge enables seamless communication across multiple platforms through NATS, with intelligent transport selection based on payload size:
| Transport | Payload Size | Method |
|-----------|--------------|--------|
| **Direct** | < 1MB | Sent directly via NATS (Base64 encoded) |
| **Link** | >= 1MB | Uploaded to HTTP file server, URL sent via NATS |
### Use Cases
- **Chat Applications**: Text, images, audio, video in a single message
- **File Transfer**: Efficient transfer of large files using claim-check pattern
- **IoT/Embedded**: Sensor data, telemetry, and analytics pipelines (MicroPython)
- **Cross-Platform Communication**: Interoperability between Julia, JavaScript, Python, and MicroPython systems
---
## Cross-Platform Support
| Platform | Implementation | Features |
|----------|----------------|----------|
| **Julia** | [`src/NATSBridge.jl`](src/NATSBridge.jl) | Full feature set, Arrow IPC, multiple dispatch |
| **JavaScript** | [`src/natbridge.js`](src/natbridge.js) | Node.js & browser, async/await |
| **Python** | [`src/natbridge.py`](src/natbridge.py) | Desktop Python, asyncio, type hints |
| **MicroPython** | [`src/natbridge_mpy.py`](src/natbridge_mpy.py) | Memory-constrained, synchronous API |
### Platform Comparison
| Feature | Julia | JavaScript | Python | MicroPython |
|---------|-------|------------|--------|-------------|
| Multiple Dispatch | ✅ Native | ❌ | ❌ | ❌ |
| Async/Await | ❌ | ✅ Native | ✅ Native | ⚠️ (uasyncio) |
| Type Safety | ✅ Strong | ⚠️ (TypeScript) | ✅ (Type hints) | ❌ |
| Memory Management | ✅ GC | ✅ GC | ✅ GC | ⚠️ (Manual) |
| Arrow IPC | ✅ Native | ✅ | ✅ | ❌ |
| Direct Transport | ✅ | ✅ | ✅ | ✅ |
| Link Transport | ✅ | ✅ | ✅ | ⚠️ (Limited) |
| Handler Functions | ✅ | ✅ | ✅ | ✅ |
| Cross-Platform API | ✅ | ✅ | ✅ | ✅ |
---
## Features
-**Cross-platform messaging** for Julia, JavaScript, Python, and MicroPython applications
-**Bi-directional messaging** with request-reply patterns
-**Multi-payload support** - send multiple payloads with different types in one message
-**Automatic transport selection** - direct vs link based on payload size
-**Claim-Check pattern** for payloads > 1MB
-**Apache Arrow IPC** support for tabular data (zero-copy reading)
-**Exponential backoff** for reliable file server downloads
-**Correlation ID tracking** for message tracing
-**Reply-to support** for request-response patterns
-**Handler function abstraction** - pluggable file server implementations (Plik, AWS S3, custom)
---
## Architecture
### System Components
```mermaid
flowchart TB
subgraph Sender["Application (Sender)"]
SenderApp[App Code]
NATSBridge_Send[NATSBridge]
NATS_Client[<b>NATS.jl</b>]
end
subgraph Receiver["Application (Receiver)"]
ReceiverApp[App Code]
NATSBridge_Recv[NATSBridge]
NATS_Client_Recv[<b>NATS.jl</b>]
end
subgraph Infrastructure["Infrastructure"]
NATS[<b>NATS Server</b><br/>Message Broker]
FileServer[<b>HTTP File Server</b><br/>Upload/Download]
end
SenderApp --> NATSBridge_Send
NATSBridge_Send --> NATS_Client
NATS_Client --> NATS
NATS --> NATS_Client_Recv
NATS_Client_Recv --> NATSBridge_Recv
NATSBridge_Recv --> ReceiverApp
NATSBridge_Send -.->|HTTP POST upload| FileServer
FileServer -.->|HTTP GET download| NATSBridge_Recv
style SenderApp fill:#e8f5e9
style ReceiverApp fill:#e8f5e9
style NATS fill:#fff3e0
style FileServer fill:#f3e5f5
```
### Message Flow
1. **Sender** creates a message envelope with payloads using `smartsend()`
2. **NATSBridge** serializes and encodes each payload based on type
3. **Transport Decision**:
- **Direct** (< 1MB): Payload encoded as Base64, published to NATS
- **Link** (≥ 1MB): Payload uploaded to HTTP file server, URL published to NATS
4. **NATS** routes message envelope to subscribers
5. **Receiver** receives message via NATS subscription callback
6. **NATSBridge** processes envelope:
- Decodes Base64 payloads from NATS message
- Fetches URLs from file server with exponential backoff
7. **Receiver** deserializes payloads based on their type
### File Server Handler Abstraction
The system uses handler functions to abstract file server operations:
| Handler | Purpose |
|---------|---------|
| `plik_oneshot_upload()` / `plikOneshotUpload()` | Uploads payload bytes to file server, returns URL |
| `_fetch_with_backoff()` / `fetchWithBackoff()` | Downloads data from URL with exponential backoff retry |
This abstraction allows support for different file server implementations (Plik, AWS S3, custom HTTP server).
### Message Envelope Schema
All platforms use identical JSON schemas for message envelopes:
```json
{
"correlation_id": "uuid-v4-string",
"msg_id": "uuid-v4-string",
"timestamp": "2024-01-15T10:30:00Z",
"send_to": "topic/subject",
"msg_purpose": "ACK | NACK | updateStatus | shutdown | chat",
"sender_name": "agent-wine-web-frontend",
"sender_id": "uuid4",
"receiver_name": "agent-backend",
"receiver_id": "uuid4",
"reply_to": "topic",
"reply_to_msg_id": "uuid4",
"broker_url": "nats://localhost:4222",
"metadata": {},
"payloads": [
{
"id": "uuid4",
"dataname": "login_image",
"payload_type": "image",
"transport": "direct",
"encoding": "base64",
"size": 15433,
"data": "base64-encoded-string"
},
{
"id": "uuid4",
"dataname": "large_table",
"payload_type": "table",
"transport": "link",
"encoding": "none",
"size": 524288,
"data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/data.arrow"
}
]
}
```
---
## Installation
### Prerequisites
- **NATS Server** (v2.10+ recommended)
- **HTTP File Server** (optional, for payloads > 1MB)
### Platform-Specific Dependencies
#### Julia
```julia
using Pkg
Pkg.add("NATS")
Pkg.add("Arrow")
Pkg.add("JSON3")
Pkg.add("HTTP")
Pkg.add("UUIDs")
Pkg.add("Dates")
```
#### JavaScript (Node.js)
```bash
npm install nats uuid apache-arrow node-fetch
# or
yarn add nats uuid apache-arrow node-fetch
```
#### JavaScript (Browser)
```bash
npm install nats uuid apache-arrow
# or use CDN:
# https://unpkg.com/nats-js/dist/bundle/nats.min.js
# https://unpkg.com/apache-arrow/arrow.min.js
```
#### Python (Desktop)
```bash
pip install nats-py aiohttp pyarrow pandas python-dateutil
```
#### MicroPython
MicroPython uses built-in modules:
- `network` - NATS connection (custom implementation)
- `time` - Timestamps
- `uos` - File operations
- `base64` - Base64 encoding
- `json` - JSON parsing
- `struct` - Binary data handling
---
## Quick Start
### Step 1: Start NATS Server
```bash
docker run -p 4222:4222 nats:latest
```
### Step 2: Start HTTP File Server (Optional)
```bash
# Create a directory for file uploads
mkdir -p /tmp/fileserver
# Start HTTP file server
python3 -m http.server 8080 --directory /tmp/fileserver
```
---
## API Reference
### Unified API Standard
All platforms use the same input/output format for payloads:
**Input format for smartsend:**
```
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
```
**Output format for smartreceive:**
```
{
"correlation_id": "...",
"msg_id": "...",
"timestamp": "...",
"send_to": "...",
"msg_purpose": "...",
"sender_name": "...",
"sender_id": "...",
"receiver_name": "...",
"receiver_id": "...",
"reply_to": "...",
"reply_to_msg_id": "...",
"broker_url": "...",
"metadata": {...},
"payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
}
```
### smartsend
Sends data either directly via NATS or via a fileserver URL, depending on payload size.
#### Julia
```julia
using NATSBridge
env, env_json_str = NATSBridge.smartsend(
subject::String, # NATS subject
data::AbstractArray{Tuple{String, Any, String}}; # List of (dataname, data, type)
broker_url::String = "nats://localhost:4222",
fileserver_url = "http://localhost:8080",
fileserver_upload_handler::Function = plik_oneshot_upload,
size_threshold::Int = 1_000_000,
correlation_id::String = string(uuid4()),
msg_purpose::String = "chat",
sender_name::String = "NATSBridge",
receiver_name::String = "",
receiver_id::String = "",
reply_to::String = "",
reply_to_msg_id::String = "",
is_publish::Bool = true,
NATS_connection::Union{NATS.Connection, Nothing} = nothing,
msg_id::String = string(uuid4()),
sender_id::String = string(uuid4())
)
# Returns: ::Tuple{msg_envelope_v1, String}
```
#### JavaScript
```javascript
const NATSBridge = require('natbridge');
const [env, env_json_str] = await NATSBridge.smartsend(
subject,
data, // Array of [dataname, data, type] tuples
{
broker_url: 'nats://localhost:4222',
fileserver_url: 'http://localhost:8080',
fileserver_upload_handler: NATSBridge.plikOneshotUpload,
size_threshold: 1_000_000,
correlation_id: uuidv4(),
msg_purpose: 'chat',
sender_name: 'NATSBridge',
receiver_name: '',
receiver_id: '',
reply_to: '',
reply_to_msg_id: '',
is_publish: true,
nats_connection: null,
msg_id: uuidv4(),
sender_id: uuidv4()
}
);
// Returns: Promise<[env, env_json_str]>
```
#### Python
```python
from natbridge import NATSBridge
env, env_json_str = await NATSBridge.smartsend(
subject: str,
data: List[Tuple[str, Any, str]],
broker_url: str = "nats://localhost:4222",
fileserver_url: str = "http://localhost:8080",
fileserver_upload_handler: Callable = plik_oneshot_upload,
size_threshold: int = 1_000_000,
correlation_id: str = None,
msg_purpose: str = "chat",
sender_name: str = "NATSBridge",
receiver_name: str = "",
receiver_id: str = "",
reply_to: str = "",
reply_to_msg_id: str = "",
is_publish: bool = True,
nats_connection: Any = None,
msg_id: str = None,
sender_id: str = None
)
# Returns: Tuple[Dict, str]
```
#### MicroPython
```python
from natbridge import NATSBridge
# Limited to direct transport (< 100KB threshold)
env, env_json_str = NATSBridge.smartsend(
subject,
data, # List of (dataname, data, type) tuples
broker_url="nats://localhost:4222",
size_threshold=100000 # Lower threshold for memory constraints
)
# Returns: Tuple[Dict, str]
```
### smartreceive
Receives and processes messages from NATS, handling both direct and link transport.
#### Julia
```julia
using NATSBridge
env = NATSBridge.smartreceive(
msg::NATS.Msg;
fileserver_download_handler::Function = _fetch_with_backoff,
max_retries::Int = 5,
base_delay::Int = 100,
max_delay::Int = 5000
)
# Returns: ::JSON.Object{String, Any}
```
#### JavaScript
```javascript
const env = await NATSBridge.smartreceive(
msg,
{
fileserver_download_handler: NATSBridge.fetchWithBackoff,
max_retries: 5,
base_delay: 100,
max_delay: 5000
}
);
// Returns: Promise<env_object>
```
#### Python
```python
env = await NATSBridge.smartreceive(
msg,
fileserver_download_handler=fetch_with_backoff,
max_retries=5,
base_delay=100,
max_delay=5000
)
# Returns: Dict with "payloads" key
```
#### MicroPython
```python
env = NATSBridge.smartreceive(
msg,
fileserver_download_handler=_sync_fileserver_download,
max_retries=3,
base_delay=100,
max_delay=1000
)
# Returns: Dict with "payloads" key
```
---
## Payload Types
| Type | Julia | JavaScript | Python | MicroPython | Description |
|------|-------|------------|--------|-------------|-------------|
| `text` | `String` | `string` | `str` | `str` | Plain text strings |
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable dictionaries |
| `table` | `DataFrame`, `Arrow.Table` | `Array<Object>` | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) |
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image data (PNG, JPG) |
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio data (WAV, MP3) |
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video data (MP4, AVI) |
| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `bytearray` | Generic binary data |
---
## Transport Strategies
### Direct Transport (Payloads < 1MB)
Small payloads are sent directly via NATS with Base64 encoding.
#### Cross-Platform
```julia
# Julia
data = [("message", "Hello", "text")]
smartsend("/topic", data)
```
```javascript
// JavaScript
const data = [["message", "Hello", "text"]];
smartsend("/topic", data);
```
```python
# Python
data = [("message", "Hello", "text")]
await smartsend("/topic", data)
```
### Link Transport (Payloads >= 1MB)
Large payloads are uploaded to an HTTP file server.
#### Cross-Platform
```julia
# Julia
data = [("file", large_data, "binary")]
smartsend("/topic", data; fileserver_url="http://localhost:8080")
```
```javascript
// JavaScript
const data = [["file", largeData, "binary"]];
smartsend("/topic", data, { fileserver_url: 'http://localhost:8080' });
```
```python
# Python
data = [("file", large_data, "binary")]
await smartsend("/topic", data, fileserver_url="http://localhost:8080")
```
---
## Cross-Platform Examples
### Example 1: Chat with Mixed Content
Send text, image, and large file in one message.
#### Julia
```julia
using NATSBridge
data = [
("message_text", "Hello!", "text"),
("user_avatar", image_data, "image"),
("large_document", large_file_data, "binary")
]
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; fileserver_url="http://localhost:8080")
```
#### JavaScript
```javascript
const NATSBridge = require('natbridge');
const data = [
["message_text", "Hello!", "text"],
["user_avatar", imageData, "image"],
["large_document", largeFileData, "binary"]
];
const [env, env_json_str] = await NATSBridge.smartsend(
"/chat/room1",
data,
{ fileserver_url: 'http://localhost:8080' }
);
```
#### Python
```python
from natbridge import NATSBridge
data = [
("message_text", "Hello!", "text"),
("user_avatar", image_data, "image"),
("large_document", large_file_data, "binary")
]
env, env_json_str = await NATSBridge.smartsend(
"/chat/room1",
data,
fileserver_url="http://localhost:8080"
)
```
### Example 2: Dictionary Exchange
Send configuration data between platforms.
#### Julia
```julia
using NATSBridge
config = Dict(
"wifi_ssid" => "MyNetwork",
"wifi_password" => "password123",
"update_interval" => 60
)
data = [("config", config, "dictionary")]
env, env_json_str = NATSBridge.smartsend("/device/config", data)
```
#### JavaScript
```javascript
const NATSBridge = require('natbridge');
const config = {
wifi_ssid: "MyNetwork",
wifi_password: "password123",
update_interval: 60
};
const [env, env_json_str] = await NATSBridge.smartsend(
"/device/config",
[["config", config, "dictionary"]]
);
```
#### Python
```python
from natbridge import NATSBridge
config = {
"wifi_ssid": "MyNetwork",
"wifi_password": "password123",
"update_interval": 60
}
data = [("config", config, "dictionary")]
env, env_json_str = await NATSBridge.smartsend("/device/config", data)
```
### Example 3: Table Data (Arrow IPC)
Send tabular data using Apache Arrow IPC format.
#### Julia
```julia
using NATSBridge
using DataFrames
df = DataFrame(
id = [1, 2, 3],
name = ["Alice", "Bob", "Charlie"],
score = [95, 88, 92]
)
data = [("students", df, "table")]
env, env_json_str = NATSBridge.smartsend("/data/analysis", data)
```
#### JavaScript
```javascript
const NATSBridge = require('natbridge');
const df = [
{ id: 1, name: "Alice", score: 95 },
{ id: 2, name: "Bob", score: 88 },
{ id: 3, name: "Charlie", score: 92 }
];
const [env, env_json_str] = await NATSBridge.smartsend(
"/data/analysis",
[["students", df, "table"]]
);
```
#### Python
```python
from natbridge import NATSBridge
import pandas as pd
df = pd.DataFrame({
"id": [1, 2, 3],
"name": ["Alice", "Bob", "Charlie"],
"score": [95, 88, 92]
})
data = [("students", df, "table")]
env, env_json_str = await NATSBridge.smartsend("/data/analysis", data)
```
### Example 4: Request-Response Pattern
Bi-directional communication with reply-to support.
#### Julia
```julia
using NATSBridge
# Requester
env, env_json_str = NATSBridge.smartsend(
"/device/command",
[("command", Dict("action" => "read_sensor"), "dictionary")];
broker_url="nats://localhost:4222",
reply_to="/device/response"
)
```
```julia
# Responder
using NATS, NATSBridge
function test_responder()
conn = NATS.connect("nats://localhost:4222")
NATS.subscribe(conn, "/device/command") do msg
env = NATSBridge.smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
reply_to = env["reply_to"]
for (dataname, data, type) in env["payloads"]
if dataname == "command" && data["action"] == "read_sensor"
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
if !isempty(reply_to)
smartsend(reply_to, [("data", response, "dictionary")])
end
end
end
end
sleep(120)
NATS.drain(conn)
end
```
#### JavaScript
```javascript
const NATSBridge = require('natbridge');
// Requester
const [env, env_json_str] = await NATSBridge.smartsend(
"/device/command",
[["command", { action: "read_sensor" }, "dictionary"]],
{ broker_url: 'nats://localhost:4222', reply_to: '/device/response' }
);
```
```javascript
// Responder
const nats = require('nats');
const NATSBridge = require('natbridge');
async function testResponder() {
const conn = await nats.connect('nats://localhost:4222');
const subscription = await conn.subscribe('/device/command');
for await (const msg of subscription) {
const env = await NATSBridge.smartreceive(msg, {
fileserver_download_handler: NATSBridge.fetchWithBackoff
});
const replyTo = env.reply_to;
for (const [dataname, data, type] of env.payloads) {
if (dataname === 'command' && data.action === 'read_sensor') {
const response = { sensor_id: 'sensor-001', value: 42.5 };
if (replyTo) {
await NATSBridge.smartsend(
replyTo,
[["data", response, "dictionary"]]
);
}
}
}
}
setTimeout(() => conn.close(), 120000);
}
```
#### Python
```python
from natbridge import NATSBridge
# Requester
env, env_json_str = await NATSBridge.smartsend(
"/device/command",
[("command", {"action": "read_sensor"}, "dictionary")],
broker_url="nats://localhost:4222",
reply_to="/device/response"
)
```
```python
# Responder
from natbridge import NATSBridge
import asyncio
import nats
async def test_responder():
nc = await nats.connect('nats://localhost:4222')
async def msg_handler(msg):
env = await NATSBridge.smartreceive(
msg,
fileserver_download_handler=fetch_with_backoff
)
reply_to = env["reply_to"]
for dataname, data, type_ in env["payloads"]:
if dataname == "command" and data["action"] == "read_sensor":
response = {"sensor_id": "sensor-001", "value": 42.5}
if reply_to:
await NATSBridge.smartsend(
reply_to,
[("data", response, "dictionary")]
)
await nc.subscribe('/device/command', cb=msg_handler)
await asyncio.sleep(120)
await nc.drain()
```
---
## Testing
### Test File Organization
| Platform | Sender Tests | Receiver Tests |
|----------|--------------|----------------|
| **Julia** | `test/test_julia_*_sender.jl` | `test/test_julia_*_receiver.jl` |
| **JavaScript** | `test/test_js_*_sender.js` | `test/test_js_*_receiver.js` |
| **Python** | `test/test_py_*_sender.py` | `test/test_py_*_receiver.py` |
### Run Tests
#### Julia
```bash
# Text message exchange
julia test/test_julia_text_sender.jl
julia test/test_julia_text_receiver.jl
# Dictionary exchange
julia test/test_julia_dict_sender.jl
julia test/test_julia_dict_receiver.jl
# File transfer
julia test/test_julia_file_sender.jl
julia test/test_julia_file_receiver.jl
# Mixed payload types
julia test/test_julia_mix_payloads_sender.jl
julia test/test_julia_mix_payloads_receiver.jl
# Table exchange
julia test/test_julia_table_sender.jl
julia test/test_julia_table_receiver.jl
```
#### JavaScript (Node.js)
```bash
# Text message exchange
node test/test_js_text_sender.js
node test/test_js_text_receiver.js
# Dictionary exchange
node test/test_js_dictionary_sender.js
node test/test_js_dictionary_receiver.js
# Binary transfer
node test/test_js_binary_sender.js
node test/test_js_binary_receiver.js
# Table exchange
node test/test_js_table_sender.js
node test/test_js_table_receiver.js
```
#### Python
```bash
# Text message exchange
python3 test/test_py_text_sender.py
python3 test/test_py_text_receiver.py
# Dictionary exchange
python3 test/test_py_dictionary_sender.py
python3 test/test_py_dictionary_receiver.py
# Binary transfer
python3 test/test_py_binary_sender.py
python3 test/test_py_binary_receiver.py
# Table exchange
python3 test/test_py_table_sender.py
python3 test/test_py_table_receiver.py
```
---
## Documentation
For detailed architecture and implementation information, see:
- [Architecture Documentation](docs/architecture.md) - Cross-platform architecture, API parity, platform-specific patterns
- [Implementation Guide](docs/implementation.md) - Detailed implementation for each platform, handler functions, testing
---
## License
MIT License
Copyright (c) 2026 NATSBridge Contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View File

@@ -1,321 +0,0 @@
# Implementation Guide: Bi-Directional Data Bridge
## Overview
This document describes the implementation of the high-performance, bi-directional data bridge between Julia and JavaScript services using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
## Architecture
The implementation follows the Claim-Check pattern:
```
┌─────────────────────────────────────────────────────────────────────────┐
│ SmartSend Function │
└─────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────┐
│ Is payload size < 1MB? │
└─────────────────────────────────────────────────────────────────────────┘
┌─────────────────┴─────────────────┐
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Direct Path │ │ Link Path │
│ (< 1MB) │ │ (> 1MB) │
│ │ │ │
│ • Serialize to │ │ • Serialize to │
│ IOBuffer │ │ IOBuffer │
│ • Base64 encode │ │ • Upload to │
│ • Publish to │ │ HTTP Server │
│ NATS │ │ • Publish to │
│ │ │ NATS with URL │
└─────────────────┘ └─────────────────┘
```
## Files
### Julia Module: [`src/julia_bridge.jl`](../src/julia_bridge.jl)
The Julia implementation provides:
- **[`MessageEnvelope`](../src/julia_bridge.jl)**: Struct for the unified JSON envelope
- **[`SmartSend()`](../src/julia_bridge.jl)**: Handles transport selection based on payload size
- **[`SmartReceive()`](../src/julia_bridge.jl)**: Handles both direct and link transport
### JavaScript Module: [`src/js_bridge.js`](../src/js_bridge.js)
The JavaScript implementation provides:
- **`MessageEnvelope` class**: For the unified JSON envelope
- **[`SmartSend()`](../src/js_bridge.js)**: Handles transport selection based on payload size
- **[`SmartReceive()`](../src/js_bridge.js)**: Handles both direct and link transport
## Installation
### Julia Dependencies
```julia
using Pkg
Pkg.add("NATS")
Pkg.add("Arrow")
Pkg.add("JSON3")
Pkg.add("HTTP")
Pkg.add("UUIDs")
Pkg.add("Dates")
```
### JavaScript Dependencies
```bash
npm install nats.js apache-arrow uuid base64-url
```
## Usage Tutorial
### Step 1: Start NATS Server
```bash
docker run -p 4222:4222 nats:latest
```
### Step 2: Start HTTP File Server (optional)
```bash
# Create a directory for file uploads
mkdir -p /tmp/fileserver
# Use any HTTP server that supports POST for file uploads
# Example: Python's built-in server
python3 -m http.server 8080 --directory /tmp/fileserver
```
### Step 3: Run Test Scenarios
```bash
# Scenario 1: Command & Control (JavaScript sender)
node test/scenario1_command_control.js
# Scenario 2: Large Arrow Table (JavaScript sender)
node test/scenario2_large_table.js
# Scenario 3: Julia-to-Julia communication
# Run both Julia and JavaScript versions
julia test/scenario3_julia_to_julia.jl
node test/scenario3_julia_to_julia.js
```
## Usage
### Scenario 1: Command & Control (Small JSON)
#### JavaScript (Sender)
```javascript
const { SmartSend } = require('./js_bridge');
const config = {
step_size: 0.01,
iterations: 1000
};
await SmartSend("control", config, "json", {
correlationId: "unique-id"
});
```
#### Julia (Receiver)
```julia
using NATS
using JSON3
# Subscribe to control subject
subscribe(nats, "control") do msg
env = MessageEnvelope(String(msg.data))
config = JSON3.read(env.payload)
# Execute simulation with parameters
step_size = config.step_size
iterations = config.iterations
# Send acknowledgment
response = Dict("status" => "Running", "correlation_id" => env.correlation_id)
publish(nats, "control_response", JSON3.stringify(response))
end
```
### Scenario 2: Deep Dive Analysis (Large Arrow Table)
#### Julia (Sender)
```julia
using Arrow
using DataFrames
# Create large DataFrame
df = DataFrame(
id = 1:10_000_000,
value = rand(10_000_000),
category = rand(["A", "B", "C"], 10_000_000)
)
# Send via SmartSend with type="table"
await SmartSend("analysis_results", df, "table");
```
#### JavaScript (Receiver)
```javascript
const { SmartReceive } = require('./js_bridge');
const result = await SmartReceive(msg);
// Use table data for visualization with Perspective.js or D3
const table = result.data;
```
### Scenario 3: Live Binary Processing
#### JavaScript (Sender)
```javascript
const { SmartSend } = require('./js_bridge');
// Capture binary chunk
const binaryData = await navigator.mediaDevices.getUserMedia({ binary: true });
await SmartSend("binary_input", binaryData, "binary", {
metadata: {
sample_rate: 44100,
channels: 1
}
});
```
#### Julia (Receiver)
```julia
using WAV
using DSP
# Receive binary data
function process_binary(data)
# Perform FFT or AI transcription
spectrum = fft(data)
# Send results back (JSON + Arrow table)
results = Dict("transcription" => "sample text", "spectrum" => spectrum)
await SmartSend("binary_output", results, "json")
end
```
### Scenario 4: Catch-Up (JetStream)
#### Julia (Producer)
```julia
using NATS
function publish_health_status(nats)
jetstream = JetStream(nats, "health_updates")
while true
status = Dict("cpu" => rand(), "memory" => rand())
publish(jetstream, "health", status)
sleep(5) # Every 5 seconds
end
end
```
#### JavaScript (Consumer)
```javascript
const { connect } = require('nats');
const nc = await connect({ servers: ['nats://localhost:4222'] });
const js = nc.jetstream();
// Request replay from last 10 minutes
const consumer = await js.pullSubscribe("health", {
durable_name: "catchup",
max_batch: 100,
max_ack_wait: 30000
});
// Process historical and real-time messages
for await (const msg of consumer) {
const result = await SmartReceive(msg);
// Process the data
msg.ack();
}
```
## Configuration
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL |
| `FILESERVER_URL` | `http://localhost:8080/upload` | HTTP file server URL |
| `SIZE_THRESHOLD` | `1_000_000` | Size threshold in bytes (1MB) |
### Message Envelope Schema
```json
{
"correlation_id": "uuid-v4-string",
"type": "json|table|binary",
"transport": "direct|link",
"payload": "base64-encoded-string", // Only if transport=direct
"url": "http://fileserver/path/to/data", // Only if transport=link
"metadata": {
"content_type": "application/octet-stream",
"content_length": 123456,
"format": "arrow_ipc_stream"
}
}
```
## Performance Considerations
### Zero-Copy Reading
- Use Arrow's memory-mapped file reading
- Avoid unnecessary data copying during deserialization
- Use Apache Arrow's native IPC reader
### Exponential Backoff
- Maximum retry count: 5
- Base delay: 100ms, max delay: 5000ms
- Implemented in both Julia and JavaScript implementations
### Correlation ID Logging
- Log correlation_id at every stage
- Include: send, receive, serialize, deserialize
- Use structured logging format
## Testing
Run the test scripts:
```bash
# Scenario 1: Command & Control (JavaScript sender)
node test/scenario1_command_control.js
# Scenario 2: Large Arrow Table (JavaScript sender)
node test/scenario2_large_table.js
```
## Troubleshooting
### Common Issues
1. **NATS Connection Failed**
- Ensure NATS server is running
- Check NATS_URL configuration
2. **HTTP Upload Failed**
- Ensure file server is running
- Check FILESERVER_URL configuration
- Verify upload permissions
3. **Arrow IPC Deserialization Error**
- Ensure data is properly serialized to Arrow format
- Check Arrow version compatibility
## License
MIT

File diff suppressed because it is too large Load Diff

1998
docs/implementation.md Normal file

File diff suppressed because it is too large Load Diff

60
etc.jl
View File

@@ -1,42 +1,38 @@
Task: Update README.md to reflect recent changes in NATSbridge package.
Context: the package has been updated with the NATS_connection keyword and the publish_message function.
Requirements:
Source of Truth: Treat the updated NATSbridge code as the definitive source. Update README.md to align exactly with these changes.
API Consistency: Ensure the Main Package API (e.g., smartsend(), publish_message()) uses consistent naming across all three supported languages.
Ecosystem Variance: Low-level native functions (e.g., NATS.connect(), JSON.read()) should follow the conventions of the specific language ecosystem and do not require cross-language consistency.
Help me expands this Julia package (NATSBridge) into a cross-platform project by adding a JavaScript and Python/MicroPython implementation. To ensure accuracy, NATSBridge.jl will serve as the ground truth, as the documentation may be outdated.
My goal is to maintain interface parity at the high-level API for a consistent user experience, while ensuring the low-level implementation adheres strictly to the idiomatic conventions of each respective language (e.g., multiple dispatch in Julia vs. asynchronous, prototype, or class-based patterns in JS and Python/MicroPython)
Now do the following:
1) check docs to see if there is any mistake.
""" fileServerURL = "http://192.168.88.104:8080"
filepath = "/home/ton/docker-apps/sendreceive/image/test.zip"
filename = basename(filepath)
filebytes = read(filepath)
plik_oneshot_upload - Upload a single file to a plik server using one-shot mode
This function uploads a raw byte array to a plik server in one-shot mode (no upload session).
It first creates a one-shot upload session by sending a POST request with `{"OneShot": true}`,
retrieves an upload ID and token, then uploads the file data as multipart form data using the token.
The function handles the entire flow:
1. Obtains an upload ID and token from the server
2. Uploads the provided binary data as a file using the `X-UploadToken` header
3. Returns identifiers and download URL for the uploaded file
# Arguments:
- `fileServerURL::String` - Base URL of the plik server (e.g., `"http://192.168.88.104:8080"`)
- `filename::String` - Name of the file being uploaded
- `data::Vector{UInt8}` - Raw byte data of the file content
# Return:
- A named tuple with fields:
- `uploadid::String` - ID of the one-shot upload session
- `fileid::String` - ID of the uploaded file within the session
- `downloadurl::String` - Full URL to download the uploaded file
# Example
```jldoctest
using HTTP, JSON
# Example data: "Hello" as bytes
data = collect("Hello World!" |> collect |> CodeUnits |> collect)
# Upload to local plik server
result = plik_oneshot_upload("http://192.168.88.104:8080", "hello.txt", data)
# Download URL for the uploaded file
println(result.downloadurl)
```
"""

731
examples/tutorial.md Normal file
View File

@@ -0,0 +1,731 @@
# Cross-Platform NATSBridge Tutorial
A step-by-step guide to get started with NATSBridge across **Julia**, **JavaScript**, and **Python/MicroPython**.
## Table of Contents
1. [Overview](#overview)
2. [Prerequisites](#prerequisites)
3. [Installation](#installation)
4. [Quick Start](#quick-start)
5. [Basic Examples](#basic-examples)
6. [Advanced Usage](#advanced-usage)
---
## Overview
NATSBridge enables seamless communication across platforms through NATS, with automatic transport selection based on payload size:
- **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded)
- **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL
### Cross-Platform API Parity
All three platforms use the same high-level API:
```
# Input format
smartsend(subject, [(dataname, data, type), ...], options)
# Output format
(env, env_json_str) = smartsend(...)
env = smartreceive(msg, options)
```
### Supported Payload Types
| Type | Julia | JavaScript | Python | MicroPython |
|------|-------|------------|--------|-------------|
| `text` | `String` | `string` | `str` | `str` |
| `dictionary` | `Dict` | `Object` | `dict` | `dict` |
| `table` | `DataFrame` | `Array<Object>` | `DataFrame` | ❌ |
| `image` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
| `audio` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
| `video` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
| `binary` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
---
## Prerequisites
Before you begin, ensure you have:
1. **NATS Server** running (or accessible)
2. **HTTP File Server** (optional, for large payloads > 1MB)
3. **Platform-specific packages** installed
---
## Installation
### Julia
```julia
using Pkg
Pkg.add("NATS")
Pkg.add("Arrow")
Pkg.add("JSON3")
Pkg.add("HTTP")
Pkg.add("UUIDs")
Pkg.add("Dates")
```
### JavaScript (Node.js)
```bash
npm install nats uuid apache-arrow node-fetch
```
### JavaScript (Browser)
```html
<script src="https://unpkg.com/nats-js/dist/bundle/nats.min.js"></script>
<script src="https://unpkg.com/apache-arrow/arrow.min.js"></script>
```
### Python (Desktop)
```bash
pip install nats-py aiohttp pyarrow pandas
```
### MicroPython
Uses built-in modules: `network`, `socket`, `time`, `json`, `base64`
---
## Quick Start
### Step 1: Start NATS Server
```bash
docker run -p 4222:4222 nats:latest
```
### Step 2: Start HTTP File Server (Optional)
```bash
mkdir -p /tmp/fileserver
python3 -m http.server 8080 --directory /tmp/fileserver
```
### Step 3: Send Your First Message
#### Julia
```julia
using NATSBridge
# Send a text message
data = [("message", "Hello World", "text")]
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222")
# env: msg_envelope_v1 struct with all metadata and payloads
# env_json_str: JSON string representation of the envelope for publishing
println("Message sent!")
# Or use is_publish=false to get envelope and JSON without publishing
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222", is_publish=false)
# env: msg_envelope_v1 struct
# env_json_str: JSON string for publishing to NATS
```
#### JavaScript
```javascript
const NATSBridge = require('./src/natbridge.js');
// Send a text message
const data = [["message", "Hello World", "text"]];
const [env, env_json_str] = await NATSBridge.smartsend(
"/chat/room1",
data,
{ broker_url: "nats://localhost:4222" }
);
// env: Object with all metadata and payloads
// env_json_str: JSON string for publishing
console.log("Message sent!");
// Or use is_publish=false
const [env, env_json_str] = await NATSBridge.smartsend(
"/chat/room1",
data,
{ broker_url: "nats://localhost:4222", is_publish: false }
);
```
#### Python
```python
from natbridge import smartsend
# Send a text message
data = [("message", "Hello World", "text")]
env, env_json_str = await smartsend(
"/chat/room1",
data,
broker_url="nats://localhost:4222"
)
# env: Dict with all metadata and payloads
# env_json_str: JSON string for publishing
print("Message sent!")
# Or use is_publish=False
env, env_json_str = await smartsend(
"/chat/room1",
data,
broker_url="nats://localhost:4222",
is_publish=False
)
# env: Dict with all metadata and payloads
# env_json_str: JSON string for publishing to NATS
```
#### MicroPython
```python
from natbridge_mpy import NATSBridge
bridge = NATSBridge()
# Send a text message (limited to small payloads)
data = [("message", "Hello World", "text")]
env, env_json_str = bridge.smartsend(
"/chat/room1",
data,
size_threshold=100000 # Lower threshold for MicroPython
)
print("Message sent!")
```
### Step 4: Receive Messages
#### Julia
```julia
using NATSBridge
# Receive and process message
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff)
# Returns: ::JSON.Object{String, Any} with "payloads" field containing Vector{Tuple{String, Any, String}}
# Access payloads: for (dataname, data, type) in env["payloads"]
for (dataname, data, type) in env["payloads"]
println("Received $dataname: $data")
end
```
#### JavaScript
```javascript
const NATSBridge = require('./src/natbridge.js');
// Receive and process message
const env = await NATSBridge.smartreceive(msg, {
fileserver_download_handler: NATSBridge.fetchWithBackoff
});
// env.payloads = [[dataname, data, type], ...]
for (const [dataname, data, type] of env.payloads) {
console.log(`Received ${dataname}:`, data);
}
```
#### Python
```python
from natbridge import smartreceive, fetch_with_backoff
# Receive and process message
env = await smartreceive(
msg,
fileserver_download_handler=fetch_with_backoff
)
# env["payloads"] = [(dataname, data, type), ...]
for dataname, data, type_ in env["payloads"]:
print(f"Received {dataname}: {data}")
```
---
## Basic Examples
### Example 1: Sending a Dictionary
#### Julia
```julia
using NATSBridge
config = Dict(
"wifi_ssid" => "MyNetwork",
"wifi_password" => "password123",
"update_interval" => 60
)
data = [("config", config, "dictionary")]
env, env_json_str = smartsend("/device/config", data, broker_url="nats://localhost:4222")
```
#### JavaScript
```javascript
const NATSBridge = require('./src/natbridge.js');
const config = {
wifi_ssid: "MyNetwork",
wifi_password: "password123",
update_interval: 60
};
const data = [["config", config, "dictionary"]];
const [env, env_json_str] = await NATSBridge.smartsend(
"/device/config",
data,
{ broker_url: "nats://localhost:4222" }
);
```
#### Python
```python
from natbridge import smartsend
config = {
"wifi_ssid": "MyNetwork",
"wifi_password": "password123",
"update_interval": 60
}
data = [("config", config, "dictionary")]
env, env_json_str = await smartsend(
"/device/config",
data,
broker_url="nats://localhost:4222"
)
```
#### MicroPython
```python
from natbridge_mpy import NATSBridge
bridge = NATSBridge()
config = {
"wifi_ssid": "MyNetwork",
"wifi_password": "password123",
"update_interval": 60
}
data = [("config", config, "dictionary")]
env, env_json_str = bridge.smartsend(
"/device/config",
data,
size_threshold=100000
)
```
### Example 2: Sending Binary Data (Image)
#### Julia
```julia
using NATSBridge
# Read image file
image_data = read("image.png")
data = [("user_image", image_data, "binary")]
env, env_json_str = smartsend("/chat/image", data, broker_url="nats://localhost:4222")
```
#### JavaScript
```javascript
const NATSBridge = require('./src/natbridge.js');
const fs = require('fs');
// Read image file
const image_data = fs.readFileSync('image.png');
const data = [["user_image", image_data, "binary"]];
const [env, env_json_str] = await NATSBridge.smartsend(
"/chat/image",
data,
{ broker_url: "nats://localhost:4222" }
);
```
#### Python
```python
from natbridge import smartsend
# Read image file
with open("image.png", "rb") as f:
image_data = f.read()
data = [("user_image", image_data, "binary")]
env, env_json_str = await smartsend(
"/chat/image",
data,
broker_url="nats://localhost:4222"
)
```
#### MicroPython
```python
from natbridge_mpy import NATSBridge
bridge = NATSBridge()
# Read image file
with open("image.png", "rb") as f:
image_data = f.read()
data = [("user_image", image_data, "binary")]
env, env_json_str = bridge.smartsend(
"/chat/image",
data,
size_threshold=100000
)
```
### Example 3: Request-Response Pattern
#### Julia (Requester)
```julia
using NATSBridge
# Send command with reply-to
data = [("command", Dict("action" => "read_sensor"), "dictionary")]
env, env_json_str = smartsend(
"/device/command",
data,
broker_url="nats://localhost:4222",
reply_to="/device/response",
reply_to_msg_id="cmd-001"
)
```
#### JavaScript (Requester)
```javascript
const NATSBridge = require('./src/natbridge.js');
// Send command with reply-to
const data = [["command", { action: "read_sensor" }, "dictionary"]];
const [env, env_json_str] = await NATSBridge.smartsend(
"/device/command",
data,
{
broker_url: "nats://localhost:4222",
reply_to: "/device/response",
reply_to_msg_id: "cmd-001"
}
);
```
#### Python (Requester)
```python
from natbridge import smartsend
# Send command with reply-to
data = [("command", {"action": "read_sensor"}, "dictionary")]
env, env_json_str = await smartsend(
"/device/command",
data,
broker_url="nats://localhost:4222",
reply_to="/device/response",
reply_to_msg_id="cmd-001"
)
```
#### Julia (Responder)
```julia
using NATSBridge, NATS
const SUBJECT = "/device/command"
const NATS_URL = "nats://localhost:4222"
function test_responder()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
env = smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
reply_to = env["reply_to"]
for (dataname, data, type) in env["payloads"]
if dataname == "command" && data["action"] == "read_sensor"
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
if !isempty(reply_to)
smartsend(reply_to, [("data", response, "dictionary")])
end
end
end
end
sleep(120)
NATS.drain(conn)
end
test_responder()
```
---
## Advanced Usage
### Example 4: Large Payloads (File Server)
For payloads larger than 1MB, NATSBridge automatically uses the file server:
#### Julia
```julia
using NATSBridge
# Create large data (> 1MB)
large_data = rand(UInt8, 2_000_000)
env, env_json_str = smartsend(
"/data/large",
[("large_file", large_data, "binary")],
broker_url="nats://localhost:4222",
fileserver_url="http://localhost:8080"
)
println("File uploaded to: $(env.payloads[1].data)")
# Note: For link transport, data field contains the URL string
```
#### JavaScript
```javascript
const NATSBridge = require('./src/natbridge.js');
// Create large data (> 1MB)
const large_data = Buffer.alloc(2_000_000);
for (let i = 0; i < large_data.length; i++) {
large_data[i] = Math.floor(Math.random() * 256);
}
const [env, env_json_str] = await NATSBridge.smartsend(
"/data/large",
[["large_file", large_data, "binary"]],
{
broker_url: "nats://localhost:4222",
fileserver_url: "http://localhost:8080"
}
);
console.log("File uploaded to:", env.payloads[0].data);
// Note: For link transport, data field contains the URL string
```
#### Python
```python
from natbridge import smartsend
# Create large data (> 1MB)
import os
large_data = os.urandom(2_000_000)
env, env_json_str = await smartsend(
"/data/large",
[("large_file", large_data, "binary")],
broker_url="nats://localhost:4222",
fileserver_url="http://localhost:8080"
)
print(f"File uploaded to: {env['payloads'][0]['data']}")
# Note: For link transport, data field contains the URL string
```
#### MicroPython
MicroPython enforces a hard limit of 50KB per payload:
```python
from natbridge_mpy import NATSBridge
bridge = NATSBridge()
# MicroPython has a hard limit of 50KB per payload
# Use streaming or chunking for larger data
small_data = bytes(1000) # 1KB
data = [("small_file", small_data, "binary")]
env, env_json_str = bridge.smartsend(
"/data/small",
data,
size_threshold=100000 # Enforced max: 50000 bytes
)
```
### Example 5: Mixed Content (Chat with Text + Image)
NATSBridge supports sending multiple payloads with different types in a single message:
#### Julia
```julia
using NATSBridge
image_data = read("avatar.png")
data = [
("message_text", "Hello with image!", "text"),
("user_avatar", image_data, "image")
]
env, env_json_str = smartsend("/chat/mixed", data, broker_url="nats://localhost:4222")
```
#### JavaScript
```javascript
const NATSBridge = require('./src/natbridge.js');
const fs = require('fs');
const image_data = fs.readFileSync('avatar.png');
const data = [
["message_text", "Hello with image!", "text"],
["user_avatar", image_data, "image"]
];
const [env, env_json_str] = await NATSBridge.smartsend(
"/chat/mixed",
data,
{ broker_url: "nats://localhost:4222" }
);
```
#### Python
```python
from natbridge import smartsend
with open("avatar.png", "rb") as f:
image_data = f.read()
data = [
("message_text", "Hello with image!", "text"),
("user_avatar", image_data, "image")
]
env, env_json_str = await smartsend(
"/chat/mixed",
data,
broker_url="nats://localhost:4222"
)
# env: Dict with all metadata and payloads
```
### Example 6: Table Data (Arrow IPC)
For tabular data, NATSBridge uses Apache Arrow IPC format:
#### Julia
```julia
using NATSBridge
using DataFrames
# Create DataFrame
df = DataFrame(
id = [1, 2, 3],
name = ["Alice", "Bob", "Charlie"],
score = [95, 88, 92]
)
data = [("students", df, "table")]
env, env_json_str = smartsend("/data/students", data, broker_url="nats://localhost:4222")
```
#### JavaScript
```javascript
const NATSBridge = require('./src/natbridge.js');
// Create table data (array of objects)
const table_data = [
{ id: 1, name: "Alice", score: 95 },
{ id: 2, name: "Bob", score: 88 },
{ id: 3, name: "Charlie", score: 92 }
];
const data = [["students", table_data, "table"]];
const [env, env_json_str] = await NATSBridge.smartsend(
"/data/students",
data,
{ broker_url: "nats://localhost:4222" }
);
```
#### Python
```python
from natbridge import smartsend
import pandas as pd
# Create DataFrame
df = pd.DataFrame({
'id': [1, 2, 3],
'name': ['Alice', 'Bob', 'Charlie'],
'score': [95, 88, 92]
})
data = [("students", df, "table")]
env, env_json_str = await smartsend(
"/data/students",
data,
broker_url="nats://localhost:4222"
)
```
#### MicroPython
MicroPython does not support table type due to memory constraints. Use dictionary or binary instead.
---
## Next Steps
1. **Explore the test directory** for more examples
2. **Check the documentation** for advanced configuration options
3. **Read the walkthrough** for building real-world applications
---
## Troubleshooting
### Connection Issues
- Ensure NATS server is running: `docker ps | grep nats`
- Check firewall settings
- Verify NATS URL configuration
### File Server Issues
- Ensure file server is running and accessible
- Check upload permissions
- Verify file server URL configuration
### Serialization Errors
- Verify data type matches the specified type
- Check that binary data is in the correct format
- MicroPython: Ensure payload size < 50KB
---
## License
MIT

1401
examples/walkthrough.md Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,14 @@
services:
plik:
image: rootgg/plik:latest
container_name: plik-server
restart: unless-stopped
ports:
- "8080:8080"
volumes:
# # Mount the config file (created below)
# - ./plikd.cfg:/home/plik/server/plikd.cfg
# Mount local folder for uploads and database
- ./plik-data:/data
# Set user to match your host UID to avoid permission issues
user: "1000:1000"

File diff suppressed because it is too large Load Diff

View File

@@ -1,245 +0,0 @@
/**
* Bi-Directional Data Bridge - JavaScript Module
* Implements SmartSend and SmartReceive for NATS communication
*/
const { v4: uuidv4 } = require('uuid');
const { decode, encode } = require('base64-url');
const Arrow = require('apache-arrow');
// Constants
const DEFAULT_SIZE_THRESHOLD = 1_000_000; // 1MB
const DEFAULT_NATS_URL = 'nats://localhost:4222';
const DEFAULT_FILESERVER_URL = 'http://localhost:8080/upload';
// Logging helper
function logTrace(correlationId, message) {
const timestamp = new Date().toISOString();
console.log(`[${timestamp}] [Correlation: ${correlationId}] ${message}`);
}
// Message Envelope Class
class MessageEnvelope {
constructor(options = {}) {
this.correlation_id = options.correlation_id || uuidv4();
this.type = options.type || 'json';
this.transport = options.transport || 'direct';
this.payload = options.payload || null;
this.url = options.url || null;
this.metadata = options.metadata || {};
}
static fromJSON(jsonStr) {
const data = JSON.parse(jsonStr);
return new MessageEnvelope({
correlation_id: data.correlation_id,
type: data.type,
transport: data.transport,
payload: data.payload || null,
url: data.url || null,
metadata: data.metadata || {}
});
}
toJSON() {
const obj = {
correlation_id: this.correlation_id,
type: this.type,
transport: this.transport
};
if (this.payload) {
obj.payload = this.payload;
}
if (this.url) {
obj.url = this.url;
}
if (Object.keys(this.metadata).length > 0) {
obj.metadata = this.metadata;
}
return JSON.stringify(obj);
}
}
// SmartSend for JavaScript - Handles transport selection based on payload size
async function SmartSend(subject, data, type = 'json', options = {}) {
const {
natsUrl = DEFAULT_NATS_URL,
fileserverUrl = DEFAULT_FILESERVER_URL,
sizeThreshold = DEFAULT_SIZE_THRESHOLD,
correlationId = uuidv4()
} = options;
logTrace(correlationId, `Starting SmartSend for subject: ${subject}`);
// Serialize data based on type
const payloadBytes = _serializeData(data, type, correlationId);
const payloadSize = payloadBytes.length;
logTrace(correlationId, `Serialized payload size: ${payloadSize} bytes`);
// Decision: Direct vs Link
if (payloadSize < sizeThreshold) {
// Direct path - Base64 encode and send via NATS
const payloadBase64 = encode(payloadBytes);
logTrace(correlationId, `Using direct transport for ${payloadSize} bytes`);
const env = new MessageEnvelope({
correlation_id: correlationId,
type: type,
transport: 'direct',
payload: payloadBase64,
metadata: {
content_length: payloadSize.toString(),
format: 'arrow_ipc_stream'
}
});
await publishMessage(natsUrl, subject, env.toJSON(), correlationId);
return env;
} else {
// Link path - Upload to HTTP server, send URL via NATS
logTrace(correlationId, `Using link transport, uploading to fileserver`);
const url = await uploadToServer(payloadBytes, fileserverUrl, correlationId);
const env = new MessageEnvelope({
correlation_id: correlationId,
type: type,
transport: 'link',
url: url,
metadata: {
content_length: payloadSize.toString(),
format: 'arrow_ipc_stream'
}
});
await publishMessage(natsUrl, subject, env.toJSON(), correlationId);
return env;
}
}
// Helper: Serialize data based on type
function _serializeData(data, type, correlationId) {
if (type === 'json') {
const jsonStr = JSON.stringify(data);
return Buffer.from(jsonStr, 'utf8');
} else if (type === 'table') {
// Table data - convert to Arrow IPC stream
const writer = new Arrow.Writer();
writer.writeTable(data);
return writer.toByteArray();
} else if (type === 'binary') {
// Binary data - treat as binary
if (data instanceof Buffer) {
return data;
} else if (Array.isArray(data)) {
return Buffer.from(data);
} else {
throw new Error('Binary data must be binary (Buffer or Array)');
}
} else {
throw new Error(`Unknown type: ${type}`);
}
}
// Helper: Publish message to NATS
async function publishMessage(natsUrl, subject, message, correlationId) {
const { connect } = require('nats');
try {
const nc = await connect({ servers: [natsUrl] });
await nc.publish(subject, message);
logTrace(correlationId, `Message published to ${subject}`);
nc.close();
} catch (error) {
logTrace(correlationId, `Failed to publish message: ${error.message}`);
throw error;
}
}
// SmartReceive for JavaScript - Handles both direct and link transport
async function SmartReceive(msg, options = {}) {
const {
fileserverUrl = DEFAULT_FILESERVER_URL,
maxRetries = 5,
baseDelay = 100,
maxDelay = 5000
} = options;
const env = MessageEnvelope.fromJSON(msg.data);
logTrace(env.correlation_id, `Processing received message`);
if (env.transport === 'direct') {
logTrace(env.correlation_id, `Direct transport - decoding payload`);
const payloadBytes = decode(env.payload);
const data = _deserializeData(payloadBytes, env.type, env.correlation_id, env.metadata);
return { data, envelope: env };
} else if (env.transport === 'link') {
logTrace(env.correlation_id, `Link transport - fetching from URL`);
const data = await _fetchWithBackoff(env.url, maxRetries, baseDelay, maxDelay, env.correlation_id);
const result = _deserializeData(data, env.type, env.correlation_id, env.metadata);
return { data: result, envelope: env };
} else {
throw new Error(`Unknown transport type: ${env.transport}`);
}
}
// Helper: Fetch with exponential backoff
async function _fetchWithBackoff(url, maxRetries, baseDelay, maxDelay, correlationId) {
let delay = baseDelay;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
const response = await fetch(url);
if (response.ok) {
const buffer = await response.arrayBuffer();
logTrace(correlationId, `Successfully fetched data from ${url} on attempt ${attempt}`);
return new Uint8Array(buffer);
} else {
throw new Error(`Failed to fetch: ${response.status}`);
}
} catch (error) {
logTrace(correlationId, `Attempt ${attempt} failed: ${error.message}`);
if (attempt < maxRetries) {
await new Promise(resolve => setTimeout(resolve, delay));
delay = Math.min(delay * 2, maxDelay);
}
}
}
throw new Error(`Failed to fetch data after ${maxRetries} attempts`);
}
// Helper: Deserialize data based on type
async function _deserializeData(data, type, correlationId, metadata) {
if (type === 'json') {
const jsonStr = new TextDecoder().decode(data);
return JSON.parse(jsonStr);
} else if (type === 'table') {
// Deserialize Arrow IPC stream to Table
const table = Arrow.Table.from(data);
return table;
} else if (type === 'binary') {
// Return binary binary data
return data;
} else {
throw new Error(`Unknown type: ${type}`);
}
}
// Export functions
module.exports = {
SmartSend,
SmartReceive,
MessageEnvelope
};

674
src/natbridge.js Normal file
View File

@@ -0,0 +1,674 @@
/**
* NATSBridge - Cross-Platform Bi-Directional Data Bridge
* JavaScript/Node.js Implementation
*
* This module provides functionality for sending and receiving data across network boundaries
* using NATS as the message bus, with support for both direct payload transport and
* URL-based transport for larger payloads.
*
* @module NATSBridge
*/
const nats = require('nats');
const { v4: uuidv4 } = require('uuid');
const fetch = require('node-fetch');
const arrow = require('apache-arrow');
// ---------------------------------------------- Constants ---------------------------------------------- //
/**
* Default size threshold for switching from direct to link transport (1MB)
*/
const DEFAULT_SIZE_THRESHOLD = 1_000_000;
/**
* Default NATS server URL
*/
const DEFAULT_BROKER_URL = 'nats://localhost:4222';
/**
* Default HTTP file server URL for link transport
*/
const DEFAULT_FILESERVER_URL = 'http://localhost:8080';
// ---------------------------------------------- Utility Functions ---------------------------------------------- //
/**
* Convert Buffer to Base64 string
* @param {Buffer} buffer - Buffer to encode
* @returns {string} Base64 encoded string
*/
function bufferToBase64(buffer) {
return buffer.toString('base64');
}
/**
* Log a trace message with correlation ID and timestamp
* @param {string} correlationId - Correlation ID for tracing
* @param {string} message - Message content to log
*/
function logTrace(correlationId, message) {
const timestamp = new Date().toISOString();
console.log(`[${timestamp}] [Correlation: ${correlationId}] ${message}`);
}
// ---------------------------------------------- Serialization Functions ---------------------------------------------- //
/**
* Serialize data according to specified format
* @param {any} data - Data to serialize
* @param {string} payloadType - Target format: "text", "dictionary", "table", "image", "audio", "video", "binary"
* @returns {Buffer} Binary representation of the serialized data
*/
async function serializeData(data, payloadType) {
if (payloadType === 'text') {
if (typeof data === 'string') {
return Buffer.from(data, 'utf8');
} else {
throw new Error('Text data must be a string');
}
} else if (payloadType === 'dictionary') {
const jsonStr = JSON.stringify(data);
return Buffer.from(jsonStr, 'utf8');
} else if (payloadType === 'table') {
// Convert array of objects to Arrow IPC format
if (!Array.isArray(data) || data.length === 0) {
throw new Error('Table data must be a non-empty array of objects');
}
return serializeArrowTable(data);
} else if (payloadType === 'image') {
if (data instanceof Uint8Array || Buffer.isBuffer(data)) {
return Buffer.from(data);
} else {
throw new Error('Image data must be Uint8Array or Buffer');
}
} else if (payloadType === 'audio') {
if (data instanceof Uint8Array || Buffer.isBuffer(data)) {
return Buffer.from(data);
} else {
throw new Error('Audio data must be Uint8Array or Buffer');
}
} else if (payloadType === 'video') {
if (data instanceof Uint8Array || Buffer.isBuffer(data)) {
return Buffer.from(data);
} else {
throw new Error('Video data must be Uint8Array or Buffer');
}
} else if (payloadType === 'binary') {
if (data instanceof Uint8Array || Buffer.isBuffer(data)) {
return Buffer.from(data);
} else {
throw new Error('Binary data must be Uint8Array or Buffer');
}
} else {
throw new Error(`Unknown payload_type: ${payloadType}`);
}
}
/**
* Helper function to properly serialize table data to Arrow IPC
* @param {Array<Object>} data - Array of objects representing table rows
* @returns {Buffer} Arrow IPC formatted buffer
*/
function serializeArrowTable(data) {
if (!Array.isArray(data) || data.length === 0) {
throw new Error('Table data must be a non-empty array of objects');
}
// Build schema from first row
const fields = Object.keys(data[0]).map(key => {
const value = data[0][key];
let arrowType;
if (typeof value === 'number') {
arrowType = Number.isInteger(value) ? arrow.Int64 : arrow.Float64;
} else if (typeof value === 'boolean') {
arrowType = arrow.Boolean;
} else if (value instanceof Date) {
arrowType = arrow.Date;
} else {
arrowType = arrow.Utf8;
}
return new arrow.Field(key, arrowType, true);
});
const schema = new arrow.Schema(fields);
const batches = [];
// Create record batches
for (const row of data) {
const batch = arrow.recordBatch.fromObjects([row], schema);
batches.push(batch);
}
// Write to buffer using IPC format
const buffers = arrow.ipc.recordBatchesToMessage(batches, schema).buffers;
const combined = new Uint8Array(buffers.reduce((acc, b) => acc + b.byteLength, 0));
let offset = 0;
for (const buf of buffers) {
combined.set(new Uint8Array(buf), offset);
offset += buf.byteLength;
}
return Buffer.from(combined);
}
/**
* Deserialize bytes to data based on type
* @param {Buffer|Uint8Array} data - Serialized data as bytes
* @param {string} payloadType - Data type
* @param {string} correlationId - Correlation ID for logging
* @returns {any} Deserialized data
*/
async function deserializeData(data, payloadType, correlationId) {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data);
if (payloadType === 'text') {
return buffer.toString('utf8');
} else if (payloadType === 'dictionary') {
const jsonStr = buffer.toString('utf8');
return JSON.parse(jsonStr);
} else if (payloadType === 'table') {
const table = arrow.tableFromRawBytes(buffer);
return table;
} else if (payloadType === 'image') {
return buffer;
} else if (payloadType === 'audio') {
return buffer;
} else if (payloadType === 'video') {
return buffer;
} else if (payloadType === 'binary') {
return buffer;
} else {
throw new Error(`Unknown payload_type: ${payloadType}`);
}
}
// ---------------------------------------------- File Server Handlers ---------------------------------------------- //
/**
* Upload data to plik server in one-shot mode
* @param {string} fileServerUrl - Base URL of the plik server
* @param {string} dataname - Name of the file being uploaded
* @param {Buffer|Uint8Array} data - Raw byte data of the file content
* @returns {Promise<{status: number, uploadid: string, fileid: string, url: string}>}
*/
async function plikOneshotUpload(fileServerUrl, dataname, data) {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data);
// Get upload id
const urlGetUploadID = `${fileServerUrl}/upload`;
const headers = { 'Content-Type': 'application/json' };
const body = JSON.stringify({ OneShot: true });
const httpResponse = await fetch(urlGetUploadID, {
method: 'POST',
headers,
body
});
const responseJson = await httpResponse.json();
const uploadid = responseJson.id;
const uploadtoken = responseJson.uploadToken;
// Upload file
const urlUpload = `${fileServerUrl}/file/${uploadid}`;
const form = new FormData();
const blob = new Blob([buffer], { type: 'application/octet-stream' });
form.append('file', blob, dataname);
const uploadHeaders = {
'X-UploadToken': uploadtoken
};
const uploadResponse = await fetch(urlUpload, {
method: 'POST',
headers: uploadHeaders,
body: form
});
const uploadJson = await uploadResponse.json();
const fileid = uploadJson.id;
const url = `${fileServerUrl}/file/${uploadid}/${fileid}/${dataname}`;
return {
status: uploadResponse.status,
uploadid,
fileid,
url
};
}
/**
* Fetch data from URL with exponential backoff
* @param {string} url - URL to fetch from
* @param {number} maxRetries - Maximum number of retry attempts
* @param {number} baseDelay - Initial delay in milliseconds
* @param {number} maxDelay - Maximum delay in milliseconds
* @param {string} correlationId - Correlation ID for logging
* @returns {Promise<Uint8Array>} Fetched data as bytes
*/
async function fetchWithBackoff(url, maxRetries, baseDelay, maxDelay, correlationId) {
let delay = baseDelay;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
const response = await fetch(url);
if (response.status === 200) {
logTrace(correlationId, `Successfully fetched data from ${url} on attempt ${attempt}`);
const arrayBuffer = await response.arrayBuffer();
return new Uint8Array(arrayBuffer);
} else {
throw new Error(`Failed to fetch: ${response.status}`);
}
} catch (e) {
logTrace(correlationId, `Attempt ${attempt} failed: ${e.constructor.name}`);
if (attempt < maxRetries) {
await new Promise(resolve => setTimeout(resolve, delay));
delay = Math.min(delay * 2, maxDelay);
}
}
}
throw new Error(`Failed to fetch data after ${maxRetries} attempts`);
}
// ---------------------------------------------- NATS Client ---------------------------------------------- //
/**
* NATS client wrapper for connection management
*/
class NATSClient {
/**
* Create a new NATS client
* @param {string} url - NATS server URL
*/
constructor(url) {
this.url = url;
this.connection = null;
}
/**
* Connect to NATS server
* @returns {Promise<NATS.Connection>}
*/
async connect() {
this.connection = await nats.connect({ servers: this.url });
return this.connection;
}
/**
* Publish message to NATS subject
* @param {string} subject - NATS subject to publish to
* @param {string} message - Message to publish
* @param {string} correlationId - Correlation ID for logging
*/
async publish(subject, message, correlationId) {
if (!this.connection) {
await this.connect();
}
await this.connection.publish(subject, message);
logTrace(correlationId, `Message published to ${subject}`);
}
/**
* Close the NATS connection
*/
async close() {
if (this.connection) {
this.connection.close();
}
}
}
// ---------------------------------------------- Core Functions ---------------------------------------------- //
/**
* Publish message to NATS
* @param {string|NATSClient|NATS.Connection} brokerUrlOrClient - NATS URL, client, or connection
* @param {string} subject - NATS subject to publish to
* @param {string} message - JSON message to publish
* @param {string} correlationId - Correlation ID for tracing
*/
async function publishMessage(brokerUrlOrClient, subject, message, correlationId) {
let conn;
if (brokerUrlOrClient instanceof NATSClient) {
conn = brokerUrlOrClient;
} else if (brokerUrlOrClient instanceof nats.Connection) {
// Create a wrapper for direct connection
conn = {
async publish(subj, msg) {
await brokerUrlOrClient.publish(subj, msg);
},
async close() {
await brokerUrlOrClient.close();
}
};
} else {
// String URL - create new client
const client = new NATSClient(brokerUrlOrClient);
conn = client;
}
await conn.publish(subject, message, correlationId);
if (conn instanceof NATSClient) {
await conn.close();
}
}
/**
* Build message envelope from payloads and metadata
* @param {string} subject - NATS subject
* @param {Array} payloads - Array of payload objects
* @param {Object} options - Envelope metadata options
* @returns {Object} Envelope object
*/
function buildEnvelope(subject, payloads, options) {
return {
correlation_id: options.correlation_id,
msg_id: options.msg_id,
timestamp: new Date().toISOString(),
send_to: subject,
msg_purpose: options.msg_purpose,
sender_name: options.sender_name,
sender_id: options.sender_id,
receiver_name: options.receiver_name,
receiver_id: options.receiver_id,
reply_to: options.reply_to,
reply_to_msg_id: options.reply_to_msg_id,
broker_url: options.broker_url,
metadata: options.metadata || {},
payloads: payloads
};
}
/**
* Build payload object from serialized data
* @param {string} dataname - Name of the payload
* @param {string} payloadType - Type of the payload
* @param {Buffer} payloadBytes - Serialized payload bytes
* @param {string} transport - Transport type ("direct" or "link")
* @param {string} data - Data (base64 for direct, URL for link)
* @returns {Object} Payload object
*/
function buildPayload(dataname, payloadType, payloadBytes, transport, data) {
return {
id: uuidv4(),
dataname,
payload_type: payloadType,
transport,
encoding: transport === 'direct' ? 'base64' : 'none',
size: payloadBytes.byteLength,
data,
metadata: transport === 'direct' ? { payload_bytes: payloadBytes.byteLength } : {}
};
}
/**
* Send data via NATS with automatic transport selection
*
* This function intelligently routes data delivery based on payload size.
* If the serialized payload is smaller than size_threshold, it encodes the data as Base64
* and publishes directly over NATS. Otherwise, it uploads the data to a fileserver
* and publishes only the download URL over NATS.
*
* @param {string} subject - NATS subject to publish the message to
* @param {Array} data - List of [dataname, data, type] tuples to send
* @param {Object} options - Optional configuration
* @param {string} [options.broker_url=DEFAULT_BROKER_URL] - URL of the NATS server
* @param {string} [options.fileserver_url=DEFAULT_FILESERVER_URL] - URL of the HTTP file server
* @param {Function} [options.fileserver_upload_handler=plikOneshotUpload] - Function to handle fileserver uploads
* @param {number} [options.size_threshold=DEFAULT_SIZE_THRESHOLD] - Threshold separating direct vs link transport
* @param {string} [options.correlation_id=uuidv4()] - Correlation ID for tracing
* @param {string} [options.msg_purpose="chat"] - Purpose of the message
* @param {string} [options.sender_name="NATSBridge"] - Name of the sender
* @param {string} [options.receiver_name=""] - Name of the receiver (empty means broadcast)
* @param {string} [options.receiver_id=""] - UUID of the receiver (empty means broadcast)
* @param {string} [options.reply_to=""] - Topic to reply to
* @param {string} [options.reply_to_msg_id=""] - Message ID this message is replying to
* @param {boolean} [options.is_publish=true] - Whether to automatically publish the message
* @param {NATSClient|NATS.Connection} [options.nats_connection=null] - Pre-existing NATS connection
* @param {string} [options.msg_id=uuidv4()] - Message ID
* @param {string} [options.sender_id=uuidv4()] - Sender ID
* @returns {Promise<[Object, string]>} Tuple of [env, env_json_str]
*
* @example
* // Send a single payload
* const [env, envJsonStr] = await smartsend(
* "/test",
* [["dataname1", data1, "dictionary"]],
* { broker_url: "nats://localhost:4222" }
* );
*
* // Send multiple payloads
* const [env, envJsonStr] = await smartsend(
* "/test",
* [
* ["dataname1", data1, "dictionary"],
* ["dataname2", data2, "table"]
* ],
* { broker_url: "nats://localhost:4222" }
* );
*
* // Send with pre-existing connection
* const client = await NATSBridge.NATSClient.connect("nats://localhost:4222");
* const [env, envJsonStr] = await smartsend(
* "/test",
* [["data", myData, "text"]],
* { nats_connection: client }
* );
*/
async function smartsend(subject, data, options = {}) {
const {
broker_url = DEFAULT_BROKER_URL,
fileserver_url = DEFAULT_FILESERVER_URL,
fileserver_upload_handler = plikOneshotUpload,
size_threshold = DEFAULT_SIZE_THRESHOLD,
correlation_id = uuidv4(),
msg_purpose = 'chat',
sender_name = 'NATSBridge',
receiver_name = '',
receiver_id = '',
reply_to = '',
reply_to_msg_id = '',
is_publish = true,
nats_connection = null,
msg_id = uuidv4(),
sender_id = uuidv4()
} = options;
logTrace(correlation_id, `Starting smartsend for subject: ${subject}`);
// Process payloads
const payloads = [];
for (const [dataname, payloadData, payloadType] of data) {
const payloadBytes = await serializeData(payloadData, payloadType);
const payloadSize = payloadBytes.byteLength;
logTrace(correlation_id, `Serialized payload '${dataname}' (type: ${payloadType}) size: ${payloadSize} bytes`);
if (payloadSize < size_threshold) {
// Direct path
const payloadB64 = bufferToBase64(payloadBytes);
logTrace(correlation_id, `Using direct transport for ${payloadSize} bytes`);
const payload = buildPayload(dataname, payloadType, payloadBytes, 'direct', payloadB64);
payloads.push(payload);
} else {
// Link path
logTrace(correlation_id, `Using link transport, uploading to fileserver`);
const response = await fileserver_upload_handler(fileserver_url, dataname, payloadBytes);
if (response.status !== 200) {
throw new Error(`Failed to upload data to fileserver: ${response.status}`);
}
logTrace(correlation_id, `Uploaded to URL: ${response.url}`);
const payload = buildPayload(dataname, payloadType, payloadBytes, 'link', response.url);
payloads.push(payload);
}
}
// Build envelope
const env = buildEnvelope(subject, payloads, {
correlation_id,
msg_id,
msg_purpose,
sender_name,
sender_id,
receiver_name,
receiver_id,
reply_to,
reply_to_msg_id,
broker_url
});
const env_json_str = JSON.stringify(env);
if (is_publish) {
if (nats_connection) {
await publishMessage(nats_connection, subject, env_json_str, correlation_id);
} else {
await publishMessage(broker_url, subject, env_json_str, correlation_id);
}
}
return [env, env_json_str];
}
/**
* Receive and process NATS message
*
* This function processes incoming NATS messages, handling both direct transport
* (base64 decoded payloads) and link transport (URL-based payloads).
* It deserializes the data based on the transport type and returns the result.
*
* @param {Object} msg - NATS message object with payload property
* @param {Object} options - Optional configuration
* @param {Function} [options.fileserver_download_handler=fetchWithBackoff] - Function to handle fileserver downloads
* @param {number} [options.max_retries=5] - Maximum retry attempts for fetching URL
* @param {number} [options.base_delay=100] - Initial delay for exponential backoff in ms
* @param {number} [options.max_delay=5000] - Maximum delay for exponential backoff in ms
* @returns {Promise<Object>} Envelope object with processed payloads
*
* @example
* // Receive and process message
* const env = await smartreceive(msg, {
* fileserver_download_handler: fetchWithBackoff,
* max_retries: 5,
* base_delay: 100,
* max_delay: 5000
* });
* // env.payloads is an Array of [dataname, data, type] arrays
* for (const [dataname, data, type] of env.payloads) {
* console.log(`${dataname}: ${data} (type: ${type})`);
* }
*/
async function smartreceive(msg, options = {}) {
const {
fileserver_download_handler = fetchWithBackoff,
max_retries = 5,
base_delay = 100,
max_delay = 5000
} = options;
// Parse the JSON envelope
const payload = typeof msg.payload === 'string' ? msg.payload : Buffer.from(msg.payload).toString('utf8');
const envJsonObj = JSON.parse(payload);
logTrace(envJsonObj.correlation_id, 'Processing received message');
// Process all payloads in the envelope
const payloadsList = [];
const numPayloads = envJsonObj.payloads.length;
for (let i = 0; i < numPayloads; i++) {
const payloadObj = envJsonObj.payloads[i];
const transport = payloadObj.transport;
const dataname = payloadObj.dataname;
if (transport === 'direct') {
logTrace(envJsonObj.correlation_id, `Direct transport - decoding payload '${dataname}'`);
// Extract base64 payload from the payload
const payloadB64 = payloadObj.data;
// Decode Base64 payload
const payloadBytes = Buffer.from(payloadB64, 'base64');
// Deserialize based on type
const dataType = payloadObj.payload_type;
const data = await deserializeData(payloadBytes, dataType, envJsonObj.correlation_id);
payloadsList.push([dataname, data, dataType]);
} else if (transport === 'link') {
// Extract download URL from the payload
const url = payloadObj.data;
logTrace(envJsonObj.correlation_id, `Link transport - fetching '${dataname}' from URL: ${url}`);
// Fetch with exponential backoff using the download handler
const downloadedData = await fileserver_download_handler(
url,
max_retries,
base_delay,
max_delay,
envJsonObj.correlation_id
);
// Deserialize based on type
const dataType = payloadObj.payload_type;
const data = await deserializeData(downloadedData, dataType, envJsonObj.correlation_id);
payloadsList.push([dataname, data, dataType]);
} else {
throw new Error(`Unknown transport type for payload '${dataname}': ${transport}`);
}
}
envJsonObj.payloads = payloadsList;
return envJsonObj;
}
// ---------------------------------------------- Module Exports ---------------------------------------------- //
const NATSBridge = {
/**
* NATS client class for connection management
*/
NATSClient,
/**
* Send data via NATS with automatic transport selection
*/
smartsend,
/**
* Receive and process NATS message
*/
smartreceive,
/**
* Upload data to plik server in one-shot mode
*/
plikOneshotUpload,
/**
* Fetch data from URL with exponential backoff
*/
fetchWithBackoff,
/**
* Default constants
*/
DEFAULT_SIZE_THRESHOLD,
DEFAULT_BROKER_URL,
DEFAULT_FILESERVER_URL
};
module.exports = NATSBridge;

815
src/natbridge.py Normal file
View File

@@ -0,0 +1,815 @@
"""
NATSBridge - Cross-Platform Bi-Directional Data Bridge
Python Desktop Implementation
This module provides functionality for sending and receiving data across network boundaries
using NATS as the message bus, with support for both direct payload transport and
URL-based transport for larger payloads.
@package natbridge
"""
import asyncio
import base64
import json
import uuid
from datetime import datetime
from typing import Any, Callable, Dict, List, Tuple, Union
import aiohttp
try:
import pyarrow as arrow
import pyarrow.ipc as ipc
ARROW_AVAILABLE = True
except ImportError:
ARROW_AVAILABLE = False
try:
import nats
from nats.aio.client import Client as NATSClient
NATS_AVAILABLE = True
except ImportError:
NATS_AVAILABLE = False
# ---------------------------------------------- Constants ---------------------------------------------- #
"""
Default size threshold for switching from direct to link transport (1MB)
"""
DEFAULT_SIZE_THRESHOLD = 1_000_000
"""
Default NATS server URL
"""
DEFAULT_BROKER_URL = "nats://localhost:4222"
"""
Default HTTP file server URL for link transport
"""
DEFAULT_FILESERVER_URL = "http://localhost:8080"
# ---------------------------------------------- Utility Functions ---------------------------------------------- #
def log_trace(correlation_id: str, message: str) -> None:
"""
Log a trace message with correlation ID and timestamp.
Args:
correlation_id: Correlation ID for tracing
message: Message content to log
"""
timestamp = datetime.utcnow().isoformat() + 'Z'
print(f"[{timestamp}] [Correlation: {correlation_id}] {message}")
# ---------------------------------------------- Serialization Functions ---------------------------------------------- #
def _serialize_data(data: Any, payload_type: str) -> bytes:
"""
Serialize data according to specified format.
Args:
data: Data to serialize (string for "text", JSON-serializable for "dictionary",
table-like for "table", binary for "image", "audio", "video", "binary")
payload_type: Target format: "text", "dictionary", "table", "image", "audio", "video", "binary"
Returns:
Binary representation of the serialized data
Raises:
Error: If payload_type is not one of the supported types
Error: If payload_type is "image", "audio", or "video" but data is not bytes
Error: If payload_type is "table" but data is not a pandas DataFrame or pyarrow Table
"""
if payload_type == 'text':
if isinstance(data, str):
return data.encode('utf-8')
else:
raise ValueError('Text data must be a string')
elif payload_type == 'dictionary':
json_str = json.dumps(data)
return json_str.encode('utf-8')
elif payload_type == 'table':
if not ARROW_AVAILABLE:
raise RuntimeError('pyarrow not available for table serialization')
import io
buf = io.BytesIO()
import pandas as pd
if isinstance(data, pd.DataFrame):
table = arrow.Table.from_pandas(data)
sink = ipc.new_file(buf, table.schema)
ipc.write_table(table, sink)
sink.close()
return buf.getvalue()
elif isinstance(data, arrow.Table):
sink = ipc.new_file(buf, data.schema)
ipc.write_table(data, sink)
sink.close()
return buf.getvalue()
else:
raise ValueError('Table data must be a pandas DataFrame or pyarrow Table')
elif payload_type == 'image':
if isinstance(data, (bytes, bytearray)):
return bytes(data)
else:
raise ValueError('Image data must be bytes')
elif payload_type == 'audio':
if isinstance(data, (bytes, bytearray)):
return bytes(data)
else:
raise ValueError('Audio data must be bytes')
elif payload_type == 'video':
if isinstance(data, (bytes, bytearray)):
return bytes(data)
else:
raise ValueError('Video data must be bytes')
elif payload_type == 'binary':
if isinstance(data, (bytes, bytearray)):
return bytes(data)
else:
raise ValueError('Binary data must be bytes')
else:
raise ValueError(f'Unknown payload_type: {payload_type}')
def _deserialize_data(data: bytes, payload_type: str, correlation_id: str) -> Any:
"""
Deserialize bytes to data based on type.
Args:
data: Serialized data as bytes
payload_type: Data type ("text", "dictionary", "table", "image", "audio", "video", "binary")
correlation_id: Correlation ID for logging
Returns:
Deserialized data (String for "text", DataFrame for "table", JSON data for "dictionary",
bytes for "image", "audio", "video", "binary")
Raises:
Error: If payload_type is not one of the supported types
"""
if payload_type == 'text':
return data.decode('utf-8')
elif payload_type == 'dictionary':
json_str = data.decode('utf-8')
return json.loads(json_str)
elif payload_type == 'table':
if not ARROW_AVAILABLE:
raise RuntimeError('pyarrow not available for table deserialization')
import io
buf = io.BytesIO(data)
reader = ipc.open_file(buf)
return reader.read_all().to_pandas()
elif payload_type == 'image':
return data
elif payload_type == 'audio':
return data
elif payload_type == 'video':
return data
elif payload_type == 'binary':
return data
else:
raise ValueError(f'Unknown payload_type: {payload_type}')
# ---------------------------------------------- File Server Handlers ---------------------------------------------- #
async def plik_oneshot_upload(
file_server_url: str,
dataname: str,
data: bytes
) -> Dict[str, Any]:
"""
Upload data to plik server in one-shot mode.
This function uploads a raw byte array to a plik server in one-shot mode (no upload session).
It first creates a one-shot upload session by sending a POST request with {"OneShot": true},
retrieves an upload ID and token, then uploads the file data as multipart form data using the token.
Args:
file_server_url: Base URL of the plik server (e.g., "http://localhost:8080")
dataname: Name of the file being uploaded
data: Raw byte data of the file content
Returns:
Dict with keys:
- "status": HTTP server response status
- "uploadid": ID of the one-shot upload session
- "fileid": ID of the uploaded file within the session
- "url": Full URL to download the uploaded file
Example:
>>> fileserver_url = "http://localhost:8080"
>>> dataname = "test.txt"
>>> data = b"hello world"
>>> result = await plik_oneshot_upload(file_server_url, dataname, data)
>>> result["status"], result["uploadid"], result["fileid"], result["url"]
"""
async with aiohttp.ClientSession() as session:
# Get upload id
url_getUploadID = f"{file_server_url}/upload"
headers = {'Content-Type': 'application/json'}
body = json.dumps({"OneShot": True})
async with session.post(url_getUploadID, headers=headers, data=body) as response:
response_json = await response.json()
uploadid = response_json['id']
uploadtoken = response_json['uploadToken']
# Upload file
url_upload = f"{file_server_url}/file/{uploadid}"
headers = {'X-UploadToken': uploadtoken}
form = aiohttp.FormData()
form.add_field('file', data, filename=dataname, content_type='application/octet-stream')
async with session.post(url_upload, headers=headers, data=form) as upload_response:
upload_json = await upload_response.json()
fileid = upload_json['id']
url = f"{file_server_url}/file/{uploadid}/{fileid}/{dataname}"
return {
'status': upload_response.status,
'uploadid': uploadid,
'fileid': fileid,
'url': url
}
async def fetch_with_backoff(
url: str,
max_retries: int,
base_delay: int,
max_delay: int,
correlation_id: str
) -> bytes:
"""
Fetch data from URL with exponential backoff.
This internal function retrieves data from a URL with retry logic using
exponential backoff to handle transient failures.
Args:
url: URL to fetch from
max_retries: Maximum number of retry attempts
base_delay: Initial delay in milliseconds
max_delay: Maximum delay in milliseconds
correlation_id: Correlation ID for logging
Returns:
Fetched data as bytes
Raises:
Error: If all retry attempts fail
Example:
>>> data = await fetch_with_backoff("http://example.com/file.zip", 5, 100, 5000, "correlation123")
"""
delay = base_delay
for attempt in range(1, max_retries + 1):
try:
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
if response.status == 200:
log_trace(correlation_id, f"Successfully fetched data from {url} on attempt {attempt}")
return await response.read()
else:
raise Exception(f"Failed to fetch: {response.status}")
except Exception as e:
log_trace(correlation_id, f"Attempt {attempt} failed: {type(e).__name__}")
if attempt < max_retries:
await asyncio.sleep(delay / 1000.0)
delay = min(delay * 2, max_delay)
raise Exception(f"Failed to fetch data after {max_retries} attempts")
# ---------------------------------------------- NATS Client ---------------------------------------------- #
class NATSClient:
"""NATS client wrapper for connection management."""
def __init__(self, url: str = DEFAULT_BROKER_URL):
"""
Create a new NATS client.
Args:
url: NATS server URL
"""
self.url = url
self._client: NATSClient = None
async def connect(self) -> NATSClient:
"""
Connect to NATS server.
Returns:
NATS client instance
"""
if NATS_AVAILABLE:
self._client = nats.connect(self.url)
await self._client
else:
raise Error('nats-py not available')
return self._client
async def publish(self, subject: str, message: str, correlation_id: str = "") -> None:
"""
Publish message to NATS subject.
Args:
subject: NATS subject to publish to
message: Message to publish
correlation_id: Correlation ID for logging
"""
if self._client:
await self._client.publish(subject, message)
if correlation_id:
log_trace(correlation_id, f"Message published to {subject}")
async def close(self) -> None:
"""Close the NATS connection."""
if self._client:
await self._client.drain()
await self._client.close()
# ---------------------------------------------- Core Functions ---------------------------------------------- #
def _build_envelope(
subject: str,
payloads: List[Dict[str, Any]],
options: Dict[str, Any]
) -> Dict[str, Any]:
"""
Build message envelope from payloads and metadata.
Args:
subject: NATS subject
payloads: Array of payload objects
options: Envelope metadata options
Returns:
Envelope object
"""
return {
'correlation_id': options['correlation_id'],
'msg_id': options['msg_id'],
'timestamp': datetime.utcnow().isoformat() + 'Z',
'send_to': subject,
'msg_purpose': options['msg_purpose'],
'sender_name': options['sender_name'],
'sender_id': options['sender_id'],
'receiver_name': options['receiver_name'],
'receiver_id': options['receiver_id'],
'reply_to': options['reply_to'],
'reply_to_msg_id': options['reply_to_msg_id'],
'broker_url': options['broker_url'],
'metadata': options.get('metadata', {}),
'payloads': payloads
}
def _build_payload(
dataname: str,
payload_type: str,
payload_bytes: bytes,
transport: str,
data: Union[str, bytes]
) -> Dict[str, Any]:
"""
Build payload object from serialized data.
Args:
dataname: Name of the payload
payload_type: Type of the payload
payload_bytes: Serialized payload bytes
transport: Transport type ("direct" or "link")
data: Data (base64 for direct, URL for link)
Returns:
Payload object
"""
return {
'id': str(uuid.uuid4()),
'dataname': dataname,
'payload_type': payload_type,
'transport': transport,
'encoding': 'base64' if transport == 'direct' else 'none',
'size': len(payload_bytes),
'data': data,
'metadata': {'payload_bytes': len(payload_bytes)} if transport == 'direct' else {}
}
async def publish_message(
broker_url_or_client: Union[str, NATSClient, Any],
subject: str,
message: str,
correlation_id: str
) -> None:
"""
Publish message to NATS.
Args:
broker_url_or_client: NATS URL, client, or connection
subject: NATS subject to publish to
message: JSON message to publish
correlation_id: Correlation ID for tracing
"""
if isinstance(broker_url_or_client, NATSClient):
client = broker_url_or_client
elif NATS_AVAILABLE and hasattr(broker_url_or_client, 'publish'):
# Direct NATS client connection
await broker_url_or_client.publish(subject, message)
log_trace(correlation_id, f"Message published to {subject}")
return
else:
# String URL - create new client
client = NATSClient(broker_url_or_client)
await client.connect()
await client.publish(subject, message, correlation_id)
if isinstance(broker_url_or_client, NATSClient):
await broker_url_or_client.close()
elif not (NATS_AVAILABLE and hasattr(broker_url_or_client, 'publish')):
await client.close()
async def smartsend(
subject: str,
data: List[Tuple[str, Any, str]],
broker_url: str = DEFAULT_BROKER_URL,
fileserver_url: str = DEFAULT_FILESERVER_URL,
fileserver_upload_handler: Callable = plik_oneshot_upload,
size_threshold: int = DEFAULT_SIZE_THRESHOLD,
correlation_id: str = None,
msg_purpose: str = "chat",
sender_name: str = "NATSBridge",
receiver_name: str = "",
receiver_id: str = "",
reply_to: str = "",
reply_to_msg_id: str = "",
is_publish: bool = True,
nats_connection: Any = None,
msg_id: str = None,
sender_id: str = None
) -> Tuple[Dict, str]:
"""
Send data via NATS with automatic transport selection.
This function intelligently routes data delivery based on payload size.
If the serialized payload is smaller than size_threshold, it encodes the data as Base64
and publishes directly over NATS. Otherwise, it uploads the data to a fileserver
and publishes only the download URL over NATS.
Args:
subject: NATS subject to publish the message to
data: List of (dataname, data, type) tuples to send
- dataname: Name of the payload
- data: The actual data to send
- type: Payload type: "text", "dictionary", "table", "image", "audio", "video", "binary"
broker_url: URL of the NATS server
fileserver_url: URL of the HTTP file server for large payloads
fileserver_upload_handler: Function to handle fileserver uploads (must return Dict with "status",
"uploadid", "fileid", "url" keys)
size_threshold: Threshold in bytes separating direct vs link transport
correlation_id: Correlation ID for tracing (auto-generated UUID if not provided)
msg_purpose: Purpose of the message: "ACK", "NACK", "updateStatus", "shutdown", "chat", etc.
sender_name: Name of the sender
receiver_name: Name of the receiver (empty string means broadcast)
receiver_id: UUID of the receiver (empty string means broadcast)
reply_to: Topic to reply to (empty string if no reply expected)
reply_to_msg_id: Message ID this message is replying to
is_publish: Whether to automatically publish the message to NATS
nats_connection: Pre-existing NATS connection (if provided, uses this connection instead of
creating a new one; saves connection establishment overhead)
msg_id: Message ID (auto-generated UUID if not provided)
sender_id: Sender ID (auto-generated UUID if not provided)
Returns:
Tuple of (env, env_json_str) where:
- env: Dict containing all metadata and payloads
- env_json_str: JSON string for publishing to NATS
Example:
>>> # Send a single payload (still wrapped in a list)
>>> data = {"key": "value"}
>>> env, env_json_str = await smartsend(
... "my.subject",
... [("dataname1", data, "dictionary")],
... broker_url="nats://localhost:4222"
... )
>>>
>>> # Send multiple payloads with different types
>>> data1 = {"key1": "value1"}
>>> data2 = [1, 2, 3, 4, 5]
>>> env, env_json_str = await smartsend(
... "my.subject",
... [("dataname1", data1, "dictionary"), ("dataname2", data2, "table")]
... )
>>>
>>> # Send a large array using fileserver upload
>>> data = list(range(10_000_000)) # ~80 MB
>>> env, env_json_str = await smartsend(
... "large.data",
... [("large_table", data, "table")]
... )
>>>
>>> # Mixed content (e.g., chat with text and image)
>>> env, env_json_str = await smartsend(
... "chat.subject",
... [
... ("message_text", "Hello!", "text"),
... ("user_image", image_data, "image"),
... ("audio_clip", audio_data, "audio")
... ]
... )
>>>
>>> # Publish the JSON string directly using NATS request-reply pattern
>>> # reply = await nats.request(broker_url, subject, env_json_str, reply_to=reply_to_topic)
"""
if correlation_id is None:
correlation_id = str(uuid.uuid4())
if msg_id is None:
msg_id = str(uuid.uuid4())
if sender_id is None:
sender_id = str(uuid.uuid4())
log_trace(correlation_id, f"Starting smartsend for subject: {subject}")
# Process payloads
payloads = []
for dataname, payload_data, payload_type in data:
payload_bytes = _serialize_data(payload_data, payload_type)
payload_size = len(payload_bytes)
log_trace(correlation_id, f"Serialized payload '{dataname}' (type: {payload_type}) size: {payload_size} bytes")
if payload_size < size_threshold:
# Direct path
payload_b64 = base64.b64encode(payload_bytes).decode('utf-8')
log_trace(correlation_id, f"Using direct transport for {payload_size} bytes")
payload = _build_payload(dataname, payload_type, payload_bytes, 'direct', payload_b64)
payloads.append(payload)
else:
# Link path
log_trace(correlation_id, "Using link transport, uploading to fileserver")
response = await fileserver_upload_handler(fileserver_url, dataname, payload_bytes)
if response['status'] != 200:
raise Exception(f"Failed to upload data to fileserver: {response['status']}")
log_trace(correlation_id, f"Uploaded to URL: {response['url']}")
payload = _build_payload(dataname, payload_type, payload_bytes, 'link', response['url'])
payloads.append(payload)
# Build envelope
env = _build_envelope(subject, payloads, {
'correlation_id': correlation_id,
'msg_id': msg_id,
'msg_purpose': msg_purpose,
'sender_name': sender_name,
'sender_id': sender_id,
'receiver_name': receiver_name,
'receiver_id': receiver_id,
'reply_to': reply_to,
'reply_to_msg_id': reply_to_msg_id,
'broker_url': broker_url
})
env_json_str = json.dumps(env)
if is_publish:
if nats_connection:
await publish_message(nats_connection, subject, env_json_str, correlation_id)
else:
await publish_message(broker_url, subject, env_json_str, correlation_id)
return env, env_json_str
async def smartreceive(
msg: Any,
fileserver_download_handler: Callable = fetch_with_backoff,
max_retries: int = 5,
base_delay: int = 100,
max_delay: int = 5000
) -> Dict[str, Any]:
"""
Receive and process NATS messages.
This function processes incoming NATS messages, handling both direct transport
(base64 decoded payloads) and link transport (URL-based payloads).
It deserializes the data based on the transport type and returns the result.
Args:
msg: NATS message to process
fileserver_download_handler: Function to handle downloading data from file server URLs
max_retries: Maximum retry attempts for fetching URL
base_delay: Initial delay for exponential backoff in ms
max_delay: Maximum delay for exponential backoff in ms
Returns:
Dict with envelope metadata and payloads field containing List[Tuple[str, Any, str]]
Example:
>>> # Receive and process message
>>> env = await smartreceive(msg, fileserver_download_handler=fetch_with_backoff)
>>> # env is a Dict with "payloads" key containing List[Tuple[str, Any, str]]
>>> # Access payloads: for dataname, data, type_ in env["payloads"]
>>> for dataname, data, type_ in env["payloads"]:
>>> print(f"{dataname}: {data} (type: {type_})")
"""
# Parse the JSON envelope
if isinstance(msg, dict):
# Already parsed
env_json_obj = msg
elif hasattr(msg, 'payload'):
# NATS message object
payload = msg.payload if isinstance(msg.payload, str) else msg.payload.decode('utf-8')
env_json_obj = json.loads(payload)
else:
# Assume it's already a JSON string or dict
env_json_obj = json.loads(msg) if isinstance(msg, str) else msg
log_trace(env_json_obj['correlation_id'], "Processing received message")
# Process all payloads in the envelope
payloads_list = []
num_payloads = len(env_json_obj['payloads'])
for i in range(num_payloads):
payload_obj = env_json_obj['payloads'][i]
transport = payload_obj['transport']
dataname = payload_obj['dataname']
if transport == 'direct':
log_trace(env_json_obj['correlation_id'], f"Direct transport - decoding payload '{dataname}'")
# Extract base64 payload from the payload
payload_b64 = payload_obj['data']
# Decode Base64 payload
payload_bytes = base64.b64decode(payload_b64)
# Deserialize based on type
data_type = payload_obj['payload_type']
data = _deserialize_data(payload_bytes, data_type, env_json_obj['correlation_id'])
payloads_list.append((dataname, data, data_type))
elif transport == 'link':
# Extract download URL from the payload
url = payload_obj['data']
log_trace(env_json_obj['correlation_id'], f"Link transport - fetching '{dataname}' from URL: {url}")
# Fetch with exponential backoff using the download handler
downloaded_data = await fileserver_download_handler(
url,
max_retries,
base_delay,
max_delay,
env_json_obj['correlation_id']
)
# Deserialize based on type
data_type = payload_obj['payload_type']
data = _deserialize_data(downloaded_data, data_type, env_json_obj['correlation_id'])
payloads_list.append((dataname, data, data_type))
else:
raise Exception(f"Unknown transport type for payload '{dataname}': {transport}")
env_json_obj['payloads'] = payloads_list
return env_json_obj
# ---------------------------------------------- Module Exports ---------------------------------------------- #
class NATSBridge:
"""
Cross-platform NATS bridge implementation.
This class provides a convenient interface for NATSBridge functionality,
encapsulating the main functions and providing a class-based API.
"""
DEFAULT_SIZE_THRESHOLD = DEFAULT_SIZE_THRESHOLD
DEFAULT_BROKER_URL = DEFAULT_BROKER_URL
DEFAULT_FILESERVER_URL = DEFAULT_FILESERVER_URL
def __init__(self, broker_url: str = None, fileserver_url: str = None):
"""
Initialize NATSBridge.
Args:
broker_url: NATS server URL (defaults to DEFAULT_BROKER_URL)
fileserver_url: HTTP file server URL (defaults to DEFAULT_FILESERVER_URL)
"""
self.broker_url = broker_url or self.DEFAULT_BROKER_URL
self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL
async def smartsend(
self,
subject: str,
data: List[Tuple[str, Any, str]],
**kwargs
) -> Tuple[Dict, str]:
"""
Send data via NATS.
Args:
subject: NATS subject to publish to
data: List of (dataname, data, type) tuples
**kwargs: Additional options passed to smartsend
Returns:
Tuple of (env, env_json_str)
"""
kwargs['broker_url'] = kwargs.get('broker_url', self.broker_url)
kwargs['fileserver_url'] = kwargs.get('fileserver_url', self.fileserver_url)
return await smartsend(subject, data, **kwargs)
async def smartreceive(
self,
msg: Any,
**kwargs
) -> Dict[str, Any]:
"""
Receive and process NATS message.
Args:
msg: NATS message to process
**kwargs: Additional options passed to smartreceive
Returns:
Dict with envelope metadata and payloads
"""
return await smartreceive(msg, **kwargs)
# Convenience functions for module-level usage
def send(
subject: str,
data: List[Tuple[str, Any, str]],
**kwargs
) -> Tuple[Dict, str]:
"""
Convenience function for sending data.
Args:
subject: NATS subject to publish to
data: List of (dataname, data, type) tuples
**kwargs: Additional options
Returns:
Tuple of (env, env_json_str)
"""
return asyncio.run(smartsend(subject, data, **kwargs))
def receive(
msg: Any,
**kwargs
) -> Dict[str, Any]:
"""
Convenience function for receiving messages.
Args:
msg: NATS message to process
**kwargs: Additional options
Returns:
Dict with envelope metadata and payloads
"""
return asyncio.run(smartreceive(msg, **kwargs))
__all__ = [
'smartsend',
'smartreceive',
'plik_oneshot_upload',
'fetch_with_backoff',
'NATSBridge',
'send',
'receive',
'DEFAULT_SIZE_THRESHOLD',
'DEFAULT_BROKER_URL',
'DEFAULT_FILESERVER_URL',
'NATSClient',
'_serialize_data',
'_deserialize_data',
'log_trace',
'publish_message'
]

673
src/natbridge_mpy.py Normal file
View File

@@ -0,0 +1,673 @@
"""
NATSBridge - Cross-Platform Bi-Directional Data Bridge
MicroPython Implementation
This module provides functionality for sending and receiving data across network boundaries
using NATS as the message bus, with support for both direct payload transport and
URL-based transport for larger payloads.
Note: MicroPython has significant constraints compared to desktop implementations:
- Limited memory (~256KB - 1MB)
- No Arrow IPC support (memory constraints)
- Synchronous API (no async/await)
- Lower size threshold for direct transport
"""
import network
import time
import json
import base64
import uos
import struct
import random
# ---------------------------------------------- Constants ---------------------------------------------- #
"""
Default size threshold for switching from direct to link transport (100KB for MicroPython)
"""
DEFAULT_SIZE_THRESHOLD = 100000
"""
Default NATS server URL
"""
DEFAULT_BROKER_URL = "nats://localhost:4222"
"""
Default HTTP file server URL for link transport
"""
DEFAULT_FILESERVER_URL = "http://localhost:8080"
"""
Hard limit for payload size in MicroPython (50KB)
"""
MAX_PAYLOAD_SIZE = 50000
# ---------------------------------------------- Utility Functions ---------------------------------------------- #
def log_trace(correlation_id, message):
"""
Log a trace message with correlation ID and timestamp.
Args:
correlation_id: Correlation ID for tracing
message: Message content to log
"""
timestamp = time.strftime('%Y-%m-%dT%H:%M:%SZ', time.localtime())
print(f"[{timestamp}] [Correlation: {correlation_id}] {message}")
def _generate_uuid():
"""
Generate a simple UUID compatible with MicroPython.
Returns:
UUID string
"""
# Generate a simple UUID-like string
# Format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
hex_chars = '0123456789abcdef'
uuid_str = ''.join([random.choice(hex_chars) for _ in range(32)])
# Insert hyphens at proper positions
return f"{uuid_str[:8]}-{uuid_str[8:12]}-{uuid_str[12:16]}-{uuid_str[16:20]}-{uuid_str[20:]}"
# ---------------------------------------------- Serialization Functions ---------------------------------------------- #
def _serialize_data(data, payload_type):
"""
Serialize data according to specified format.
Args:
data: Data to serialize (string for "text", dict for "dictionary",
bytes for "image", "audio", "video", "binary")
payload_type: Target format: "text", "dictionary", "image", "audio", "video", "binary"
Returns:
Binary representation of the serialized data
Note:
MicroPython does not support "table" type due to memory constraints.
Raises:
ValueError: If payload_type is not one of the supported types
"""
if payload_type == 'text':
if isinstance(data, str):
return data.encode('utf-8')
else:
raise ValueError('Text data must be a string')
elif payload_type == 'dictionary':
json_str = json.dumps(data)
return json_str.encode('utf-8')
elif payload_type in ('image', 'audio', 'video', 'binary'):
if isinstance(data, (bytes, bytearray, memoryview)):
return bytes(data)
else:
raise ValueError(f'{payload_type} data must be bytes')
else:
raise ValueError(f'Unknown payload_type: {payload_type}')
def _deserialize_data(data, payload_type):
"""
Deserialize bytes to data based on type.
Args:
data: Serialized data as bytes
payload_type: Data type ("text", "dictionary", "image", "audio", "video", "binary")
Returns:
Deserialized data (String for "text", dict for "dictionary", bytes for others)
Note:
MicroPython does not support "table" type due to memory constraints.
Raises:
ValueError: If payload_type is not one of the supported types
"""
if payload_type == 'text':
return data.decode('utf-8')
elif payload_type == 'dictionary':
json_str = data.decode('utf-8')
return json.loads(json_str)
elif payload_type in ('image', 'audio', 'video', 'binary'):
return data
else:
raise ValueError(f'Unknown payload_type: {payload_type}')
# ---------------------------------------------- File Server Handlers ---------------------------------------------- #
def _sync_fileserver_upload(file_server_url, dataname, data):
"""
Synchronous file upload to HTTP server.
Note:
This is a simplified implementation for MicroPython.
In practice, would use network.HTTP or similar.
Currently raises NotImplementedError as file upload is not fully supported.
Args:
file_server_url: Base URL of the file server
dataname: Name of the file being uploaded
data: Raw byte data of the file content
Returns:
Dict with keys: 'status', 'url'
Raises:
NotImplementedError: File upload is not implemented in MicroPython
"""
raise NotImplementedError("File upload not fully implemented in MicroPython. "
"Use direct transport only for memory-constrained devices.")
def _sync_fileserver_download(url, max_retries, base_delay, max_delay, correlation_id):
"""
Synchronous file download with exponential backoff.
Note:
This is a simplified implementation for MicroPython.
In practice, would use network.HTTP or similar.
Currently raises NotImplementedError as file download is not fully supported.
Args:
url: URL to download from
max_retries: Maximum retry attempts
base_delay: Initial delay in ms
max_delay: Maximum delay in ms
correlation_id: Correlation ID for logging
Returns:
Downloaded bytes
Raises:
NotImplementedError: File download is not implemented in MicroPython
"""
raise NotImplementedError("File download not fully implemented in MicroPython. "
"Use direct transport only for memory-constrained devices.")
# ---------------------------------------------- NATS Client ---------------------------------------------- #
class NATSClient:
"""
NATS client wrapper for MicroPython.
Note:
This is a simplified implementation for MicroPython.
Full NATS client implementation would require additional network stack support.
"""
def __init__(self, url=DEFAULT_BROKER_URL):
"""
Initialize NATS client.
Args:
url: NATS server URL
"""
self.url = url
self._connected = False
def connect(self):
"""
Connect to NATS server.
Note:
This is a placeholder implementation.
Actual NATS client would require network stack support.
Returns:
True if connected, False otherwise
"""
# Placeholder - actual implementation would connect to NATS server
self._connected = True
return self._connected
def publish(self, subject, message):
"""
Publish message to NATS subject.
Note:
This is a placeholder implementation.
Actual NATS client would require network stack support.
Args:
subject: NATS subject to publish to
message: Message to publish
"""
if not self._connected:
raise RuntimeError("Not connected to NATS server")
# Placeholder - actual implementation would publish to NATS
print(f"[NATS] Publish to {subject}: {message[:50]}...")
def close(self):
"""Close the NATS connection."""
self._connected = False
# ---------------------------------------------- Core Functions ---------------------------------------------- #
def _build_envelope(subject, payloads, options):
"""
Build message envelope from payloads and metadata.
Args:
subject: NATS subject
payloads: Array of payload objects
options: Envelope metadata options
Returns:
Envelope dict
"""
return {
'correlation_id': options['correlation_id'],
'msg_id': options['msg_id'],
'timestamp': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.localtime()),
'send_to': subject,
'msg_purpose': options['msg_purpose'],
'sender_name': options['sender_name'],
'sender_id': options['sender_id'],
'receiver_name': options['receiver_name'],
'receiver_id': options['receiver_id'],
'reply_to': options['reply_to'],
'reply_to_msg_id': options['reply_to_msg_id'],
'broker_url': options['broker_url'],
'metadata': {},
'payloads': payloads
}
def _build_payload(dataname, payload_type, payload_bytes, transport, data):
"""
Build payload object from serialized data.
Args:
dataname: Name of the payload
payload_type: Type of the payload
payload_bytes: Serialized payload bytes
transport: Transport type ("direct" or "link")
data: Data (base64 for direct, URL for link)
Returns:
Payload dict
"""
return {
'id': _generate_uuid(),
'dataname': dataname,
'payload_type': payload_type,
'transport': transport,
'encoding': 'base64' if transport == 'direct' else 'none',
'size': len(payload_bytes),
'data': data,
'metadata': {'payload_bytes': len(payload_bytes)} if transport == 'direct' else {}
}
def _publish(subject, message, correlation_id):
"""
Publish message to NATS.
Note:
This is a simplified implementation for MicroPython.
Args:
subject: NATS subject to publish to
message: JSON message to publish
correlation_id: Correlation ID for logging
"""
log_trace(correlation_id, f"Publishing to {subject}")
# Placeholder - actual implementation would use NATSClient
# client = NATSClient()
# client.connect()
# client.publish(subject, message)
# client.close()
def smartsend(subject, data, **kwargs):
"""
Send data via NATS with automatic transport selection.
This function intelligently routes data delivery based on payload size.
If the serialized payload is smaller than size_threshold, it encodes the data as Base64
and publishes directly over NATS. Otherwise, it uploads the data to a fileserver
and publishes only the download URL over NATS.
Note:
MicroPython has memory constraints, so the default size_threshold is lower (100KB).
Table type is not supported due to memory constraints.
Args:
subject: NATS subject to publish the message to
data: List of (dataname, data, type) tuples to send
- dataname: Name of the payload
- data: The actual data to send
- type: Payload type: "text", "dictionary", "image", "audio", "video", "binary"
broker_url: NATS server URL (default: DEFAULT_BROKER_URL)
fileserver_url: HTTP file server URL (default: DEFAULT_FILESERVER_URL)
fileserver_upload_handler: Function to handle fileserver uploads (default: _sync_fileserver_upload)
size_threshold: Threshold in bytes separating direct vs link transport (default: 100000)
correlation_id: Correlation ID for tracing (auto-generated if not provided)
msg_purpose: Purpose of the message (default: "chat")
sender_name: Name of the sender (default: "NATSBridge")
receiver_name: Name of the receiver (empty means broadcast)
receiver_id: UUID of the receiver (empty means broadcast)
reply_to: Topic to reply to (empty if no reply expected)
reply_to_msg_id: Message ID this message is replying to
is_publish: Whether to automatically publish the message (default: True)
msg_id: Message ID (auto-generated if not provided)
sender_id: Sender ID (auto-generated if not provided)
Returns:
Tuple of (env, env_json_str) where:
- env: Dict containing all metadata and payloads
- env_json_str: JSON string for publishing to NATS
Example:
>>> # Send text payload
>>> env, env_json_str = NATSBridge.smartsend(
... "/chat",
... [("message", "Hello!", "text")],
... broker_url="nats://localhost:4222"
... )
>>>
>>> # Send dictionary payload
>>> env, env_json_str = NATSBridge.smartsend(
... "/config",
... [("config", {"key": "value"}, "dictionary")],
... broker_url="nats://localhost:4222"
... )
>>>
>>> # Send binary payload (image, audio, video)
>>> env, env_json_str = NATSBridge.smartsend(
... "/media",
... [("image", image_bytes, "image")],
... broker_url="nats://localhost:4222"
... )
"""
# Extract options with defaults
correlation_id = kwargs.get('correlation_id', _generate_uuid())
msg_id = kwargs.get('msg_id', _generate_uuid())
sender_id = kwargs.get('sender_id', _generate_uuid())
broker_url = kwargs.get('broker_url', DEFAULT_BROKER_URL)
fileserver_url = kwargs.get('fileserver_url', DEFAULT_FILESERVER_URL)
size_threshold = kwargs.get('size_threshold', DEFAULT_SIZE_THRESHOLD)
msg_purpose = kwargs.get('msg_purpose', 'chat')
sender_name = kwargs.get('sender_name', 'NATSBridge')
receiver_name = kwargs.get('receiver_name', '')
receiver_id = kwargs.get('receiver_id', '')
reply_to = kwargs.get('reply_to', '')
reply_to_msg_id = kwargs.get('reply_to_msg_id', '')
is_publish = kwargs.get('is_publish', True)
fileserver_upload_handler = kwargs.get('fileserver_upload_handler', _sync_fileserver_upload)
log_trace(correlation_id, f"Starting smartsend for subject: {subject}")
# Process payloads
payloads = []
for dataname, payload_data, payload_type in data:
payload_bytes = _serialize_data(payload_data, payload_type)
payload_size = len(payload_bytes)
# Check against hard limit for MicroPython
if payload_size > MAX_PAYLOAD_SIZE:
raise MemoryError(f"Payload '{dataname}' exceeds max size {MAX_PAYLOAD_SIZE} bytes")
log_trace(correlation_id, f"Serialized payload '{dataname}' (type: {payload_type}) size: {payload_size} bytes")
if payload_size < size_threshold:
# Direct path
payload_b64 = base64.b64encode(payload_bytes).decode('ascii')
log_trace(correlation_id, f"Using direct transport for {payload_size} bytes")
payload = _build_payload(dataname, payload_type, payload_bytes, 'direct', payload_b64)
payloads.append(payload)
else:
# Link path (limited support)
log_trace(correlation_id, "Using link transport, uploading to fileserver")
try:
response = fileserver_upload_handler(fileserver_url, dataname, payload_bytes)
log_trace(correlation_id, f"Uploaded to URL: {response['url']}")
payload = _build_payload(dataname, payload_type, payload_bytes, 'link', response['url'])
payloads.append(payload)
except NotImplementedError:
# Fall back to direct transport if file upload not available
log_trace(correlation_id, "File upload not available, using direct transport")
payload_b64 = base64.b64encode(payload_bytes).decode('ascii')
payload = _build_payload(dataname, payload_type, payload_bytes, 'direct', payload_b64)
payloads.append(payload)
# Build envelope
env = _build_envelope(subject, payloads, {
'correlation_id': correlation_id,
'msg_id': msg_id,
'msg_purpose': msg_purpose,
'sender_name': sender_name,
'sender_id': sender_id,
'receiver_name': receiver_name,
'receiver_id': receiver_id,
'reply_to': reply_to,
'reply_to_msg_id': reply_to_msg_id,
'broker_url': broker_url
})
env_json_str = json.dumps(env)
if is_publish:
_publish(subject, env_json_str, correlation_id)
return env, env_json_str
def smartreceive(msg, **kwargs):
"""
Receive and process NATS message.
This function processes incoming NATS messages, handling both direct transport
(base64 decoded payloads) and link transport (URL-based payloads).
It deserializes the data based on the transport type and returns the result.
Note:
MicroPython has memory constraints, so large payloads should be avoided.
Table type is not supported due to memory constraints.
Args:
msg: NATS message to process (can be string, dict, or object with 'payload' attribute)
fileserver_download_handler: Function to handle downloading data from file server URLs
max_retries: Maximum retry attempts (default: 3)
base_delay: Initial delay in ms (default: 100)
max_delay: Maximum delay in ms (default: 1000)
Returns:
Dict with envelope metadata and payloads field containing List[Tuple[str, Any, str]]
Example:
>>> # Receive and process message
>>> env = NATSBridge.smartreceive(msg, fileserver_download_handler=_sync_fileserver_download)
>>> # env is a Dict with "payloads" key containing List[Tuple[str, Any, str]]
>>> for dataname, data, type_ in env["payloads"]:
... print(f"{dataname}: {data} (type: {type_})")
"""
# Parse the JSON envelope
if isinstance(msg, dict):
# Already parsed
env_json_obj = msg
elif hasattr(msg, 'payload'):
# Object with payload attribute
payload = msg.payload if isinstance(msg.payload, str) else msg.payload.decode('utf-8')
env_json_obj = json.loads(payload)
else:
# Assume it's already a JSON string or dict
env_json_obj = json.loads(msg) if isinstance(msg, str) else msg
correlation_id = env_json_obj['correlation_id']
log_trace(correlation_id, "Processing received message")
# Process all payloads in the envelope
payloads_list = []
num_payloads = len(env_json_obj['payloads'])
for i in range(num_payloads):
payload_obj = env_json_obj['payloads'][i]
transport = payload_obj['transport']
dataname = payload_obj['dataname']
if transport == 'direct':
log_trace(correlation_id, f"Direct transport - decoding payload '{dataname}'")
# Extract base64 payload from the payload
payload_b64 = payload_obj['data']
# Decode Base64 payload
payload_bytes = base64.b64decode(payload_b64)
# Deserialize based on type
data_type = payload_obj['payload_type']
data = _deserialize_data(payload_bytes, data_type)
payloads_list.append((dataname, data, data_type))
elif transport == 'link':
# Extract download URL from the payload
url = payload_obj['data']
log_trace(correlation_id, f"Link transport - fetching '{dataname}' from URL: {url}")
# Fetch with exponential backoff using the download handler
fileserver_download_handler = kwargs.get('fileserver_download_handler', _sync_fileserver_download)
max_retries = kwargs.get('max_retries', 3)
base_delay = kwargs.get('base_delay', 100)
max_delay = kwargs.get('max_delay', 1000)
downloaded_data = fileserver_download_handler(
url,
max_retries,
base_delay,
max_delay,
correlation_id
)
# Deserialize based on type
data_type = payload_obj['payload_type']
data = _deserialize_data(downloaded_data, data_type)
payloads_list.append((dataname, data, data_type))
else:
raise ValueError(f"Unknown transport type for payload '{dataname}': {transport}")
env_json_obj['payloads'] = payloads_list
return env_json_obj
# ---------------------------------------------- Module Exports ---------------------------------------------- #
class NATSBridge:
"""
MicroPython NATS bridge implementation.
This class provides a convenient interface for NATSBridge functionality,
encapsulating the main functions and providing a class-based API.
Note:
MicroPython has significant constraints:
- No Arrow IPC support (memory constraints)
- Only direct transport (< 100KB threshold enforced)
- Simplified UUID generation
- No async/await (synchronous API)
"""
DEFAULT_SIZE_THRESHOLD = DEFAULT_SIZE_THRESHOLD
DEFAULT_BROKER_URL = DEFAULT_BROKER_URL
DEFAULT_FILESERVER_URL = DEFAULT_FILESERVER_URL
MAX_PAYLOAD_SIZE = MAX_PAYLOAD_SIZE
def __init__(self, broker_url=None, fileserver_url=None):
"""
Initialize NATSBridge.
Args:
broker_url: NATS server URL (defaults to DEFAULT_BROKER_URL)
fileserver_url: HTTP file server URL (defaults to DEFAULT_FILESERVER_URL)
"""
self.broker_url = broker_url or self.DEFAULT_BROKER_URL
self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL
def smartsend(self, subject, data, **kwargs):
"""
Send data via NATS.
Args:
subject: NATS subject to publish to
data: List of (dataname, data, type) tuples
**kwargs: Additional options passed to smartsend
Returns:
Tuple of (env, env_json_str)
"""
kwargs['broker_url'] = kwargs.get('broker_url', self.broker_url)
kwargs['fileserver_url'] = kwargs.get('fileserver_url', self.fileserver_url)
return smartsend(subject, data, **kwargs)
def smartreceive(self, msg, **kwargs):
"""
Receive and process NATS message.
Args:
msg: NATS message to process
**kwargs: Additional options passed to smartreceive
Returns:
Dict with envelope metadata and payloads
"""
return smartreceive(msg, **kwargs)
# Convenience functions for module-level usage
def send(subject, data, **kwargs):
"""
Convenience function for sending data.
Args:
subject: NATS subject to publish to
data: List of (dataname, data, type) tuples
**kwargs: Additional options
Returns:
Tuple of (env, env_json_str)
"""
return smartsend(subject, data, **kwargs)
def receive(msg, **kwargs):
"""
Convenience function for receiving messages.
Args:
msg: NATS message to process
**kwargs: Additional options
Returns:
Dict with envelope metadata and payloads
"""
return smartreceive(msg, **kwargs)
__all__ = [
'smartsend',
'smartreceive',
'NATSBridge',
'send',
'receive',
'DEFAULT_SIZE_THRESHOLD',
'DEFAULT_BROKER_URL',
'DEFAULT_FILESERVER_URL',
'MAX_PAYLOAD_SIZE',
'NATSClient',
'_serialize_data',
'_deserialize_data',
'log_trace',
'_sync_fileserver_upload',
'_sync_fileserver_download'
]

View File

@@ -1,67 +0,0 @@
#!/usr/bin/env julia
# Scenario 1: Command & Control (Small JSON)
# Tests small JSON payloads (< 1MB) sent directly via NATS
using NATS
using JSON3
using UUIDs
# Include the bridge module
include("../src/julia_bridge.jl")
using .BiDirectionalBridge
# Configuration
const CONTROL_SUBJECT = "control"
const RESPONSE_SUBJECT = "control_response"
const NATS_URL = "nats://localhost:4222"
# Create correlation ID for tracing
correlation_id = string(uuid4())
# Receiver: Listen for control commands
function start_control_listener()
conn = NATS.Connection(NATS_URL)
try
NATS.subscribe(conn, CONTROL_SUBJECT) do msg
log_trace(msg.data)
# Parse the envelope
env = MessageEnvelope(String(msg.data))
# Parse JSON payload
config = JSON3.read(env.payload)
# Execute simulation with parameters
step_size = config.step_size
iterations = config.iterations
# Simulate processing
sleep(0.1) # Simulate some work
# Send acknowledgment
response = Dict(
"status" => "Running",
"correlation_id" => env.correlation_id,
"step_size" => step_size,
"iterations" => iterations
)
NATS.publish(conn, RESPONSE_SUBJECT, JSON3.stringify(response))
log_trace("Sent response: $(JSON3.stringify(response))")
end
# Keep listening for 5 seconds
sleep(5)
finally
NATS.close(conn)
end
end
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] [Correlation: $correlation_id] $message")
end
# Run the listener
start_control_listener()

View File

@@ -1,34 +0,0 @@
#!/usr/bin/env node
// Scenario 1: Command & Control (Small JSON)
// Tests small JSON payloads (< 1MB) sent directly via NATS
const { SmartSend } = require('../js_bridge');
// Configuration
const CONTROL_SUBJECT = "control";
const NATS_URL = "nats://localhost:4222";
// Create correlation ID for tracing
const correlationId = require('uuid').v4();
// Sender: Send control command to Julia
async function sendControlCommand() {
const config = {
step_size: 0.01,
iterations: 1000
};
// Send via SmartSend with type="json"
const env = await SmartSend(
CONTROL_SUBJECT,
config,
"json",
{ correlationId }
);
console.log(`Sent control command with correlation_id: ${correlationId}`);
console.log(`Envelope: ${JSON.stringify(env, null, 2)}`);
}
// Run the sender
sendControlCommand().catch(console.error);

View File

@@ -1,66 +0,0 @@
#!/usr/bin/env julia
# Scenario 2: Deep Dive Analysis (Large Arrow Table)
# Tests large Arrow tables (> 1MB) sent via HTTP fileserver
using NATS
using Arrow
using DataFrames
using JSON3
using UUIDs
# Include the bridge module
include("../src/julia_bridge.jl")
using .BiDirectionalBridge
# Configuration
const ANALYSIS_SUBJECT = "analysis_results"
const RESPONSE_SUBJECT = "analysis_response"
const NATS_URL = "nats://localhost:4222"
# Create correlation ID for tracing
correlation_id = string(uuid4())
# Receiver: Listen for analysis results
function start_analysis_listener()
conn = NATS.Connection(NATS_URL)
try
NATS.subscribe(conn, ANALYSIS_SUBJECT) do msg
log_trace("Received message from $(msg.subject)")
# Parse the envelope
env = MessageEnvelope(String(msg.data))
# Use SmartReceive to handle the data
result = SmartReceive(msg)
# Process the data based on type
if result.envelope.type == "table"
df = result.data
log_trace("Received DataFrame with $(nrows(df)) rows")
log_trace("DataFrame columns: $(names(df))")
# Send acknowledgment
response = Dict(
"status" => "Processed",
"correlation_id" => env.correlation_id,
"row_count" => nrows(df)
)
NATS.publish(conn, RESPONSE_SUBJECT, JSON3.stringify(response))
end
end
# Keep listening for 10 seconds
sleep(10)
finally
NATS.close(conn)
end
end
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] [Correlation: $correlation_id] $message")
end
# Run the listener
start_analysis_listener()

View File

@@ -1,54 +0,0 @@
#!/usr/bin/env node
// Scenario 2: Deep Dive Analysis (Large Arrow Table)
// Tests large Arrow tables (> 1MB) sent via HTTP fileserver
const { SmartSend } = require('../js_bridge');
// Configuration
const ANALYSIS_SUBJECT = "analysis_results";
const NATS_URL = "nats://localhost:4222";
// Create correlation ID for tracing
const correlationId = require('uuid').v4();
// Sender: Send large Arrow table to Julia
async function sendLargeTable() {
// Create a large DataFrame-like structure (10 million rows)
// For testing, we'll create a smaller but still large table
const numRows = 1000000; // 1 million rows
const data = {
id: Array.from({ length: numRows }, (_, i) => i + 1),
value: Array.from({ length: numRows }, () => Math.random()),
category: Array.from({ length: numRows }, () => ['A', 'B', 'C'][Math.floor(Math.random() * 3)])
};
// Convert to Arrow Table
const { Table, Vector, RecordBatch } = require('apache-arrow');
const idVector = Vector.from(data.id);
const valueVector = Vector.from(data.value);
const categoryVector = Vector.from(data.category);
const table = Table.from({
id: idVector,
value: valueVector,
category: categoryVector
});
// Send via SmartSend with type="table"
const env = await SmartSend(
ANALYSIS_SUBJECT,
table,
"table",
{ correlationId }
);
console.log(`Sent large table with ${numRows} rows`);
console.log(`Correlation ID: ${correlationId}`);
console.log(`Transport: ${env.transport}`);
console.log(`URL: ${env.url || 'N/A'}`);
}
// Run the sender
sendLargeTable().catch(console.error);

View File

@@ -0,0 +1,215 @@
/**
* JavaScript Binary Receiver Test
* Tests the smartreceive function with binary/image/audio/video payloads
*/
const NATSBridge = require('../src/natbridge.js');
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
async function runTest() {
console.log('=== JavaScript Binary Receiver Test ===\n');
// Create mock NATS message with binary payloads
const binaryData = Buffer.from([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]); // PNG header
const audioData = Buffer.from([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]); // FLAC header
const videoData = Buffer.from([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]); // MP4 header
const genericBinary = Buffer.from([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE]);
const testData = {
correlation_id: 'js-binary-receiver-' + Date.now(),
msg_id: 'msg-' + Date.now(),
timestamp: new Date().toISOString(),
send_to: '/test/binary',
msg_purpose: 'test',
sender_name: 'js-binary-test',
sender_id: 'sender-' + Date.now(),
receiver_name: 'js-receiver',
receiver_id: 'receiver-' + Date.now(),
reply_to: '',
reply_to_msg_id: '',
broker_url: TEST_BROKER_URL,
metadata: {},
payloads: [
{
id: 'payload-1',
dataname: 'image',
payload_type: 'image',
transport: 'direct',
encoding: 'base64',
size: binaryData.length,
data: binaryData.toString('base64'),
metadata: { payload_bytes: binaryData.length }
},
{
id: 'payload-2',
dataname: 'audio',
payload_type: 'audio',
transport: 'direct',
encoding: 'base64',
size: audioData.length,
data: audioData.toString('base64'),
metadata: { payload_bytes: audioData.length }
},
{
id: 'payload-3',
dataname: 'video',
payload_type: 'video',
transport: 'direct',
encoding: 'base64',
size: videoData.length,
data: videoData.toString('base64'),
metadata: { payload_bytes: videoData.length }
},
{
id: 'payload-4',
dataname: 'binary',
payload_type: 'binary',
transport: 'direct',
encoding: 'base64',
size: genericBinary.length,
data: genericBinary.toString('base64'),
metadata: { payload_bytes: genericBinary.length }
}
]
};
const mockMsg = {
payload: JSON.stringify(testData)
};
console.log('Mock Message Created:');
console.log(` Correlation ID: ${testData.correlation_id}`);
console.log(` Payloads: ${testData.payloads.length}`);
console.log(` Payload types: ${testData.payloads.map(p => p.payload_type).join(', ')}\n`);
try {
// Receive and process the message
console.log('Receiving and processing message...');
const env = await NATSBridge.smartreceive(
mockMsg,
{
max_retries: 3,
base_delay: 100,
max_delay: 1000
}
);
console.log('\n=== Received Envelope ===');
console.log(`Correlation ID: ${env.correlation_id}`);
console.log(`Message ID: ${env.msg_id}`);
console.log(`Timestamp: ${env.timestamp}`);
console.log(`Subject: ${env.send_to}`);
console.log(`Payloads: ${env.payloads.length}\n`);
// Validate received data
console.log('=== Validation ===');
let passed = true;
if (!env.correlation_id) {
console.log('❌ correlation_id is missing');
passed = false;
} else {
console.log('✅ correlation_id present');
}
if (env.payloads.length !== 4) {
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
passed = false;
} else {
console.log('✅ Correct number of payloads');
}
// Expected data
const expectedData = [
['image', binaryData, 'image'],
['audio', audioData, 'audio'],
['video', videoData, 'video'],
['binary', genericBinary, 'binary']
];
for (let i = 0; i < env.payloads.length; i++) {
const payload = env.payloads[i];
const expected = expectedData[i];
if (payload[0] !== expected[0]) {
console.log(`❌ Payload ${i + 1}: Expected dataname '${expected[0]}', got '${payload[0]}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct dataname`);
}
if (payload[2] !== expected[2]) {
console.log(`❌ Payload ${i + 1}: Expected type '${expected[2]}', got '${payload[2]}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct type`);
}
// Verify binary data integrity
const receivedData = payload[1];
if (!(receivedData instanceof Buffer || receivedData instanceof Uint8Array)) {
console.log(`❌ Payload ${i + 1}: Expected Buffer/Uint8Array, got ${typeof receivedData}`);
passed = false;
} else if (Buffer.isBuffer(receivedData)) {
if (receivedData.length !== expected[1].length) {
console.log(`❌ Payload ${i + 1}: Length mismatch`);
passed = false;
} else {
let dataMatch = true;
for (let j = 0; j < expected[1].length; j++) {
if (receivedData[j] !== expected[1][j]) {
dataMatch = false;
break;
}
}
if (dataMatch) {
console.log(`✅ Payload ${i + 1}: Data correctly deserialized`);
} else {
console.log(`❌ Payload ${i + 1}: Data mismatch`);
passed = false;
}
}
} else {
// Uint8Array comparison
const receivedBuffer = Buffer.from(receivedData);
if (receivedBuffer.length !== expected[1].length) {
console.log(`❌ Payload ${i + 1}: Length mismatch`);
passed = false;
} else {
let dataMatch = true;
for (let j = 0; j < expected[1].length; j++) {
if (receivedBuffer[j] !== expected[1][j]) {
dataMatch = false;
break;
}
}
if (dataMatch) {
console.log(`✅ Payload ${i + 1}: Data correctly deserialized`);
} else {
console.log(`❌ Payload ${i + 1}: Data mismatch`);
passed = false;
}
}
}
}
// Final result
console.log('\n=== Test Result ===');
if (passed) {
console.log('✅ ALL TESTS PASSED');
process.exit(0);
} else {
console.log('❌ SOME TESTS FAILED');
process.exit(1);
}
} catch (error) {
console.error('❌ Test failed with error:', error.message);
console.error(error.stack);
process.exit(1);
}
}
runTest();

View File

@@ -0,0 +1,173 @@
/**
* JavaScript Binary Sender Test
* Tests the smartsend function with binary/image/audio/video payloads
*/
const NATSBridge = require('../src/natbridge.js');
const TEST_SUBJECT = '/test/binary';
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
async function runTest() {
console.log('=== JavaScript Binary Sender Test ===\n');
const correlationId = NATSBridge.uuidv4();
console.log(`Correlation ID: ${correlationId}`);
console.log(`Subject: ${TEST_SUBJECT}`);
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
// Test data - binary data for different types
const binaryData = Buffer.from([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]); // PNG header
const audioData = Buffer.from([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]); // FLAC header
const videoData = Buffer.from([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]); // MP4 header
const genericBinary = Buffer.from([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE]);
const testData = [
['image', binaryData, 'image'],
['audio', audioData, 'audio'],
['video', videoData, 'video'],
['binary', genericBinary, 'binary']
];
try {
// Send the message
console.log('Sending binary payloads...');
const [env, envJsonStr] = await NATSBridge.smartsend(
TEST_SUBJECT,
testData,
{
broker_url: TEST_BROKER_URL,
fileserver_url: TEST_FILESERVER_URL,
correlation_id: correlationId,
msg_purpose: 'test',
sender_name: 'js-binary-test',
is_publish: false
}
);
console.log('\n=== Envelope Created ===');
console.log(`Correlation ID: ${env.correlation_id}`);
console.log(`Message ID: ${env.msg_id}`);
console.log(`Timestamp: ${env.timestamp}`);
console.log(`Subject: ${env.send_to}`);
console.log(`Purpose: ${env.msg_purpose}`);
console.log(`Sender: ${env.sender_name}`);
console.log(`Payloads: ${env.payloads.length}\n`);
// Validate envelope structure
console.log('=== Validation ===');
let passed = true;
if (env.payloads.length !== 4) {
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
passed = false;
} else {
console.log('✅ Correct number of payloads');
}
// Test each payload
const expectedDatanames = ['image', 'audio', 'video', 'binary'];
const expectedTypes = ['image', 'audio', 'video', 'binary'];
const expectedData = [binaryData, audioData, videoData, genericBinary];
for (let i = 0; i < env.payloads.length; i++) {
const payload = env.payloads[i];
if (payload.dataname !== expectedDatanames[i]) {
console.log(`❌ Payload ${i + 1}: Expected dataname '${expectedDatanames[i]}', got '${payload.dataname}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct dataname`);
}
if (payload.payload_type !== expectedTypes[i]) {
console.log(`❌ Payload ${i + 1}: Expected type '${expectedTypes[i]}', got '${payload.payload_type}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct type`);
}
if (payload.transport !== 'direct') {
console.log(`❌ Payload ${i + 1}: Expected transport 'direct', got '${payload.transport}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct transport`);
}
if (payload.encoding !== 'base64') {
console.log(`❌ Payload ${i + 1}: Expected encoding 'base64', got '${payload.encoding}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct encoding`);
}
// Decode and verify the data
const decodedData = Buffer.from(payload.data, 'base64');
const originalData = expectedData[i];
if (decodedData.length !== originalData.length) {
console.log(`❌ Payload ${i + 1}: Length mismatch (${decodedData.length} vs ${originalData.length})`);
passed = false;
} else {
let dataMatch = true;
for (let j = 0; j < originalData.length; j++) {
if (decodedData[j] !== originalData[j]) {
dataMatch = false;
break;
}
}
if (dataMatch) {
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
} else {
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
passed = false;
}
}
console.log(` Size: ${payload.size} bytes\n`);
}
// Test with larger binary data (simulating file upload scenario)
console.log('=== Large Binary Data Test ===');
const largeData = Buffer.alloc(10000, 0xFF); // 10KB of binary data
const largeTestData = [
['large_binary', largeData, 'binary']
];
const [largeEnv, _] = await NATSBridge.smartsend(
TEST_SUBJECT,
largeTestData,
{
broker_url: TEST_BROKER_URL,
fileserver_url: TEST_FILESERVER_URL,
correlation_id: 'large-' + correlationId,
is_publish: false
}
);
if (largeEnv.payloads.length === 1 && largeEnv.payloads[0].size === 10000) {
console.log('✅ Large binary data handled correctly');
} else {
console.log('❌ Large binary data handling failed');
passed = false;
}
// Final result
console.log('\n=== Test Result ===');
if (passed) {
console.log('✅ ALL TESTS PASSED');
process.exit(0);
} else {
console.log('❌ SOME TESTS FAILED');
process.exit(1);
}
} catch (error) {
console.error('❌ Test failed with error:', error.message);
console.error(error.stack);
process.exit(1);
}
}
runTest();

View File

@@ -0,0 +1,220 @@
/**
* JavaScript Dictionary Receiver Test
* Tests the smartreceive function with dictionary payloads
*/
const NATSBridge = require('../src/natbridge.js');
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
async function runTest() {
console.log('=== JavaScript Dictionary Receiver Test ===\n');
// Create a mock NATS message with dictionary payloads
const simpleDict = { key1: 'value1', key2: 'value2' };
const nestedDict = { outer: { inner: 'value', number: 42 } };
const arrayDict = { items: [1, 2, 3, 'four', 'five'] };
const mixedDict = { string: 'text', number: 123, boolean: true, null_val: null };
const testData = {
correlation_id: 'test-receiver-dict-' + Date.now(),
msg_id: 'msg-' + Date.now(),
timestamp: new Date().toISOString(),
send_to: '/test/dictionary',
msg_purpose: 'test',
sender_name: 'js-dict-test',
sender_id: 'sender-' + Date.now(),
receiver_name: 'js-receiver',
receiver_id: 'receiver-' + Date.now(),
reply_to: '',
reply_to_msg_id: '',
broker_url: TEST_BROKER_URL,
metadata: {},
payloads: [
{
id: 'payload-1',
dataname: 'simple_dict',
payload_type: 'dictionary',
transport: 'direct',
encoding: 'base64',
size: Buffer.from(JSON.stringify(simpleDict)).length,
data: Buffer.from(JSON.stringify(simpleDict)).toString('base64'),
metadata: { payload_bytes: Buffer.from(JSON.stringify(simpleDict)).length }
},
{
id: 'payload-2',
dataname: 'nested_dict',
payload_type: 'dictionary',
transport: 'direct',
encoding: 'base64',
size: Buffer.from(JSON.stringify(nestedDict)).length,
data: Buffer.from(JSON.stringify(nestedDict)).toString('base64'),
metadata: { payload_bytes: Buffer.from(JSON.stringify(nestedDict)).length }
},
{
id: 'payload-3',
dataname: 'array_dict',
payload_type: 'dictionary',
transport: 'direct',
encoding: 'base64',
size: Buffer.from(JSON.stringify(arrayDict)).length,
data: Buffer.from(JSON.stringify(arrayDict)).toString('base64'),
metadata: { payload_bytes: Buffer.from(JSON.stringify(arrayDict)).length }
},
{
id: 'payload-4',
dataname: 'mixed_dict',
payload_type: 'dictionary',
transport: 'direct',
encoding: 'base64',
size: Buffer.from(JSON.stringify(mixedDict)).length,
data: Buffer.from(JSON.stringify(mixedDict)).toString('base64'),
metadata: { payload_bytes: Buffer.from(JSON.stringify(mixedDict)).length }
}
]
};
const mockMsg = {
payload: JSON.stringify(testData)
};
console.log('Mock Message Created:');
console.log(` Correlation ID: ${testData.correlation_id}`);
console.log(` Payloads: ${testData.payloads.length}`);
console.log(` Payload types: ${testData.payloads.map(p => p.payload_type).join(', ')}\n`);
try {
// Receive and process the message
console.log('Receiving and processing message...');
const env = await NATSBridge.smartreceive(
mockMsg,
{
max_retries: 3,
base_delay: 100,
max_delay: 1000
}
);
console.log('\n=== Received Envelope ===');
console.log(`Correlation ID: ${env.correlation_id}`);
console.log(`Message ID: ${env.msg_id}`);
console.log(`Timestamp: ${env.timestamp}`);
console.log(`Subject: ${env.send_to}`);
console.log(`Payloads: ${env.payloads.length}\n`);
// Validate received data
console.log('=== Validation ===');
let passed = true;
if (!env.correlation_id) {
console.log('❌ correlation_id is missing');
passed = false;
} else {
console.log('✅ correlation_id present');
}
if (env.payloads.length !== 4) {
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
passed = false;
} else {
console.log('✅ Correct number of payloads');
}
// Expected data
const expectedData = [
['simple_dict', simpleDict, 'dictionary'],
['nested_dict', nestedDict, 'dictionary'],
['array_dict', arrayDict, 'dictionary'],
['mixed_dict', mixedDict, 'dictionary']
];
for (let i = 0; i < env.payloads.length; i++) {
const payload = env.payloads[i];
const expected = expectedData[i];
if (payload[0] !== expected[0]) {
console.log(`❌ Payload ${i + 1}: Expected dataname '${expected[0]}', got '${payload[0]}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct dataname`);
}
if (payload[2] !== expected[2]) {
console.log(`❌ Payload ${i + 1}: Expected type '${expected[2]}', got '${payload[2]}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct type`);
}
const dataMatch = JSON.stringify(payload[1]) === JSON.stringify(expected[1]);
if (!dataMatch) {
console.log(`❌ Payload ${i + 1}: Data mismatch`);
console.log(` Expected: ${JSON.stringify(expected[1])}`);
console.log(` Got: ${JSON.stringify(payload[1])}`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Data correctly deserialized`);
}
}
// Test round-trip with receive
console.log('\n=== Round-trip Test ===');
const roundTripData = {
correlation_id: 'roundtrip-' + Date.now(),
msg_id: 'msg-' + Date.now(),
timestamp: new Date().toISOString(),
send_to: '/test/dictionary',
msg_purpose: 'test',
sender_name: 'js-test',
sender_id: 'sender-' + Date.now(),
receiver_name: 'js-receiver',
receiver_id: 'receiver-' + Date.now(),
reply_to: '',
reply_to_msg_id: '',
broker_url: TEST_BROKER_URL,
metadata: {},
payloads: [
{
id: 'payload-rt',
dataname: 'roundtrip',
payload_type: 'dictionary',
transport: 'direct',
encoding: 'base64',
size: Buffer.from(JSON.stringify({ test: 'data', nested: { a: 1, b: 2 } })).length,
data: Buffer.from(JSON.stringify({ test: 'data', nested: { a: 1, b: 2 } })).toString('base64'),
metadata: { payload_bytes: Buffer.from(JSON.stringify({ test: 'data', nested: { a: 1, b: 2 } })).length }
}
]
};
const mockRtMsg = { payload: JSON.stringify(roundTripData) };
const rtEnv = await NATSBridge.smartreceive(mockRtMsg);
if (rtEnv.payloads.length === 1 &&
rtEnv.payloads[0][0] === 'roundtrip' &&
rtEnv.payloads[0][2] === 'dictionary') {
console.log('✅ Round-trip test successful');
} else {
console.log('❌ Round-trip test failed');
passed = false;
}
// Final result
console.log('\n=== Test Result ===');
if (passed) {
console.log('✅ ALL TESTS PASSED');
process.exit(0);
} else {
console.log('❌ SOME TESTS FAILED');
process.exit(1);
}
} catch (error) {
console.error('❌ Test failed with error:', error.message);
console.error(error.stack);
process.exit(1);
}
}
runTest();

View File

@@ -0,0 +1,178 @@
/**
* JavaScript Dictionary Sender Test
* Tests the smartsend function with dictionary payloads
*/
const NATSBridge = require('../src/natbridge.js');
const TEST_SUBJECT = '/test/dictionary';
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
async function runTest() {
console.log('=== JavaScript Dictionary Sender Test ===\n');
const correlationId = NATSBridge.uuidv4();
console.log(`Correlation ID: ${correlationId}`);
console.log(`Subject: ${TEST_SUBJECT}`);
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
// Test data - various dictionary structures
const testData = [
['simple_dict', { key1: 'value1', key2: 'value2' }, 'dictionary'],
['nested_dict', { outer: { inner: 'value', number: 42 } }, 'dictionary'],
['array_dict', { items: [1, 2, 3, 'four', 'five'] }, 'dictionary'],
['mixed_dict', { string: 'text', number: 123, boolean: true, null_val: null }, 'dictionary']
];
try {
// Send the message
console.log('Sending dictionary payloads...');
const [env, envJsonStr] = await NATSBridge.smartsend(
TEST_SUBJECT,
testData,
{
broker_url: TEST_BROKER_URL,
fileserver_url: TEST_FILESERVER_URL,
correlation_id: correlationId,
msg_purpose: 'test',
sender_name: 'js-dict-test',
is_publish: false
}
);
console.log('\n=== Envelope Created ===');
console.log(`Correlation ID: ${env.correlation_id}`);
console.log(`Message ID: ${env.msg_id}`);
console.log(`Timestamp: ${env.timestamp}`);
console.log(`Subject: ${env.send_to}`);
console.log(`Purpose: ${env.msg_purpose}`);
console.log(`Sender: ${env.sender_name}`);
console.log(`Payloads: ${env.payloads.length}\n`);
// Validate envelope structure
console.log('=== Validation ===');
let passed = true;
if (env.payloads.length !== 4) {
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
passed = false;
} else {
console.log('✅ Correct number of payloads');
}
// Test each payload
const expectedDatanames = ['simple_dict', 'nested_dict', 'array_dict', 'mixed_dict'];
const expectedTypes = ['dictionary', 'dictionary', 'dictionary', 'dictionary'];
for (let i = 0; i < env.payloads.length; i++) {
const payload = env.payloads[i];
if (payload.dataname !== expectedDatanames[i]) {
console.log(`❌ Payload ${i + 1}: Expected dataname '${expectedDatanames[i]}', got '${payload.dataname}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct dataname`);
}
if (payload.payload_type !== expectedTypes[i]) {
console.log(`❌ Payload ${i + 1}: Expected type '${expectedTypes[i]}', got '${payload.payload_type}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct type`);
}
if (payload.transport !== 'direct') {
console.log(`❌ Payload ${i + 1}: Expected transport 'direct', got '${payload.transport}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct transport`);
}
if (payload.encoding !== 'base64') {
console.log(`❌ Payload ${i + 1}: Expected encoding 'base64', got '${payload.encoding}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct encoding`);
}
// Decode and verify the data
const decodedData = JSON.parse(Buffer.from(payload.data, 'base64').toString('utf8'));
const originalData = testData[i][1];
const originalJson = JSON.stringify(originalData);
const decodedJson = JSON.stringify(decodedData);
if (originalJson !== decodedJson) {
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
console.log(` Expected: ${originalJson}`);
console.log(` Got: ${decodedJson}`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
}
console.log(` Size: ${payload.size} bytes\n`);
}
// Test round-trip serialization
console.log('=== Round-trip Serialization Test ===');
const roundTripTestData = [
['roundtrip', { test: 'data', numbers: [1, 2, 3], nested: { a: 1, b: 2 } }, 'dictionary']
];
const [rtEnv, rtEnvJsonStr] = await NATSBridge.smartsend(
TEST_SUBJECT,
roundTripTestData,
{
broker_url: TEST_BROKER_URL,
fileserver_url: TEST_FILESERVER_URL,
correlation_id: 'roundtrip-' + correlationId,
is_publish: false
}
);
const rtPayload = rtEnv.payloads[0];
const rtDecoded = JSON.parse(Buffer.from(rtPayload.data, 'base64').toString('utf8'));
if (JSON.stringify(rtDecoded) === JSON.stringify(roundTripTestData[0][1])) {
console.log('✅ Round-trip serialization successful');
} else {
console.log('❌ Round-trip serialization failed');
passed = false;
}
// Test JSON string output
console.log('\n=== JSON String Output Test ===');
try {
const parsed = JSON.parse(envJsonStr);
if (parsed.correlation_id === env.correlation_id &&
parsed.payloads.length === env.payloads.length) {
console.log('✅ JSON string is valid and matches envelope');
} else {
console.log('❌ JSON string does not match envelope');
passed = false;
}
} catch (e) {
console.log('❌ JSON string is invalid:', e.message);
passed = false;
}
// Final result
console.log('\n=== Test Result ===');
if (passed) {
console.log('✅ ALL TESTS PASSED');
process.exit(0);
} else {
console.log('❌ SOME TESTS FAILED');
process.exit(1);
}
} catch (error) {
console.error('❌ Test failed with error:', error.message);
console.error(error.stack);
process.exit(1);
}
}
runTest();

View File

@@ -0,0 +1,204 @@
/**
* JavaScript Mix Payloads Sender Test
* Tests the smartsend function with mixed payload types
*/
const NATSBridge = require('../src/natbridge.js');
const TEST_SUBJECT = '/test/mix';
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
async function runTest() {
console.log('=== JavaScript Mix Payloads Sender Test ===\n');
const correlationId = NATSBridge.uuidv4();
console.log(`Correlation ID: ${correlationId}`);
console.log(`Subject: ${TEST_SUBJECT}`);
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
// Test data - mixed payload types
const textData = 'Hello, NATSBridge!';
const dictData = { key1: 'value1', key2: 42, nested: { a: 1, b: 2 } };
const binaryData = Buffer.from([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]); // PNG header
// Table data
const tableData = [
{ id: 1, name: 'Alice', age: 30 },
{ id: 2, name: 'Bob', age: 25 },
{ id: 3, name: 'Charlie', age: 35 }
];
const testData = [
['message', textData, 'text'],
['config', dictData, 'dictionary'],
['image', binaryData, 'image'],
['users', tableData, 'table']
];
try {
// Send the message
console.log('Sending mixed payloads...');
const [env, envJsonStr] = await NATSBridge.smartsend(
TEST_SUBJECT,
testData,
{
broker_url: TEST_BROKER_URL,
fileserver_url: TEST_FILESERVER_URL,
correlation_id: correlationId,
msg_purpose: 'test',
sender_name: 'js-mix-test',
is_publish: false
}
);
console.log('\n=== Envelope Created ===');
console.log(`Correlation ID: ${env.correlation_id}`);
console.log(`Message ID: ${env.msg_id}`);
console.log(`Timestamp: ${env.timestamp}`);
console.log(`Subject: ${env.send_to}`);
console.log(`Purpose: ${env.msg_purpose}`);
console.log(`Sender: ${env.sender_name}`);
console.log(`Payloads: ${env.payloads.length}\n`);
// Validate envelope structure
console.log('=== Validation ===');
let passed = true;
if (env.payloads.length !== 4) {
console.log(`❌ Expected 4 payloads, got ${env.payloads.length}`);
passed = false;
} else {
console.log('✅ Correct number of payloads');
}
// Test each payload
const expectedDatanames = ['message', 'config', 'image', 'users'];
const expectedTypes = ['text', 'dictionary', 'image', 'table'];
const expectedData = [textData, dictData, binaryData, tableData];
for (let i = 0; i < env.payloads.length; i++) {
const payload = env.payloads[i];
if (payload.dataname !== expectedDatanames[i]) {
console.log(`❌ Payload ${i + 1}: Expected dataname '${expectedDatanames[i]}', got '${payload.dataname}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct dataname`);
}
if (payload.payload_type !== expectedTypes[i]) {
console.log(`❌ Payload ${i + 1}: Expected type '${expectedTypes[i]}', got '${payload.payload_type}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct type`);
}
if (payload.transport !== 'direct') {
console.log(`❌ Payload ${i + 1}: Expected transport 'direct', got '${payload.transport}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct transport`);
}
if (payload.encoding !== 'base64') {
console.log(`❌ Payload ${i + 1}: Expected encoding 'base64', got '${payload.encoding}'`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Correct encoding`);
}
// Verify data integrity based on type
if (expectedTypes[i] === 'text') {
const decodedData = Buffer.from(payload.data, 'base64').toString('utf8');
if (decodedData !== expectedData[i]) {
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
}
} else if (expectedTypes[i] === 'dictionary') {
const decodedData = JSON.parse(Buffer.from(payload.data, 'base64').toString('utf8'));
if (JSON.stringify(decodedData) !== JSON.stringify(expectedData[i])) {
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
passed = false;
} else {
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
}
} else if (expectedTypes[i] === 'image') {
const decodedData = Buffer.from(payload.data, 'base64');
if (decodedData.length !== expectedData[i].length) {
console.log(`❌ Payload ${i + 1}: Length mismatch`);
passed = false;
} else {
let dataMatch = true;
for (let j = 0; j < expectedData[i].length; j++) {
if (decodedData[j] !== expectedData[i][j]) {
dataMatch = false;
break;
}
}
if (dataMatch) {
console.log(`✅ Payload ${i + 1}: Data integrity verified`);
} else {
console.log(`❌ Payload ${i + 1}: Data integrity mismatch`);
passed = false;
}
}
} else if (expectedTypes[i] === 'table') {
const decodedData = Buffer.from(payload.data, 'base64');
if (decodedData.length > 0) {
console.log(`✅ Payload ${i + 1}: Arrow IPC data present (${decodedData.length} bytes)`);
} else {
console.log(`❌ Payload ${i + 1}: Arrow IPC data is empty`);
passed = false;
}
}
console.log(` Size: ${payload.size} bytes\n`);
}
// Test with chat-like payload (text + image + audio)
console.log('=== Chat-like Payload Test ===');
const chatData = [
['text', 'Hello!', 'text'],
['image', Buffer.from([0xFF, 0xD8, 0xFF, 0xE0]), 'image'],
['audio', Buffer.from([0x46, 0x4C, 0x41, 0x43]), 'audio']
];
const [chatEnv, _] = await NATSBridge.smartsend(
TEST_SUBJECT,
chatData,
{
broker_url: TEST_BROKER_URL,
fileserver_url: TEST_FILESERVER_URL,
correlation_id: 'chat-' + correlationId,
is_publish: false
}
);
if (chatEnv.payloads.length === 3) {
console.log('✅ Chat-like payloads handled correctly');
} else {
console.log('❌ Chat-like payloads handling failed');
passed = false;
}
// Final result
console.log('\n=== Test Result ===');
if (passed) {
console.log('✅ ALL TESTS PASSED');
process.exit(0);
} else {
console.log('❌ SOME TESTS FAILED');
process.exit(1);
}
} catch (error) {
console.error('❌ Test failed with error:', error.message);
console.error(error.stack);
process.exit(1);
}
}
runTest();

View File

@@ -0,0 +1,172 @@
/**
* JavaScript Table Receiver Test
* Tests the smartreceive function with table (Arrow IPC) payloads
*/
const NATSBridge = require('../src/natbridge.js');
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
async function runTest() {
console.log('=== JavaScript Table Receiver Test ===\n');
// Create a mock NATS message with table payload
const tableData = [
{ id: 1, name: 'Alice', age: 30, active: true },
{ id: 2, name: 'Bob', age: 25, active: false },
{ id: 3, name: 'Charlie', age: 35, active: true }
];
// Convert to Arrow IPC format
const arrow = require('apache-arrow');
const fields = [
new arrow.Field('id', arrow.Int64, true),
new arrow.Field('name', arrow.Utf8, true),
new arrow.Field('age', arrow.Int64, true),
new arrow.Field('active', arrow.Boolean, true)
];
const schema = new arrow.Schema(fields);
const batches = [];
for (const row of tableData) {
const batch = arrow.recordBatch.fromObjects([row], schema);
batches.push(batch);
}
const buffers = arrow.ipc.recordBatchesToMessage(batches, schema).buffers;
const combined = new Uint8Array(buffers.reduce((acc, b) => acc + b.byteLength, 0));
let offset = 0;
for (const buf of buffers) {
combined.set(new Uint8Array(buf), offset);
offset += buf.byteLength;
}
const arrowBuffer = Buffer.from(combined);
const testData = {
correlation_id: 'js-table-receiver-' + Date.now(),
msg_id: 'msg-' + Date.now(),
timestamp: new Date().toISOString(),
send_to: '/test/table',
msg_purpose: 'test',
sender_name: 'js-table-test',
sender_id: 'sender-' + Date.now(),
receiver_name: 'js-receiver',
receiver_id: 'receiver-' + Date.now(),
reply_to: '',
reply_to_msg_id: '',
broker_url: TEST_BROKER_URL,
metadata: {},
payloads: [
{
id: 'payload-1',
dataname: 'users_table',
payload_type: 'table',
transport: 'direct',
encoding: 'base64',
size: arrowBuffer.length,
data: arrowBuffer.toString('base64'),
metadata: { payload_bytes: arrowBuffer.length }
}
]
};
const mockMsg = {
payload: JSON.stringify(testData)
};
console.log('Mock Message Created:');
console.log(` Correlation ID: ${testData.correlation_id}`);
console.log(` Payloads: ${testData.payloads.length}`);
console.log(` Payload type: ${testData.payloads[0].payload_type}`);
console.log(` Transport: ${testData.payloads[0].transport}\n`);
try {
// Receive and process the message
console.log('Receiving and processing message...');
const env = await NATSBridge.smartreceive(
mockMsg,
{
max_retries: 3,
base_delay: 100,
max_delay: 1000
}
);
console.log('\n=== Received Envelope ===');
console.log(`Correlation ID: ${env.correlation_id}`);
console.log(`Message ID: ${env.msg_id}`);
console.log(`Timestamp: ${env.timestamp}`);
console.log(`Subject: ${env.send_to}`);
console.log(`Payloads: ${env.payloads.length}\n`);
// Validate received data
console.log('=== Validation ===');
let passed = true;
if (!env.correlation_id) {
console.log('❌ correlation_id is missing');
passed = false;
} else {
console.log('✅ correlation_id present');
}
if (env.payloads.length !== 1) {
console.log(`❌ Expected 1 payload, got ${env.payloads.length}`);
passed = false;
} else {
console.log('✅ Correct number of payloads');
}
const payload = env.payloads[0];
if (payload[0] !== 'users_table') {
console.log(`❌ Expected dataname 'users_table', got '${payload[0]}'`);
passed = false;
} else {
console.log('✅ Correct dataname');
}
if (payload[2] !== 'table') {
console.log(`❌ Expected type 'table', got '${payload[2]}'`);
passed = false;
} else {
console.log('✅ Correct type');
}
// Verify table data is a Buffer (Arrow IPC format)
if (payload[1] instanceof Buffer || payload[1] instanceof Uint8Array) {
console.log('✅ Table data is Arrow IPC buffer');
console.log(` Buffer size: ${payload[1].length} bytes`);
} else {
console.log(`❌ Expected Buffer/Uint8Array, got ${typeof payload[1]}`);
passed = false;
}
// Test round-trip with Arrow deserialization
console.log('\n=== Arrow Deserialization Test ===');
try {
const table = arrow.tableFromRawBytes(payload[1]);
console.log(`✅ Arrow table deserialized successfully`);
console.log(` Schema: ${table.schema.fields.map(f => f.name).join(', ')}`);
console.log(` Num rows: ${table.numRows}`);
} catch (e) {
console.log('❌ Arrow deserialization failed:', e.message);
passed = false;
}
// Final result
console.log('\n=== Test Result ===');
if (passed) {
console.log('✅ ALL TESTS PASSED');
process.exit(0);
} else {
console.log('❌ SOME TESTS FAILED');
process.exit(1);
}
} catch (error) {
console.error('❌ Test failed with error:', error.message);
console.error(error.stack);
process.exit(1);
}
}
runTest();

View File

@@ -0,0 +1,179 @@
/**
* JavaScript Table Sender Test
* Tests the smartsend function with table (Arrow IPC) payloads
*/
const NATSBridge = require('../src/natbridge.js');
const TEST_SUBJECT = '/test/table';
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
async function runTest() {
console.log('=== JavaScript Table Sender Test ===\n');
const correlationId = NATSBridge.uuidv4();
console.log(`Correlation ID: ${correlationId}`);
console.log(`Subject: ${TEST_SUBJECT}`);
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
// Test data - table data as array of objects
const tableData = [
{ id: 1, name: 'Alice', age: 30, active: true },
{ id: 2, name: 'Bob', age: 25, active: false },
{ id: 3, name: 'Charlie', age: 35, active: true },
{ id: 4, name: 'Diana', age: 28, active: true },
{ id: 5, name: 'Eve', age: 32, active: false }
];
const testData = [
['users_table', tableData, 'table']
];
try {
// Send the message
console.log('Sending table payload...');
const [env, envJsonStr] = await NATSBridge.smartsend(
TEST_SUBJECT,
testData,
{
broker_url: TEST_BROKER_URL,
fileserver_url: TEST_FILESERVER_URL,
correlation_id: correlationId,
msg_purpose: 'test',
sender_name: 'js-table-test',
is_publish: false
}
);
console.log('\n=== Envelope Created ===');
console.log(`Correlation ID: ${env.correlation_id}`);
console.log(`Message ID: ${env.msg_id}`);
console.log(`Timestamp: ${env.timestamp}`);
console.log(`Subject: ${env.send_to}`);
console.log(`Purpose: ${env.msg_purpose}`);
console.log(`Sender: ${env.sender_name}`);
console.log(`Payloads: ${env.payloads.length}\n`);
// Validate envelope structure
console.log('=== Validation ===');
let passed = true;
if (env.payloads.length !== 1) {
console.log(`❌ Expected 1 payload, got ${env.payloads.length}`);
passed = false;
} else {
console.log('✅ Correct number of payloads');
}
const payload = env.payloads[0];
if (payload.dataname !== 'users_table') {
console.log(`❌ Expected dataname 'users_table', got '${payload.dataname}'`);
passed = false;
} else {
console.log('✅ Correct dataname');
}
if (payload.payload_type !== 'table') {
console.log(`❌ Expected payload_type 'table', got '${payload.payload_type}'`);
passed = false;
} else {
console.log('✅ Correct payload_type');
}
if (payload.transport !== 'direct') {
console.log(`❌ Expected transport 'direct', got '${payload.transport}'`);
passed = false;
} else {
console.log('✅ Correct transport');
}
if (payload.encoding !== 'base64') {
console.log(`❌ Expected encoding 'base64', got '${payload.encoding}'`);
passed = false;
} else {
console.log('✅ Correct encoding');
}
// Verify Arrow IPC data can be decoded
console.log('\n=== Arrow IPC Verification ===');
const decodedData = Buffer.from(payload.data, 'base64');
console.log(`Arrow IPC buffer size: ${decodedData.length} bytes`);
if (decodedData.length > 0) {
console.log('✅ Arrow IPC data present');
} else {
console.log('❌ Arrow IPC data is empty');
passed = false;
}
// Test with larger table
console.log('\n=== Larger Table Test ===');
const largeTableData = [];
for (let i = 1; i <= 100; i++) {
largeTableData.push({
id: i,
name: `User${i}`,
age: Math.floor(Math.random() * 100),
active: Math.random() > 0.5,
score: Math.random() * 100
});
}
const largeTestData = [
['large_table', largeTableData, 'table']
];
const [largeEnv, _] = await NATSBridge.smartsend(
TEST_SUBJECT,
largeTestData,
{
broker_url: TEST_BROKER_URL,
fileserver_url: TEST_FILESERVER_URL,
correlation_id: 'large-' + correlationId,
is_publish: false
}
);
if (largeEnv.payloads.length === 1) {
console.log('✅ Large table handled correctly');
console.log(` Size: ${largeEnv.payloads[0].size} bytes`);
} else {
console.log('❌ Large table handling failed');
passed = false;
}
// Test JSON string output
console.log('\n=== JSON String Output Test ===');
try {
const parsed = JSON.parse(envJsonStr);
if (parsed.correlation_id === env.correlation_id &&
parsed.payloads.length === env.payloads.length) {
console.log('✅ JSON string is valid and matches envelope');
} else {
console.log('❌ JSON string does not match envelope');
passed = false;
}
} catch (e) {
console.log('❌ JSON string is invalid:', e.message);
passed = false;
}
// Final result
console.log('\n=== Test Result ===');
if (passed) {
console.log('✅ ALL TESTS PASSED');
process.exit(0);
} else {
console.log('❌ SOME TESTS FAILED');
process.exit(1);
}
} catch (error) {
console.error('❌ Test failed with error:', error.message);
console.error(error.stack);
process.exit(1);
}
}
runTest();

View File

@@ -0,0 +1,206 @@
/**
* JavaScript Text Receiver Test
* Tests the smartreceive function with text payloads
*/
const NATSBridge = require('../src/natbridge.js');
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
async function runTest() {
console.log('=== JavaScript Text Receiver Test ===\n');
// Create a mock NATS message with text payload
const testData = {
correlation_id: 'test-receiver-' + Date.now(),
msg_id: 'msg-' + Date.now(),
timestamp: new Date().toISOString(),
send_to: '/test/text',
msg_purpose: 'test',
sender_name: 'js-text-test',
sender_id: 'sender-' + Date.now(),
receiver_name: 'js-receiver',
receiver_id: 'receiver-' + Date.now(),
reply_to: '',
reply_to_msg_id: '',
broker_url: TEST_BROKER_URL,
metadata: {},
payloads: [
{
id: 'payload-' + Date.now(),
dataname: 'message',
payload_type: 'text',
transport: 'direct',
encoding: 'base64',
size: 38,
data: Buffer.from('Hello, NATSBridge! This is a test message.').toString('base64'),
metadata: { payload_bytes: 38 }
}
]
};
const mockMsg = {
payload: JSON.stringify(testData)
};
console.log('Mock Message Created:');
console.log(` Correlation ID: ${testData.correlation_id}`);
console.log(` Payloads: ${testData.payloads.length}`);
console.log(` Payload dataname: ${testData.payloads[0].dataname}`);
console.log(` Payload type: ${testData.payloads[0].payload_type}`);
console.log(` Transport: ${testData.payloads[0].transport}\n`);
try {
// Receive and process the message
console.log('Receiving and processing message...');
const env = await NATSBridge.smartreceive(
mockMsg,
{
max_retries: 3,
base_delay: 100,
max_delay: 1000
}
);
console.log('\n=== Received Envelope ===');
console.log(`Correlation ID: ${env.correlation_id}`);
console.log(`Message ID: ${env.msg_id}`);
console.log(`Timestamp: ${env.timestamp}`);
console.log(`Subject: ${env.send_to}`);
console.log(`Payloads: ${env.payloads.length}\n`);
// Validate received data
console.log('=== Validation ===');
let passed = true;
if (!env.correlation_id) {
console.log('❌ correlation_id is missing');
passed = false;
} else {
console.log('✅ correlation_id present');
}
if (env.payloads.length !== 1) {
console.log(`❌ Expected 1 payload, got ${env.payloads.length}`);
passed = false;
} else {
console.log('✅ Correct number of payloads');
}
const payload = env.payloads[0];
if (payload[0] !== 'message') {
console.log(`❌ Expected dataname 'message', got '${payload[0]}'`);
passed = false;
} else {
console.log('✅ Correct dataname');
}
if (payload[2] !== 'text') {
console.log(`❌ Expected type 'text', got '${payload[2]}'`);
passed = false;
} else {
console.log('✅ Correct type');
}
if (payload[1] !== 'Hello, NATSBridge! This is a test message.') {
console.log(`❌ Data mismatch`);
console.log(` Expected: Hello, NATSBridge! This is a test message.`);
console.log(` Got: ${payload[1]}`);
passed = false;
} else {
console.log('✅ Data correctly deserialized');
}
// Test with multiple text payloads
console.log('\n=== Multiple Text Payloads Test ===');
const multiTestData = {
correlation_id: 'multi-receiver-' + Date.now(),
msg_id: 'msg-' + Date.now(),
timestamp: new Date().toISOString(),
send_to: '/test/text',
msg_purpose: 'test',
sender_name: 'js-text-test',
sender_id: 'sender-' + Date.now(),
receiver_name: 'js-receiver',
receiver_id: 'receiver-' + Date.now(),
reply_to: '',
reply_to_msg_id: '',
broker_url: TEST_BROKER_URL,
metadata: {},
payloads: [
{
id: 'payload-1',
dataname: 'msg1',
payload_type: 'text',
transport: 'direct',
encoding: 'base64',
size: 16,
data: Buffer.from('First message').toString('base64'),
metadata: { payload_bytes: 16 }
},
{
id: 'payload-2',
dataname: 'msg2',
payload_type: 'text',
transport: 'direct',
encoding: 'base64',
size: 16,
data: Buffer.from('Second message').toString('base64'),
metadata: { payload_bytes: 16 }
},
{
id: 'payload-3',
dataname: 'msg3',
payload_type: 'text',
transport: 'direct',
encoding: 'base64',
size: 16,
data: Buffer.from('Third message').toString('base64'),
metadata: { payload_bytes: 16 }
}
]
};
const mockMultiMsg = {
payload: JSON.stringify(multiTestData)
};
const multiEnv = await NATSBridge.smartreceive(mockMultiMsg);
if (multiEnv.payloads.length === 3) {
console.log('✅ Multiple payloads handled correctly');
// Verify each payload
const expectedMessages = ['First message', 'Second message', 'Third message'];
for (let i = 0; i < 3; i++) {
if (multiEnv.payloads[i][1] === expectedMessages[i]) {
console.log(`✅ Payload ${i + 1} correctly deserialized`);
} else {
console.log(`❌ Payload ${i + 1} mismatch`);
passed = false;
}
}
} else {
console.log(`❌ Expected 3 payloads, got ${multiEnv.payloads.length}`);
passed = false;
}
// Final result
console.log('\n=== Test Result ===');
if (passed) {
console.log('✅ ALL TESTS PASSED');
process.exit(0);
} else {
console.log('❌ SOME TESTS FAILED');
process.exit(1);
}
} catch (error) {
console.error('❌ Test failed with error:', error.message);
console.error(error.stack);
process.exit(1);
}
}
runTest();

169
test/test_js_text_sender.js Normal file
View File

@@ -0,0 +1,169 @@
/**
* JavaScript Text Sender Test
* Tests the smartsend function with text payloads
*/
const NATSBridge = require('../src/natbridge.js');
const TEST_SUBJECT = '/test/text';
const TEST_BROKER_URL = process.env.NATS_URL || 'nats://localhost:4222';
const TEST_FILESERVER_URL = process.env.FILESERVER_URL || 'http://localhost:8080';
async function runTest() {
console.log('=== JavaScript Text Sender Test ===\n');
const correlationId = NATSBridge.uuidv4();
console.log(`Correlation ID: ${correlationId}`);
console.log(`Subject: ${TEST_SUBJECT}`);
console.log(`Broker URL: ${TEST_BROKER_URL}\n`);
// Test data
const textData = 'Hello, NATSBridge! This is a test message.';
const testData = [
['message', textData, 'text']
];
try {
// Send the message
console.log('Sending text payload...');
const [env, envJsonStr] = await NATSBridge.smartsend(
TEST_SUBJECT,
testData,
{
broker_url: TEST_BROKER_URL,
fileserver_url: TEST_FILESERVER_URL,
correlation_id: correlationId,
msg_purpose: 'test',
sender_name: 'js-text-test',
is_publish: false // Don't actually publish for this test
}
);
console.log('\n=== Envelope Created ===');
console.log(`Correlation ID: ${env.correlation_id}`);
console.log(`Message ID: ${env.msg_id}`);
console.log(`Timestamp: ${env.timestamp}`);
console.log(`Subject: ${env.send_to}`);
console.log(`Purpose: ${env.msg_purpose}`);
console.log(`Sender: ${env.sender_name}`);
console.log(`Payloads: ${env.payloads.length}\n`);
// Validate envelope structure
console.log('=== Validation ===');
let passed = true;
if (!env.correlation_id) {
console.log('❌ correlation_id is missing');
passed = false;
} else {
console.log('✅ correlation_id present');
}
if (!env.msg_id) {
console.log('❌ msg_id is missing');
passed = false;
} else {
console.log('✅ msg_id present');
}
if (!env.timestamp) {
console.log('❌ timestamp is missing');
passed = false;
} else {
console.log('✅ timestamp present');
}
if (env.payloads.length !== 1) {
console.log(`❌ Expected 1 payload, got ${env.payloads.length}`);
passed = false;
} else {
console.log('✅ Correct number of payloads');
}
const payload = env.payloads[0];
if (payload.dataname !== 'message') {
console.log(`❌ Expected dataname 'message', got '${payload.dataname}'`);
passed = false;
} else {
console.log('✅ Correct dataname');
}
if (payload.payload_type !== 'text') {
console.log(`❌ Expected payload_type 'text', got '${payload.payload_type}'`);
passed = false;
} else {
console.log('✅ Correct payload_type');
}
if (payload.transport !== 'direct') {
console.log(`❌ Expected transport 'direct', got '${payload.transport}'`);
passed = false;
} else {
console.log('✅ Correct transport');
}
if (payload.encoding !== 'base64') {
console.log(`❌ Expected encoding 'base64', got '${payload.encoding}'`);
passed = false;
} else {
console.log('✅ Correct encoding');
}
// Decode and verify the data
const decodedData = Buffer.from(payload.data, 'base64').toString('utf8');
if (decodedData !== textData) {
console.log(`❌ Decoded data mismatch`);
console.log(` Expected: ${textData}`);
console.log(` Got: ${decodedData}`);
passed = false;
} else {
console.log('✅ Data integrity verified');
}
console.log(`\nPayload size: ${payload.size} bytes`);
console.log(`Base64 data length: ${payload.data.length} chars`);
// Test with multiple text payloads
console.log('\n=== Multiple Text Payloads Test ===');
const multiTestData = [
['msg1', 'First message', 'text'],
['msg2', 'Second message', 'text'],
['msg3', 'Third message', 'text']
];
const [multiEnv, multiEnvJsonStr] = await NATSBridge.smartsend(
TEST_SUBJECT,
multiTestData,
{
broker_url: TEST_BROKER_URL,
fileserver_url: TEST_FILESERVER_URL,
correlation_id: 'multi-test-' + correlationId,
is_publish: false
}
);
if (multiEnv.payloads.length === 3) {
console.log('✅ Multiple payloads handled correctly');
} else {
console.log(`❌ Expected 3 payloads, got ${multiEnv.payloads.length}`);
passed = false;
}
// Final result
console.log('\n=== Test Result ===');
if (passed) {
console.log('✅ ALL TESTS PASSED');
process.exit(0);
} else {
console.log('❌ SOME TESTS FAILED');
process.exit(1);
}
} catch (error) {
console.error('❌ Test failed with error:', error.message);
console.error(error.stack);
process.exit(1);
}
}
runTest();

View File

@@ -0,0 +1,82 @@
#!/usr/bin/env julia
# Test script for Dictionary transport testing
# Tests receiving 1 large and 1 small Dictionaries via direct and link transport
# Uses NATSBridge.jl smartreceive with "dictionary" type
using NATS, JSON, UUIDs, Dates, PrettyPrinting, DataFrames, Arrow, HTTP
# Include the bridge module
include("../src/NATSBridge.jl")
using .NATSBridge
# Configuration
const SUBJECT = "/NATSBridge_dict_test"
const NATS_URL = "nats.yiem.cc"
const FILESERVER_URL = "http://192.168.88.104:8080"
# ------------------------------------------------------------------------------------------------ #
# test dictionary transfer #
# ------------------------------------------------------------------------------------------------ #
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] $message")
end
# Receiver: Listen for messages and verify Dictionary handling
function test_dict_receive()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
log_trace("Received message on $(msg.subject)")
# Use NATSBridge.smartreceive to handle the data
# API: smartreceive(msg, download_handler; max_retries, base_delay, max_delay)
result = NATSBridge.smartreceive(
msg;
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result["payloads"]
if isa(data, JSON.Object{String, Any})
log_trace("Received Dictionary '$dataname' of type $data_type")
# Display dictionary contents
println(" Contents:")
for (key, value) in data
println(" $key => $value")
end
# Save to JSON file
output_path = "./received_$dataname.json"
json_str = JSON.json(data, 2)
write(output_path, json_str)
log_trace("Saved Dictionary to $output_path")
else
log_trace("Received unexpected data type for '$dataname': $(typeof(data))")
end
end
end
# Keep listening for 10 seconds
sleep(120)
NATS.drain(conn)
end
# Run the test
println("Starting Dictionary transport test...")
println("Note: This receiver will wait for messages from the sender.")
println("Run test_julia_to_julia_dict_sender.jl first to send test data.")
# Run receiver
println("testing smartreceive")
test_dict_receive()
println("Test completed.")

View File

@@ -0,0 +1,137 @@
#!/usr/bin/env julia
# Test script for Dictionary transport testing
# Tests sending 1 large and 1 small Dictionaries via direct and link transport
# Uses NATSBridge.jl smartsend with "dictionary" type
using NATS, JSON, UUIDs, Dates, PrettyPrinting, DataFrames, Arrow, HTTP
# Include the bridge module
include("../src/NATSBridge.jl")
using .NATSBridge
# Configuration
const SUBJECT = "/NATSBridge_dict_test"
const NATS_URL = "nats.yiem.cc"
const FILESERVER_URL = "http://192.168.88.104:8080"
# Create correlation ID for tracing
correlation_id = string(uuid4())
# ------------------------------------------------------------------------------------------------ #
# test dictionary transfer #
# ------------------------------------------------------------------------------------------------ #
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] [Correlation: $correlation_id] $message")
end
# File upload handler for plik server
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
# Get upload ID
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
# Upload file
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
# Sender: Send Dictionaries via smartsend
function test_dict_send()
# Create a small Dictionary (will use direct transport)
small_dict = Dict(
"name" => "Alice",
"age" => 30,
"scores" => [95, 88, 92],
"metadata" => Dict(
"height" => 155,
"weight" => 55
)
)
# Create a large Dictionary (will use link transport if > 1MB)
# Generate a larger dataset (~2MB to ensure link transport)
large_dict = Dict(
"ids" => collect(1:50000),
"names" => ["User_$i" for i in 1:50000],
"scores" => rand(1:100, 50000),
"categories" => ["Category_$(rand(1:10))" for i in 1:50000],
"metadata" => Dict(
"source" => "test_generator",
"timestamp" => string(Dates.now())
)
)
# Test data 1: small Dictionary
data1 = ("small_dict", small_dict, "dictionary")
# Test data 2: large Dictionary
data2 = ("large_dict", large_dict, "dictionary")
# Use smartsend with dictionary type
# For small Dictionary: will use direct transport (JSON encoded)
# For large Dictionary: will use link transport (uploaded to fileserver)
env, env_json_str = NATSBridge.smartsend(
SUBJECT,
[data1, data2]; # List of (dataname, data, type) tuples
broker_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserver_upload_handler = plik_upload_handler,
size_threshold = 1_000_000, # 1MB threshold
correlation_id = correlation_id,
msg_purpose = "chat",
sender_name = "dict_sender",
receiver_name = "",
receiver_id = "",
reply_to = "",
reply_to_msg_id = "",
is_publish = true # Publish the message to NATS
)
log_trace("Sent message with $(length(env.payloads)) payloads")
# Log transport type for each payload
for (i, payload) in enumerate(env.payloads)
log_trace("Payload $i ('$payload.dataname'):")
log_trace(" Transport: $(payload.transport)")
log_trace(" Type: $(payload.payload_type)")
log_trace(" Size: $(payload.size) bytes")
log_trace(" Encoding: $(payload.encoding)")
if payload.transport == "link"
log_trace(" URL: $(payload.data)")
end
end
end
# Run the test
println("Starting Dictionary transport test...")
println("Correlation ID: $correlation_id")
# Run sender
println("start smartsend for dictionaries")
test_dict_send()
println("Test completed.")

View File

@@ -0,0 +1,84 @@
#!/usr/bin/env julia
# Test script for large payload testing using binary transport
# Tests sending a large file (> 1MB) via smartsend with binary type
# Updated to match NATSBridge.jl API
using NATS, JSON, UUIDs, Dates, PrettyPrinting, DataFrames, Arrow, HTTP
# workdir =
# Include the bridge module
include("../src/NATSBridge.jl")
using .NATSBridge
# Configuration
const SUBJECT = "/NATSBridge_test"
const NATS_URL = "nats.yiem.cc"
const FILESERVER_URL = "http://192.168.88.104:8080"
# ------------------------------------------------------------------------------------------------ #
# test file transfer #
# ------------------------------------------------------------------------------------------------ #
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] $message")
end
# Receiver: Listen for messages and verify large payload handling
function test_large_binary_receive()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
log_trace("Received message on $(msg.subject)")
# Use NATSBridge.smartreceive to handle the data
# API: smartreceive(msg, download_handler; max_retries, base_delay, max_delay)
result = NATSBridge.smartreceive(
msg;
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result["payloads"]
# Check transport type from the envelope
# For link transport, data is the URL string
# For direct transport, data is the actual payload bytes
if isa(data, Vector{UInt8})
file_size = length(data)
log_trace("Received $(file_size) bytes of binary data for '$dataname' of type $data_type")
# Save received data to a test file
output_path = "./new_$dataname"
write(output_path, data)
log_trace("Saved received data to $output_path")
else
log_trace("Received $(file_size) bytes of binary data for '$dataname' of type $data_type")
end
end
end
# Keep listening for 10 seconds
sleep(120)
NATS.drain(conn)
end
# Run the test
println("Starting large binary payload test...")
# # Run sender first
# println("start smartsend")
# test_large_binary_send()
# Run receiver
println("testing smartreceive")
test_large_binary_receive()
println("Test completed.")

View File

@@ -0,0 +1,123 @@
#!/usr/bin/env julia
# Test script for large payload testing using binary transport
# Tests sending a large file (> 1MB) via smartsend with binary type
# Updated to match NATSBridge.jl API
using NATS, JSON, UUIDs, Dates, PrettyPrinting, DataFrames, Arrow, HTTP
# workdir =
# Include the bridge module
include("../src/NATSBridge.jl")
using .NATSBridge
# Configuration
const SUBJECT = "/NATSBridge_test"
const NATS_URL = "nats.yiem.cc"
const FILESERVER_URL = "http://192.168.88.104:8080"
# Create correlation ID for tracing
correlation_id = string(uuid4())
# ------------------------------------------------------------------------------------------------ #
# test file transfer #
# ------------------------------------------------------------------------------------------------ #
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] [Correlation: $correlation_id] $message")
end
# File upload handler for plik server
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
# Get upload ID
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
# Upload file
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
# Sender: Send large binary file via smartsend
function test_large_binary_send()
# Read the large file as binary data
# test data 1
file_path1 = "./testFile_large.zip"
file_data1 = read(file_path1)
filename1 = basename(file_path1)
data1 = (filename1, file_data1, "binary")
# test data 2
file_path2 = "./testFile_small.zip"
file_data2 = read(file_path2)
filename2 = basename(file_path2)
data2 = (filename2, file_data2, "binary")
# Use smartsend with binary type - will automatically use link transport
# if file size exceeds the threshold (1MB by default)
# API: smartsend(subject, [(dataname, data, type), ...]; keywords...)
env, env_json_str = NATSBridge.smartsend(
SUBJECT,
[data1, data2]; # List of (dataname, data, type) tuples
broker_url = NATS_URL;
fileserver_url = FILESERVER_URL,
fileserver_upload_handler = plik_upload_handler,
size_threshold = 1_000_000,
correlation_id = correlation_id,
msg_purpose = "chat",
sender_name = "sender",
receiver_name = "",
receiver_id = "",
reply_to = "",
reply_to_msg_id = "",
is_publish = true # Publish the message to NATS
)
log_trace("Sent message with transport: $(env.payloads[1].transport)")
log_trace("Envelope type: $(env.payloads[1].payload_type)")
# Check if link transport was used
if env.payloads[1].transport == "link"
log_trace("Using link transport - file uploaded to HTTP server")
log_trace("URL: $(env.payloads[1].data)")
else
log_trace("Using direct transport - payload sent via NATS")
end
end
# Run the test
println("Starting large binary payload test...")
println("Correlation ID: $correlation_id")
# Run sender first
println("start smartsend")
test_large_binary_send()
# Run receiver
# println("testing smartreceive")
# test_large_binary_receive()
println("Test completed.")

View File

@@ -1,190 +0,0 @@
#!/usr/bin/env julia
# Test script for large payload testing using binary transport
# Tests sending a large file (> 1MB) via smartsend with binary type
using NATS, JSON, UUIDs, Dates
# Include the bridge module
include("../src/NATSBridge.jl")
using .NATSBridge
# Configuration
const SUBJECT = "/large_binary_test"
const NATS_URL = "nats.yiem.cc"
const FILESERVER_URL = "http://192.168.88.104:8080"
# Create correlation ID for tracing
correlation_id = string(uuid4())
# ------------------------------------------------------------------------------------------------ #
# test file transfer #
# ------------------------------------------------------------------------------------------------ #
# File path for large binary payload test
const FILE_PATH = "./testFile_small.zip"
const filename = basename(FILE_PATH)
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] [Correlation: $correlation_id] $message")
end
# Sender: Send large binary file via smartsend
function test_large_binary_send()
conn = NATS.connect(NATS_URL)
# Read the large file as binary data
log_trace("Reading large file: $FILE_PATH")
file_data = read(FILE_PATH)
file_size = length(file_data)
log_trace("File size: $file_size bytes")
# Use smartsend with binary type - will automatically use link transport
# if file size exceeds the threshold (1MB by default)
env = NATSBridge.smartsend(
SUBJECT,
file_data,
"binary",
nats_url = NATS_URL,
fileserver_url = FILESERVER_URL;
dataname=filename
)
log_trace("Sent message with transport: $(env.transport)")
log_trace("Envelope type: $(env.type)")
# Check if link transport was used
if env.transport == "link"
log_trace("Using link transport - file uploaded to HTTP server")
log_trace("URL: $(env.url)")
else
log_trace("Using direct transport - payload sent via NATS")
end
NATS.drain(conn)
end
# Receiver: Listen for messages and verify large payload handling
function test_large_binary_receive()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
log_trace("Received message on $(msg.subject)")
# Use NATSBridge.smartreceive to handle the data
result = NATSBridge.smartreceive(msg)
# Check transport type
if result.envelope.transport == "direct"
log_trace("Received direct transport")
else
# For link transport, result.data is the URL
log_trace("Received link transport")
end
# Verify the received data matches the original
if result.envelope.type == "binary"
if isa(result.data, Vector{UInt8})
file_size = length(result.data)
log_trace("Received $(file_size) bytes of binary data")
# Save received data to a test file
println("metadata ", result.envelope.metadata)
dataname = result.envelope.metadata["dataname"]
if dataname != "NA"
output_path = "./new_$dataname"
write(output_path, result.data)
log_trace("Saved received data to $output_path")
end
# Verify file size
original_size = length(read(FILE_PATH))
if file_size == result.envelope.metadata["content_length"]
log_trace("SUCCESS: File size matches! Original: $(result.envelope.metadata["content_length"]) bytes")
else
log_trace("WARNING: File size mismatch! Original: $(result.envelope.metadata["content_length"]), Received: $file_size")
end
end
end
end
# Keep listening for 10 seconds
sleep(120)
NATS.drain(conn)
end
# Run the test
println("Starting large binary payload test...")
println("Correlation ID: $correlation_id")
println("File: $FILE_PATH")
# Run sender first
println("start smartsend")
test_large_binary_send()
# # Run receiver
# println("testing smartreceive")
# test_large_binary_receive()
println("Test completed.")

View File

@@ -0,0 +1,228 @@
#!/usr/bin/env julia
# Test script for mixed-content message testing
# Tests receiving a mix of text, json, table, image, audio, video, and binary data
# from Julia serviceA to Julia serviceB using NATSBridge.jl smartreceive
#
# This test demonstrates that any combination and any number of mixed content
# can be sent and received correctly.
using NATS, JSON, UUIDs, Dates, PrettyPrinting, DataFrames, Arrow, HTTP, Base64
# Include the bridge module
include("../src/NATSBridge.jl")
using .NATSBridge
# Configuration
const SUBJECT = "/NATSBridge_mix_test"
const NATS_URL = "nats.yiem.cc"
const FILESERVER_URL = "http://192.168.88.104:8080"
# ------------------------------------------------------------------------------------------------ #
# test mixed content transfer #
# ------------------------------------------------------------------------------------------------ #
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] $message")
end
# Receiver: Listen for messages and verify mixed content handling
function test_mix_receive()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
log_trace("Received message on $(msg.subject)")
# Use NATSBridge.smartreceive to handle the data
# API: smartreceive(msg, download_handler; max_retries, base_delay, max_delay)
result = NATSBridge.smartreceive(
msg;
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
log_trace("Received $(length(result["payloads"])) payloads")
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result["payloads"]
log_trace("\n=== Payload: $dataname (type: $data_type) ===")
# Handle different data types
if data_type == "text"
# Text data - should be a String
if isa(data, String)
log_trace(" Type: String")
log_trace(" Length: $(length(data)) characters")
# Display first 200 characters
if length(data) > 200
log_trace(" First 200 chars: $(data[1:200])...")
else
log_trace(" Content: $data")
end
# Save to file
output_path = "./received_$dataname.txt"
write(output_path, data)
log_trace(" Saved to: $output_path")
else
log_trace(" ERROR: Expected String, got $(typeof(data))")
end
elseif data_type == "dictionary"
# Dictionary data - should be JSON object
if isa(data, JSON.Object{String, Any})
log_trace(" Type: Dict")
log_trace(" Keys: $(keys(data))")
# Display nested content
for (key, value) in data
log_trace(" $key => $value")
end
# Save to JSON file
output_path = "./received_$dataname.json"
json_str = JSON.json(data, 2)
write(output_path, json_str)
log_trace(" Saved to: $output_path")
else
log_trace(" ERROR: Expected Dict, got $(typeof(data))")
end
elseif data_type == "table"
# Table data - should be a DataFrame
data = DataFrame(data)
if isa(data, DataFrame)
log_trace(" Type: DataFrame")
log_trace(" Dimensions: $(size(data, 1)) rows x $(size(data, 2)) columns")
log_trace(" Columns: $(names(data))")
# Display first few rows
log_trace(" First 5 rows:")
display(data[1:min(5, size(data, 1)), :])
# Save to Arrow file
output_path = "./received_$dataname.arrow"
io = IOBuffer()
Arrow.write(io, data)
write(output_path, take!(io))
log_trace(" Saved to: $output_path")
else
log_trace(" ERROR: Expected DataFrame, got $(typeof(data))")
end
elseif data_type == "image"
# Image data - should be Vector{UInt8}
if isa(data, Vector{UInt8})
log_trace(" Type: Vector{UInt8} (binary)")
log_trace(" Size: $(length(data)) bytes")
# Save to file
output_path = "./received_$dataname.bin"
write(output_path, data)
log_trace(" Saved to: $output_path")
else
log_trace(" ERROR: Expected Vector{UInt8}, got $(typeof(data))")
end
elseif data_type == "audio"
# Audio data - should be Vector{UInt8}
if isa(data, Vector{UInt8})
log_trace(" Type: Vector{UInt8} (binary)")
log_trace(" Size: $(length(data)) bytes")
# Save to file
output_path = "./received_$dataname.bin"
write(output_path, data)
log_trace(" Saved to: $output_path")
else
log_trace(" ERROR: Expected Vector{UInt8}, got $(typeof(data))")
end
elseif data_type == "video"
# Video data - should be Vector{UInt8}
if isa(data, Vector{UInt8})
log_trace(" Type: Vector{UInt8} (binary)")
log_trace(" Size: $(length(data)) bytes")
# Save to file
output_path = "./received_$dataname.bin"
write(output_path, data)
log_trace(" Saved to: $output_path")
else
log_trace(" ERROR: Expected Vector{UInt8}, got $(typeof(data))")
end
elseif data_type == "binary"
# Binary data - should be Vector{UInt8}
if isa(data, Vector{UInt8})
log_trace(" Type: Vector{UInt8} (binary)")
log_trace(" Size: $(length(data)) bytes")
# Save to file
output_path = "./received_$dataname.bin"
write(output_path, data)
log_trace(" Saved to: $output_path")
else
log_trace(" ERROR: Expected Vector{UInt8}, got $(typeof(data))")
end
else
log_trace(" ERROR: Unknown data type '$data_type'")
end
end
# Summary
println("\n=== Verification Summary ===")
text_count = count(x -> x[3] == "text", result["payloads"])
dict_count = count(x -> x[3] == "dictionary", result["payloads"])
table_count = count(x -> x[3] == "table", result["payloads"])
image_count = count(x -> x[3] == "image", result["payloads"])
audio_count = count(x -> x[3] == "audio", result["payloads"])
video_count = count(x -> x[3] == "video", result["payloads"])
binary_count = count(x -> x[3] == "binary", result["payloads"])
log_trace("Text payloads: $text_count")
log_trace("Dictionary payloads: $dict_count")
log_trace("Table payloads: $table_count")
log_trace("Image payloads: $image_count")
log_trace("Audio payloads: $audio_count")
log_trace("Video payloads: $video_count")
log_trace("Binary payloads: $binary_count")
# Print transport type info for each payload if available
println("\n=== Payload Details ===")
for (dataname, data, data_type) in result["payloads"]
if data_type in ["image", "audio", "video", "binary"]
log_trace("$dataname: $(length(data)) bytes (binary)")
elseif data_type == "table"
data = DataFrame(data)
log_trace("$dataname: $(size(data, 1)) rows x $(size(data, 2)) columns (DataFrame)")
elseif data_type == "dictionary"
log_trace("$dataname: $(length(JSON.json(data))) bytes (Dict)")
elseif data_type == "text"
log_trace("$dataname: $(length(data)) characters (String)")
end
end
end
# Keep listening for 2 minutes
sleep(120)
NATS.drain(conn)
end
# Run the test
println("Starting mixed-content transport test...")
println("Note: This receiver will wait for messages from the sender.")
println("Run test_julia_to_julia_mix_sender.jl first to send test data.")
# Run receiver
println("\ntesting smartreceive for mixed content")
test_mix_receive()
println("\nTest completed.")

View File

@@ -0,0 +1,240 @@
#!/usr/bin/env julia
# Test script for mixed-content message testing
# Tests sending a mix of text, json, table, image, audio, video, and binary data
# from Julia serviceA to Julia serviceB using NATSBridge.jl smartsend
#
# This test demonstrates that any combination and any number of mixed content
# can be sent and received correctly.
using NATS, JSON, UUIDs, Dates, PrettyPrinting, DataFrames, Arrow, HTTP, Base64
# Include the bridge module
include("../src/NATSBridge.jl")
using .NATSBridge
# Configuration
const SUBJECT = "/NATSBridge_mix_test"
const NATS_URL = "nats.yiem.cc"
const FILESERVER_URL = "http://192.168.88.104:8080"
# Create correlation ID for tracing
correlation_id = string(uuid4())
# ------------------------------------------------------------------------------------------------ #
# test mixed content transfer #
# ------------------------------------------------------------------------------------------------ #
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] [Correlation: $correlation_id] $message")
end
# File upload handler for plik server
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
# Get upload ID
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
# Upload file
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
# Helper: Create sample data for each type
function create_sample_data()
# Text data (small - direct transport)
text_data = "Hello! This is a test chat message. 🎉\nHow are you doing today? 😊"
# Dictionary/JSON data (medium - could be direct or link)
dict_data = Dict(
"type" => "chat",
"sender" => "serviceA",
"receiver" => "serviceB",
"metadata" => Dict(
"timestamp" => string(Dates.now()),
"priority" => "high",
"tags" => ["urgent", "chat", "test"]
),
"content" => Dict(
"text" => "This is a JSON-formatted chat message with nested structure.",
"format" => "markdown",
"mentions" => ["user1", "user2"]
)
)
# Table data (DataFrame - small - direct transport)
table_data_small = DataFrame(
id = 1:10,
message = ["msg_$i" for i in 1:10],
sender = ["sender_$i" for i in 1:10],
timestamp = [string(Dates.now()) for _ in 1:10],
priority = rand(1:3, 10)
)
# Table data (DataFrame - large - link transport)
# ~1.5MB of data (150,000 rows) - should trigger link transport
table_data_large = DataFrame(
id = 1:150_000,
message = ["msg_$i" for i in 1:150_000],
sender = ["sender_$i" for i in 1:150_000],
timestamp = [string(Dates.now()) for i in 1:150_000],
priority = rand(1:3, 150_000)
)
# Image data (small binary - direct transport)
# Create a simple 10x10 pixel PNG-like data (128 bytes header + 100 pixels = 112 bytes)
# Using simple RGB data (10*10*3 = 300 bytes of pixel data)
image_width = 10
image_height = 10
image_data = UInt8[]
# PNG header (simplified)
push!(image_data, 0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A)
# Simple RGB data (RGBRGBRGB...)
for i in 1:image_width*image_height
push!(image_data, 0xFF, 0x00, 0x00) # Red pixel
end
# Image data (large - link transport)
# Create a larger image (~1.5MB) to test link transport
large_image_width = 500
large_image_height = 1000
large_image_data = UInt8[]
# PNG header (simplified for 500x1000)
push!(large_image_data, 0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A)
# RGB data (500*1000*3 = 1,500,000 bytes)
for i in 1:large_image_width*large_image_height
push!(large_image_data, rand(1:255), rand(1:255), rand(1:255)) # Random color pixels
end
# Audio data (small binary - direct transport)
audio_data = UInt8[rand(1:255) for _ in 1:100]
# Audio data (large - link transport)
# ~1.5MB of audio-like data
large_audio_data = UInt8[rand(1:255) for _ in 1:1_500_000]
# Video data (small binary - direct transport)
video_data = UInt8[rand(1:255) for _ in 1:150]
# Video data (large - link transport)
# ~1.5MB of video-like data
large_video_data = UInt8[rand(1:255) for _ in 1:1_500_000]
# Binary data (small - direct transport)
binary_data = UInt8[rand(1:255) for _ in 1:200]
# Binary data (large - link transport)
# ~1.5MB of binary data
large_binary_data = UInt8[rand(1:255) for _ in 1:1_500_000]
return (
text_data,
dict_data,
table_data_small,
table_data_large,
image_data,
large_image_data,
audio_data,
large_audio_data,
video_data,
large_video_data,
binary_data,
large_binary_data
)
end
# Sender: Send mixed content via smartsend
function test_mix_send()
# Create sample data
(text_data, dict_data, table_data_small, table_data_large, image_data, large_image_data, audio_data, large_audio_data, video_data, large_video_data, binary_data, large_binary_data) = create_sample_data()
# Create payloads list - mixed content with both small and large data
# Small data uses direct transport, large data uses link transport
payloads = [
# Small data (direct transport) - text, dictionary, small table
("chat_text", text_data, "text"),
("chat_json", dict_data, "dictionary"),
("chat_table_small", table_data_small, "table"),
# Large data (link transport) - large table, large image, large audio, large video, large binary
("chat_table_large", table_data_large, "table"),
("user_image_large", large_image_data, "image"),
("audio_clip_large", large_audio_data, "audio"),
("video_clip_large", large_video_data, "video"),
("binary_file_large", large_binary_data, "binary")
]
# Use smartsend with mixed content
sendinfo = NATSBridge.smartsend(
SUBJECT,
payloads; # List of (dataname, data, type) tuples
broker_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserver_upload_handler = plik_upload_handler,
size_threshold = 1_000_000, # 1MB threshold
correlation_id = correlation_id,
msg_purpose = "chat",
sender_name = "mix_sender",
receiver_name = "",
receiver_id = "",
reply_to = "",
reply_to_msg_id = "",
is_publish = true # Publish the message to NATS
)
env, env_json_str = sendinfo
log_trace("Sent message with $(length(env.payloads)) payloads")
# Log transport type for each payload
for (i, payload) in enumerate(env.payloads)
log_trace("Payload $i ('$payload.dataname'):")
log_trace(" Transport: $(payload.transport)")
log_trace(" Type: $(payload.payload_type)")
log_trace(" Size: $(payload.size) bytes")
log_trace(" Encoding: $(payload.encoding)")
if payload.transport == "link"
log_trace(" URL: $(payload.data)")
end
end
# Summary
println("\n--- Transport Summary ---")
direct_count = count(p -> p.transport == "direct", env.payloads)
link_count = count(p -> p.transport == "link", env.payloads)
log_trace("Direct transport: $direct_count payloads")
log_trace("Link transport: $link_count payloads")
end
# Run the test
println("Starting mixed-content transport test...")
println("Correlation ID: $correlation_id")
# Run sender
println("start smartsend for mixed content")
test_mix_send()
println("\nTest completed.")
println("Note: Run test_julia_to_julia_mix_receiver.jl to receive the messages.")

View File

@@ -0,0 +1,84 @@
#!/usr/bin/env julia
# Test script for DataFrame table transport testing
# Tests receiving 1 large and 1 small DataFrames via direct and link transport
# Uses NATSBridge.jl smartreceive with "table" type
using NATS, JSON, UUIDs, Dates, PrettyPrinting, DataFrames, Arrow, HTTP
# Include the bridge module
include("../src/NATSBridge.jl")
using .NATSBridge
# Configuration
const SUBJECT = "/NATSBridge_table_test"
const NATS_URL = "nats.yiem.cc"
const FILESERVER_URL = "http://192.168.88.104:8080"
# ------------------------------------------------------------------------------------------------ #
# test table transfer #
# ------------------------------------------------------------------------------------------------ #
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] $message")
end
# Receiver: Listen for messages and verify DataFrame table handling
function test_table_receive()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
log_trace("Received message on $(msg.subject)")
# Use NATSBridge.smartreceive to handle the data
# API: smartreceive(msg, download_handler; max_retries, base_delay, max_delay)
result = NATSBridge.smartreceive(
msg;
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result["payloads"]
data = DataFrame(data)
if isa(data, DataFrame)
log_trace("Received DataFrame '$dataname' of type $data_type")
log_trace(" Dimensions: $(size(data, 1)) rows x $(size(data, 2)) columns")
log_trace(" Column names: $(names(data))")
# Display first few rows
println(" First 5 rows:")
display(data[1:min(5, size(data, 1)), :])
# Save to file
output_path = "./received_$dataname.arrow"
io = IOBuffer()
Arrow.write(io, data)
write(output_path, take!(io))
log_trace("Saved DataFrame to $output_path")
else
log_trace("Received unexpected data type for '$dataname': $(typeof(data))")
end
end
end
# Keep listening for 10 seconds
sleep(120)
NATS.drain(conn)
end
# Run the test
println("Starting DataFrame table transport test...")
println("Note: This receiver will wait for messages from the sender.")
println("Run test_julia_to_julia_table_sender.jl first to send test data.")
# Run receiver
println("testing smartreceive")
test_table_receive()
println("Test completed.")

View File

@@ -0,0 +1,135 @@
#!/usr/bin/env julia
# Test script for DataFrame table transport testing
# Tests sending 1 large and 1 small DataFrames via direct and link transport
# Uses NATSBridge.jl smartsend with "table" type
using NATS, JSON, UUIDs, Dates, PrettyPrinting, DataFrames, Arrow, HTTP
# Include the bridge module
include("../src/NATSBridge.jl")
using .NATSBridge
# Configuration
const SUBJECT = "/NATSBridge_table_test"
const NATS_URL = "nats.yiem.cc"
const FILESERVER_URL = "http://192.168.88.104:8080"
# Create correlation ID for tracing
correlation_id = string(uuid4())
# ------------------------------------------------------------------------------------------------ #
# test table transfer #
# ------------------------------------------------------------------------------------------------ #
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] [Correlation: $correlation_id] $message")
end
# File upload handler for plik server
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
# Get upload ID
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
# Upload file
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
# Sender: Send DataFrame tables via smartsend
function test_table_send()
# Create a small DataFrame (will use direct transport)
small_df = DataFrame(
id = 1:10,
name = ["Alice", "Bob", "Charlie", "Diana", "Eve", "Frank", "Grace", "Henry", "Ivy", "Jack"],
score = [95, 88, 92, 85, 90, 78, 95, 88, 92, 85],
category = ["A", "B", "A", "B", "A", "B", "A", "B", "A", "B"]
)
# Create a large DataFrame (will use link transport if > 1MB)
# Generate a larger dataset (~2MB to ensure link transport)
large_ids = 1:50000
large_names = ["User_$i" for i in 1:50000]
large_scores = rand(1:100, 50000)
large_categories = ["Category_$(rand(1:10))" for i in 1:50000]
large_df = DataFrame(
id = large_ids,
name = large_names,
score = large_scores,
category = large_categories
)
# Test data 1: small DataFrame
data1 = ("small_table", small_df, "table")
# Test data 2: large DataFrame
data2 = ("large_table", large_df, "table")
# Use smartsend with table type
# For small DataFrame: will use direct transport (Base64 encoded Arrow IPC)
# For large DataFrame: will use link transport (uploaded to fileserver)
env, env_json_str = NATSBridge.smartsend(
SUBJECT,
[data1, data2]; # List of (dataname, data, type) tuples
broker_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserver_upload_handler = plik_upload_handler,
size_threshold = 1_000_000, # 1MB threshold
correlation_id = correlation_id,
msg_purpose = "chat",
sender_name = "table_sender",
receiver_name = "",
receiver_id = "",
reply_to = "",
reply_to_msg_id = "",
is_publish = true # Publish the message to NATS
)
log_trace("Sent message with $(length(env.payloads)) payloads")
# Log transport type for each payload
for (i, payload) in enumerate(env.payloads)
log_trace("Payload $i ('$payload.dataname'):")
log_trace(" Transport: $(payload.transport)")
log_trace(" Type: $(payload.payload_type)")
log_trace(" Size: $(payload.size) bytes")
log_trace(" Encoding: $(payload.encoding)")
if payload.transport == "link"
log_trace(" URL: $(payload.data)")
end
end
end
# Run the test
println("Starting DataFrame table transport test...")
println("Correlation ID: $correlation_id")
# Run sender
println("start smartsend for tables")
test_table_send()
println("Test completed.")

View File

@@ -0,0 +1,83 @@
#!/usr/bin/env julia
# Test script for text transport testing
# Tests receiving 1 large and 1 small text from Julia serviceA to Julia serviceB
# Uses NATSBridge.jl smartreceive with "text" type
using NATS, JSON, UUIDs, Dates, PrettyPrinting, DataFrames, Arrow, HTTP
# Include the bridge module
include("../src/NATSBridge.jl")
using .NATSBridge
# Configuration
const SUBJECT = "/NATSBridge_text_test"
const NATS_URL = "nats.yiem.cc"
const FILESERVER_URL = "http://192.168.88.104:8080"
# ------------------------------------------------------------------------------------------------ #
# test text transfer #
# ------------------------------------------------------------------------------------------------ #
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] $message")
end
# Receiver: Listen for messages and verify text handling
function test_text_receive()
conn = NATS.connect(NATS_URL)
NATS.subscribe(conn, SUBJECT) do msg
log_trace("Received message on $(msg.subject)")
# Use NATSBridge.smartreceive to handle the data
# API: smartreceive(msg, download_handler; max_retries, base_delay, max_delay)
result = NATSBridge.smartreceive(
msg;
max_retries = 5,
base_delay = 100,
max_delay = 5000
)
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
for (dataname, data, data_type) in result["payloads"]
if isa(data, String)
log_trace("Received text '$dataname' of type $data_type")
log_trace(" Length: $(length(data)) characters")
# Display first 100 characters
if length(data) > 100
log_trace(" First 100 characters: $(data[1:100])...")
else
log_trace(" Content: $data")
end
# Save to file
output_path = "./received_$dataname.txt"
write(output_path, data)
log_trace("Saved text to $output_path")
else
log_trace("Received unexpected data type for '$dataname': $(typeof(data))")
end
end
end
# Keep listening for 10 seconds
sleep(120)
NATS.drain(conn)
end
# Run the test
println("Starting text transport test...")
println("Note: This receiver will wait for messages from the sender.")
println("Run test_julia_to_julia_text_sender.jl first to send test data.")
# Run receiver
println("testing smartreceive for text")
test_text_receive()
println("Test completed.")

View File

@@ -0,0 +1,120 @@
#!/usr/bin/env julia
# Test script for text transport testing
# Tests sending 1 large and 1 small text from Julia serviceA to Julia serviceB
# Uses NATSBridge.jl smartsend with "text" type
using NATS, JSON, UUIDs, Dates, PrettyPrinting, DataFrames, Arrow, HTTP
# Include the bridge module
include("../src/NATSBridge.jl")
using .NATSBridge
# Configuration
const SUBJECT = "/NATSBridge_text_test"
const NATS_URL = "nats.yiem.cc"
const FILESERVER_URL = "http://192.168.88.104:8080"
# Create correlation ID for tracing
correlation_id = string(uuid4())
# ------------------------------------------------------------------------------------------------ #
# test text transfer #
# ------------------------------------------------------------------------------------------------ #
# Helper: Log with correlation ID
function log_trace(message)
timestamp = Dates.now()
println("[$timestamp] [Correlation: $correlation_id] $message")
end
# File upload handler for plik server
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
# Get upload ID
url_getUploadID = "$fileserver_url/upload"
headers = ["Content-Type" => "application/json"]
body = """{ "OneShot" : true }"""
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
responseJson = JSON.parse(String(httpResponse.body))
uploadid = responseJson["id"]
uploadtoken = responseJson["uploadToken"]
# Upload file
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
url_upload = "$fileserver_url/file/$uploadid"
headers = ["X-UploadToken" => uploadtoken]
form = HTTP.Form(Dict("file" => file_multipart))
httpResponse = HTTP.post(url_upload, headers, form)
responseJson = JSON.parse(String(httpResponse.body))
fileid = responseJson["id"]
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
end
# Sender: Send text via smartsend
function test_text_send()
# Create a small text (will use direct transport)
small_text = "Hello, this is a small text message. Testing direct transport via NATS."
# Create a large text (will use link transport if > 1MB)
# Generate a larger text (~2MB to ensure link transport)
large_text = join(["Line $i: This is a sample text line with some content to pad the size. " for i in 1:50000], "")
# Test data 1: small text
data1 = ("small_text", small_text, "text")
# Test data 2: large text
data2 = ("large_text", large_text, "text")
# Use smartsend with text type
# For small text: will use direct transport (Base64 encoded UTF-8)
# For large text: will use link transport (uploaded to fileserver)
env, env_json_str = NATSBridge.smartsend(
SUBJECT,
[data1, data2]; # List of (dataname, data, type) tuples
broker_url = NATS_URL,
fileserver_url = FILESERVER_URL,
fileserver_upload_handler = plik_upload_handler,
size_threshold = 1_000_000, # 1MB threshold
correlation_id = correlation_id,
msg_purpose = "chat",
sender_name = "text_sender",
receiver_name = "",
receiver_id = "",
reply_to = "",
reply_to_msg_id = "",
is_publish = true # Publish the message to NATS
)
log_trace("Sent message with $(length(env.payloads)) payloads")
# Log transport type for each payload
for (i, payload) in enumerate(env.payloads)
log_trace("Payload $i ('$payload.dataname'):")
log_trace(" Transport: $(payload.transport)")
log_trace(" Type: $(payload.payload_type)")
log_trace(" Size: $(payload.size) bytes")
log_trace(" Encoding: $(payload.encoding)")
if payload.transport == "link"
log_trace(" URL: $(payload.data)")
end
end
end
# Run the test
println("Starting text transport test...")
println("Correlation ID: $correlation_id")
# Run sender
println("start smartsend for text")
test_text_send()
println("Test completed.")

View File

@@ -0,0 +1,185 @@
"""
MicroPython Binary Receiver Test
Tests the smartreceive function with binary/image/audio/video payloads
Note: This test is designed for both MicroPython and desktop Python
for compatibility testing.
"""
import sys
import os
import json
import base64
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge_mpy import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
def run_test():
print('=== MicroPython Binary Receiver Test ===\n')
from natbridge_mpy import _generate_uuid
# Create mock NATS message with binary payloads
image_data = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
audio_data = bytes([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]) # FLAC header
video_data = bytes([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]) # MP4 header
generic_binary = bytes([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE])
test_data = {
'correlation_id': 'mpy-binary-receiver-' + _generate_uuid(),
'msg_id': _generate_uuid(),
'timestamp': '2024-01-15T10:30:00Z',
'send_to': '/test/binary',
'msg_purpose': 'test',
'sender_name': 'mpy-binary-test',
'sender_id': _generate_uuid(),
'receiver_name': 'mpy-receiver',
'receiver_id': _generate_uuid(),
'reply_to': '',
'reply_to_msg_id': '',
'broker_url': TEST_BROKER_URL,
'metadata': {},
'payloads': [
{
'id': _generate_uuid(),
'dataname': 'image',
'payload_type': 'image',
'transport': 'direct',
'encoding': 'base64',
'size': len(image_data),
'data': base64.b64encode(image_data).decode('ascii'),
'metadata': {'payload_bytes': len(image_data)}
},
{
'id': _generate_uuid(),
'dataname': 'audio',
'payload_type': 'audio',
'transport': 'direct',
'encoding': 'base64',
'size': len(audio_data),
'data': base64.b64encode(audio_data).decode('ascii'),
'metadata': {'payload_bytes': len(audio_data)}
},
{
'id': _generate_uuid(),
'dataname': 'video',
'payload_type': 'video',
'transport': 'direct',
'encoding': 'base64',
'size': len(video_data),
'data': base64.b64encode(video_data).decode('ascii'),
'metadata': {'payload_bytes': len(video_data)}
},
{
'id': _generate_uuid(),
'dataname': 'binary',
'payload_type': 'binary',
'transport': 'direct',
'encoding': 'base64',
'size': len(generic_binary),
'data': base64.b64encode(generic_binary).decode('ascii'),
'metadata': {'payload_bytes': len(generic_binary)}
}
]
}
mock_msg = {
'payload': json.dumps(test_data)
}
print('Mock Message Created:')
print(f' Correlation ID: {test_data["correlation_id"]}')
print(f' Payloads: {len(test_data["payloads"])}')
print(f' Payload types: {", ".join(p["payload_type"] for p in test_data["payloads"])}\n')
try:
# Receive and process the message
print('Receiving and processing message...')
env = smartreceive(
mock_msg,
max_retries=3,
base_delay=100,
max_delay=1000
)
print('\n=== Received Envelope ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate received data
print('=== Validation ===')
passed = True
if not env.get('correlation_id'):
print('❌ correlation_id is missing')
passed = False
else:
print('✅ correlation_id present')
if len(env['payloads']) != 4:
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
# Expected data
expected_data = [
('image', image_data, 'image'),
('audio', audio_data, 'audio'),
('video', video_data, 'video'),
('binary', generic_binary, 'binary')
]
for i in range(len(env['payloads'])):
payload = env['payloads'][i]
expected = expected_data[i]
if payload[0] != expected[0]:
print(f"❌ Payload {i + 1}: Expected dataname '{expected[0]}', got '{payload[0]}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct dataname')
if payload[2] != expected[2]:
print(f"❌ Payload {i + 1}: Expected type '{expected[2]}', got '{payload[2]}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct type')
# Verify binary data integrity
received_data = payload[1]
if received_data != expected[1]:
print(f'❌ Payload {i + 1}: Data mismatch')
print(f' Expected: {expected[1]}')
print(f' Got: {received_data}')
passed = False
else:
print(f'✅ Payload {i + 1}: Data correctly deserialized')
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
run_test()

View File

@@ -0,0 +1,163 @@
"""
MicroPython Binary Sender Test
Tests the smartsend function with binary/image/audio/video payloads
Note: This test is designed for both MicroPython and desktop Python
for compatibility testing.
"""
import sys
import os
import base64
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge_mpy import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL, DEFAULT_SIZE_THRESHOLD, MAX_PAYLOAD_SIZE
TEST_SUBJECT = '/test/binary'
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
def run_test():
print('=== MicroPython Binary Sender Test ===\n')
from natbridge_mpy import _generate_uuid
correlation_id = 'mpy-binary-test-' + _generate_uuid()
print(f'Correlation ID: {correlation_id}')
print(f'Subject: {TEST_SUBJECT}')
print(f'Broker URL: {TEST_BROKER_URL}')
print(f'Default Size Threshold: {DEFAULT_SIZE_THRESHOLD} bytes')
print(f'Max Payload Size: {MAX_PAYLOAD_SIZE} bytes\n')
# Test data - binary data for different types
image_data = bytearray([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
audio_data = bytearray([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]) # FLAC header
video_data = bytearray([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]) # MP4 header
generic_binary = bytearray([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE])
test_data = [
('image', bytes(image_data), 'image'),
('audio', bytes(audio_data), 'audio'),
('video', bytes(video_data), 'video'),
('binary', bytes(generic_binary), 'binary')
]
try:
# Send the message
print('Sending binary payloads...')
env, env_json_str = smartsend(
TEST_SUBJECT,
test_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id=correlation_id,
msg_purpose='test',
sender_name='mpy-binary-test',
is_publish=False
)
print('\n=== Envelope Created ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Purpose: {env["msg_purpose"]}')
print(f'Sender: {env["sender_name"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate envelope structure
print('=== Validation ===')
passed = True
if len(env['payloads']) != 4:
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
# Test each payload
expected_datanames = ['image', 'audio', 'video', 'binary']
expected_types = ['image', 'audio', 'video', 'binary']
expected_data = [bytes(image_data), bytes(audio_data), bytes(video_data), bytes(generic_binary)]
for i in range(len(env['payloads'])):
payload = env['payloads'][i]
if payload['dataname'] != expected_datanames[i]:
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct dataname')
if payload['payload_type'] != expected_types[i]:
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct type')
if payload['transport'] != 'direct':
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct transport')
if payload['encoding'] != 'base64':
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct encoding')
# Decode and verify the data
decoded_data = base64.b64decode(payload['data'])
original_data = expected_data[i]
if decoded_data != original_data:
print(f'❌ Payload {i + 1}: Data integrity mismatch')
passed = False
else:
print(f'✅ Payload {i + 1}: Data integrity verified')
print(f' Size: {payload["size"]} bytes\n')
# Test with larger binary data
print('=== Large Binary Data Test ===')
large_data = bytes([0xFF] * 1000) # 1KB of binary data
large_test_data = [
('large_binary', large_data, 'binary')
]
large_env, _ = smartsend(
TEST_SUBJECT,
large_test_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id='large-' + correlation_id,
is_publish=False
)
if len(large_env['payloads']) == 1 and large_env['payloads'][0]['size'] == 1000:
print('✅ Large binary data handled correctly')
else:
print('❌ Large binary data handling failed')
passed = False
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
run_test()

View File

@@ -0,0 +1,224 @@
"""
MicroPython Dictionary Receiver Test
Tests the smartreceive function with dictionary payloads
Note: This test is designed for both MicroPython and desktop Python
for compatibility testing.
"""
import sys
import os
import json
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge_mpy import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
def run_test():
print('=== MicroPython Dictionary Receiver Test ===\n')
from natbridge_mpy import _generate_uuid
# Create a mock NATS message with dictionary payloads
import base64
simple_dict = {'key1': 'value1', 'key2': 'value2'}
nested_dict = {'outer': {'inner': 'value', 'number': 42}}
array_dict = {'items': [1, 2, 3, 'four', 'five']}
mixed_dict = {'string': 'text', 'number': 123, 'boolean': True, 'null_val': None}
test_data = {
'correlation_id': 'mpy-receiver-dict-' + _generate_uuid(),
'msg_id': _generate_uuid(),
'timestamp': '2024-01-15T10:30:00Z',
'send_to': '/test/dictionary',
'msg_purpose': 'test',
'sender_name': 'mpy-dict-test',
'sender_id': _generate_uuid(),
'receiver_name': 'mpy-receiver',
'receiver_id': _generate_uuid(),
'reply_to': '',
'reply_to_msg_id': '',
'broker_url': TEST_BROKER_URL,
'metadata': {},
'payloads': [
{
'id': _generate_uuid(),
'dataname': 'simple_dict',
'payload_type': 'dictionary',
'transport': 'direct',
'encoding': 'base64',
'size': len(json.dumps(simple_dict).encode('utf8')),
'data': base64.b64encode(json.dumps(simple_dict).encode('utf8')).decode('ascii'),
'metadata': {'payload_bytes': len(json.dumps(simple_dict).encode('utf8'))}
},
{
'id': _generate_uuid(),
'dataname': 'nested_dict',
'payload_type': 'dictionary',
'transport': 'direct',
'encoding': 'base64',
'size': len(json.dumps(nested_dict).encode('utf8')),
'data': base64.b64encode(json.dumps(nested_dict).encode('utf8')).decode('ascii'),
'metadata': {'payload_bytes': len(json.dumps(nested_dict).encode('utf8'))}
},
{
'id': _generate_uuid(),
'dataname': 'array_dict',
'payload_type': 'dictionary',
'transport': 'direct',
'encoding': 'base64',
'size': len(json.dumps(array_dict).encode('utf8')),
'data': base64.b64encode(json.dumps(array_dict).encode('utf8')).decode('ascii'),
'metadata': {'payload_bytes': len(json.dumps(array_dict).encode('utf8'))}
},
{
'id': _generate_uuid(),
'dataname': 'mixed_dict',
'payload_type': 'dictionary',
'transport': 'direct',
'encoding': 'base64',
'size': len(json.dumps(mixed_dict).encode('utf8')),
'data': base64.b64encode(json.dumps(mixed_dict).encode('utf8')).decode('ascii'),
'metadata': {'payload_bytes': len(json.dumps(mixed_dict).encode('utf8'))}
}
]
}
mock_msg = {
'payload': json.dumps(test_data)
}
print('Mock Message Created:')
print(f' Correlation ID: {test_data["correlation_id"]}')
print(f' Payloads: {len(test_data["payloads"])}')
print(f' Payload types: {", ".join(p["payload_type"] for p in test_data["payloads"])}\n')
try:
# Receive and process the message
print('Receiving and processing message...')
env = smartreceive(
mock_msg,
max_retries=3,
base_delay=100,
max_delay=1000
)
print('\n=== Received Envelope ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate received data
print('=== Validation ===')
passed = True
if not env.get('correlation_id'):
print('❌ correlation_id is missing')
passed = False
else:
print('✅ correlation_id present')
if len(env['payloads']) != 4:
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
# Expected data
expected_data = [
('simple_dict', simple_dict, 'dictionary'),
('nested_dict', nested_dict, 'dictionary'),
('array_dict', array_dict, 'dictionary'),
('mixed_dict', mixed_dict, 'dictionary')
]
for i in range(len(env['payloads'])):
payload = env['payloads'][i]
expected = expected_data[i]
if payload[0] != expected[0]:
print(f"❌ Payload {i + 1}: Expected dataname '{expected[0]}', got '{payload[0]}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct dataname')
if payload[2] != expected[2]:
print(f"❌ Payload {i + 1}: Expected type '{expected[2]}', got '{payload[2]}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct type')
data_match = json.dumps(payload[1], sort_keys=True) == json.dumps(expected[1], sort_keys=True)
if not data_match:
print(f'❌ Payload {i + 1}: Data mismatch')
print(f' Expected: {json.dumps(expected[1], sort_keys=True)}')
print(f' Got: {json.dumps(payload[1], sort_keys=True)}')
passed = False
else:
print(f'✅ Payload {i + 1}: Data correctly deserialized')
# Test round-trip with receive
print('\n=== Round-trip Test ===')
round_trip_data = {
'correlation_id': 'roundtrip-' + _generate_uuid(),
'msg_id': _generate_uuid(),
'timestamp': '2024-01-15T10:30:00Z',
'send_to': '/test/dictionary',
'msg_purpose': 'test',
'sender_name': 'mpy-test',
'sender_id': _generate_uuid(),
'receiver_name': 'mpy-receiver',
'receiver_id': _generate_uuid(),
'reply_to': '',
'reply_to_msg_id': '',
'broker_url': TEST_BROKER_URL,
'metadata': {},
'payloads': [
{
'id': _generate_uuid(),
'dataname': 'roundtrip',
'payload_type': 'dictionary',
'transport': 'direct',
'encoding': 'base64',
'size': len(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8')),
'data': base64.b64encode(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8')).decode('ascii'),
'metadata': {'payload_bytes': len(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8'))}
}
]
}
mock_rt_msg = {'payload': json.dumps(round_trip_data)}
rt_env = smartreceive(mock_rt_msg)
if rt_env['payloads'][0][0] == 'roundtrip' and rt_env['payloads'][0][2] == 'dictionary':
print('✅ Round-trip test successful')
else:
print('❌ Round-trip test failed')
passed = False
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
run_test()

View File

@@ -0,0 +1,177 @@
"""
MicroPython Dictionary Sender Test
Tests the smartsend function with dictionary payloads
Note: This test is designed for both MicroPython and desktop Python
for compatibility testing.
"""
import sys
import os
import json
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge_mpy import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL, DEFAULT_SIZE_THRESHOLD, MAX_PAYLOAD_SIZE
TEST_SUBJECT = '/test/dictionary'
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
def run_test():
print('=== MicroPython Dictionary Sender Test ===\n')
from natbridge_mpy import _generate_uuid
correlation_id = 'mpy-dict-test-' + _generate_uuid()
print(f'Correlation ID: {correlation_id}')
print(f'Subject: {TEST_SUBJECT}')
print(f'Broker URL: {TEST_BROKER_URL}')
print(f'Default Size Threshold: {DEFAULT_SIZE_THRESHOLD} bytes')
print(f'Max Payload Size: {MAX_PAYLOAD_SIZE} bytes\n')
# Test data - various dictionary structures
test_data = [
('simple_dict', {'key1': 'value1', 'key2': 'value2'}, 'dictionary'),
('nested_dict', {'outer': {'inner': 'value', 'number': 42}}, 'dictionary'),
('array_dict', {'items': [1, 2, 3, 'four', 'five']}, 'dictionary'),
('mixed_dict', {'string': 'text', 'number': 123, 'boolean': True, 'null_val': None}, 'dictionary')
]
try:
# Send the message
print('Sending dictionary payloads...')
env, env_json_str = smartsend(
TEST_SUBJECT,
test_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id=correlation_id,
msg_purpose='test',
sender_name='mpy-dict-test',
is_publish=False
)
print('\n=== Envelope Created ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Purpose: {env["msg_purpose"]}')
print(f'Sender: {env["sender_name"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate envelope structure
print('=== Validation ===')
passed = True
if len(env['payloads']) != 4:
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
# Test each payload
expected_datanames = ['simple_dict', 'nested_dict', 'array_dict', 'mixed_dict']
expected_types = ['dictionary', 'dictionary', 'dictionary', 'dictionary']
for i in range(len(env['payloads'])):
payload = env['payloads'][i]
if payload['dataname'] != expected_datanames[i]:
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct dataname')
if payload['payload_type'] != expected_types[i]:
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct type')
if payload['transport'] != 'direct':
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct transport')
if payload['encoding'] != 'base64':
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct encoding')
# Decode and verify the data
import base64
decoded_data = json.loads(base64.b64decode(payload['data']).decode('ascii'))
original_data = test_data[i][1]
# Normalize for comparison
if json.dumps(decoded_data, sort_keys=True) != json.dumps(original_data, sort_keys=True):
print(f'❌ Payload {i + 1}: Data integrity mismatch')
print(f' Expected: {json.dumps(original_data)}')
print(f' Got: {json.dumps(decoded_data)}')
passed = False
else:
print(f'✅ Payload {i + 1}: Data integrity verified')
print(f' Size: {payload["size"]} bytes\n')
# Test round-trip serialization
print('=== Round-trip Serialization Test ===')
round_trip_data = [
('roundtrip', {'test': 'data', 'numbers': [1, 2, 3], 'nested': {'a': 1, 'b': 2}}, 'dictionary')
]
rt_env, _ = smartsend(
TEST_SUBJECT,
round_trip_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id='roundtrip-' + correlation_id,
is_publish=False
)
rt_payload = rt_env['payloads'][0]
rt_decoded = json.loads(base64.b64decode(rt_payload['data']).decode('ascii'))
if json.dumps(rt_decoded, sort_keys=True) == json.dumps(round_trip_data[0][1], sort_keys=True):
print('✅ Round-trip serialization successful')
else:
print('❌ Round-trip serialization failed')
passed = False
# Test JSON string output
print('\n=== JSON String Output Test ===')
try:
parsed = json.loads(env_json_str)
if parsed['correlation_id'] == env['correlation_id'] and \
len(parsed['payloads']) == len(env['payloads']):
print('✅ JSON string is valid and matches envelope')
else:
print('❌ JSON string does not match envelope')
passed = False
except json.JSONDecodeError as e:
print(f'❌ JSON string is invalid: {e}')
passed = False
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
run_test()

View File

@@ -0,0 +1,209 @@
"""
MicroPython Text Receiver Test
Tests the smartreceive function with text payloads
Note: This test is designed for both MicroPython and desktop Python
for compatibility testing.
"""
import sys
import os
import json
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge_mpy import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
def run_test():
print('=== MicroPython Text Receiver Test ===\n')
from natbridge_mpy import _generate_uuid
# Create a mock NATS message with text payload
test_text = 'Hello, NATSBridge! This is a test message.'
import base64
test_data = {
'correlation_id': 'mpy-receiver-test-' + _generate_uuid(),
'msg_id': _generate_uuid(),
'timestamp': '2024-01-15T10:30:00Z',
'send_to': '/test/text',
'msg_purpose': 'test',
'sender_name': 'mpy-text-test',
'sender_id': _generate_uuid(),
'receiver_name': 'mpy-receiver',
'receiver_id': _generate_uuid(),
'reply_to': '',
'reply_to_msg_id': '',
'broker_url': TEST_BROKER_URL,
'metadata': {},
'payloads': [
{
'id': _generate_uuid(),
'dataname': 'message',
'payload_type': 'text',
'transport': 'direct',
'encoding': 'base64',
'size': len(test_text.encode('utf8')),
'data': base64.b64encode(test_text.encode('utf8')).decode('ascii'),
'metadata': {'payload_bytes': len(test_text.encode('utf8'))}
}
]
}
mock_msg = {
'payload': json.dumps(test_data)
}
print('Mock Message Created:')
print(f' Correlation ID: {test_data["correlation_id"]}')
print(f' Payloads: {len(test_data["payloads"])}')
print(f' Payload dataname: {test_data["payloads"][0]["dataname"]}')
print(f' Payload type: {test_data["payloads"][0]["payload_type"]}')
print(f' Transport: {test_data["payloads"][0]["transport"]}\n')
try:
# Receive and process the message
print('Receiving and processing message...')
env = smartreceive(
mock_msg,
max_retries=3,
base_delay=100,
max_delay=1000
)
print('\n=== Received Envelope ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate received data
print('=== Validation ===')
passed = True
if not env.get('correlation_id'):
print('❌ correlation_id is missing')
passed = False
else:
print('✅ correlation_id present')
if len(env['payloads']) != 1:
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
payload = env['payloads'][0]
if payload[0] != 'message':
print(f"❌ Expected dataname 'message', got '{payload[0]}'")
passed = False
else:
print('✅ Correct dataname')
if payload[2] != 'text':
print(f"❌ Expected type 'text', got '{payload[2]}'")
passed = False
else:
print('✅ Correct type')
if payload[1] != test_text:
print('❌ Data mismatch')
print(f' Expected: {test_text}')
print(f' Got: {payload[1]}')
passed = False
else:
print('✅ Data correctly deserialized')
# Test with multiple text payloads
print('\n=== Multiple Text Payloads Test ===')
multi_test_data = {
'correlation_id': 'multi-receiver-' + _generate_uuid(),
'msg_id': _generate_uuid(),
'timestamp': '2024-01-15T10:30:00Z',
'send_to': '/test/text',
'msg_purpose': 'test',
'sender_name': 'mpy-text-test',
'sender_id': _generate_uuid(),
'receiver_name': 'mpy-receiver',
'receiver_id': _generate_uuid(),
'reply_to': '',
'reply_to_msg_id': '',
'broker_url': TEST_BROKER_URL,
'metadata': {},
'payloads': [
{
'id': _generate_uuid(),
'dataname': 'msg1',
'payload_type': 'text',
'transport': 'direct',
'encoding': 'base64',
'size': 16,
'data': base64.b64encode(b'First message').decode('ascii'),
'metadata': {'payload_bytes': 16}
},
{
'id': _generate_uuid(),
'dataname': 'msg2',
'payload_type': 'text',
'transport': 'direct',
'encoding': 'base64',
'size': 16,
'data': base64.b64encode(b'Second message').decode('ascii'),
'metadata': {'payload_bytes': 16}
},
{
'id': _generate_uuid(),
'dataname': 'msg3',
'payload_type': 'text',
'transport': 'direct',
'encoding': 'base64',
'size': 16,
'data': base64.b64encode(b'Third message').decode('ascii'),
'metadata': {'payload_bytes': 16}
}
]
}
mock_multi_msg = {'payload': json.dumps(multi_test_data)}
multi_env = smartreceive(mock_multi_msg)
if len(multi_env['payloads']) == 3:
print('✅ Multiple payloads handled correctly')
# Verify each payload
expected_messages = ['First message', 'Second message', 'Third message']
for i in range(3):
if multi_env['payloads'][i][1] == expected_messages[i]:
print(f'✅ Payload {i + 1} correctly deserialized')
else:
print(f'❌ Payload {i + 1} mismatch')
passed = False
else:
print(f'❌ Expected 3 payloads, got {len(multi_env["payloads"])}')
passed = False
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
run_test()

View File

@@ -0,0 +1,205 @@
"""
MicroPython Text Sender Test
Tests the smartsend function with text payloads
Note: This test is designed for both MicroPython and desktop Python
for compatibility testing.
"""
import sys
import os
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge_mpy import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL, DEFAULT_SIZE_THRESHOLD, MAX_PAYLOAD_SIZE
TEST_SUBJECT = '/test/text'
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
def run_test():
print('=== MicroPython Text Sender Test ===\n')
from natbridge_mpy import _generate_uuid
correlation_id = 'mpy-text-test-' + _generate_uuid()
print(f'Correlation ID: {correlation_id}')
print(f'Subject: {TEST_SUBJECT}')
print(f'Broker URL: {TEST_BROKER_URL}')
print(f'Default Size Threshold: {DEFAULT_SIZE_THRESHOLD} bytes')
print(f'Max Payload Size: {MAX_PAYLOAD_SIZE} bytes\n')
# Test data
text_data = 'Hello, NATSBridge! This is a test message.'
test_data = [
('message', text_data, 'text')
]
try:
# Send the message
print('Sending text payload...')
env, env_json_str = smartsend(
TEST_SUBJECT,
test_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id=correlation_id,
msg_purpose='test',
sender_name='mpy-text-test',
is_publish=False # Don't actually publish for this test
)
print('\n=== Envelope Created ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Purpose: {env["msg_purpose"]}')
print(f'Sender: {env["sender_name"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate envelope structure
print('=== Validation ===')
passed = True
if not env.get('correlation_id'):
print('❌ correlation_id is missing')
passed = False
else:
print('✅ correlation_id present')
if not env.get('msg_id'):
print('❌ msg_id is missing')
passed = False
else:
print('✅ msg_id present')
if not env.get('timestamp'):
print('❌ timestamp is missing')
passed = False
else:
print('✅ timestamp present')
if len(env['payloads']) != 1:
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
payload = env['payloads'][0]
if payload['dataname'] != 'message':
print(f"❌ Expected dataname 'message', got '{payload['dataname']}'")
passed = False
else:
print('✅ Correct dataname')
if payload['payload_type'] != 'text':
print(f"❌ Expected payload_type 'text', got '{payload['payload_type']}'")
passed = False
else:
print('✅ Correct payload_type')
if payload['transport'] != 'direct':
print(f"❌ Expected transport 'direct', got '{payload['transport']}'")
passed = False
else:
print('✅ Correct transport')
if payload['encoding'] != 'base64':
print(f"❌ Expected encoding 'base64', got '{payload['encoding']}'")
passed = False
else:
print('✅ Correct encoding')
# Decode and verify the data
import base64
decoded_data = base64.b64decode(payload['data']).decode('ascii')
if decoded_data != text_data:
print('❌ Decoded data mismatch')
print(f' Expected: {text_data}')
print(f' Got: {decoded_data}')
passed = False
else:
print('✅ Data integrity verified')
print(f'\nPayload size: {payload["size"]} bytes')
print(f'Base64 data length: {len(payload["data"])} chars')
# Test with multiple text payloads
print('\n=== Multiple Text Payloads Test ===')
multi_test_data = [
('msg1', 'First message', 'text'),
('msg2', 'Second message', 'text'),
('msg3', 'Third message', 'text')
]
multi_env, _ = smartsend(
TEST_SUBJECT,
multi_test_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id='multi-test-' + correlation_id,
is_publish=False
)
if len(multi_env['payloads']) == 3:
print('✅ Multiple payloads handled correctly')
else:
print(f'❌ Expected 3 payloads, got {len(multi_env["payloads"])}')
passed = False
# Test size threshold enforcement
print('\n=== Size Threshold Test ===')
small_text = 'small'
large_text = 'x' * (DEFAULT_SIZE_THRESHOLD - 100) # Just under threshold
small_env, _ = smartsend(
TEST_SUBJECT,
[('small', small_text, 'text')],
broker_url=TEST_BROKER_URL,
is_publish=False
)
if small_env['payloads'][0]['transport'] == 'direct':
print('✅ Small payload uses direct transport')
else:
print('❌ Small payload should use direct transport')
passed = False
# Test that large text (> MAX_PAYLOAD_SIZE) raises error
print('\n=== Max Payload Size Test ===')
try:
too_large_text = 'x' * (MAX_PAYLOAD_SIZE + 1000)
large_env, _ = smartsend(
TEST_SUBJECT,
[('large', too_large_text, 'text')],
broker_url=TEST_BROKER_URL,
is_publish=False
)
print('❌ Should have raised MemoryError for payload exceeding MAX_PAYLOAD_SIZE')
passed = False
except MemoryError as e:
print(f'✅ Correctly raised MemoryError: {e}')
except Exception as e:
print(f'❌ Unexpected error: {type(e).__name__}: {e}')
passed = False
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
run_test()

View File

@@ -0,0 +1,184 @@
"""
Python Binary Receiver Test
Tests the smartreceive function with binary/image/audio/video/table payloads
"""
import asyncio
import sys
import os
import json
import base64
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
async def run_test():
print('=== Python Binary Receiver Test ===\n')
# Create mock NATS message with binary payloads
image_data = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
audio_data = bytes([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]) # FLAC header
video_data = bytes([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]) # MP4 header
generic_binary = bytes([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE])
test_data = {
'correlation_id': 'py-binary-receiver-' + str(asyncio.get_event_loop().time() * 1000000),
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
'timestamp': asyncio.get_event_loop().time().isoformat(),
'send_to': '/test/binary',
'msg_purpose': 'test',
'sender_name': 'py-binary-test',
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
'receiver_name': 'py-receiver',
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
'reply_to': '',
'reply_to_msg_id': '',
'broker_url': TEST_BROKER_URL,
'metadata': {},
'payloads': [
{
'id': 'payload-1',
'dataname': 'image',
'payload_type': 'image',
'transport': 'direct',
'encoding': 'base64',
'size': len(image_data),
'data': base64.b64encode(image_data).decode('ascii'),
'metadata': {'payload_bytes': len(image_data)}
},
{
'id': 'payload-2',
'dataname': 'audio',
'payload_type': 'audio',
'transport': 'direct',
'encoding': 'base64',
'size': len(audio_data),
'data': base64.b64encode(audio_data).decode('ascii'),
'metadata': {'payload_bytes': len(audio_data)}
},
{
'id': 'payload-3',
'dataname': 'video',
'payload_type': 'video',
'transport': 'direct',
'encoding': 'base64',
'size': len(video_data),
'data': base64.b64encode(video_data).decode('ascii'),
'metadata': {'payload_bytes': len(video_data)}
},
{
'id': 'payload-4',
'dataname': 'binary',
'payload_type': 'binary',
'transport': 'direct',
'encoding': 'base64',
'size': len(generic_binary),
'data': base64.b64encode(generic_binary).decode('ascii'),
'metadata': {'payload_bytes': len(generic_binary)}
}
]
}
mock_msg = {
'payload': json.dumps(test_data)
}
print('Mock Message Created:')
print(f' Correlation ID: {test_data["correlation_id"]}')
print(f' Payloads: {len(test_data["payloads"])}')
print(f' Payload types: {", ".join(p["payload_type"] for p in test_data["payloads"])}\n')
try:
# Receive and process the message
print('Receiving and processing message...')
env = await smartreceive(
mock_msg,
max_retries=3,
base_delay=100,
max_delay=1000
)
print('\n=== Received Envelope ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate received data
print('=== Validation ===')
passed = True
if not env.get('correlation_id'):
print('❌ correlation_id is missing')
passed = False
else:
print('✅ correlation_id present')
if len(env['payloads']) != 4:
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
# Expected data
expected_data = [
('image', image_data, 'image'),
('audio', audio_data, 'audio'),
('video', video_data, 'video'),
('binary', generic_binary, 'binary')
]
for i in range(len(env['payloads'])):
payload = env['payloads'][i]
expected = expected_data[i]
if payload[0] != expected[0]:
print(f"❌ Payload {i + 1}: Expected dataname '{expected[0]}', got '{payload[0]}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct dataname')
if payload[2] != expected[2]:
print(f"❌ Payload {i + 1}: Expected type '{expected[2]}', got '{payload[2]}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct type')
# Verify binary data integrity
received_data = payload[1]
if not isinstance(received_data, (bytes, bytearray)):
print(f'❌ Payload {i + 1}: Expected bytes/bytearray, got {type(received_data)}')
passed = False
elif received_data != expected[1]:
print(f'❌ Payload {i + 1}: Data mismatch')
print(f' Expected: {expected[1]}')
print(f' Got: {received_data}')
passed = False
else:
print(f'✅ Payload {i + 1}: Data correctly deserialized')
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
asyncio.run(run_test())

View File

@@ -0,0 +1,183 @@
"""
Python Binary Sender Test
Tests the smartsend function with binary/image/audio/video/table payloads
"""
import asyncio
import sys
import os
import base64
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
TEST_SUBJECT = '/test/binary'
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
async def run_test():
print('=== Python Binary Sender Test ===\n')
correlation_id = 'py-binary-test-' + str(asyncio.get_event_loop().time() * 1000000)
print(f'Correlation ID: {correlation_id}')
print(f'Subject: {TEST_SUBJECT}')
print(f'Broker URL: {TEST_BROKER_URL}\n')
# Test data - binary data for different types
image_data = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
audio_data = bytes([0x46, 0x4C, 0x41, 0x43, 0x00, 0x00, 0x00, 0x00]) # FLAC header
video_data = bytes([0x00, 0x00, 0x00, 0x18, 0x66, 0x74, 0x79, 0x70]) # MP4 header
generic_binary = bytes([0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE, 0xBA, 0xBE])
# Test table data
try:
import pandas as pd
table_data = pd.DataFrame({
'id': [1, 2, 3, 4, 5],
'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
'value': [10.5, 20.3, 30.1, 40.9, 50.7]
})
table_available = True
except ImportError:
table_available = False
table_data = None
test_data = [
('image', image_data, 'image'),
('audio', audio_data, 'audio'),
('video', video_data, 'video'),
('binary', generic_binary, 'binary')
]
if table_available:
test_data.append(('table', table_data, 'table'))
try:
# Send the message
print('Sending binary payloads...')
env, env_json_str = await smartsend(
TEST_SUBJECT,
test_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id=correlation_id,
msg_purpose='test',
sender_name='py-binary-test',
is_publish=False
)
print('\n=== Envelope Created ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Purpose: {env["msg_purpose"]}')
print(f'Sender: {env["sender_name"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate envelope structure
print('=== Validation ===')
passed = True
expected_count = 5 if table_available else 4
if len(env['payloads']) != expected_count:
print(f'❌ Expected {expected_count} payloads, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
# Test each payload
expected_datanames = ['image', 'audio', 'video', 'binary']
expected_types = ['image', 'audio', 'video', 'binary']
expected_data = [image_data, audio_data, video_data, generic_binary]
if table_available:
expected_datanames.append('table')
expected_types.append('table')
for i in range(len(env['payloads'])):
payload = env['payloads'][i]
if payload['dataname'] != expected_datanames[i]:
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct dataname')
if payload['payload_type'] != expected_types[i]:
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct type')
if payload['transport'] != 'direct':
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct transport')
if payload['encoding'] != 'base64':
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct encoding')
# Decode and verify the data
decoded_data = base64.b64decode(payload['data'])
if i < len(expected_data):
original_data = expected_data[i]
if decoded_data != original_data:
print(f'❌ Payload {i + 1}: Data integrity mismatch')
passed = False
else:
print(f'✅ Payload {i + 1}: Data integrity verified')
else:
# Table payload - just verify it's present
print(f'✅ Payload {i + 1}: Table data present (size: {payload["size"]} bytes)')
print(f' Size: {payload["size"]} bytes\n')
# Test with larger binary data
print('=== Large Binary Data Test ===')
large_data = bytes([0xFF] * 10000) # 10KB of binary data
large_test_data = [
('large_binary', large_data, 'binary')
]
large_env, _ = await smartsend(
TEST_SUBJECT,
large_test_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id='large-' + correlation_id,
is_publish=False
)
if len(large_env['payloads']) == 1 and large_env['payloads'][0]['size'] == 10000:
print('✅ Large binary data handled correctly')
else:
print('❌ Large binary data handling failed')
passed = False
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
asyncio.run(run_test())

View File

@@ -0,0 +1,220 @@
"""
Python Dictionary Receiver Test
Tests the smartreceive function with dictionary payloads
"""
import asyncio
import sys
import os
import json
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
async def run_test():
print('=== Python Dictionary Receiver Test ===\n')
# Create a mock NATS message with dictionary payloads
import base64
simple_dict = {'key1': 'value1', 'key2': 'value2'}
nested_dict = {'outer': {'inner': 'value', 'number': 42}}
array_dict = {'items': [1, 2, 3, 'four', 'five']}
mixed_dict = {'string': 'text', 'number': 123, 'boolean': True, 'null_val': None}
test_data = {
'correlation_id': 'py-receiver-dict-' + str(asyncio.get_event_loop().time() * 1000000),
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
'timestamp': asyncio.get_event_loop().time().isoformat(),
'send_to': '/test/dictionary',
'msg_purpose': 'test',
'sender_name': 'py-dict-test',
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
'receiver_name': 'py-receiver',
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
'reply_to': '',
'reply_to_msg_id': '',
'broker_url': TEST_BROKER_URL,
'metadata': {},
'payloads': [
{
'id': 'payload-1',
'dataname': 'simple_dict',
'payload_type': 'dictionary',
'transport': 'direct',
'encoding': 'base64',
'size': len(json.dumps(simple_dict).encode('utf8')),
'data': base64.b64encode(json.dumps(simple_dict).encode('utf8')).decode('ascii'),
'metadata': {'payload_bytes': len(json.dumps(simple_dict).encode('utf8'))}
},
{
'id': 'payload-2',
'dataname': 'nested_dict',
'payload_type': 'dictionary',
'transport': 'direct',
'encoding': 'base64',
'size': len(json.dumps(nested_dict).encode('utf8')),
'data': base64.b64encode(json.dumps(nested_dict).encode('utf8')).decode('ascii'),
'metadata': {'payload_bytes': len(json.dumps(nested_dict).encode('utf8'))}
},
{
'id': 'payload-3',
'dataname': 'array_dict',
'payload_type': 'dictionary',
'transport': 'direct',
'encoding': 'base64',
'size': len(json.dumps(array_dict).encode('utf8')),
'data': base64.b64encode(json.dumps(array_dict).encode('utf8')).decode('ascii'),
'metadata': {'payload_bytes': len(json.dumps(array_dict).encode('utf8'))}
},
{
'id': 'payload-4',
'dataname': 'mixed_dict',
'payload_type': 'dictionary',
'transport': 'direct',
'encoding': 'base64',
'size': len(json.dumps(mixed_dict).encode('utf8')),
'data': base64.b64encode(json.dumps(mixed_dict).encode('utf8')).decode('ascii'),
'metadata': {'payload_bytes': len(json.dumps(mixed_dict).encode('utf8'))}
}
]
}
mock_msg = {
'payload': json.dumps(test_data)
}
print('Mock Message Created:')
print(f' Correlation ID: {test_data["correlation_id"]}')
print(f' Payloads: {len(test_data["payloads"])}')
print(f' Payload types: {", ".join(p["payload_type"] for p in test_data["payloads"])}\n')
try:
# Receive and process the message
print('Receiving and processing message...')
env = await smartreceive(
mock_msg,
max_retries=3,
base_delay=100,
max_delay=1000
)
print('\n=== Received Envelope ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate received data
print('=== Validation ===')
passed = True
if not env.get('correlation_id'):
print('❌ correlation_id is missing')
passed = False
else:
print('✅ correlation_id present')
if len(env['payloads']) != 4:
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
# Expected data
expected_data = [
('simple_dict', simple_dict, 'dictionary'),
('nested_dict', nested_dict, 'dictionary'),
('array_dict', array_dict, 'dictionary'),
('mixed_dict', mixed_dict, 'dictionary')
]
for i in range(len(env['payloads'])):
payload = env['payloads'][i]
expected = expected_data[i]
if payload[0] != expected[0]:
print(f"❌ Payload {i + 1}: Expected dataname '{expected[0]}', got '{payload[0]}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct dataname')
if payload[2] != expected[2]:
print(f"❌ Payload {i + 1}: Expected type '{expected[2]}', got '{payload[2]}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct type')
data_match = json.dumps(payload[1], sort_keys=True) == json.dumps(expected[1], sort_keys=True)
if not data_match:
print(f'❌ Payload {i + 1}: Data mismatch')
print(f' Expected: {json.dumps(expected[1], sort_keys=True)}')
print(f' Got: {json.dumps(payload[1], sort_keys=True)}')
passed = False
else:
print(f'✅ Payload {i + 1}: Data correctly deserialized')
# Test round-trip with receive
print('\n=== Round-trip Test ===')
round_trip_data = {
'correlation_id': 'roundtrip-' + str(asyncio.get_event_loop().time() * 1000000),
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
'timestamp': asyncio.get_event_loop().time().isoformat(),
'send_to': '/test/dictionary',
'msg_purpose': 'test',
'sender_name': 'py-test',
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
'receiver_name': 'py-receiver',
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
'reply_to': '',
'reply_to_msg_id': '',
'broker_url': TEST_BROKER_URL,
'metadata': {},
'payloads': [
{
'id': 'payload-rt',
'dataname': 'roundtrip',
'payload_type': 'dictionary',
'transport': 'direct',
'encoding': 'base64',
'size': len(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8')),
'data': base64.b64encode(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8')).decode('ascii'),
'metadata': {'payload_bytes': len(json.dumps({'test': 'data', 'nested': {'a': 1, 'b': 2}}).encode('utf8'))}
}
]
}
mock_rt_msg = {'payload': json.dumps(round_trip_data)}
rt_env = await smartreceive(mock_rt_msg)
if rt_env['payloads'][0][0] == 'roundtrip' and rt_env['payloads'][0][2] == 'dictionary':
print('✅ Round-trip test successful')
else:
print('❌ Round-trip test failed')
passed = False
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
asyncio.run(run_test())

View File

@@ -0,0 +1,172 @@
"""
Python Dictionary Sender Test
Tests the smartsend function with dictionary payloads
"""
import asyncio
import sys
import os
import json
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
TEST_SUBJECT = '/test/dictionary'
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
async def run_test():
print('=== Python Dictionary Sender Test ===\n')
correlation_id = 'py-dict-test-' + str(asyncio.get_event_loop().time() * 1000000)
print(f'Correlation ID: {correlation_id}')
print(f'Subject: {TEST_SUBJECT}')
print(f'Broker URL: {TEST_BROKER_URL}\n')
# Test data - various dictionary structures
test_data = [
('simple_dict', {'key1': 'value1', 'key2': 'value2'}, 'dictionary'),
('nested_dict', {'outer': {'inner': 'value', 'number': 42}}, 'dictionary'),
('array_dict', {'items': [1, 2, 3, 'four', 'five']}, 'dictionary'),
('mixed_dict', {'string': 'text', 'number': 123, 'boolean': True, 'null_val': None}, 'dictionary')
]
try:
# Send the message
print('Sending dictionary payloads...')
env, env_json_str = await smartsend(
TEST_SUBJECT,
test_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id=correlation_id,
msg_purpose='test',
sender_name='py-dict-test',
is_publish=False
)
print('\n=== Envelope Created ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Purpose: {env["msg_purpose"]}')
print(f'Sender: {env["sender_name"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate envelope structure
print('=== Validation ===')
passed = True
if len(env['payloads']) != 4:
print(f'❌ Expected 4 payloads, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
# Test each payload
expected_datanames = ['simple_dict', 'nested_dict', 'array_dict', 'mixed_dict']
expected_types = ['dictionary', 'dictionary', 'dictionary', 'dictionary']
for i in range(len(env['payloads'])):
payload = env['payloads'][i]
if payload['dataname'] != expected_datanames[i]:
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct dataname')
if payload['payload_type'] != expected_types[i]:
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct type')
if payload['transport'] != 'direct':
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct transport')
if payload['encoding'] != 'base64':
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct encoding')
# Decode and verify the data
import base64
decoded_data = json.loads(base64.b64decode(payload['data']).decode('utf8'))
original_data = test_data[i][1]
# Normalize for comparison (None vs null, True vs true, etc.)
if json.dumps(decoded_data, sort_keys=True) != json.dumps(original_data, sort_keys=True):
print(f'❌ Payload {i + 1}: Data integrity mismatch')
print(f' Expected: {json.dumps(original_data)}')
print(f' Got: {json.dumps(decoded_data)}')
passed = False
else:
print(f'✅ Payload {i + 1}: Data integrity verified')
print(f' Size: {payload["size"]} bytes\n')
# Test round-trip serialization
print('=== Round-trip Serialization Test ===')
round_trip_data = [
('roundtrip', {'test': 'data', 'numbers': [1, 2, 3], 'nested': {'a': 1, 'b': 2}}, 'dictionary')
]
rt_env, _ = await smartsend(
TEST_SUBJECT,
round_trip_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id='roundtrip-' + correlation_id,
is_publish=False
)
rt_payload = rt_env['payloads'][0]
rt_decoded = json.loads(base64.b64decode(rt_payload['data']).decode('utf8'))
if json.dumps(rt_decoded, sort_keys=True) == json.dumps(round_trip_data[0][1], sort_keys=True):
print('✅ Round-trip serialization successful')
else:
print('❌ Round-trip serialization failed')
passed = False
# Test JSON string output
print('\n=== JSON String Output Test ===')
try:
parsed = json.loads(env_json_str)
if parsed['correlation_id'] == env['correlation_id'] and \
len(parsed['payloads']) == len(env['payloads']):
print('✅ JSON string is valid and matches envelope')
else:
print('❌ JSON string does not match envelope')
passed = False
except json.JSONDecodeError as e:
print(f'❌ JSON string is invalid: {e}')
passed = False
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
asyncio.run(run_test())

View File

@@ -0,0 +1,199 @@
"""
Python Mix Payloads Sender Test
Tests the smartsend function with mixed payload types
"""
import asyncio
import sys
import os
import base64
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
TEST_SUBJECT = '/test/mix'
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
async def run_test():
print('=== Python Mix Payloads Sender Test ===\n')
correlation_id = 'py-mix-test-' + str(asyncio.get_event_loop().time() * 1000000)
print(f'Correlation ID: {correlation_id}')
print(f'Subject: {TEST_SUBJECT}')
print(f'Broker URL: {TEST_BROKER_URL}\n')
# Test data - mixed payload types
text_data = 'Hello, NATSBridge!'
dict_data = {'key1': 'value1', 'key2': 42, 'nested': {'a': 1, 'b': 2}}
image_data = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]) # PNG header
# Table data
try:
import pandas as pd
table_data = pd.DataFrame({
'id': [1, 2, 3],
'name': ['Alice', 'Bob', 'Charlie'],
'age': [30, 25, 35]
})
table_available = True
except ImportError:
table_available = False
table_data = None
test_data = [
('message', text_data, 'text'),
('config', dict_data, 'dictionary'),
('image', image_data, 'image')
]
if table_available:
test_data.append(('users', table_data, 'table'))
try:
# Send the message
print('Sending mixed payloads...')
env, env_json_str = await smartsend(
TEST_SUBJECT,
test_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id=correlation_id,
msg_purpose='test',
sender_name='py-mix-test',
is_publish=False
)
print('\n=== Envelope Created ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Purpose: {env["msg_purpose"]}')
print(f'Sender: {env["sender_name"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate envelope structure
print('=== Validation ===')
passed = True
expected_count = 4 if table_available else 3
if len(env['payloads']) != expected_count:
print(f'❌ Expected {expected_count} payloads, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
# Test each payload
expected_datanames = ['message', 'config', 'image']
expected_types = ['text', 'dictionary', 'image']
expected_data = [text_data, dict_data, image_data]
if table_available:
expected_datanames.append('users')
expected_types.append('table')
for i in range(len(env['payloads'])):
payload = env['payloads'][i]
if payload['dataname'] != expected_datanames[i]:
print(f"❌ Payload {i + 1}: Expected dataname '{expected_datanames[i]}', got '{payload['dataname']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct dataname')
if payload['payload_type'] != expected_types[i]:
print(f"❌ Payload {i + 1}: Expected type '{expected_types[i]}', got '{payload['payload_type']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct type')
if payload['transport'] != 'direct':
print(f"❌ Payload {i + 1}: Expected transport 'direct', got '{payload['transport']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct transport')
if payload['encoding'] != 'base64':
print(f"❌ Payload {i + 1}: Expected encoding 'base64', got '{payload['encoding']}'")
passed = False
else:
print(f'✅ Payload {i + 1}: Correct encoding')
# Verify data integrity based on type
decoded_data = base64.b64decode(payload['data'])
if expected_types[i] == 'text':
decoded_text = decoded_data.decode('utf8')
if decoded_text != expected_data[i]:
print(f'❌ Payload {i + 1}: Data integrity mismatch')
passed = False
else:
print(f'✅ Payload {i + 1}: Data integrity verified')
elif expected_types[i] == 'dictionary':
import json
decoded_dict = json.loads(decoded_data.decode('utf8'))
if json.dumps(decoded_dict, sort_keys=True) != json.dumps(expected_data[i], sort_keys=True):
print(f'❌ Payload {i + 1}: Data integrity mismatch')
passed = False
else:
print(f'✅ Payload {i + 1}: Data integrity verified')
elif expected_types[i] == 'image':
if decoded_data != expected_data[i]:
print(f'❌ Payload {i + 1}: Data integrity mismatch')
passed = False
else:
print(f'✅ Payload {i + 1}: Data integrity verified')
elif expected_types[i] == 'table':
if len(decoded_data) > 0:
print(f'✅ Payload {i + 1}: Arrow IPC data present ({len(decoded_data)} bytes)')
else:
print(f'❌ Payload {i + 1}: Arrow IPC data is empty')
passed = False
print(f' Size: {payload["size"]} bytes\n')
# Test with chat-like payload (text + image + audio)
print('=== Chat-like Payload Test ===')
chat_data = [
('text', 'Hello!', 'text'),
('image', bytes([0xFF, 0xD8, 0xFF, 0xE0]), 'image'),
('audio', bytes([0x46, 0x4C, 0x41, 0x43]), 'audio')
]
chat_env, _ = await smartsend(
TEST_SUBJECT,
chat_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id='chat-' + correlation_id,
is_publish=False
)
if len(chat_env['payloads']) == 3:
print('✅ Chat-like payloads handled correctly')
else:
print('❌ Chat-like payloads handling failed')
passed = False
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
asyncio.run(run_test())

View File

@@ -0,0 +1,167 @@
"""
Python Table Sender Test
Tests the smartsend function with table (Arrow IPC) payloads
"""
import asyncio
import sys
import os
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
TEST_SUBJECT = '/test/table'
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
async def run_test():
print('=== Python Table Sender Test ===\n')
correlation_id = 'py-table-test-' + str(asyncio.get_event_loop().time() * 1000000)
print(f'Correlation ID: {correlation_id}')
print(f'Subject: {TEST_SUBJECT}')
print(f'Broker URL: {TEST_BROKER_URL}\n')
# Test data - pandas DataFrame
try:
import pandas as pd
table_data = pd.DataFrame({
'id': [1, 2, 3, 4, 5],
'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
'age': [30, 25, 35, 28, 32],
'active': [True, False, True, True, False]
})
table_available = True
except ImportError:
print('❌ pandas not available - skipping table tests')
sys.exit(0)
test_data = [
('users_table', table_data, 'table')
]
try:
# Send the message
print('Sending table payload...')
env, env_json_str = await smartsend(
TEST_SUBJECT,
test_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id=correlation_id,
msg_purpose='test',
sender_name='py-table-test',
is_publish=False
)
print('\n=== Envelope Created ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Purpose: {env["msg_purpose"]}')
print(f'Sender: {env["sender_name"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate envelope structure
print('=== Validation ===')
passed = True
if len(env['payloads']) != 1:
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
payload = env['payloads'][0]
if payload['dataname'] != 'users_table':
print(f"❌ Expected dataname 'users_table', got '{payload['dataname']}'")
passed = False
else:
print('✅ Correct dataname')
if payload['payload_type'] != 'table':
print(f"❌ Expected payload_type 'table', got '{payload['payload_type']}'")
passed = False
else:
print('✅ Correct payload_type')
if payload['transport'] != 'direct':
print(f"❌ Expected transport 'direct', got '{payload['transport']}'")
passed = False
else:
print('✅ Correct transport')
if payload['encoding'] != 'base64':
print(f"❌ Expected encoding 'base64', got '{payload['encoding']}'")
passed = False
else:
print('✅ Correct encoding')
print(f'\nPayload size: {payload["size"]} bytes')
# Test with larger table
print('\n=== Larger Table Test ===')
large_table_data = pd.DataFrame({
'id': range(100),
'name': [f'User{i}' for i in range(100)],
'age': [20 + (i % 50) for i in range(100)],
'active': [i % 2 == 0 for i in range(100)]
})
large_test_data = [
('large_table', large_table_data, 'table')
]
large_env, _ = await smartsend(
TEST_SUBJECT,
large_test_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id='large-' + correlation_id,
is_publish=False
)
if len(large_env['payloads']) == 1:
print('✅ Large table handled correctly')
print(f' Size: {large_env["payloads"][0]["size"]} bytes')
else:
print('❌ Large table handling failed')
passed = False
# Test JSON string output
print('\n=== JSON String Output Test ===')
import json
try:
parsed = json.loads(env_json_str)
if parsed['correlation_id'] == env['correlation_id'] and \
len(parsed['payloads']) == len(env['payloads']):
print('✅ JSON string is valid and matches envelope')
else:
print('❌ JSON string does not match envelope')
passed = False
except json.JSONDecodeError as e:
print(f'❌ JSON string is invalid: {e}')
passed = False
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
asyncio.run(run_test())

View File

@@ -0,0 +1,205 @@
"""
Python Text Receiver Test
Tests the smartreceive function with text payloads
"""
import asyncio
import sys
import os
import json
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge import smartreceive, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
async def run_test():
print('=== Python Text Receiver Test ===\n')
# Create a mock NATS message with text payload
test_text = 'Hello, NATSBridge! This is a test message.'
import base64
test_data = {
'correlation_id': 'py-receiver-test-' + str(asyncio.get_event_loop().time() * 1000000),
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
'timestamp': asyncio.get_event_loop().time().isoformat(),
'send_to': '/test/text',
'msg_purpose': 'test',
'sender_name': 'py-text-test',
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
'receiver_name': 'py-receiver',
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
'reply_to': '',
'reply_to_msg_id': '',
'broker_url': TEST_BROKER_URL,
'metadata': {},
'payloads': [
{
'id': 'payload-' + str(asyncio.get_event_loop().time() * 1000000),
'dataname': 'message',
'payload_type': 'text',
'transport': 'direct',
'encoding': 'base64',
'size': len(test_text.encode('utf8')),
'data': base64.b64encode(test_text.encode('utf8')).decode('ascii'),
'metadata': {'payload_bytes': len(test_text.encode('utf8'))}
}
]
}
mock_msg = {
'payload': json.dumps(test_data)
}
print('Mock Message Created:')
print(f' Correlation ID: {test_data["correlation_id"]}')
print(f' Payloads: {len(test_data["payloads"])}')
print(f' Payload dataname: {test_data["payloads"][0]["dataname"]}')
print(f' Payload type: {test_data["payloads"][0]["payload_type"]}')
print(f' Transport: {test_data["payloads"][0]["transport"]}\n')
try:
# Receive and process the message
print('Receiving and processing message...')
env = await smartreceive(
mock_msg,
max_retries=3,
base_delay=100,
max_delay=1000
)
print('\n=== Received Envelope ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate received data
print('=== Validation ===')
passed = True
if not env.get('correlation_id'):
print('❌ correlation_id is missing')
passed = False
else:
print('✅ correlation_id present')
if len(env['payloads']) != 1:
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
payload = env['payloads'][0]
if payload[0] != 'message':
print(f"❌ Expected dataname 'message', got '{payload[0]}'")
passed = False
else:
print('✅ Correct dataname')
if payload[2] != 'text':
print(f"❌ Expected type 'text', got '{payload[2]}'")
passed = False
else:
print('✅ Correct type')
if payload[1] != test_text:
print('❌ Data mismatch')
print(f' Expected: {test_text}')
print(f' Got: {payload[1]}')
passed = False
else:
print('✅ Data correctly deserialized')
# Test with multiple text payloads
print('\n=== Multiple Text Payloads Test ===')
multi_test_data = {
'correlation_id': 'multi-receiver-' + str(asyncio.get_event_loop().time() * 1000000),
'msg_id': 'msg-' + str(asyncio.get_event_loop().time() * 1000000),
'timestamp': asyncio.get_event_loop().time().isoformat(),
'send_to': '/test/text',
'msg_purpose': 'test',
'sender_name': 'py-text-test',
'sender_id': 'sender-' + str(asyncio.get_event_loop().time() * 1000000),
'receiver_name': 'py-receiver',
'receiver_id': 'receiver-' + str(asyncio.get_event_loop().time() * 1000000),
'reply_to': '',
'reply_to_msg_id': '',
'broker_url': TEST_BROKER_URL,
'metadata': {},
'payloads': [
{
'id': 'payload-1',
'dataname': 'msg1',
'payload_type': 'text',
'transport': 'direct',
'encoding': 'base64',
'size': 16,
'data': base64.b64encode(b'First message').decode('ascii'),
'metadata': {'payload_bytes': 16}
},
{
'id': 'payload-2',
'dataname': 'msg2',
'payload_type': 'text',
'transport': 'direct',
'encoding': 'base64',
'size': 16,
'data': base64.b64encode(b'Second message').decode('ascii'),
'metadata': {'payload_bytes': 16}
},
{
'id': 'payload-3',
'dataname': 'msg3',
'payload_type': 'text',
'transport': 'direct',
'encoding': 'base64',
'size': 16,
'data': base64.b64encode(b'Third message').decode('ascii'),
'metadata': {'payload_bytes': 16}
}
]
}
mock_multi_msg = {'payload': json.dumps(multi_test_data)}
multi_env = await smartreceive(mock_multi_msg)
if len(multi_env['payloads']) == 3:
print('✅ Multiple payloads handled correctly')
# Verify each payload
expected_messages = ['First message', 'Second message', 'Third message']
for i in range(3):
if multi_env['payloads'][i][1] == expected_messages[i]:
print(f'✅ Payload {i + 1} correctly deserialized')
else:
print(f'❌ Payload {i + 1} mismatch')
passed = False
else:
print(f"❌ Expected 3 payloads, got {len(multi_env['payloads'])}")
passed = False
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
asyncio.run(run_test())

164
test/test_py_text_sender.py Normal file
View File

@@ -0,0 +1,164 @@
"""
Python Text Sender Test
Tests the smartsend function with text payloads
"""
import asyncio
import sys
import os
# Add parent directory to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from natbridge import smartsend, DEFAULT_BROKER_URL, DEFAULT_FILESERVER_URL
TEST_SUBJECT = '/test/text'
TEST_BROKER_URL = os.environ.get('NATS_URL', 'nats://localhost:4222')
TEST_FILESERVER_URL = os.environ.get('FILESERVER_URL', 'http://localhost:8080')
async def run_test():
print('=== Python Text Sender Test ===\n')
correlation_id = 'py-text-test-' + str(asyncio.get_event_loop().time() * 1000000)
print(f'Correlation ID: {correlation_id}')
print(f'Subject: {TEST_SUBJECT}')
print(f'Broker URL: {TEST_BROKER_URL}\n')
# Test data
text_data = 'Hello, NATSBridge! This is a test message.'
test_data = [
('message', text_data, 'text')
]
try:
# Send the message
print('Sending text payload...')
env, env_json_str = await smartsend(
TEST_SUBJECT,
test_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id=correlation_id,
msg_purpose='test',
sender_name='py-text-test',
is_publish=False # Don't actually publish for this test
)
print('\n=== Envelope Created ===')
print(f'Correlation ID: {env["correlation_id"]}')
print(f'Message ID: {env["msg_id"]}')
print(f'Timestamp: {env["timestamp"]}')
print(f'Subject: {env["send_to"]}')
print(f'Purpose: {env["msg_purpose"]}')
print(f'Sender: {env["sender_name"]}')
print(f'Payloads: {len(env["payloads"])}\n')
# Validate envelope structure
print('=== Validation ===')
passed = True
if not env.get('correlation_id'):
print('❌ correlation_id is missing')
passed = False
else:
print('✅ correlation_id present')
if not env.get('msg_id'):
print('❌ msg_id is missing')
passed = False
else:
print('✅ msg_id present')
if not env.get('timestamp'):
print('❌ timestamp is missing')
passed = False
else:
print('✅ timestamp present')
if len(env['payloads']) != 1:
print(f'❌ Expected 1 payload, got {len(env["payloads"])}')
passed = False
else:
print('✅ Correct number of payloads')
payload = env['payloads'][0]
if payload['dataname'] != 'message':
print(f"❌ Expected dataname 'message', got '{payload['dataname']}'")
passed = False
else:
print('✅ Correct dataname')
if payload['payload_type'] != 'text':
print(f"❌ Expected payload_type 'text', got '{payload['payload_type']}'")
passed = False
else:
print('✅ Correct payload_type')
if payload['transport'] != 'direct':
print(f"❌ Expected transport 'direct', got '{payload['transport']}'")
passed = False
else:
print('✅ Correct transport')
if payload['encoding'] != 'base64':
print(f"❌ Expected encoding 'base64', got '{payload['encoding']}'")
passed = False
else:
print('✅ Correct encoding')
# Decode and verify the data
import base64
decoded_data = base64.b64decode(payload['data']).decode('utf8')
if decoded_data != text_data:
print('❌ Decoded data mismatch')
print(f' Expected: {text_data}')
print(f' Got: {decoded_data}')
passed = False
else:
print('✅ Data integrity verified')
print(f"\nPayload size: {payload['size']} bytes")
print(f'Base64 data length: {len(payload["data"])} chars')
# Test with multiple text payloads
print('\n=== Multiple Text Payloads Test ===')
multi_test_data = [
('msg1', 'First message', 'text'),
('msg2', 'Second message', 'text'),
('msg3', 'Third message', 'text')
]
multi_env, _ = await smartsend(
TEST_SUBJECT,
multi_test_data,
broker_url=TEST_BROKER_URL,
fileserver_url=TEST_FILESERVER_URL,
correlation_id='multi-test-' + correlation_id,
is_publish=False
)
if len(multi_env['payloads']) == 3:
print('✅ Multiple payloads handled correctly')
else:
print(f"❌ Expected 3 payloads, got {len(multi_env['payloads'])}")
passed = False
# Final result
print('\n=== Test Result ===')
if passed:
print('✅ ALL TESTS PASSED')
sys.exit(0)
else:
print('❌ SOME TESTS FAILED')
sys.exit(1)
except Exception as e:
print(f'❌ Test failed with error: {e}')
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == '__main__':
asyncio.run(run_test())

Binary file not shown.

Binary file not shown.