Compare commits
46 Commits
v0.4.1
...
split_smar
| Author | SHA1 | Date | |
|---|---|---|---|
| 1299febcdc | |||
| be94c62760 | |||
| 6a862ef243 | |||
| ae2de5fc62 | |||
| df0bbc7327 | |||
| d94761c866 | |||
| f8235e1a59 | |||
| 647cadf497 | |||
| 8c793a81b6 | |||
| 6a42ba7e43 | |||
| 14b3790251 | |||
| 61d81bed62 | |||
| 1a10bc1a5f | |||
| 7f68d08134 | |||
| ab20cd896f | |||
| 5a9e93d6e7 | |||
| b51641dc7e | |||
| 45f1257896 | |||
| 3e2b8b1e3a | |||
| 90d81617ef | |||
| 64c62e616b | |||
| 2c340e37c7 | |||
| 7853e94d2e | |||
| 99bf57b154 | |||
| 0fa6eaf95b | |||
| 76f42be740 | |||
| d99dc41be9 | |||
| 263508b8f7 | |||
| 0c2cca30ed | |||
| 46fdf668c6 | |||
| f8a92a45a0 | |||
| cec70e6036 | |||
| f9e08ba628 | |||
| c12a078149 | |||
| dedd803dc3 | |||
| e8e927a491 | |||
| d950bbac23 | |||
| fc8da2ebf5 | |||
| f6e50c405f | |||
| c06f508e8f | |||
| 97bf1e47f4 | |||
| ef47fddd56 | |||
| 896dd84d2a | |||
| def75d8f86 | |||
| 69f2173f75 | |||
| 075d355c58 |
@@ -13,3 +13,23 @@ Role: Principal Systems Architect & Lead Software Engineer.Objective: Implement
|
|||||||
|
|
||||||
|
|
||||||
Create a walkthrough for Julia service-A service sending a mix-content chat message to Julia service-B. the chat message must includes
|
Create a walkthrough for Julia service-A service sending a mix-content chat message to Julia service-B. the chat message must includes
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
I updated the following:
|
||||||
|
- NATSBridge.jl. Essentially I add NATS_connection keyword and new publish_message function to support the keyword.
|
||||||
|
|
||||||
|
Use them and ONLY them as ground truth.
|
||||||
|
|
||||||
|
Then update the following files accordingly:
|
||||||
|
- architecture.md
|
||||||
|
- implementation.md
|
||||||
|
|
||||||
|
All API should be semantically consistent and naming should be consistent across the board.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
name = "NATSBridge"
|
name = "NATSBridge"
|
||||||
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
|
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
|
||||||
version = "0.4.1"
|
version = "0.4.3"
|
||||||
authors = ["narawat <narawat@gmail.com>"]
|
authors = ["narawat <narawat@gmail.com>"]
|
||||||
|
|
||||||
[deps]
|
[deps]
|
||||||
|
|||||||
@@ -1,8 +1,13 @@
|
|||||||
# Architecture Documentation: Bi-Directional Data Bridge (Julia ↔ JavaScript)
|
# Architecture Documentation: Bi-Directional Data Bridge
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
This document describes the architecture for a high-performance, bi-directional data bridge between a Julia service and a JavaScript (Node.js) service using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
This document describes the architecture for a high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||||
|
|
||||||
|
The system enables seamless communication across all three platforms:
|
||||||
|
- **Julia ↔ JavaScript** bi-directional messaging
|
||||||
|
- **JavaScript ↔ Python/Micropython** bi-directional messaging
|
||||||
|
- **Julia ↔ Python/Micropython** bi-directional messaging (via JSON serialization)
|
||||||
|
|
||||||
### File Server Handler Architecture
|
### File Server Handler Architecture
|
||||||
|
|
||||||
@@ -12,16 +17,16 @@ The system uses **handler functions** to abstract file server operations, allowi
|
|||||||
|
|
||||||
```julia
|
```julia
|
||||||
# Upload handler - uploads data to file server and returns URL
|
# Upload handler - uploads data to file server and returns URL
|
||||||
# The handler is passed to smartsend as fileserverUploadHandler parameter
|
# The handler is passed to smartsend as fileserver_upload_handler parameter
|
||||||
# It receives: (fileserver_url::String, dataname::String, data::Vector{UInt8})
|
# It receives: (fileserver_url::String, dataname::String, data::Vector{UInt8})
|
||||||
# Returns: Dict{String, Any} with keys: "status", "uploadid", "fileid", "url"
|
# Returns: Dict{String, Any} with keys: "status", "uploadid", "fileid", "url"
|
||||||
fileserverUploadHandler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
fileserver_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
||||||
|
|
||||||
# Download handler - fetches data from file server URL with exponential backoff
|
# Download handler - fetches data from file server URL with exponential backoff
|
||||||
# The handler is passed to smartreceive as fileserverDownloadHandler parameter
|
# The handler is passed to smartreceive as fileserver_download_handler parameter
|
||||||
# It receives: (url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)
|
# It receives: (url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)
|
||||||
# Returns: Vector{UInt8} (the downloaded data)
|
# Returns: Vector{UInt8} (the downloaded data)
|
||||||
fileserverDownloadHandler(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)::Vector{UInt8}
|
fileserver_download_handler(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)::Vector{UInt8}
|
||||||
```
|
```
|
||||||
|
|
||||||
This design allows the system to support multiple file server backends without changing the core messaging logic.
|
This design allows the system to support multiple file server backends without changing the core messaging logic.
|
||||||
@@ -35,8 +40,24 @@ The system uses a **standardized list-of-tuples format** for all payload operati
|
|||||||
# Input format for smartsend (always a list of tuples with type info)
|
# Input format for smartsend (always a list of tuples with type info)
|
||||||
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||||
|
|
||||||
# Output format for smartreceive (always returns a list of tuples)
|
# Output format for smartreceive (returns a dictionary-like object with payloads field containing list of tuples)
|
||||||
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
# Returns: Dict-like object with envelope metadata and payloads field containing Vector{Tuple{String, Any, String}}
|
||||||
|
# {
|
||||||
|
# "correlation_id": "...",
|
||||||
|
# "msg_id": "...",
|
||||||
|
# "timestamp": "...",
|
||||||
|
# "send_to": "...",
|
||||||
|
# "msg_purpose": "...",
|
||||||
|
# "sender_name": "...",
|
||||||
|
# "sender_id": "...",
|
||||||
|
# "receiver_name": "...",
|
||||||
|
# "receiver_id": "...",
|
||||||
|
# "reply_to": "...",
|
||||||
|
# "reply_to_msg_id": "...",
|
||||||
|
# "broker_url": "...",
|
||||||
|
# "metadata": {...},
|
||||||
|
# "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||||
|
# }
|
||||||
```
|
```
|
||||||
|
|
||||||
**Supported Types:**
|
**Supported Types:**
|
||||||
@@ -57,17 +78,16 @@ This design allows per-payload type specification, enabling **mixed-content mess
|
|||||||
smartsend(
|
smartsend(
|
||||||
"/test",
|
"/test",
|
||||||
[("dataname1", data1, "dictionary")], # List with one tuple (data, type)
|
[("dataname1", data1, "dictionary")], # List with one tuple (data, type)
|
||||||
nats_url="nats://localhost:4222",
|
broker_url="nats://localhost:4222",
|
||||||
fileserverUploadHandler=plik_oneshot_upload,
|
fileserver_upload_handler=plik_oneshot_upload
|
||||||
metadata=user_provided_envelope_level_metadata
|
|
||||||
)
|
)
|
||||||
|
|
||||||
# Multiple payloads in one message with different types
|
# Multiple payloads in one message with different types
|
||||||
smartsend(
|
smartsend(
|
||||||
"/test",
|
"/test",
|
||||||
[("dataname1", data1, "dictionary"), ("dataname2", data2, "table")],
|
[("dataname1", data1, "dictionary"), ("dataname2", data2, "table")],
|
||||||
nats_url="nats://localhost:4222",
|
broker_url="nats://localhost:4222",
|
||||||
fileserverUploadHandler=plik_oneshot_upload
|
fileserver_upload_handler=plik_oneshot_upload
|
||||||
)
|
)
|
||||||
|
|
||||||
# Mixed content (e.g., chat with text, image, audio)
|
# Mixed content (e.g., chat with text, image, audio)
|
||||||
@@ -78,12 +98,14 @@ smartsend(
|
|||||||
("user_image", image_data, "image"),
|
("user_image", image_data, "image"),
|
||||||
("audio_clip", audio_data, "audio")
|
("audio_clip", audio_data, "audio")
|
||||||
],
|
],
|
||||||
nats_url="nats://localhost:4222"
|
broker_url="nats://localhost:4222"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Receive always returns a list
|
# Receive returns a dictionary envelope with all metadata and deserialized payloads
|
||||||
payloads = smartreceive(msg, fileserverDownloadHandler, max_retries, base_delay, max_delay)
|
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff, max_retries=5, base_delay=100, max_delay=5000)
|
||||||
# payloads = [("dataname1", data1, type1), ("dataname2", data2, type2), ...]
|
# env["payloads"] = [("dataname1", data1, type1), ("dataname2", data2, type2), ...]
|
||||||
|
# env["correlation_id"], env["msg_id"], etc.
|
||||||
|
# env is a dictionary containing envelope metadata and payloads field
|
||||||
```
|
```
|
||||||
|
|
||||||
## Architecture Diagram
|
## Architecture Diagram
|
||||||
@@ -116,48 +138,48 @@ flowchart TD
|
|||||||
|
|
||||||
## System Components
|
## System Components
|
||||||
|
|
||||||
### 1. msgEnvelope_v1 - Message Envelope
|
### 1. msg_envelope_v1 - Message Envelope
|
||||||
|
|
||||||
The `msgEnvelope_v1` structure provides a comprehensive message format for bidirectional communication between Julia and JavaScript services.
|
The `msg_envelope_v1` structure provides a comprehensive message format for bidirectional communication between Julia, JavaScript, and Python/Micropython applications.
|
||||||
|
|
||||||
**Julia Structure:**
|
**Julia Structure:**
|
||||||
```julia
|
```julia
|
||||||
struct msgEnvelope_v1
|
struct msg_envelope_v1
|
||||||
correlationId::String # Unique identifier to track messages across systems
|
correlation_id::String # Unique identifier to track messages across systems
|
||||||
msgId::String # This message id
|
msg_id::String # This message id
|
||||||
timestamp::String # Message published timestamp
|
timestamp::String # Message published timestamp
|
||||||
|
|
||||||
sendTo::String # Topic/subject the sender sends to
|
send_to::String # Topic/subject the sender sends to
|
||||||
msgPurpose::String # Purpose of this message (ACK | NACK | updateStatus | shutdown | ...)
|
msg_purpose::String # Purpose of this message (ACK | NACK | updateStatus | shutdown | ...)
|
||||||
senderName::String # Sender name (e.g., "agent-wine-web-frontend")
|
sender_name::String # Sender name (e.g., "agent-wine-web-frontend")
|
||||||
senderId::String # Sender id (uuid4)
|
sender_id::String # Sender id (uuid4)
|
||||||
receiverName::String # Message receiver name (e.g., "agent-backend")
|
receiver_name::String # Message receiver name (e.g., "agent-backend")
|
||||||
receiverId::String # Message receiver id (uuid4 or nothing for broadcast)
|
receiver_id::String # Message receiver id (uuid4 or nothing for broadcast)
|
||||||
replyTo::String # Topic to reply to
|
reply_to::String # Topic to reply to
|
||||||
replyToMsgId::String # Message id this message is replying to
|
reply_to_msg_id::String # Message id this message is replying to
|
||||||
brokerURL::String # NATS server address
|
broker_url::String # NATS server address
|
||||||
|
|
||||||
metadata::Dict{String, Any}
|
metadata::Dict{String, Any}
|
||||||
payloads::AbstractArray{msgPayload_v1} # Multiple payloads stored here
|
payloads::Vector{msg_payload_v1} # Multiple payloads stored here
|
||||||
end
|
end
|
||||||
```
|
```
|
||||||
|
|
||||||
**JSON Schema:**
|
**JSON Schema:**
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"correlationId": "uuid-v4-string",
|
"correlation_id": "uuid-v4-string",
|
||||||
"msgId": "uuid-v4-string",
|
"msg_id": "uuid-v4-string",
|
||||||
"timestamp": "2024-01-15T10:30:00Z",
|
"timestamp": "2024-01-15T10:30:00Z",
|
||||||
|
|
||||||
"sendTo": "topic/subject",
|
"send_to": "topic/subject",
|
||||||
"msgPurpose": "ACK | NACK | updateStatus | shutdown | chat",
|
"msg_purpose": "ACK | NACK | updateStatus | shutdown | chat",
|
||||||
"senderName": "agent-wine-web-frontend",
|
"sender_name": "agent-wine-web-frontend",
|
||||||
"senderId": "uuid4",
|
"sender_id": "uuid4",
|
||||||
"receiverName": "agent-backend",
|
"receiver_name": "agent-backend",
|
||||||
"receiverId": "uuid4",
|
"receiver_id": "uuid4",
|
||||||
"replyTo": "topic",
|
"reply_to": "topic",
|
||||||
"replyToMsgId": "uuid4",
|
"reply_to_msg_id": "uuid4",
|
||||||
"brokerURL": "nats://localhost:4222",
|
"broker_url": "nats://localhost:4222",
|
||||||
|
|
||||||
"metadata": {
|
"metadata": {
|
||||||
|
|
||||||
@@ -167,7 +189,7 @@ end
|
|||||||
{
|
{
|
||||||
"id": "uuid4",
|
"id": "uuid4",
|
||||||
"dataname": "login_image",
|
"dataname": "login_image",
|
||||||
"type": "image",
|
"payload_type": "image",
|
||||||
"transport": "direct",
|
"transport": "direct",
|
||||||
"encoding": "base64",
|
"encoding": "base64",
|
||||||
"size": 15433,
|
"size": 15433,
|
||||||
@@ -179,7 +201,7 @@ end
|
|||||||
{
|
{
|
||||||
"id": "uuid4",
|
"id": "uuid4",
|
||||||
"dataname": "large_data",
|
"dataname": "large_data",
|
||||||
"type": "table",
|
"payload_type": "table",
|
||||||
"transport": "link",
|
"transport": "link",
|
||||||
"encoding": "none",
|
"encoding": "none",
|
||||||
"size": 524288,
|
"size": 524288,
|
||||||
@@ -192,16 +214,16 @@ end
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. msgPayload_v1 - Payload Structure
|
### 2. msg_payload_v1 - Payload Structure
|
||||||
|
|
||||||
The `msgPayload_v1` structure provides flexible payload handling for various data types.
|
The `msg_payload_v1` structure provides flexible payload handling for various data types across all supported platforms.
|
||||||
|
|
||||||
**Julia Structure:**
|
**Julia Structure:**
|
||||||
```julia
|
```julia
|
||||||
struct msgPayload_v1
|
struct msg_payload_v1
|
||||||
id::String # Id of this payload (e.g., "uuid4")
|
id::String # Id of this payload (e.g., "uuid4")
|
||||||
dataname::String # Name of this payload (e.g., "login_image")
|
dataname::String # Name of this payload (e.g., "login_image")
|
||||||
type::String # "text | dictionary | table | image | audio | video | binary"
|
payload_type::String # "text | dictionary | table | image | audio | video | binary"
|
||||||
transport::String # "direct | link"
|
transport::String # "direct | link"
|
||||||
encoding::String # "none | json | base64 | arrow-ipc"
|
encoding::String # "none | json | base64 | arrow-ipc"
|
||||||
size::Integer # Data size in bytes
|
size::Integer # Data size in bytes
|
||||||
@@ -222,7 +244,7 @@ end
|
|||||||
┌─────────────────────────────────────────────────────────────┐
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
│ smartsend Function │
|
│ smartsend Function │
|
||||||
│ Accepts: [(dataname1, data1, type1), ...] │
|
│ Accepts: [(dataname1, data1, type1), ...] │
|
||||||
│ (No standalone type parameter - type per payload) │
|
│ (Type is per payload, not standalone) │
|
||||||
└─────────────────────────────────────────────────────────────┘
|
└─────────────────────────────────────────────────────────────┘
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
@@ -249,19 +271,77 @@ end
|
|||||||
└─────────────────┘ └─────────────────┘
|
└─────────────────┘ └─────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
### 4. Julia Module Architecture
|
### 4. Cross-Platform Architecture
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
subgraph PythonMicropython
|
||||||
|
Py[Python/Micropython]
|
||||||
|
PySmartSend[smartsend]
|
||||||
|
PySmartReceive[smartreceive]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph JavaScript
|
||||||
|
JS[JavaScript]
|
||||||
|
JSSmartSend[smartsend]
|
||||||
|
JSSmartReceive[smartreceive]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Julia
|
||||||
|
Julia[Julia]
|
||||||
|
JuliaSmartSend[smartsend]
|
||||||
|
JuliaSmartReceive[smartreceive]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph NATS
|
||||||
|
NATSServer[NATS Server]
|
||||||
|
end
|
||||||
|
|
||||||
|
PySmartSend --> NATSServer
|
||||||
|
JSSmartSend --> NATSServer
|
||||||
|
JuliaSmartSend --> NATSServer
|
||||||
|
NATSServer --> PySmartReceive
|
||||||
|
NATSServer --> JSSmartReceive
|
||||||
|
NATSServer --> JuliaSmartReceive
|
||||||
|
|
||||||
|
style PythonMicropython fill:#e1f5fe
|
||||||
|
style JavaScript fill:#f3e5f5
|
||||||
|
style Julia fill:#e8f5e9
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Python/Micropython Module Architecture
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
graph TD
|
graph TD
|
||||||
subgraph JuliaModule
|
subgraph PyModule
|
||||||
smartsendJulia[smartsend Julia]
|
PySmartSend[smartsend]
|
||||||
SizeCheck[Size Check]
|
SizeCheck[Size Check]
|
||||||
DirectPath[Direct Path]
|
DirectPath[Direct Path]
|
||||||
LinkPath[Link Path]
|
LinkPath[Link Path]
|
||||||
HTTPClient[HTTP Client]
|
HTTPClient[HTTP Client]
|
||||||
end
|
end
|
||||||
|
|
||||||
smartsendJulia --> SizeCheck
|
PySmartSend --> SizeCheck
|
||||||
|
SizeCheck -->|< 1MB| DirectPath
|
||||||
|
SizeCheck -->|>= 1MB| LinkPath
|
||||||
|
LinkPath --> HTTPClient
|
||||||
|
|
||||||
|
style PyModule fill:#b3e5fc
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Julia Module Architecture
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph TD
|
||||||
|
subgraph JuliaModule
|
||||||
|
JuliaSmartSend[smartsend]
|
||||||
|
SizeCheck[Size Check]
|
||||||
|
DirectPath[Direct Path]
|
||||||
|
LinkPath[Link Path]
|
||||||
|
HTTPClient[HTTP Client]
|
||||||
|
end
|
||||||
|
|
||||||
|
JuliaSmartSend --> SizeCheck
|
||||||
SizeCheck -->|< 1MB| DirectPath
|
SizeCheck -->|< 1MB| DirectPath
|
||||||
SizeCheck -->|>= 1MB| LinkPath
|
SizeCheck -->|>= 1MB| LinkPath
|
||||||
LinkPath --> HTTPClient
|
LinkPath --> HTTPClient
|
||||||
@@ -269,19 +349,19 @@ graph TD
|
|||||||
style JuliaModule fill:#c5e1a5
|
style JuliaModule fill:#c5e1a5
|
||||||
```
|
```
|
||||||
|
|
||||||
### 5. JavaScript Module Architecture
|
### 7. JavaScript Module Architecture
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
graph TD
|
graph TD
|
||||||
subgraph JSModule
|
subgraph JSModule
|
||||||
smartsendJS[smartsend JS]
|
JSSmartSend[smartsend]
|
||||||
smartreceiveJS[smartreceive JS]
|
JSSmartReceive[smartreceive]
|
||||||
JetStreamConsumer[JetStream Pull Consumer]
|
JetStreamConsumer[JetStream Pull Consumer]
|
||||||
ApacheArrow[Apache Arrow]
|
ApacheArrow[Apache Arrow]
|
||||||
end
|
end
|
||||||
|
|
||||||
smartsendJS --> NATS
|
JSSmartSend --> NATS
|
||||||
smartreceiveJS --> JetStreamConsumer
|
JSSmartReceive --> JetStreamConsumer
|
||||||
JetStreamConsumer --> ApacheArrow
|
JetStreamConsumer --> ApacheArrow
|
||||||
|
|
||||||
style JSModule fill:#f3e5f5
|
style JSModule fill:#f3e5f5
|
||||||
@@ -289,6 +369,25 @@ graph TD
|
|||||||
|
|
||||||
## Implementation Details
|
## Implementation Details
|
||||||
|
|
||||||
|
### API Consistency Across Languages
|
||||||
|
|
||||||
|
**High-Level API (Consistent Across All Languages):**
|
||||||
|
- `smartsend(subject, data, ...)` - Main publishing function
|
||||||
|
- `smartreceive(msg, ...)` - Main receiving function
|
||||||
|
- Message envelope structure (`msg_envelope_v1` / `MessageEnvelope`)
|
||||||
|
- Payload structure (`msg_payload_v1` / `MessagePayload`)
|
||||||
|
- Transport strategy (direct vs link based on size threshold)
|
||||||
|
- Supported payload types: text, dictionary, table, image, audio, video, binary
|
||||||
|
|
||||||
|
**Low-Level Native Functions (Language-Specific Conventions):**
|
||||||
|
- Julia: `NATS.connect()`, `publish_message()`, function overloading
|
||||||
|
- JavaScript: `nats.js` client, native async/await patterns
|
||||||
|
- Python: `nats-python` client, native async/await patterns
|
||||||
|
|
||||||
|
**Connection Reuse Pattern:**
|
||||||
|
- **Julia:** Uses `NATS_connection` keyword parameter with function overloading
|
||||||
|
- **JavaScript/Python:** Achieved by creating NATS client outside the function and reusing it in custom handlers
|
||||||
|
|
||||||
### Julia Implementation
|
### Julia Implementation
|
||||||
|
|
||||||
#### Dependencies
|
#### Dependencies
|
||||||
@@ -303,13 +402,47 @@ graph TD
|
|||||||
```julia
|
```julia
|
||||||
function smartsend(
|
function smartsend(
|
||||||
subject::String,
|
subject::String,
|
||||||
data::AbstractArray{Tuple{String, Any, String}}; # No standalone type parameter
|
data::AbstractArray{Tuple{String, Any, String}, 1}; # List of (dataname, data, type) tuples
|
||||||
nats_url::String = "nats://localhost:4222",
|
broker_url::String = DEFAULT_BROKER_URL, # NATS server URL
|
||||||
fileserverUploadHandler::Function = plik_oneshot_upload,
|
fileserver_url = DEFAULT_FILESERVER_URL,
|
||||||
size_threshold::Int = 1_000_000 # 1MB
|
fileserver_upload_handler::Function = plik_oneshot_upload,
|
||||||
|
size_threshold::Int = DEFAULT_SIZE_THRESHOLD,
|
||||||
|
correlation_id::Union{String, Nothing} = nothing,
|
||||||
|
msg_purpose::String = "chat",
|
||||||
|
sender_name::String = "NATSBridge",
|
||||||
|
receiver_name::String = "",
|
||||||
|
receiver_id::String = "",
|
||||||
|
reply_to::String = "",
|
||||||
|
reply_to_msg_id::String = "",
|
||||||
|
is_publish::Bool = true, # Whether to automatically publish to NATS
|
||||||
|
NATS_connection::Union{NATS.Connection, Nothing} = nothing # Pre-existing NATS connection (optional, saves connection overhead)
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Keyword Parameter - NATS_connection:**
|
||||||
|
- `NATS_connection::Union{NATS.Connection, Nothing} = nothing` - Pre-existing NATS connection. When provided, `smartsend` uses this connection instead of creating a new one, avoiding the overhead of connection establishment. This is useful for high-frequency publishing scenarios where connection reuse provides performance benefits.
|
||||||
|
|
||||||
|
**Connection Handling Logic:**
|
||||||
|
```julia
|
||||||
|
if is_publish == false
|
||||||
|
# skip publish a message
|
||||||
|
elseif is_publish == true && NATS_connection === nothing
|
||||||
|
publish_message(broker_url, subject, env_json_str, cid) # Creates new connection
|
||||||
|
elseif is_publish == true && NATS_connection !== nothing
|
||||||
|
publish_message(NATS_connection, subject, env_json_str, cid) # Uses provided connection
|
||||||
|
end
|
||||||
|
```
|
||||||
|
|
||||||
|
**Return Value:**
|
||||||
|
- Returns a tuple `(env, env_json_str)` where:
|
||||||
|
- `env::msg_envelope_v1` - The envelope object containing all metadata and payloads
|
||||||
|
- `env_json_str::String` - JSON string representation of the envelope for publishing
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `is_publish::Bool = true` - When `true` (default), the message is automatically published to NATS. When `false`, the function returns the envelope and JSON string without publishing, allowing manual publishing via NATS request-reply pattern.
|
||||||
|
|
||||||
|
The envelope object can be accessed directly for programmatic use, while the JSON string can be published directly to NATS using the request-reply pattern.
|
||||||
|
|
||||||
**Input Format:**
|
**Input Format:**
|
||||||
- `data::AbstractArray{Tuple{String, Any, String}}` - **Must be a list of (dataname, data, type) tuples**: `[("dataname1", data1, "type1"), ("dataname2", data2, "type2"), ...]`
|
- `data::AbstractArray{Tuple{String, Any, String}}` - **Must be a list of (dataname, data, type) tuples**: `[("dataname1", data1, "type1"), ("dataname2", data2, "type2"), ...]`
|
||||||
- Even for single payloads: `[(dataname1, data1, "type1")]`
|
- Even for single payloads: `[(dataname1, data1, "type1")]`
|
||||||
@@ -326,8 +459,8 @@ function smartsend(
|
|||||||
|
|
||||||
```julia
|
```julia
|
||||||
function smartreceive(
|
function smartreceive(
|
||||||
msg::NATS.Message,
|
msg::NATS.Msg;
|
||||||
fileserverDownloadHandler::Function;
|
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||||
max_retries::Int = 5,
|
max_retries::Int = 5,
|
||||||
base_delay::Int = 100,
|
base_delay::Int = 100,
|
||||||
max_delay::Int = 5000
|
max_delay::Int = 5000
|
||||||
@@ -336,86 +469,276 @@ function smartreceive(
|
|||||||
# Iterate through all payloads
|
# Iterate through all payloads
|
||||||
# For each payload: check transport type
|
# For each payload: check transport type
|
||||||
# If direct: decode Base64 payload
|
# If direct: decode Base64 payload
|
||||||
# If link: fetch from URL with exponential backoff using fileserverDownloadHandler
|
# If link: fetch from URL with exponential backoff using fileserver_download_handler
|
||||||
# Deserialize payload based on type
|
# Deserialize payload based on type
|
||||||
# Return list of (dataname, data, type) tuples
|
# Return envelope dictionary with all metadata and deserialized payloads
|
||||||
end
|
end
|
||||||
```
|
```
|
||||||
|
|
||||||
**Output Format:**
|
**Output Format:**
|
||||||
- Always returns a list of tuples: `[(dataname1, data1, type1), (dataname2, data2, type2), ...]`
|
- Returns a JSON object (dictionary) containing all envelope fields:
|
||||||
- Even for single payloads: `[(dataname1, data1, type1)]`
|
- `correlation_id`, `msg_id`, `timestamp`, `send_to`, `msg_purpose`, `sender_name`, `sender_id`, `receiver_name`, `receiver_id`, `reply_to`, `reply_to_msg_id`, `broker_url`
|
||||||
|
- `metadata` - Message-level metadata dictionary
|
||||||
|
- `payloads` - List of tuples, each containing `(dataname, data, type)` with deserialized payload data
|
||||||
|
|
||||||
**Process Flow:**
|
**Process Flow:**
|
||||||
1. Parse the JSON envelope to extract the `payloads` array
|
1. Parse the JSON envelope to extract all fields
|
||||||
2. Iterate through each payload in `payloads`
|
2. Iterate through each payload in `payloads`
|
||||||
3. For each payload:
|
3. For each payload:
|
||||||
- Determine transport type (`direct` or `link`)
|
- Determine transport type (`direct` or `link`)
|
||||||
- If `direct`: decode Base64 data from the message
|
- If `direct`: decode Base64 data from the message
|
||||||
- If `link`: fetch data from URL using exponential backoff (via `fileserverDownloadHandler`)
|
- If `link`: fetch data from URL using exponential backoff (via `fileserver_download_handler`)
|
||||||
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
|
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
|
||||||
4. Return list of `(dataname, data, type)` tuples
|
4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples
|
||||||
|
|
||||||
**Note:** The `fileserverDownloadHandler` receives `(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)` and returns `Vector{UInt8}`.
|
**Note:** The `fileserver_download_handler` receives `(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)` and returns `Vector{UInt8}`.
|
||||||
|
|
||||||
|
#### publish_message Function
|
||||||
|
|
||||||
|
The `publish_message` function provides two overloads for publishing messages to NATS:
|
||||||
|
|
||||||
|
**Overload 1 - URL-based publishing (creates new connection):**
|
||||||
|
```julia
|
||||||
|
function publish_message(broker_url::String, subject::String, message::String, correlation_id::String)
|
||||||
|
conn = NATS.connect(broker_url) # Create NATS connection
|
||||||
|
publish_message(conn, subject, message, correlation_id)
|
||||||
|
end
|
||||||
|
```
|
||||||
|
|
||||||
|
**Overload 2 - Connection-based publishing (uses pre-existing connection):**
|
||||||
|
```julia
|
||||||
|
function publish_message(conn::NATS.Connection, subject::String, message::String, correlation_id::String)
|
||||||
|
try
|
||||||
|
NATS.publish(conn, subject, message) # Publish message to NATS
|
||||||
|
log_trace(correlation_id, "Message published to $subject") # Log successful publish
|
||||||
|
finally
|
||||||
|
NATS.drain(conn) # Ensure connection is closed properly
|
||||||
|
end
|
||||||
|
end
|
||||||
|
```
|
||||||
|
|
||||||
|
**Use Case:** Use the connection-based overload when you already have an established NATS connection and want to publish multiple messages without the overhead of creating a new connection for each publish. This is a Julia-specific optimization that leverages function overloading.
|
||||||
|
|
||||||
|
**Integration with smartsend:**
|
||||||
|
```julia
|
||||||
|
# When NATS_connection is provided to smartsend, it uses the connection-based publish_message
|
||||||
|
env, env_json_str = smartsend(
|
||||||
|
"my.subject",
|
||||||
|
[("data", payload_data, "type")],
|
||||||
|
NATS_connection=my_connection, # Pre-existing connection
|
||||||
|
is_publish=true
|
||||||
|
)
|
||||||
|
# Uses: publish_message(NATS_connection, subject, env_json_str, cid)
|
||||||
|
|
||||||
|
# When NATS_connection is not provided, it uses the URL-based publish_message
|
||||||
|
env, env_json_str = smartsend(
|
||||||
|
"my.subject",
|
||||||
|
[("data", payload_data, "type")],
|
||||||
|
broker_url="nats://localhost:4222",
|
||||||
|
is_publish=true
|
||||||
|
)
|
||||||
|
# Uses: publish_message(broker_url, subject, env_json_str, cid)
|
||||||
|
```
|
||||||
|
|
||||||
|
**API Consistency Note:**
|
||||||
|
- **High-level API (smartsend, smartreceive):** Uses consistent naming across all three languages (Julia, JavaScript, Python/Micropython)
|
||||||
|
- **Low-level native functions (NATS.connect(), publish_message()):** Follow the conventions of the specific language ecosystem and do not require cross-language consistency
|
||||||
|
|
||||||
### JavaScript Implementation
|
### JavaScript Implementation
|
||||||
|
|
||||||
#### Dependencies
|
#### Dependencies
|
||||||
- `nats.js` - Core NATS functionality
|
- `nats.js` - Core NATS functionality
|
||||||
- `apache-arrow` - Arrow IPC serialization
|
- `apache-arrow` - Arrow IPC serialization
|
||||||
- `uuid` - Correlation ID generation
|
- `uuid` - Correlation ID and message ID generation
|
||||||
|
- `base64-arraybuffer` - Base64 encoding/decoding
|
||||||
|
- `node-fetch` or `fetch` - HTTP client for file server
|
||||||
|
|
||||||
#### smartsend Function
|
#### smartsend Function
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
async function smartsend(subject, data, options = {})
|
async function smartsend(
|
||||||
// data format: [(dataname, data, type), ...]
|
subject,
|
||||||
// options object should include:
|
data, // List of (dataname, data, type) tuples: [(dataname1, data1, type1), ...]
|
||||||
// - natsUrl: NATS server URL
|
options = {}
|
||||||
// - fileserverUrl: base URL of the file server
|
)
|
||||||
// - sizeThreshold: threshold in bytes for transport selection
|
|
||||||
// - correlationId: optional correlation ID for tracing
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `broker_url` (String) - NATS server URL (default: `"nats://localhost:4222"`)
|
||||||
|
- `fileserver_url` (String) - Base URL of the file server (default: `"http://localhost:8080"`)
|
||||||
|
- `size_threshold` (Number) - Threshold in bytes for transport selection (default: `1048576` = 1MB)
|
||||||
|
- `correlation_id` (String) - Optional correlation ID for tracing
|
||||||
|
- `msg_purpose` (String) - Purpose of the message (default: `"chat"`)
|
||||||
|
- `sender_name` (String) - Sender name (default: `"NATSBridge"`)
|
||||||
|
- `receiver_name` (String) - Message receiver name (default: `""`)
|
||||||
|
- `receiver_id` (String) - Message receiver ID (default: `""`)
|
||||||
|
- `reply_to` (String) - Topic to reply to (default: `""`)
|
||||||
|
- `reply_to_msg_id` (String) - Message ID this message is replying to (default: `""`)
|
||||||
|
- `is_publish` (Boolean) - Whether to automatically publish the message to NATS (default: `true`)
|
||||||
|
- `fileserver_upload_handler` (Function) - Custom upload handler function
|
||||||
|
|
||||||
|
**Note:** JavaScript uses `is_publish` option (instead of `NATS_connection` keyword) to control automatic publishing behavior. Connection reuse can be achieved by creating a NATS client outside the function and reusing it in a custom `fileserver_upload_handler` or custom publish implementation.
|
||||||
|
|
||||||
|
**Return Value:**
|
||||||
|
- Returns a Promise that resolves to an object containing:
|
||||||
|
- `env` - The envelope object containing all metadata and payloads
|
||||||
|
- `env_json_str` - JSON string representation of the envelope for publishing
|
||||||
|
|
||||||
**Input Format:**
|
**Input Format:**
|
||||||
- `data` - **Must be a list of (dataname, data, type) tuples**: `[(dataname1, data1, "type1"), (dataname2, data2, "type2"), ...]`
|
- `data` - **Must be a list of (dataname, data, type) tuples**: `[(dataname1, data1, "type1"), (dataname2, data2, "type2"), ...]`
|
||||||
- Even for single payloads: `[(dataname1, data1, "type1")]`
|
- Even for single payloads: `[(dataname1, data1, "type1")]`
|
||||||
- Each payload can have a different type, enabling mixed-content messages
|
- Each payload can have a different type, enabling mixed-content messages
|
||||||
|
- Supported types: `"text"`, `"dictionary"`, `"table"`, `"image"`, `"audio"`, `"video"`, `"binary"`
|
||||||
|
|
||||||
**Flow:**
|
**Flow:**
|
||||||
1. Iterate through the list of (dataname, data, type) tuples
|
1. Generate correlation ID and message ID if not provided
|
||||||
2. For each payload: extract the type from the tuple and serialize accordingly
|
2. Iterate through the list of `(dataname, data, type)` tuples
|
||||||
3. Check payload size
|
3. For each payload:
|
||||||
4. If < threshold: publish directly to NATS
|
- Serialize based on payload type
|
||||||
5. If >= threshold: upload to HTTP server, publish NATS with URL
|
- Check payload size
|
||||||
|
- If < threshold: Base64 encode and include in envelope
|
||||||
|
- If >= threshold: Upload to HTTP server, store URL in envelope
|
||||||
|
4. Publish the JSON envelope to NATS
|
||||||
|
5. Return envelope object and JSON string
|
||||||
|
|
||||||
#### smartreceive Handler
|
#### smartreceive Handler
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
async function smartreceive(msg, options = {})
|
async function smartreceive(msg, options = {})
|
||||||
// options object should include:
|
|
||||||
// - fileserverDownloadHandler: function to fetch data from file server URL
|
|
||||||
// - max_retries: maximum retry attempts for fetching URL
|
|
||||||
// - base_delay: initial delay for exponential backoff in ms
|
|
||||||
// - max_delay: maximum delay for exponential backoff in ms
|
|
||||||
// - correlationId: optional correlation ID for tracing
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `fileserver_download_handler` (Function) - Custom download handler function
|
||||||
|
- `max_retries` (Number) - Maximum retry attempts for fetching URL (default: `5`)
|
||||||
|
- `base_delay` (Number) - Initial delay for exponential backoff in ms (default: `100`)
|
||||||
|
- `max_delay` (Number) - Maximum delay for exponential backoff in ms (default: `5000`)
|
||||||
|
- `correlation_id` (String) - Optional correlation ID for tracing
|
||||||
|
|
||||||
|
**Output Format:**
|
||||||
|
- Returns a Promise that resolves to an object containing all envelope fields:
|
||||||
|
- `correlation_id`, `msg_id`, `timestamp`, `send_to`, `msg_purpose`, `sender_name`, `sender_id`, `receiver_name`, `receiver_id`, `reply_to`, `reply_to_msg_id`, `broker_url`
|
||||||
|
- `metadata` - Message-level metadata dictionary
|
||||||
|
- `payloads` - List of tuples, each containing `(dataname, data, type)` with deserialized payload data
|
||||||
|
|
||||||
**Process Flow:**
|
**Process Flow:**
|
||||||
1. Parse the JSON envelope to extract the `payloads` array
|
1. Parse the JSON envelope to extract all fields
|
||||||
2. Iterate through each payload in `payloads`
|
2. Iterate through each payload in `payloads` array
|
||||||
3. For each payload:
|
3. For each payload:
|
||||||
- Determine transport type (`direct` or `link`)
|
- Determine transport type (`direct` or `link`)
|
||||||
- If `direct`: decode Base64 data from the message
|
- If `direct`: Base64 decode the data from the message
|
||||||
- If `link`: fetch data from URL using exponential backoff
|
- If `link`: Fetch data from URL using exponential backoff (via `fileserver_download_handler`)
|
||||||
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
|
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
|
||||||
4. Return list of `(dataname, data, type)` tuples
|
4. Return envelope object with `payloads` field containing list of `(dataname, data, type)` tuples
|
||||||
|
|
||||||
|
**Note:** The `fileserver_download_handler` receives `(url, max_retries, base_delay, max_delay, correlation_id)` and returns `ArrayBuffer` or `Uint8Array`.
|
||||||
|
|
||||||
|
### Python/Micropython Implementation
|
||||||
|
|
||||||
|
#### Dependencies
|
||||||
|
- `nats-python` - Core NATS functionality
|
||||||
|
- `pyarrow` - Arrow IPC serialization
|
||||||
|
- `uuid` - Correlation ID and message ID generation
|
||||||
|
- `base64` - Base64 encoding/decoding
|
||||||
|
- `requests` or `aiohttp` - HTTP client for file server
|
||||||
|
|
||||||
|
#### smartsend Function
|
||||||
|
|
||||||
|
```python
|
||||||
|
def smartsend(
|
||||||
|
subject: str,
|
||||||
|
data: List[Tuple[str, Any, str]], # List of (dataname, data, type) tuples
|
||||||
|
broker_url: str = DEFAULT_BROKER_URL,
|
||||||
|
fileserver_url: str = DEFAULT_FILESERVER_URL,
|
||||||
|
fileserver_upload_handler: Callable = plik_oneshot_upload,
|
||||||
|
size_threshold: int = DEFAULT_SIZE_THRESHOLD,
|
||||||
|
correlation_id: Union[str, None] = None,
|
||||||
|
msg_purpose: str = "chat",
|
||||||
|
sender_name: str = "NATSBridge",
|
||||||
|
receiver_name: str = "",
|
||||||
|
receiver_id: str = "",
|
||||||
|
reply_to: str = "",
|
||||||
|
reply_to_msg_id: str = "",
|
||||||
|
is_publish: bool = True
|
||||||
|
) -> Tuple[MessageEnvelope, str]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `broker_url` (str) - NATS server URL (default: `"nats://localhost:4222"`)
|
||||||
|
- `fileserver_url` (str) - Base URL of the file server (default: `"http://localhost:8080"`)
|
||||||
|
- `size_threshold` (int) - Threshold in bytes for transport selection (default: `1048576` = 1MB)
|
||||||
|
- `correlation_id` (str) - Optional correlation ID for tracing (auto-generated if None)
|
||||||
|
- `msg_purpose` (str) - Purpose of the message (default: `"chat"`)
|
||||||
|
- `sender_name` (str) - Sender name (default: `"NATSBridge"`)
|
||||||
|
- `receiver_name` (str) - Message receiver name (default: `""`)
|
||||||
|
- `receiver_id` (str) - Message receiver ID (default: `""`)
|
||||||
|
- `reply_to` (str) - Topic to reply to (default: `""`)
|
||||||
|
- `reply_to_msg_id` (str) - Message ID this message is replying to (default: `""`)
|
||||||
|
- `is_publish` (bool) - Whether to automatically publish the message to NATS (default: `True`)
|
||||||
|
- `fileserver_upload_handler` (Callable) - Custom upload handler function
|
||||||
|
|
||||||
|
**Note:** Python uses `is_publish` parameter (instead of `NATS_connection` keyword) to control automatic publishing behavior. Connection reuse can be achieved by creating a NATS client outside the function and reusing it in a custom `fileserver_upload_handler` or custom publish implementation.
|
||||||
|
|
||||||
|
**Return Value:**
|
||||||
|
- Returns a tuple `(env, env_json_str)` where:
|
||||||
|
- `env` - The envelope dictionary containing all metadata and payloads
|
||||||
|
- `env_json_str` - JSON string representation of the envelope for publishing
|
||||||
|
|
||||||
|
**Input Format:**
|
||||||
|
- `data` - **Must be a list of (dataname, data, type) tuples**: `[(dataname1, data1, "type1"), (dataname2, data2, "type2"), ...]`
|
||||||
|
- Even for single payloads: `[(dataname1, data1, "type1")]`
|
||||||
|
- Each payload can have a different type, enabling mixed-content messages
|
||||||
|
- Supported types: `"text"`, `"dictionary"`, `"table"`, `"image"`, `"audio"`, `"video"`, `"binary"`
|
||||||
|
|
||||||
|
**Flow:**
|
||||||
|
1. Generate correlation ID and message ID if not provided
|
||||||
|
2. Iterate through the list of `(dataname, data, type)` tuples
|
||||||
|
3. For each payload:
|
||||||
|
- Serialize based on payload type
|
||||||
|
- Check payload size
|
||||||
|
- If < threshold: Base64 encode and include in envelope
|
||||||
|
- If >= threshold: Upload to HTTP server, store URL in envelope
|
||||||
|
4. Publish the JSON envelope to NATS
|
||||||
|
5. Return envelope dictionary and JSON string
|
||||||
|
|
||||||
|
#### smartreceive Handler
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def smartreceive(
|
||||||
|
msg: NATS.Message,
|
||||||
|
options: Dict = {}
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `fileserver_download_handler` (Callable) - Custom download handler function
|
||||||
|
- `max_retries` (int) - Maximum retry attempts for fetching URL (default: `5`)
|
||||||
|
- `base_delay` (int) - Initial delay for exponential backoff in ms (default: `100`)
|
||||||
|
- `max_delay` (int) - Maximum delay for exponential backoff in ms (default: `5000`)
|
||||||
|
- `correlation_id` (str) - Optional correlation ID for tracing
|
||||||
|
|
||||||
|
**Output Format:**
|
||||||
|
- Returns a JSON object (dictionary) containing all envelope fields:
|
||||||
|
- `correlation_id`, `msg_id`, `timestamp`, `send_to`, `msg_purpose`, `sender_name`, `sender_id`, `receiver_name`, `receiver_id`, `reply_to`, `reply_to_msg_id`, `broker_url`
|
||||||
|
- `metadata` - Message-level metadata dictionary
|
||||||
|
- `payloads` - List of tuples, each containing `(dataname, data, payload_type)` with deserialized payload data
|
||||||
|
|
||||||
|
**Process Flow:**
|
||||||
|
1. Parse the JSON envelope to extract all fields
|
||||||
|
2. Iterate through each payload in `payloads` list
|
||||||
|
3. For each payload:
|
||||||
|
- Determine transport type (`direct` or `link`)
|
||||||
|
- If `direct`: Base64 decode the data from the message
|
||||||
|
- If `link`: Fetch data from URL using exponential backoff (via `fileserver_download_handler`)
|
||||||
|
- Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.)
|
||||||
|
4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples
|
||||||
|
|
||||||
|
**Note:** The `fileserver_download_handler` receives `(url: str, max_retries: int, base_delay: int, max_delay: int, correlation_id: str)` and returns `bytes`.
|
||||||
|
|
||||||
## Scenario Implementations
|
## Scenario Implementations
|
||||||
|
|
||||||
### Scenario 1: Command & Control (Small Dictionary)
|
### Scenario 1: Command & Control (Small Dictionary)
|
||||||
|
|
||||||
**Julia (Receiver):**
|
**Julia (Sender/Receiver):**
|
||||||
```julia
|
```julia
|
||||||
# Subscribe to control subject
|
# Subscribe to control subject
|
||||||
# Parse JSON envelope
|
# Parse JSON envelope
|
||||||
@@ -423,15 +746,21 @@ async function smartreceive(msg, options = {})
|
|||||||
# Send acknowledgment
|
# Send acknowledgment
|
||||||
```
|
```
|
||||||
|
|
||||||
**JavaScript (Sender):**
|
**JavaScript (Sender/Receiver):**
|
||||||
```javascript
|
```javascript
|
||||||
// Create small dictionary config
|
// Create small dictionary config
|
||||||
// Send via smartsend with type="dictionary"
|
// Send via smartsend with type="dictionary"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Python/Micropython (Sender/Receiver):**
|
||||||
|
```python
|
||||||
|
# Create small dictionary config
|
||||||
|
# Send via smartsend with type="dictionary"
|
||||||
|
```
|
||||||
|
|
||||||
### Scenario 2: Deep Dive Analysis (Large Arrow Table)
|
### Scenario 2: Deep Dive Analysis (Large Arrow Table)
|
||||||
|
|
||||||
**Julia (Sender):**
|
**Julia (Sender/Receiver):**
|
||||||
```julia
|
```julia
|
||||||
# Create large DataFrame
|
# Create large DataFrame
|
||||||
# Convert to Arrow IPC stream
|
# Convert to Arrow IPC stream
|
||||||
@@ -440,7 +769,7 @@ async function smartreceive(msg, options = {})
|
|||||||
# Publish NATS with URL
|
# Publish NATS with URL
|
||||||
```
|
```
|
||||||
|
|
||||||
**JavaScript (Receiver):**
|
**JavaScript (Sender/Receiver):**
|
||||||
```javascript
|
```javascript
|
||||||
// Receive NATS message with URL
|
// Receive NATS message with URL
|
||||||
// Fetch data from HTTP server
|
// Fetch data from HTTP server
|
||||||
@@ -448,42 +777,64 @@ async function smartreceive(msg, options = {})
|
|||||||
// Load into Perspective.js or D3
|
// Load into Perspective.js or D3
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Python/Micropython (Sender/Receiver):**
|
||||||
|
```python
|
||||||
|
# Create large DataFrame
|
||||||
|
# Convert to Arrow IPC stream
|
||||||
|
# Check size (> 1MB)
|
||||||
|
# Upload to HTTP server
|
||||||
|
# Publish NATS with URL
|
||||||
|
```
|
||||||
|
|
||||||
### Scenario 3: Live Audio Processing
|
### Scenario 3: Live Audio Processing
|
||||||
|
|
||||||
**JavaScript (Sender):**
|
**JavaScript (Sender/Receiver):**
|
||||||
```javascript
|
```javascript
|
||||||
// Capture audio chunk
|
// Capture audio chunk
|
||||||
// Send as binary with metadata headers
|
// Send as binary with metadata headers
|
||||||
// Use smartsend with type="audio"
|
// Use smartsend with type="audio"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Julia (Receiver):**
|
**Julia (Sender/Receiver):**
|
||||||
```julia
|
```julia
|
||||||
// Receive audio data
|
# Receive audio data
|
||||||
// Perform FFT or AI transcription
|
# Perform FFT or AI transcription
|
||||||
// Send results back (JSON + Arrow table)
|
# Send results back (JSON + Arrow table)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Python/Micropython (Sender/Receiver):**
|
||||||
|
```python
|
||||||
|
# Capture audio chunk
|
||||||
|
# Send as binary with metadata headers
|
||||||
|
# Use smartsend with type="audio"
|
||||||
```
|
```
|
||||||
|
|
||||||
### Scenario 4: Catch-Up (JetStream)
|
### Scenario 4: Catch-Up (JetStream)
|
||||||
|
|
||||||
**Julia (Producer):**
|
**Julia (Producer/Consumer):**
|
||||||
```julia
|
```julia
|
||||||
# Publish to JetStream
|
# Publish to JetStream
|
||||||
# Include metadata for temporal tracking
|
# Include metadata for temporal tracking
|
||||||
```
|
```
|
||||||
|
|
||||||
**JavaScript (Consumer):**
|
**JavaScript (Producer/Consumer):**
|
||||||
```javascript
|
```javascript
|
||||||
// Connect to JetStream
|
// Connect to JetStream
|
||||||
// Request replay from last 10 minutes
|
// Request replay from last 10 minutes
|
||||||
// Process historical and real-time messages
|
// Process historical and real-time messages
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Python/Micropython (Producer/Consumer):**
|
||||||
|
```python
|
||||||
|
# Publish to JetStream
|
||||||
|
# Include metadata for temporal tracking
|
||||||
|
```
|
||||||
|
|
||||||
### Scenario 5: Selection (Low Bandwidth)
|
### Scenario 5: Selection (Low Bandwidth)
|
||||||
|
|
||||||
**Focus:** Small Arrow tables, Julia to JavaScript. The Action: Julia wants to send a small DataFrame to show on a JavaScript dashboard for the user to choose.
|
**Focus:** Small Arrow tables, cross-platform communication. The Action: Any platform wants to send a small DataFrame to show on any receiving application for the user to choose.
|
||||||
|
|
||||||
**Julia (Sender):**
|
**Julia (Sender/Receiver):**
|
||||||
```julia
|
```julia
|
||||||
# Create small DataFrame (e.g., 50KB - 500KB)
|
# Create small DataFrame (e.g., 50KB - 500KB)
|
||||||
# Convert to Arrow IPC stream
|
# Convert to Arrow IPC stream
|
||||||
@@ -492,7 +843,7 @@ async function smartreceive(msg, options = {})
|
|||||||
# Include metadata for dashboard selection context
|
# Include metadata for dashboard selection context
|
||||||
```
|
```
|
||||||
|
|
||||||
**JavaScript (Receiver):**
|
**JavaScript (Sender/Receiver):**
|
||||||
```javascript
|
```javascript
|
||||||
// Receive NATS message with direct transport
|
// Receive NATS message with direct transport
|
||||||
// Decode Base64 payload
|
// Decode Base64 payload
|
||||||
@@ -502,11 +853,20 @@ async function smartreceive(msg, options = {})
|
|||||||
// Send selection back to Julia
|
// Send selection back to Julia
|
||||||
```
|
```
|
||||||
|
|
||||||
**Use Case:** Julia server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to JavaScript dashboard for user selection. The selection is then sent back to Julia for processing.
|
**Python/Micropython (Sender/Receiver):**
|
||||||
|
```python
|
||||||
|
# Create small DataFrame (e.g., 50KB - 500KB)
|
||||||
|
# Convert to Arrow IPC stream
|
||||||
|
# Check payload size (< 1MB threshold)
|
||||||
|
# Publish directly to NATS with Base64-encoded payload
|
||||||
|
# Include metadata for dashboard selection context
|
||||||
|
```
|
||||||
|
|
||||||
|
**Use Case:** Any server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to any receiving application for user selection. The selection is then sent back to the sender for processing.
|
||||||
|
|
||||||
### Scenario 6: Chat System
|
### Scenario 6: Chat System
|
||||||
|
|
||||||
**Focus:** Every conversational message is composed of any number and any combination of components, spanning the full spectrum from small to large. This includes text, images, audio, video, tables, and files—specifically accommodating everything from brief snippets to high-resolution images, large audio files, extensive tables, and massive documents. Support for claim-check delivery and full bi-directional messaging.
|
**Focus:** Every conversational message is composed of any number and any combination of components, spanning the full spectrum from small to large. This includes text, images, audio, video, tables, and files—specifically accommodating everything from brief snippets to high-resolution images, large audio files, extensive tables, and massive documents. Support for claim-check delivery and full bi-directional messaging across all platforms.
|
||||||
|
|
||||||
**Multi-Payload Support:** The system supports mixed-payload messages where a single message can contain multiple payloads with different transport strategies. The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type.
|
**Multi-Payload Support:** The system supports mixed-payload messages where a single message can contain multiple payloads with different transport strategies. The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type.
|
||||||
|
|
||||||
@@ -545,9 +905,27 @@ async function smartreceive(msg, options = {})
|
|||||||
// Support bidirectional reply with claim-check delivery confirmation
|
// Support bidirectional reply with claim-check delivery confirmation
|
||||||
```
|
```
|
||||||
|
|
||||||
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components.
|
**Python/Micropython (Sender/Receiver):**
|
||||||
|
```python
|
||||||
|
# Build chat message with mixed payloads:
|
||||||
|
# - Text: direct transport (Base64)
|
||||||
|
# - Small images: direct transport (Base64)
|
||||||
|
# - Large images: link transport (HTTP URL)
|
||||||
|
# - Audio/video: link transport (HTTP URL)
|
||||||
|
# - Tables: direct or link depending on size
|
||||||
|
# - Files: link transport (HTTP URL)
|
||||||
|
#
|
||||||
|
# Each payload uses appropriate transport strategy:
|
||||||
|
# - Size < 1MB → direct (NATS + Base64)
|
||||||
|
# - Size >= 1MB → link (HTTP upload + NATS URL)
|
||||||
|
#
|
||||||
|
# Include claim-check metadata for delivery tracking
|
||||||
|
# Support bidirectional messaging with replyTo fields
|
||||||
|
```
|
||||||
|
|
||||||
**Implementation Note:** The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. See the standard API format in Section 1: `msgEnvelope_v1` supports `AbstractArray{msgPayload_v1}` for multiple payloads.
|
**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components across all platforms.
|
||||||
|
|
||||||
|
**Implementation Note:** The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. See the standard API format in Section 1: `msg_envelope_v1` supports `Vector{msg_payload_v1}` for multiple payloads.
|
||||||
|
|
||||||
## Performance Considerations
|
## Performance Considerations
|
||||||
|
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
24
etc.jl
24
etc.jl
@@ -1,21 +1,9 @@
|
|||||||
Check architecture.jl, NATSBridge.jl and its test files:
|
Task: Update README.md to reflect recent changes in NATSbridge package.
|
||||||
- test_julia_to_julia_table_receiver.jl
|
|
||||||
- test_julia_to_julia_table_sender.jl.
|
|
||||||
|
|
||||||
Now I want to test sending a mix-content message from Julia serviceA to Julia serviceB, for example, a chat system.
|
|
||||||
The test message must show that any combination and any number and any data size of text | json | table | image | audio | video | binary can be send and receive.
|
|
||||||
|
|
||||||
Can you write me the following test files:
|
|
||||||
- test_julia_to_julia_mix_receiver.jl
|
|
||||||
- test_julia_to_julia_mix_sender.jl
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
1. create a tutorial file "tutorial_julia.md" for NATSBridge.jl
|
|
||||||
2. create a walkthrough file "walkthrough_julia.md" for NATSBridge.jl
|
|
||||||
|
|
||||||
You may consult architecture.md for more info.
|
|
||||||
|
|
||||||
|
Context: the package has been updated with the NATS_connection keyword and the publish_message function.
|
||||||
|
|
||||||
|
Requirements:
|
||||||
|
|
||||||
|
Source of Truth: Treat the updated NATSbridge code as the definitive source. Update README.md to align exactly with these changes.
|
||||||
|
API Consistency: Ensure the Main Package API (e.g., smartsend(), publish_message()) uses consistent naming across all three supported languages.
|
||||||
|
Ecosystem Variance: Low-level native functions (e.g., NATS.connect(), JSON.read()) should follow the conventions of the specific language ecosystem and do not require cross-language consistency.
|
||||||
@@ -1,221 +0,0 @@
|
|||||||
"""
|
|
||||||
Micropython NATS Bridge - Simple Example
|
|
||||||
|
|
||||||
This example demonstrates the basic usage of the NATSBridge for Micropython.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, "../src")
|
|
||||||
|
|
||||||
from nats_bridge import smartsend, smartreceive, log_trace
|
|
||||||
import json
|
|
||||||
|
|
||||||
|
|
||||||
def example_simple_chat():
|
|
||||||
"""
|
|
||||||
Simple chat example: Send text messages via NATS.
|
|
||||||
|
|
||||||
Sender (this script):
|
|
||||||
- Sends a text message to NATS
|
|
||||||
- Uses direct transport (no fileserver needed)
|
|
||||||
|
|
||||||
Receiver (separate script):
|
|
||||||
- Listens to NATS
|
|
||||||
- Receives and processes the message
|
|
||||||
"""
|
|
||||||
print("=== Simple Chat Example ===")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Define the message data as list of (dataname, data, type) tuples
|
|
||||||
data = [
|
|
||||||
("message", "Hello from Micropython!", "text")
|
|
||||||
]
|
|
||||||
|
|
||||||
# Send the message
|
|
||||||
env = smartsend(
|
|
||||||
"/chat/room1",
|
|
||||||
data,
|
|
||||||
nats_url="nats://localhost:4222",
|
|
||||||
msg_purpose="chat",
|
|
||||||
sender_name="micropython-client"
|
|
||||||
)
|
|
||||||
|
|
||||||
print("Message sent!")
|
|
||||||
print(" Subject: {}".format(env.send_to))
|
|
||||||
print(" Correlation ID: {}".format(env.correlation_id))
|
|
||||||
print(" Payloads: {}".format(len(env.payloads)))
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Expected receiver output:
|
|
||||||
print("Expected receiver output:")
|
|
||||||
print(" [timestamp] [Correlation: ...] Starting smartsend for subject: /chat/room1")
|
|
||||||
print(" [timestamp] [Correlation: ...] Serialized payload 'message' (type: text) size: 22 bytes")
|
|
||||||
print(" [timestamp] [Correlation: ...] Using direct transport for 22 bytes")
|
|
||||||
print(" [timestamp] [Correlation: ...] Message published to /chat/room1")
|
|
||||||
print()
|
|
||||||
|
|
||||||
return env
|
|
||||||
|
|
||||||
|
|
||||||
def example_send_json():
|
|
||||||
"""
|
|
||||||
Example: Send JSON configuration to a Micropython device.
|
|
||||||
|
|
||||||
This demonstrates sending structured data (dictionary type).
|
|
||||||
"""
|
|
||||||
print("\n=== Send JSON Configuration ===")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Define configuration as dictionary
|
|
||||||
config = {
|
|
||||||
"wifi_ssid": "MyNetwork",
|
|
||||||
"wifi_password": "password123",
|
|
||||||
"server_host": "mqtt.example.com",
|
|
||||||
"server_port": 1883,
|
|
||||||
"update_interval": 60
|
|
||||||
}
|
|
||||||
|
|
||||||
# Send configuration
|
|
||||||
data = [
|
|
||||||
("device_config", config, "dictionary")
|
|
||||||
]
|
|
||||||
|
|
||||||
env = smartsend(
|
|
||||||
"/device/config",
|
|
||||||
data,
|
|
||||||
nats_url="nats://localhost:4222",
|
|
||||||
msg_purpose="updateStatus",
|
|
||||||
sender_name="server"
|
|
||||||
)
|
|
||||||
|
|
||||||
print("Configuration sent!")
|
|
||||||
print(" Subject: {}".format(env.send_to))
|
|
||||||
print(" Payloads: {}".format(len(env.payloads)))
|
|
||||||
print()
|
|
||||||
|
|
||||||
return env
|
|
||||||
|
|
||||||
|
|
||||||
def example_receive_message(msg):
|
|
||||||
"""
|
|
||||||
Example: Receive and process a NATS message.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
msg: The NATS message received (should be dict or JSON string)
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
list: List of (dataname, data, type) tuples
|
|
||||||
"""
|
|
||||||
print("\n=== Receive Message ===")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Process the message
|
|
||||||
payloads = smartreceive(
|
|
||||||
msg,
|
|
||||||
fileserver_download_handler=None, # Not needed for direct transport
|
|
||||||
max_retries=3,
|
|
||||||
base_delay=100,
|
|
||||||
max_delay=1000
|
|
||||||
)
|
|
||||||
|
|
||||||
print("Received {} payload(s):".format(len(payloads)))
|
|
||||||
for dataname, data, type in payloads:
|
|
||||||
print(" - {}: {} (type: {})".format(dataname, data, type))
|
|
||||||
|
|
||||||
return payloads
|
|
||||||
|
|
||||||
|
|
||||||
def example_mixed_content():
|
|
||||||
"""
|
|
||||||
Example: Send mixed content (text + dictionary + binary).
|
|
||||||
|
|
||||||
This demonstrates the multi-payload capability.
|
|
||||||
"""
|
|
||||||
print("\n=== Mixed Content Example ===")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Create mixed content
|
|
||||||
image_data = b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR" # Example PNG header
|
|
||||||
|
|
||||||
data = [
|
|
||||||
("message_text", "Hello with image!", "text"),
|
|
||||||
("user_config", {"theme": "dark", "notifications": True}, "dictionary"),
|
|
||||||
("user_avatar", image_data, "binary")
|
|
||||||
]
|
|
||||||
|
|
||||||
env = smartsend(
|
|
||||||
"/chat/mixed",
|
|
||||||
data,
|
|
||||||
nats_url="nats://localhost:4222",
|
|
||||||
msg_purpose="chat",
|
|
||||||
sender_name="micropython-client"
|
|
||||||
)
|
|
||||||
|
|
||||||
print("Mixed content sent!")
|
|
||||||
print(" Subject: {}".format(env.send_to))
|
|
||||||
print(" Payloads:")
|
|
||||||
for p in env.payloads:
|
|
||||||
print(" - {} (transport: {}, type: {}, size: {} bytes)".format(
|
|
||||||
p.dataname, p.transport, p.type, p.size))
|
|
||||||
|
|
||||||
return env
|
|
||||||
|
|
||||||
|
|
||||||
def example_reply():
|
|
||||||
"""
|
|
||||||
Example: Send a message with reply-to functionality.
|
|
||||||
|
|
||||||
This demonstrates request-response pattern.
|
|
||||||
"""
|
|
||||||
print("\n=== Request-Response Example ===")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Send command
|
|
||||||
data = [
|
|
||||||
("command", {"action": "read_sensor", "sensor_id": "temp1"}, "dictionary")
|
|
||||||
]
|
|
||||||
|
|
||||||
env = smartsend(
|
|
||||||
"/device/command",
|
|
||||||
data,
|
|
||||||
nats_url="nats://localhost:4222",
|
|
||||||
msg_purpose="command",
|
|
||||||
sender_name="server",
|
|
||||||
reply_to="/device/response",
|
|
||||||
reply_to_msg_id="cmd-001"
|
|
||||||
)
|
|
||||||
|
|
||||||
print("Command sent!")
|
|
||||||
print(" Subject: {}".format(env.send_to))
|
|
||||||
print(" Reply To: {}".format(env.reply_to))
|
|
||||||
print(" Reply To Msg ID: {}".format(env.reply_to_msg_id))
|
|
||||||
print()
|
|
||||||
|
|
||||||
print("Expected receiver behavior:")
|
|
||||||
print(" 1. Receive command on /device/command")
|
|
||||||
print(" 2. Process command")
|
|
||||||
print(" 3. Send response to /device/response")
|
|
||||||
print(" 4. Include replyToMsgId in response")
|
|
||||||
|
|
||||||
return env
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
print("Micropython NATS Bridge Examples")
|
|
||||||
print("================================")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Run examples
|
|
||||||
example_simple_chat()
|
|
||||||
example_send_json()
|
|
||||||
example_mixed_content()
|
|
||||||
example_reply()
|
|
||||||
|
|
||||||
print("\n=== Examples Completed ===")
|
|
||||||
print()
|
|
||||||
print("To run these examples, you need:")
|
|
||||||
print(" 1. A running NATS server at nats://localhost:4222")
|
|
||||||
print(" 2. Import the nats_bridge module")
|
|
||||||
print(" 3. Call the desired example function")
|
|
||||||
print()
|
|
||||||
print("For more examples, see test/test_micropython_basic.py")
|
|
||||||
622
examples/tutorial.md
Normal file
622
examples/tutorial.md
Normal file
@@ -0,0 +1,622 @@
|
|||||||
|
# NATSBridge Tutorial
|
||||||
|
|
||||||
|
A step-by-step guide to get started with NATSBridge - a high-performance, bi-directional data bridge for **Julia**, **JavaScript**, and **Python/Micropython**.
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
1. [Overview](#overview)
|
||||||
|
2. [Prerequisites](#prerequisites)
|
||||||
|
3. [Installation](#installation)
|
||||||
|
4. [Quick Start](#quick-start)
|
||||||
|
5. [Basic Examples](#basic-examples)
|
||||||
|
6. [Advanced Usage](#advanced-usage)
|
||||||
|
7. [Cross-Platform Communication](#cross-platform-communication)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
NATSBridge enables seamless communication between Julia, JavaScript, and Python/Micropython applications through NATS, with automatic transport selection based on payload size:
|
||||||
|
|
||||||
|
- **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded)
|
||||||
|
- **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL
|
||||||
|
|
||||||
|
### Supported Payload Types
|
||||||
|
|
||||||
|
| Type | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `text` | Plain text strings |
|
||||||
|
| `dictionary` | JSON-serializable dictionaries |
|
||||||
|
| `table` | Tabular data (Arrow IPC format) |
|
||||||
|
| `image` | Image data (PNG, JPG bytes) |
|
||||||
|
| `audio` | Audio data (WAV, MP3 bytes) |
|
||||||
|
| `video` | Video data (MP4, AVI bytes) |
|
||||||
|
| `binary` | Generic binary data |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
Before you begin, ensure you have:
|
||||||
|
|
||||||
|
1. **NATS Server** running (or accessible)
|
||||||
|
2. **HTTP File Server** (optional, for large payloads > 1MB)
|
||||||
|
3. **One of the supported platforms**: Julia, JavaScript (Node.js), or Python/Micropython
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using Pkg
|
||||||
|
Pkg.add("NATS")
|
||||||
|
Pkg.add("Arrow")
|
||||||
|
Pkg.add("JSON3")
|
||||||
|
Pkg.add("HTTP")
|
||||||
|
Pkg.add("UUIDs")
|
||||||
|
Pkg.add("Dates")
|
||||||
|
```
|
||||||
|
|
||||||
|
### JavaScript
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install nats.js apache-arrow uuid base64-url
|
||||||
|
```
|
||||||
|
|
||||||
|
### Python/Micropython
|
||||||
|
|
||||||
|
1. Copy `src/nats_bridge.py` to your device
|
||||||
|
2. Install dependencies:
|
||||||
|
|
||||||
|
**For Python (desktop):**
|
||||||
|
```bash
|
||||||
|
pip install nats-py
|
||||||
|
```
|
||||||
|
|
||||||
|
**For Micropython:**
|
||||||
|
- `urequests` for HTTP requests
|
||||||
|
- `base64` for base64 encoding (built-in)
|
||||||
|
- `json` for JSON handling (built-in)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Step 1: Start NATS Server
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run -p 4222:4222 nats:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Start HTTP File Server (Optional)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create a directory for file uploads
|
||||||
|
mkdir -p /tmp/fileserver
|
||||||
|
|
||||||
|
# Use Python's built-in server
|
||||||
|
python3 -m http.server 8080 --directory /tmp/fileserver
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Send Your First Message
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
|
# Send a text message (is_publish=True by default)
|
||||||
|
data = [("message", "Hello World", "text")]
|
||||||
|
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222")
|
||||||
|
print("Message sent!")
|
||||||
|
|
||||||
|
# Or use is_publish=False to get envelope and JSON without publishing
|
||||||
|
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222", is_publish=False)
|
||||||
|
# env: MessageEnvelope object
|
||||||
|
# env_json_str: JSON string for publishing to NATS
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
// Send a text message (isPublish=true by default)
|
||||||
|
await smartsend("/chat/room1", [
|
||||||
|
{ dataname: "message", data: "Hello World", type: "text" }
|
||||||
|
], { brokerUrl: "nats://localhost:4222" });
|
||||||
|
|
||||||
|
console.log("Message sent!");
|
||||||
|
|
||||||
|
// Or use isPublish=false to get envelope and JSON without publishing
|
||||||
|
const { env, env_json_str } = await smartsend("/chat/room1", [
|
||||||
|
{ dataname: "message", data: "Hello World", type: "text" }
|
||||||
|
], { brokerUrl: "nats://localhost:4222", isPublish: false });
|
||||||
|
// env: MessageEnvelope object
|
||||||
|
// env_json_str: JSON string for publishing to NATS
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
# Send a text message
|
||||||
|
data = [("message", "Hello World", "text")]
|
||||||
|
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222")
|
||||||
|
# env: msg_envelope_v1 object with all metadata and payloads
|
||||||
|
# env_json_str: JSON string representation of the envelope for publishing
|
||||||
|
println("Message sent!")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 4: Receive Messages
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartreceive
|
||||||
|
|
||||||
|
# Receive and process message
|
||||||
|
env = smartreceive(msg)
|
||||||
|
for dataname, data, type in env["payloads"]:
|
||||||
|
print(f"Received {dataname}: {data}")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const { smartreceive } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
// Receive and process message
|
||||||
|
const env = await smartreceive(msg);
|
||||||
|
for (const payload of env.payloads) {
|
||||||
|
console.log(`Received ${payload.dataname}: ${payload.data}`);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
# Receive and process message
|
||||||
|
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff)
|
||||||
|
for (dataname, data, type) in env["payloads"]
|
||||||
|
println("Received $dataname: $data")
|
||||||
|
end
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Basic Examples
|
||||||
|
|
||||||
|
### Example 1: Sending a Dictionary
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
|
# Create configuration dictionary
|
||||||
|
config = {
|
||||||
|
"wifi_ssid": "MyNetwork",
|
||||||
|
"wifi_password": "password123",
|
||||||
|
"update_interval": 60
|
||||||
|
}
|
||||||
|
|
||||||
|
# Send as dictionary type
|
||||||
|
data = [("config", config, "dictionary")]
|
||||||
|
env, env_json_str = smartsend("/device/config", data, broker_url="nats://localhost:4222")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
const config = {
|
||||||
|
wifi_ssid: "MyNetwork",
|
||||||
|
wifi_password: "password123",
|
||||||
|
update_interval: 60
|
||||||
|
};
|
||||||
|
|
||||||
|
const { env, env_json_str } = await smartsend("/device/config", [
|
||||||
|
{ dataname: "config", data: config, type: "dictionary" }
|
||||||
|
], { brokerUrl: "nats://localhost:4222" });
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
config = Dict(
|
||||||
|
"wifi_ssid" => "MyNetwork",
|
||||||
|
"wifi_password" => "password123",
|
||||||
|
"update_interval" => 60
|
||||||
|
)
|
||||||
|
|
||||||
|
data = [("config", config, "dictionary")]
|
||||||
|
env, env_json_str = smartsend("/device/config", data, broker_url="nats://localhost:4222")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example 2: Sending Binary Data (Image)
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
|
# Read image file
|
||||||
|
with open("image.png", "rb") as f:
|
||||||
|
image_data = f.read()
|
||||||
|
|
||||||
|
# Send as binary type
|
||||||
|
data = [("user_image", image_data, "binary")]
|
||||||
|
env, env_json_str = smartsend("/chat/image", data, broker_url="nats://localhost:4222")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
// Read image file (Node.js)
|
||||||
|
const fs = require('fs');
|
||||||
|
const image_data = fs.readFileSync('image.png');
|
||||||
|
|
||||||
|
const { env, env_json_str } = await smartsend("/chat/image", [
|
||||||
|
{ dataname: "user_image", data: image_data, type: "binary" }
|
||||||
|
], { brokerUrl: "nats://localhost:4222" });
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
# Read image file
|
||||||
|
image_data = read("image.png")
|
||||||
|
|
||||||
|
data = [("user_image", image_data, "binary")]
|
||||||
|
env, env_json_str = smartsend("/chat/image", data, broker_url="nats://localhost:4222")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example 3: Request-Response Pattern
|
||||||
|
|
||||||
|
#### Python/Micropython (Requester)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
|
# Send command with reply-to
|
||||||
|
data = [("command", {"action": "read_sensor"}, "dictionary")]
|
||||||
|
env, env_json_str = smartsend(
|
||||||
|
"/device/command",
|
||||||
|
data,
|
||||||
|
broker_url="nats://localhost:4222",
|
||||||
|
reply_to="/device/response",
|
||||||
|
reply_to_msg_id="cmd-001"
|
||||||
|
)
|
||||||
|
# env: MessageEnvelope object
|
||||||
|
# env_json_str: JSON string for publishing to NATS
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript (Responder)
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const { smartreceive, smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
// Subscribe to command topic
|
||||||
|
const sub = nc.subscribe("/device/command");
|
||||||
|
|
||||||
|
for await (const msg of sub) {
|
||||||
|
const env = await smartreceive(msg);
|
||||||
|
|
||||||
|
// Process command
|
||||||
|
for (const payload of env.payloads) {
|
||||||
|
if (payload.dataname === "command") {
|
||||||
|
const command = payload.data;
|
||||||
|
|
||||||
|
if (command.action === "read_sensor") {
|
||||||
|
// Read sensor and send response
|
||||||
|
const response = {
|
||||||
|
sensor_id: "sensor-001",
|
||||||
|
value: 42.5,
|
||||||
|
timestamp: new Date().toISOString()
|
||||||
|
};
|
||||||
|
|
||||||
|
await smartsend("/device/response", [
|
||||||
|
{ dataname: "sensor_data", data: response, type: "dictionary" }
|
||||||
|
], {
|
||||||
|
reply_to: env.replyTo,
|
||||||
|
reply_to_msg_id: env.msgId
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Advanced Usage
|
||||||
|
|
||||||
|
### Example 4: Large Payloads (File Server)
|
||||||
|
|
||||||
|
For payloads larger than 1MB, NATSBridge automatically uses the file server:
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
import os
|
||||||
|
|
||||||
|
# Create large data (> 1MB)
|
||||||
|
large_data = os.urandom(2_000_000) # 2MB of random data
|
||||||
|
|
||||||
|
# Send with file server URL
|
||||||
|
env, env_json_str = smartsend(
|
||||||
|
"/data/large",
|
||||||
|
[("large_file", large_data, "binary")],
|
||||||
|
broker_url="nats://localhost:4222",
|
||||||
|
fileserver_url="http://localhost:8080",
|
||||||
|
size_threshold=1_000_000
|
||||||
|
)
|
||||||
|
|
||||||
|
# The envelope will contain the download URL
|
||||||
|
print(f"File uploaded to: {env.payloads[0].data}")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
// Create large data (> 1MB)
|
||||||
|
const largeData = new ArrayBuffer(2_000_000);
|
||||||
|
const view = new Uint8Array(largeData);
|
||||||
|
view.fill(42); // Fill with some data
|
||||||
|
|
||||||
|
const { env, env_json_str } = await smartsend("/data/large", [
|
||||||
|
{ dataname: "large_file", data: largeData, type: "binary" }
|
||||||
|
], {
|
||||||
|
brokerUrl: "nats://localhost:4222",
|
||||||
|
fileserverUrl: "http://localhost:8080",
|
||||||
|
sizeThreshold: 1_000_000
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
# Create large data (> 1MB)
|
||||||
|
large_data = rand(UInt8, 2_000_000)
|
||||||
|
|
||||||
|
env, env_json_str = smartsend(
|
||||||
|
"/data/large",
|
||||||
|
[("large_file", large_data, "binary")],
|
||||||
|
broker_url="nats://localhost:4222",
|
||||||
|
fileserver_url="http://localhost:8080"
|
||||||
|
)
|
||||||
|
|
||||||
|
# The envelope will contain the download URL
|
||||||
|
println("File uploaded to: $(env.payloads[1].data)")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example 5: Mixed Content (Chat with Text + Image)
|
||||||
|
|
||||||
|
NATSBridge supports sending multiple payloads with different types in a single message:
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
|
# Read image file
|
||||||
|
with open("avatar.png", "rb") as f:
|
||||||
|
image_data = f.read()
|
||||||
|
|
||||||
|
# Send mixed content
|
||||||
|
data = [
|
||||||
|
("message_text", "Hello with image!", "text"),
|
||||||
|
("user_avatar", image_data, "image")
|
||||||
|
]
|
||||||
|
|
||||||
|
env, env_json_str = smartsend("/chat/mixed", data, broker_url="nats://localhost:4222")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
const fs = require('fs');
|
||||||
|
|
||||||
|
const { env, env_json_str } = await smartsend("/chat/mixed", [
|
||||||
|
{
|
||||||
|
dataname: "message_text",
|
||||||
|
data: "Hello with image!",
|
||||||
|
type: "text"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
dataname: "user_avatar",
|
||||||
|
data: fs.readFileSync("avatar.png"),
|
||||||
|
type: "image"
|
||||||
|
}
|
||||||
|
], { brokerUrl: "nats://localhost:4222" });
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
image_data = read("avatar.png")
|
||||||
|
|
||||||
|
data = [
|
||||||
|
("message_text", "Hello with image!", "text"),
|
||||||
|
("user_avatar", image_data, "image")
|
||||||
|
]
|
||||||
|
|
||||||
|
env, env_json_str = smartsend("/chat/mixed", data, broker_url="nats://localhost:4222")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example 6: Table Data (Arrow IPC)
|
||||||
|
|
||||||
|
For tabular data, NATSBridge uses Apache Arrow IPC format:
|
||||||
|
|
||||||
|
#### Python/Micropython
|
||||||
|
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
# Create DataFrame
|
||||||
|
df = pd.DataFrame({
|
||||||
|
"id": [1, 2, 3],
|
||||||
|
"name": ["Alice", "Bob", "Charlie"],
|
||||||
|
"score": [95, 88, 92]
|
||||||
|
})
|
||||||
|
|
||||||
|
# Send as table type
|
||||||
|
data = [("students", df, "table")]
|
||||||
|
env, env_json_str = smartsend("/data/students", data, broker_url="nats://localhost:4222")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
using DataFrames
|
||||||
|
|
||||||
|
# Create DataFrame
|
||||||
|
df = DataFrame(
|
||||||
|
id = [1, 2, 3],
|
||||||
|
name = ["Alice", "Bob", "Charlie"],
|
||||||
|
score = [95, 88, 92]
|
||||||
|
)
|
||||||
|
|
||||||
|
data = [("students", df, "table")]
|
||||||
|
env, env_json_str = smartsend("/data/students", data, broker_url="nats://localhost:4222")
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cross-Platform Communication
|
||||||
|
|
||||||
|
NATSBridge enables seamless communication between different platforms:
|
||||||
|
|
||||||
|
### Julia ↔ JavaScript
|
||||||
|
|
||||||
|
#### Julia Sender
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
# Send dictionary from Julia to JavaScript
|
||||||
|
config = Dict("step_size" => 0.01, "iterations" => 1000)
|
||||||
|
data = [("config", config, "dictionary")]
|
||||||
|
env, env_json_str = smartsend("/analysis/config", data, broker_url="nats://localhost:4222")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript Receiver
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const { smartreceive } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
// Receive dictionary from Julia
|
||||||
|
const env = await smartreceive(msg);
|
||||||
|
for (const payload of env.payloads) {
|
||||||
|
if (payload.type === "dictionary") {
|
||||||
|
console.log("Received config:", payload.data);
|
||||||
|
// payload.data = { step_size: 0.01, iterations: 1000 }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### JavaScript ↔ Python
|
||||||
|
|
||||||
|
#### JavaScript Sender
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const { smartsend } = require('./src/NATSBridge');
|
||||||
|
|
||||||
|
const { env, env_json_str } = await smartsend("/data/transfer", [
|
||||||
|
{ dataname: "message", data: "Hello from JS!", type: "text" }
|
||||||
|
], { brokerUrl: "nats://localhost:4222" });
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Python Receiver
|
||||||
|
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartreceive
|
||||||
|
|
||||||
|
env = smartreceive(msg)
|
||||||
|
for dataname, data, type in env["payloads"]:
|
||||||
|
if type == "text":
|
||||||
|
print(f"Received from JS: {data}")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Python ↔ Julia
|
||||||
|
|
||||||
|
#### Python Sender
|
||||||
|
|
||||||
|
```python
|
||||||
|
from nats_bridge import smartsend
|
||||||
|
|
||||||
|
data = [("message", "Hello from Python!", "text")]
|
||||||
|
env, env_json_str = smartsend("/chat/python", data, broker_url="nats://localhost:4222")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Julia Receiver
|
||||||
|
|
||||||
|
```julia
|
||||||
|
using NATSBridge
|
||||||
|
|
||||||
|
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff)
|
||||||
|
for (dataname, data, type) in env["payloads"]
|
||||||
|
if type == "text"
|
||||||
|
println("Received from Python: $data")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
1. **Explore the test directory** for more examples
|
||||||
|
2. **Check the documentation** for advanced configuration options
|
||||||
|
3. **Join the community** to share your use cases
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Connection Issues
|
||||||
|
|
||||||
|
- Ensure NATS server is running: `docker ps | grep nats`
|
||||||
|
- Check firewall settings
|
||||||
|
- Verify NATS URL configuration
|
||||||
|
|
||||||
|
### File Server Issues
|
||||||
|
|
||||||
|
- Ensure file server is running and accessible
|
||||||
|
- Check upload permissions
|
||||||
|
- Verify file server URL configuration
|
||||||
|
|
||||||
|
### Serialization Errors
|
||||||
|
|
||||||
|
- Verify data type matches the specified type
|
||||||
|
- Check that binary data is in the correct format (bytes/Vector{UInt8})
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT
|
||||||
1073
examples/walkthrough.md
Normal file
1073
examples/walkthrough.md
Normal file
File diff suppressed because it is too large
Load Diff
14
plik_fileserver/docker-compose.yml
Normal file
14
plik_fileserver/docker-compose.yml
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
services:
|
||||||
|
plik:
|
||||||
|
image: rootgg/plik:latest
|
||||||
|
container_name: plik-server
|
||||||
|
restart: unless-stopped
|
||||||
|
ports:
|
||||||
|
- "8080:8080"
|
||||||
|
volumes:
|
||||||
|
# # Mount the config file (created below)
|
||||||
|
# - ./plikd.cfg:/home/plik/server/plikd.cfg
|
||||||
|
# Mount local folder for uploads and database
|
||||||
|
- ./plik-data:/data
|
||||||
|
# Set user to match your host UID to avoid permission issues
|
||||||
|
user: "1000:1000"
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -98,6 +98,26 @@ function base64ToArrayBuffer(base64) {
|
|||||||
return bytes.buffer;
|
return bytes.buffer;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Helper: Convert Uint8Array to Base64 string
|
||||||
|
function uint8ArrayToBase64(uint8array) {
|
||||||
|
let binary = '';
|
||||||
|
for (let i = 0; i < uint8array.byteLength; i++) {
|
||||||
|
binary += String.fromCharCode(uint8array[i]);
|
||||||
|
}
|
||||||
|
return btoa(binary);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Helper: Convert Base64 string to Uint8Array
|
||||||
|
function base64ToUint8Array(base64) {
|
||||||
|
const binaryString = atob(base64);
|
||||||
|
const len = binaryString.length;
|
||||||
|
const bytes = new Uint8Array(len);
|
||||||
|
for (let i = 0; i < len; i++) {
|
||||||
|
bytes[i] = binaryString.charCodeAt(i);
|
||||||
|
}
|
||||||
|
return bytes;
|
||||||
|
}
|
||||||
|
|
||||||
// Helper: Serialize data based on type
|
// Helper: Serialize data based on type
|
||||||
function _serialize_data(data, type) {
|
function _serialize_data(data, type) {
|
||||||
/**
|
/**
|
||||||
@@ -114,39 +134,39 @@ function _serialize_data(data, type) {
|
|||||||
*/
|
*/
|
||||||
if (type === "text") {
|
if (type === "text") {
|
||||||
if (typeof data === 'string') {
|
if (typeof data === 'string') {
|
||||||
return new TextEncoder().encode(data).buffer;
|
return new TextEncoder().encode(data);
|
||||||
} else {
|
} else {
|
||||||
throw new Error("Text data must be a String");
|
throw new Error("Text data must be a String");
|
||||||
}
|
}
|
||||||
} else if (type === "dictionary") {
|
} else if (type === "dictionary") {
|
||||||
// JSON data - serialize directly
|
// JSON data - serialize directly
|
||||||
const jsonStr = JSON.stringify(data);
|
const jsonStr = JSON.stringify(data);
|
||||||
return new TextEncoder().encode(jsonStr).buffer;
|
return new TextEncoder().encode(jsonStr);
|
||||||
} else if (type === "table") {
|
} else if (type === "table") {
|
||||||
// Table data - convert to Arrow IPC stream (NOT IMPLEMENTED in pure JavaScript)
|
// Table data - convert to Arrow IPC stream (NOT IMPLEMENTED in pure JavaScript)
|
||||||
// This would require the apache-arrow library
|
// This would require the apache-arrow library
|
||||||
throw new Error("Table serialization requires apache-arrow library");
|
throw new Error("Table serialization requires apache-arrow library");
|
||||||
} else if (type === "image") {
|
} else if (type === "image") {
|
||||||
if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
||||||
return data instanceof ArrayBuffer ? data : data.buffer;
|
return data instanceof ArrayBuffer ? new Uint8Array(data) : data;
|
||||||
} else {
|
} else {
|
||||||
throw new Error("Image data must be ArrayBuffer or Uint8Array");
|
throw new Error("Image data must be ArrayBuffer or Uint8Array");
|
||||||
}
|
}
|
||||||
} else if (type === "audio") {
|
} else if (type === "audio") {
|
||||||
if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
||||||
return data instanceof ArrayBuffer ? data : data.buffer;
|
return data instanceof ArrayBuffer ? new Uint8Array(data) : data;
|
||||||
} else {
|
} else {
|
||||||
throw new Error("Audio data must be ArrayBuffer or Uint8Array");
|
throw new Error("Audio data must be ArrayBuffer or Uint8Array");
|
||||||
}
|
}
|
||||||
} else if (type === "video") {
|
} else if (type === "video") {
|
||||||
if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
||||||
return data instanceof ArrayBuffer ? data : data.buffer;
|
return data instanceof ArrayBuffer ? new Uint8Array(data) : data;
|
||||||
} else {
|
} else {
|
||||||
throw new Error("Video data must be ArrayBuffer or Uint8Array");
|
throw new Error("Video data must be ArrayBuffer or Uint8Array");
|
||||||
}
|
}
|
||||||
} else if (type === "binary") {
|
} else if (type === "binary") {
|
||||||
if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
if (data instanceof ArrayBuffer || data instanceof Uint8Array) {
|
||||||
return data instanceof ArrayBuffer ? data : data.buffer;
|
return data instanceof ArrayBuffer ? new Uint8Array(data) : data;
|
||||||
} else {
|
} else {
|
||||||
throw new Error("Binary data must be ArrayBuffer or Uint8Array");
|
throw new Error("Binary data must be ArrayBuffer or Uint8Array");
|
||||||
}
|
}
|
||||||
@@ -171,10 +191,10 @@ function _deserialize_data(data, type, correlation_id) {
|
|||||||
*/
|
*/
|
||||||
if (type === "text") {
|
if (type === "text") {
|
||||||
const decoder = new TextDecoder();
|
const decoder = new TextDecoder();
|
||||||
return decoder.decode(new Uint8Array(data));
|
return decoder.decode(data);
|
||||||
} else if (type === "dictionary") {
|
} else if (type === "dictionary") {
|
||||||
const decoder = new TextDecoder();
|
const decoder = new TextDecoder();
|
||||||
const jsonStr = decoder.decode(new Uint8Array(data));
|
const jsonStr = decoder.decode(data);
|
||||||
return JSON.parse(jsonStr);
|
return JSON.parse(jsonStr);
|
||||||
} else if (type === "table") {
|
} else if (type === "table") {
|
||||||
// Table data - deserialize Arrow IPC stream (NOT IMPLEMENTED in pure JavaScript)
|
// Table data - deserialize Arrow IPC stream (NOT IMPLEMENTED in pure JavaScript)
|
||||||
@@ -230,7 +250,7 @@ async function _upload_to_fileserver(fileserver_url, dataname, data, correlation
|
|||||||
|
|
||||||
// Create multipart form data
|
// Create multipart form data
|
||||||
const formData = new FormData();
|
const formData = new FormData();
|
||||||
// Create a Blob from the ArrayBuffer
|
// Create a Blob from the Uint8Array
|
||||||
const blob = new Blob([data], { type: "application/octet-stream" });
|
const blob = new Blob([data], { type: "application/octet-stream" });
|
||||||
formData.append("file", blob, dataname);
|
formData.append("file", blob, dataname);
|
||||||
|
|
||||||
@@ -276,7 +296,7 @@ async function _fetch_with_backoff(url, max_retries, base_delay, max_delay, corr
|
|||||||
if (response.status === 200) {
|
if (response.status === 200) {
|
||||||
log_trace(correlation_id, `Successfully fetched data from ${url} on attempt ${attempt}`);
|
log_trace(correlation_id, `Successfully fetched data from ${url} on attempt ${attempt}`);
|
||||||
const arrayBuffer = await response.arrayBuffer();
|
const arrayBuffer = await response.arrayBuffer();
|
||||||
return arrayBuffer;
|
return new Uint8Array(arrayBuffer);
|
||||||
} else {
|
} else {
|
||||||
throw new Error(`Failed to fetch: ${response.status} ${response.statusText}`);
|
throw new Error(`Failed to fetch: ${response.status} ${response.statusText}`);
|
||||||
}
|
}
|
||||||
@@ -306,25 +326,26 @@ function _get_payload_bytes(data) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// MessagePayload class
|
// MessagePayload class - matches msg_payload_v1 Julia struct
|
||||||
class MessagePayload {
|
class MessagePayload {
|
||||||
/**
|
/**
|
||||||
* Represents a single payload in the message envelope
|
* Represents a single payload in the message envelope
|
||||||
|
* Matches Julia's msg_payload_v1 struct
|
||||||
*
|
*
|
||||||
* @param {Object} options - Payload options
|
* @param {Object} options - Payload options
|
||||||
* @param {string} options.id - ID of this payload (e.g., "uuid4")
|
* @param {string} options.id - ID of this payload (e.g., "uuid4")
|
||||||
* @param {string} options.dataname - Name of this payload (e.g., "login_image")
|
* @param {string} options.dataname - Name of this payload (e.g., "login_image")
|
||||||
* @param {string} options.type - Payload type: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
* @param {string} options.payload_type - Payload type: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||||
* @param {string} options.transport - "direct" or "link"
|
* @param {string} options.transport - "direct" or "link"
|
||||||
* @param {string} options.encoding - "none", "json", "base64", "arrow-ipc"
|
* @param {string} options.encoding - "none", "json", "base64", "arrow-ipc"
|
||||||
* @param {number} options.size - Data size in bytes
|
* @param {number} options.size - Data size in bytes
|
||||||
* @param {string|ArrayBuffer} options.data - Payload data (direct) or URL (link)
|
* @param {string|Uint8Array} options.data - Payload data (Uint8Array for direct, URL string for link)
|
||||||
* @param {Object} options.metadata - Metadata for this payload
|
* @param {Object} options.metadata - Metadata for this payload
|
||||||
*/
|
*/
|
||||||
constructor(options) {
|
constructor(options) {
|
||||||
this.id = options.id || uuid4();
|
this.id = options.id || uuid4();
|
||||||
this.dataname = options.dataname;
|
this.dataname = options.dataname;
|
||||||
this.type = options.type;
|
this.payload_type = options.payload_type;
|
||||||
this.transport = options.transport;
|
this.transport = options.transport;
|
||||||
this.encoding = options.encoding;
|
this.encoding = options.encoding;
|
||||||
this.size = options.size;
|
this.size = options.size;
|
||||||
@@ -332,27 +353,27 @@ class MessagePayload {
|
|||||||
this.metadata = options.metadata || {};
|
this.metadata = options.metadata || {};
|
||||||
}
|
}
|
||||||
|
|
||||||
// Convert to JSON object
|
// Convert to JSON object - uses snake_case to match Julia API
|
||||||
toJSON() {
|
toJSON() {
|
||||||
const obj = {
|
const obj = {
|
||||||
id: this.id,
|
id: this.id,
|
||||||
dataname: this.dataname,
|
dataname: this.dataname,
|
||||||
type: this.type,
|
payload_type: this.payload_type,
|
||||||
transport: this.transport,
|
transport: this.transport,
|
||||||
encoding: this.encoding,
|
encoding: this.encoding,
|
||||||
size: this.size
|
size: this.size
|
||||||
};
|
};
|
||||||
|
|
||||||
// Include data based on transport type
|
// Include data based on transport type
|
||||||
if (this.transport === "direct" && this.data !== null) {
|
if (this.transport === "direct" && this.data !== null && this.data !== undefined) {
|
||||||
if (this.encoding === "base64" || this.encoding === "json") {
|
if (this.encoding === "base64" || this.encoding === "json") {
|
||||||
obj.data = this.data;
|
obj.data = this.data;
|
||||||
} else {
|
} else {
|
||||||
// For other encodings, use base64
|
// For other encodings, use base64
|
||||||
const payloadBytes = _get_payload_bytes(this.data);
|
const payloadBytes = _get_payload_bytes(this.data);
|
||||||
obj.data = arrayBufferToBase64(payloadBytes);
|
obj.data = uint8ArrayToBase64(payloadBytes);
|
||||||
}
|
}
|
||||||
} else if (this.transport === "link" && this.data !== null) {
|
} else if (this.transport === "link" && this.data !== null && this.data !== undefined) {
|
||||||
// For link transport, data is a URL string
|
// For link transport, data is a URL string
|
||||||
obj.data = this.data;
|
obj.data = this.data;
|
||||||
}
|
}
|
||||||
@@ -365,59 +386,60 @@ class MessagePayload {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// MessageEnvelope class
|
// MessageEnvelope class - matches msg_envelope_v1 Julia struct
|
||||||
class MessageEnvelope {
|
class MessageEnvelope {
|
||||||
/**
|
/**
|
||||||
* Represents the message envelope containing metadata and payloads
|
* Represents the message envelope containing metadata and payloads
|
||||||
|
* Matches Julia's msg_envelope_v1 struct
|
||||||
*
|
*
|
||||||
* @param {Object} options - Envelope options
|
* @param {Object} options - Envelope options
|
||||||
* @param {string} options.sendTo - Topic/subject the sender sends to
|
* @param {string} options.correlation_id - Unique identifier to track messages
|
||||||
* @param {Array<MessagePayload>} options.payloads - Array of payloads
|
* @param {string} options.msg_id - This message id
|
||||||
* @param {string} options.correlationId - Unique identifier to track messages
|
|
||||||
* @param {string} options.msgId - This message id
|
|
||||||
* @param {string} options.timestamp - Message published timestamp
|
* @param {string} options.timestamp - Message published timestamp
|
||||||
* @param {string} options.msgPurpose - Purpose of this message
|
* @param {string} options.send_to - Topic/subject the sender sends to
|
||||||
* @param {string} options.senderName - Name of the sender
|
* @param {string} options.msg_purpose - Purpose of this message
|
||||||
* @param {string} options.senderId - UUID of the sender
|
* @param {string} options.sender_name - Name of the sender
|
||||||
* @param {string} options.receiverName - Name of the receiver
|
* @param {string} options.sender_id - UUID of the sender
|
||||||
* @param {string} options.receiverId - UUID of the receiver
|
* @param {string} options.receiver_name - Name of the receiver
|
||||||
* @param {string} options.replyTo - Topic to reply to
|
* @param {string} options.receiver_id - UUID of the receiver
|
||||||
* @param {string} options.replyToMsgId - Message id this message is replying to
|
* @param {string} options.reply_to - Topic to reply to
|
||||||
* @param {string} options.brokerURL - NATS server address
|
* @param {string} options.reply_to_msg_id - Message id this message is replying to
|
||||||
|
* @param {string} options.broker_url - NATS server address
|
||||||
* @param {Object} options.metadata - Metadata for the envelope
|
* @param {Object} options.metadata - Metadata for the envelope
|
||||||
|
* @param {Array<MessagePayload>} options.payloads - Array of payloads
|
||||||
*/
|
*/
|
||||||
constructor(options) {
|
constructor(options) {
|
||||||
this.correlationId = options.correlationId || uuid4();
|
this.correlation_id = options.correlation_id || uuid4();
|
||||||
this.msgId = options.msgId || uuid4();
|
this.msg_id = options.msg_id || uuid4();
|
||||||
this.timestamp = options.timestamp || new Date().toISOString();
|
this.timestamp = options.timestamp || new Date().toISOString();
|
||||||
this.sendTo = options.sendTo;
|
this.send_to = options.send_to;
|
||||||
this.msgPurpose = options.msgPurpose || "";
|
this.msg_purpose = options.msg_purpose || "";
|
||||||
this.senderName = options.senderName || "";
|
this.sender_name = options.sender_name || "";
|
||||||
this.senderId = options.senderId || uuid4();
|
this.sender_id = options.sender_id || uuid4();
|
||||||
this.receiverName = options.receiverName || "";
|
this.receiver_name = options.receiver_name || "";
|
||||||
this.receiverId = options.receiverId || "";
|
this.receiver_id = options.receiver_id || "";
|
||||||
this.replyTo = options.replyTo || "";
|
this.reply_to = options.reply_to || "";
|
||||||
this.replyToMsgId = options.replyToMsgId || "";
|
this.reply_to_msg_id = options.reply_to_msg_id || "";
|
||||||
this.brokerURL = options.brokerURL || DEFAULT_NATS_URL;
|
this.broker_url = options.broker_url || DEFAULT_NATS_URL;
|
||||||
this.metadata = options.metadata || {};
|
this.metadata = options.metadata || {};
|
||||||
this.payloads = options.payloads || [];
|
this.payloads = options.payloads || [];
|
||||||
}
|
}
|
||||||
|
|
||||||
// Convert to JSON string
|
// Convert to JSON object - uses snake_case to match Julia API
|
||||||
toJSON() {
|
toJSON() {
|
||||||
const obj = {
|
const obj = {
|
||||||
correlationId: this.correlationId,
|
correlation_id: this.correlation_id,
|
||||||
msgId: this.msgId,
|
msg_id: this.msg_id,
|
||||||
timestamp: this.timestamp,
|
timestamp: this.timestamp,
|
||||||
sendTo: this.sendTo,
|
send_to: this.send_to,
|
||||||
msgPurpose: this.msgPurpose,
|
msg_purpose: this.msg_purpose,
|
||||||
senderName: this.senderName,
|
sender_name: this.sender_name,
|
||||||
senderId: this.senderId,
|
sender_id: this.sender_id,
|
||||||
receiverName: this.receiverName,
|
receiver_name: this.receiver_name,
|
||||||
receiverId: this.receiverId,
|
receiver_id: this.receiver_id,
|
||||||
replyTo: this.replyTo,
|
reply_to: this.reply_to,
|
||||||
replyToMsgId: this.replyToMsgId,
|
reply_to_msg_id: this.reply_to_msg_id,
|
||||||
brokerURL: this.brokerURL
|
broker_url: this.broker_url
|
||||||
};
|
};
|
||||||
|
|
||||||
if (Object.keys(this.metadata).length > 0) {
|
if (Object.keys(this.metadata).length > 0) {
|
||||||
@@ -437,7 +459,7 @@ class MessageEnvelope {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// SmartSend function
|
// SmartSend function - matches Julia smartsend signature and behavior
|
||||||
async function smartsend(subject, data, options = {}) {
|
async function smartsend(subject, data, options = {}) {
|
||||||
/**
|
/**
|
||||||
* Send data either directly via NATS or via a fileserver URL, depending on payload size
|
* Send data either directly via NATS or via a fileserver URL, depending on payload size
|
||||||
@@ -447,40 +469,42 @@ async function smartsend(subject, data, options = {}) {
|
|||||||
* Otherwise, it uploads the data to a fileserver and publishes only the download URL over NATS.
|
* Otherwise, it uploads the data to a fileserver and publishes only the download URL over NATS.
|
||||||
*
|
*
|
||||||
* @param {string} subject - NATS subject to publish the message to
|
* @param {string} subject - NATS subject to publish the message to
|
||||||
* @param {Array} data - List of {dataname, data, type} objects to send
|
* @param {Array} data - List of {dataname, data, type} objects to send (must be a list, even for single payload)
|
||||||
* @param {Object} options - Additional options
|
* @param {Object} options - Additional options
|
||||||
* @param {string} options.natsUrl - URL of the NATS server (default: "nats://localhost:4222")
|
* @param {string} options.broker_url - URL of the NATS server (default: "nats://localhost:4222")
|
||||||
* @param {string} options.fileserverUrl - Base URL of the file server (default: "http://localhost:8080")
|
* @param {string} options.fileserver_url - Base URL of the file server (default: "http://localhost:8080")
|
||||||
* @param {Function} options.fileserverUploadHandler - Function to handle fileserver uploads
|
* @param {Function} options.fileserver_upload_handler - Function to handle fileserver uploads
|
||||||
* @param {number} options.sizeThreshold - Threshold in bytes separating direct vs link transport (default: 1MB)
|
* @param {number} options.size_threshold - Threshold in bytes separating direct vs link transport (default: 1MB)
|
||||||
* @param {string} options.correlationId - Optional correlation ID for tracing
|
* @param {string} options.correlation_id - Optional correlation ID for tracing
|
||||||
* @param {string} options.msgPurpose - Purpose of the message (default: "chat")
|
* @param {string} options.msg_purpose - Purpose of the message (default: "chat")
|
||||||
* @param {string} options.senderName - Name of the sender (default: "NATSBridge")
|
* @param {string} options.sender_name - Name of the sender (default: "NATSBridge")
|
||||||
* @param {string} options.receiverName - Name of the receiver (default: "")
|
* @param {string} options.receiver_name - Name of the receiver (default: "")
|
||||||
* @param {string} options.receiverId - UUID of the receiver (default: "")
|
* @param {string} options.receiver_id - UUID of the receiver (default: "")
|
||||||
* @param {string} options.replyTo - Topic to reply to (default: "")
|
* @param {string} options.reply_to - Topic to reply to (default: "")
|
||||||
* @param {string} options.replyToMsgId - Message ID this message is replying to (default: "")
|
* @param {string} options.reply_to_msg_id - Message ID this message is replying to (default: "")
|
||||||
|
* @param {boolean} options.is_publish - Whether to automatically publish the message to NATS (default: true)
|
||||||
*
|
*
|
||||||
* @returns {Promise<MessageEnvelope>} - The envelope for tracking
|
* @returns {Promise<Object>} - A tuple-like object with { env: MessageEnvelope, env_json_str: string }
|
||||||
*/
|
*/
|
||||||
const {
|
const {
|
||||||
natsUrl = DEFAULT_NATS_URL,
|
broker_url = DEFAULT_NATS_URL,
|
||||||
fileserverUrl = DEFAULT_FILESERVER_URL,
|
fileserver_url = DEFAULT_FILESERVER_URL,
|
||||||
fileserverUploadHandler = _upload_to_fileserver,
|
fileserver_upload_handler = _upload_to_fileserver,
|
||||||
sizeThreshold = DEFAULT_SIZE_THRESHOLD,
|
size_threshold = DEFAULT_SIZE_THRESHOLD,
|
||||||
correlationId = uuid4(),
|
correlation_id = uuid4(),
|
||||||
msgPurpose = "chat",
|
msg_purpose = "chat",
|
||||||
senderName = "NATSBridge",
|
sender_name = "NATSBridge",
|
||||||
receiverName = "",
|
receiver_name = "",
|
||||||
receiverId = "",
|
receiver_id = "",
|
||||||
replyTo = "",
|
reply_to = "",
|
||||||
replyToMsgId = ""
|
reply_to_msg_id = "",
|
||||||
|
is_publish = true // Whether to automatically publish the message to NATS
|
||||||
} = options;
|
} = options;
|
||||||
|
|
||||||
log_trace(correlationId, `Starting smartsend for subject: ${subject}`);
|
log_trace(correlation_id, `Starting smartsend for subject: ${subject}`);
|
||||||
|
|
||||||
// Generate message metadata
|
// Generate message metadata
|
||||||
const msgId = uuid4();
|
const msg_id = uuid4();
|
||||||
|
|
||||||
// Process each payload in the list
|
// Process each payload in the list
|
||||||
const payloads = [];
|
const payloads = [];
|
||||||
@@ -494,18 +518,18 @@ async function smartsend(subject, data, options = {}) {
|
|||||||
const payloadBytes = _serialize_data(payloadData, payloadType);
|
const payloadBytes = _serialize_data(payloadData, payloadType);
|
||||||
const payloadSize = payloadBytes.byteLength;
|
const payloadSize = payloadBytes.byteLength;
|
||||||
|
|
||||||
log_trace(correlationId, `Serialized payload '${dataname}' (type: ${payloadType}) size: ${payloadSize} bytes`);
|
log_trace(correlation_id, `Serialized payload '${dataname}' (payload_type: ${payloadType}) size: ${payloadSize} bytes`);
|
||||||
|
|
||||||
// Decision: Direct vs Link
|
// Decision: Direct vs Link
|
||||||
if (payloadSize < sizeThreshold) {
|
if (payloadSize < size_threshold) {
|
||||||
// Direct path - Base64 encode and send via NATS
|
// Direct path - Base64 encode and send via NATS
|
||||||
const payloadB64 = arrayBufferToBase64(payloadBytes);
|
const payloadB64 = uint8ArrayToBase64(payloadBytes);
|
||||||
log_trace(correlationId, `Using direct transport for ${payloadSize} bytes`);
|
log_trace(correlation_id, `Using direct transport for ${payloadSize} bytes`);
|
||||||
|
|
||||||
// Create MessagePayload for direct transport
|
// Create MessagePayload for direct transport
|
||||||
const payloadObj = new MessagePayload({
|
const payloadObj = new MessagePayload({
|
||||||
dataname: dataname,
|
dataname: dataname,
|
||||||
type: payloadType,
|
payload_type: payloadType,
|
||||||
transport: "direct",
|
transport: "direct",
|
||||||
encoding: "base64",
|
encoding: "base64",
|
||||||
size: payloadSize,
|
size: payloadSize,
|
||||||
@@ -515,22 +539,22 @@ async function smartsend(subject, data, options = {}) {
|
|||||||
payloads.push(payloadObj);
|
payloads.push(payloadObj);
|
||||||
} else {
|
} else {
|
||||||
// Link path - Upload to HTTP server, send URL via NATS
|
// Link path - Upload to HTTP server, send URL via NATS
|
||||||
log_trace(correlationId, `Using link transport, uploading to fileserver`);
|
log_trace(correlation_id, `Using link transport, uploading to fileserver`);
|
||||||
|
|
||||||
// Upload to HTTP server
|
// Upload to HTTP server
|
||||||
const response = await fileserverUploadHandler(fileserverUrl, dataname, payloadBytes, correlationId);
|
const response = await fileserver_upload_handler(fileserver_url, dataname, payloadBytes, correlation_id);
|
||||||
|
|
||||||
if (response.status !== 200) {
|
if (response.status !== 200) {
|
||||||
throw new Error(`Failed to upload data to fileserver: ${response.status}`);
|
throw new Error(`Failed to upload data to fileserver: ${response.status}`);
|
||||||
}
|
}
|
||||||
|
|
||||||
const url = response.url;
|
const url = response.url;
|
||||||
log_trace(correlationId, `Uploaded to URL: ${url}`);
|
log_trace(correlation_id, `Uploaded to URL: ${url}`);
|
||||||
|
|
||||||
// Create MessagePayload for link transport
|
// Create MessagePayload for link transport
|
||||||
const payloadObj = new MessagePayload({
|
const payloadObj = new MessagePayload({
|
||||||
dataname: dataname,
|
dataname: dataname,
|
||||||
type: payloadType,
|
payload_type: payloadType,
|
||||||
transport: "link",
|
transport: "link",
|
||||||
encoding: "none",
|
encoding: "none",
|
||||||
size: payloadSize,
|
size: payloadSize,
|
||||||
@@ -543,31 +567,40 @@ async function smartsend(subject, data, options = {}) {
|
|||||||
|
|
||||||
// Create MessageEnvelope with all payloads
|
// Create MessageEnvelope with all payloads
|
||||||
const env = new MessageEnvelope({
|
const env = new MessageEnvelope({
|
||||||
correlationId: correlationId,
|
correlation_id: correlation_id,
|
||||||
msgId: msgId,
|
msg_id: msg_id,
|
||||||
sendTo: subject,
|
send_to: subject,
|
||||||
msgPurpose: msgPurpose,
|
msg_purpose: msg_purpose,
|
||||||
senderName: senderName,
|
sender_name: sender_name,
|
||||||
receiverName: receiverName,
|
receiver_name: receiver_name,
|
||||||
receiverId: receiverId,
|
receiver_id: receiver_id,
|
||||||
replyTo: replyTo,
|
reply_to: reply_to,
|
||||||
replyToMsgId: replyToMsgId,
|
reply_to_msg_id: reply_to_msg_id,
|
||||||
brokerURL: natsUrl,
|
broker_url: broker_url,
|
||||||
payloads: payloads
|
payloads: payloads
|
||||||
});
|
});
|
||||||
|
|
||||||
// Publish message to NATS
|
// Convert envelope to JSON string
|
||||||
await publish_message(natsUrl, subject, env.toString(), correlationId);
|
const env_json_str = env.toString();
|
||||||
|
|
||||||
return env;
|
// Publish to NATS if isPublish is true
|
||||||
|
if (is_publish) {
|
||||||
|
await publish_message(broker_url, subject, env_json_str, correlation_id);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Return both envelope and JSON string (tuple-like structure, matching Julia API)
|
||||||
|
return {
|
||||||
|
env: env,
|
||||||
|
env_json_str: env_json_str
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
// Helper: Publish message to NATS
|
// Helper: Publish message to NATS
|
||||||
async function publish_message(natsUrl, subject, message, correlation_id) {
|
async function publish_message(broker_url, subject, message, correlation_id) {
|
||||||
/**
|
/**
|
||||||
* Publish a message to a NATS subject with proper connection management
|
* Publish a message to a NATS subject with proper connection management
|
||||||
*
|
*
|
||||||
* @param {string} natsUrl - NATS server URL
|
* @param {string} broker_url - NATS server URL
|
||||||
* @param {string} subject - NATS subject to publish to
|
* @param {string} subject - NATS subject to publish to
|
||||||
* @param {string} message - JSON message to publish
|
* @param {string} message - JSON message to publish
|
||||||
* @param {string} correlation_id - Correlation ID for logging
|
* @param {string} correlation_id - Correlation ID for logging
|
||||||
@@ -580,7 +613,7 @@ async function publish_message(natsUrl, subject, message, correlation_id) {
|
|||||||
|
|
||||||
// Example with nats.js:
|
// Example with nats.js:
|
||||||
// import { connect } from 'nats';
|
// import { connect } from 'nats';
|
||||||
// const nc = await connect({ servers: [natsUrl] });
|
// const nc = await connect({ servers: [broker_url] });
|
||||||
// await nc.publish(subject, message);
|
// await nc.publish(subject, message);
|
||||||
// nc.close();
|
// nc.close();
|
||||||
|
|
||||||
@@ -588,7 +621,7 @@ async function publish_message(natsUrl, subject, message, correlation_id) {
|
|||||||
console.log(`[NATS PUBLISH] Subject: ${subject}, Message: ${message.substring(0, 100)}...`);
|
console.log(`[NATS PUBLISH] Subject: ${subject}, Message: ${message.substring(0, 100)}...`);
|
||||||
}
|
}
|
||||||
|
|
||||||
// SmartReceive function
|
// SmartReceive function - matches Julia smartreceive signature and behavior
|
||||||
async function smartreceive(msg, options = {}) {
|
async function smartreceive(msg, options = {}) {
|
||||||
/**
|
/**
|
||||||
* Receive and process messages from NATS
|
* Receive and process messages from NATS
|
||||||
@@ -598,25 +631,25 @@ async function smartreceive(msg, options = {}) {
|
|||||||
*
|
*
|
||||||
* @param {Object} msg - NATS message object with payload property
|
* @param {Object} msg - NATS message object with payload property
|
||||||
* @param {Object} options - Additional options
|
* @param {Object} options - Additional options
|
||||||
* @param {Function} options.fileserverDownloadHandler - Function to handle downloading data from file server URLs
|
* @param {Function} options.fileserver_download_handler - Function to handle downloading data from file server URLs
|
||||||
* @param {number} options.maxRetries - Maximum retry attempts for fetching URL (default: 5)
|
* @param {number} options.max_retries - Maximum retry attempts for fetching URL (default: 5)
|
||||||
* @param {number} options.baseDelay - Initial delay for exponential backoff in ms (default: 100)
|
* @param {number} options.base_delay - Initial delay for exponential backoff in ms (default: 100)
|
||||||
* @param {number} options.maxDelay - Maximum delay for exponential backoff in ms (default: 5000)
|
* @param {number} options.max_delay - Maximum delay for exponential backoff in ms (default: 5000)
|
||||||
*
|
*
|
||||||
* @returns {Promise<Array>} - List of {dataname, data, type} objects
|
* @returns {Promise<Object>} - JSON object of envelope with payloads field containing list of {dataname, data, type} tuples
|
||||||
*/
|
*/
|
||||||
const {
|
const {
|
||||||
fileserverDownloadHandler = _fetch_with_backoff,
|
fileserver_download_handler = _fetch_with_backoff,
|
||||||
maxRetries = 5,
|
max_retries = 5,
|
||||||
baseDelay = 100,
|
base_delay = 100,
|
||||||
maxDelay = 5000
|
max_delay = 5000
|
||||||
} = options;
|
} = options;
|
||||||
|
|
||||||
// Parse the JSON envelope
|
// Parse the JSON envelope
|
||||||
const jsonStr = typeof msg.payload === 'string' ? msg.payload : new TextDecoder().decode(msg.payload);
|
const jsonStr = typeof msg.payload === 'string' ? msg.payload : new TextDecoder().decode(msg.payload);
|
||||||
const json_data = JSON.parse(jsonStr);
|
const json_data = JSON.parse(jsonStr);
|
||||||
|
|
||||||
log_trace(json_data.correlationId, `Processing received message`);
|
log_trace(json_data.correlation_id, `Processing received message`);
|
||||||
|
|
||||||
// Process all payloads in the envelope
|
// Process all payloads in the envelope
|
||||||
const payloads_list = [];
|
const payloads_list = [];
|
||||||
@@ -631,32 +664,32 @@ async function smartreceive(msg, options = {}) {
|
|||||||
|
|
||||||
if (transport === "direct") {
|
if (transport === "direct") {
|
||||||
// Direct transport - payload is in the message
|
// Direct transport - payload is in the message
|
||||||
log_trace(json_data.correlationId, `Direct transport - decoding payload '${dataname}'`);
|
log_trace(json_data.correlation_id, `Direct transport - decoding payload '${dataname}'`);
|
||||||
|
|
||||||
// Extract base64 payload from the payload
|
// Extract base64 payload from the payload
|
||||||
const payload_b64 = payload.data;
|
const payload_b64 = payload.data;
|
||||||
|
|
||||||
// Decode Base64 payload
|
// Decode Base64 payload
|
||||||
const payload_bytes = base64ToArrayBuffer(payload_b64);
|
const payload_bytes = base64ToUint8Array(payload_b64);
|
||||||
|
|
||||||
// Deserialize based on type
|
// Deserialize based on type
|
||||||
const data_type = payload.type;
|
const data_type = payload.payload_type;
|
||||||
const data = _deserialize_data(payload_bytes, data_type, json_data.correlationId);
|
const data = _deserialize_data(payload_bytes, data_type, json_data.correlation_id);
|
||||||
|
|
||||||
payloads_list.push({ dataname, data, type: data_type });
|
payloads_list.push({ dataname, data, type: data_type });
|
||||||
} else if (transport === "link") {
|
} else if (transport === "link") {
|
||||||
// Link transport - payload is at URL
|
// Link transport - payload is at URL
|
||||||
const url = payload.data;
|
const url = payload.data;
|
||||||
log_trace(json_data.correlationId, `Link transport - fetching '${dataname}' from URL: ${url}`);
|
log_trace(json_data.correlation_id, `Link transport - fetching '${dataname}' from URL: ${url}`);
|
||||||
|
|
||||||
// Fetch with exponential backoff using the download handler
|
// Fetch with exponential backoff using the download handler
|
||||||
const downloaded_data = await fileserverDownloadHandler(
|
const downloaded_data = await fileserver_download_handler(
|
||||||
url, maxRetries, baseDelay, maxDelay, json_data.correlationId
|
url, max_retries, base_delay, max_delay, json_data.correlation_id
|
||||||
);
|
);
|
||||||
|
|
||||||
// Deserialize based on type
|
// Deserialize based on type
|
||||||
const data_type = payload.type;
|
const data_type = payload.payload_type;
|
||||||
const data = _deserialize_data(downloaded_data, data_type, json_data.correlationId);
|
const data = _deserialize_data(downloaded_data, data_type, json_data.correlation_id);
|
||||||
|
|
||||||
payloads_list.push({ dataname, data, type: data_type });
|
payloads_list.push({ dataname, data, type: data_type });
|
||||||
} else {
|
} else {
|
||||||
@@ -664,7 +697,68 @@ async function smartreceive(msg, options = {}) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return payloads_list;
|
// Replace payloads array with the processed list of {dataname, data, type} tuples
|
||||||
|
// This matches Julia's smartreceive return format
|
||||||
|
json_data.payloads = payloads_list;
|
||||||
|
|
||||||
|
return json_data;
|
||||||
|
}
|
||||||
|
|
||||||
|
// plik_oneshot_upload - matches Julia plik_oneshot_upload function
|
||||||
|
async function plik_oneshot_upload(file_server_url, dataname, data) {
|
||||||
|
/**
|
||||||
|
* Upload a single file to a plik server using one-shot mode
|
||||||
|
* This function uploads raw byte array to a plik server in one-shot mode (no upload session).
|
||||||
|
* It first creates a one-shot upload session by sending a POST request with {"OneShot": true},
|
||||||
|
* retrieves an upload ID and token, then uploads the file data as multipart form data using the token.
|
||||||
|
*
|
||||||
|
* @param {string} file_server_url - Base URL of the plik server (e.g., "http://localhost:8080")
|
||||||
|
* @param {string} dataname - Name of the file being uploaded
|
||||||
|
* @param {Uint8Array} data - Raw byte data of the file content
|
||||||
|
* @returns {Promise<Object>} - Dictionary with keys: status, uploadid, fileid, url
|
||||||
|
*/
|
||||||
|
|
||||||
|
// Step 1: Get upload ID and token
|
||||||
|
const url_getUploadID = `${file_server_url}/upload`;
|
||||||
|
const headers = { "Content-Type": "application/json" };
|
||||||
|
const body = JSON.stringify({ OneShot: true });
|
||||||
|
|
||||||
|
let http_response = await fetch(url_getUploadID, {
|
||||||
|
method: "POST",
|
||||||
|
headers: headers,
|
||||||
|
body: body
|
||||||
|
});
|
||||||
|
|
||||||
|
const response_json = await http_response.json();
|
||||||
|
const uploadid = response_json.id;
|
||||||
|
const uploadtoken = response_json.uploadToken;
|
||||||
|
|
||||||
|
// Step 2: Upload file data
|
||||||
|
const url_upload = `${file_server_url}/file/${uploadid}`;
|
||||||
|
|
||||||
|
// Create multipart form data
|
||||||
|
const formData = new FormData();
|
||||||
|
const blob = new Blob([data], { type: "application/octet-stream" });
|
||||||
|
formData.append("file", blob, dataname);
|
||||||
|
|
||||||
|
http_response = await fetch(url_upload, {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "X-UploadToken": uploadtoken },
|
||||||
|
body: formData
|
||||||
|
});
|
||||||
|
|
||||||
|
const fileResponseJson = await http_response.json();
|
||||||
|
const fileid = fileResponseJson.id;
|
||||||
|
|
||||||
|
// URL of the uploaded data e.g. "http://192.168.1.20:8080/file/3F62E/4AgGT/test.zip"
|
||||||
|
const url = `${file_server_url}/file/${uploadid}/${fileid}/${encodeURIComponent(dataname)}`;
|
||||||
|
|
||||||
|
return {
|
||||||
|
status: http_response.status,
|
||||||
|
uploadid: uploadid,
|
||||||
|
fileid: fileid,
|
||||||
|
url: url
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
// Export for Node.js
|
// Export for Node.js
|
||||||
@@ -678,6 +772,7 @@ if (typeof module !== 'undefined' && module.exports) {
|
|||||||
_deserialize_data,
|
_deserialize_data,
|
||||||
_fetch_with_backoff,
|
_fetch_with_backoff,
|
||||||
_upload_to_fileserver,
|
_upload_to_fileserver,
|
||||||
|
plik_oneshot_upload,
|
||||||
DEFAULT_SIZE_THRESHOLD,
|
DEFAULT_SIZE_THRESHOLD,
|
||||||
DEFAULT_NATS_URL,
|
DEFAULT_NATS_URL,
|
||||||
DEFAULT_FILESERVER_URL,
|
DEFAULT_FILESERVER_URL,
|
||||||
@@ -697,6 +792,7 @@ if (typeof window !== 'undefined') {
|
|||||||
_deserialize_data,
|
_deserialize_data,
|
||||||
_fetch_with_backoff,
|
_fetch_with_backoff,
|
||||||
_upload_to_fileserver,
|
_upload_to_fileserver,
|
||||||
|
plik_oneshot_upload,
|
||||||
DEFAULT_SIZE_THRESHOLD,
|
DEFAULT_SIZE_THRESHOLD,
|
||||||
DEFAULT_NATS_URL,
|
DEFAULT_NATS_URL,
|
||||||
DEFAULT_FILESERVER_URL,
|
DEFAULT_FILESERVER_URL,
|
||||||
|
|||||||
212
src/README.md
212
src/README.md
@@ -1,212 +0,0 @@
|
|||||||
# NATSBridge for Micropython
|
|
||||||
|
|
||||||
A high-performance, bi-directional data bridge for Micropython devices using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
This module provides functionality for sending and receiving data over NATS with automatic transport selection based on payload size:
|
|
||||||
|
|
||||||
- **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded)
|
|
||||||
- **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL
|
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
- ✅ Bi-directional NATS communication
|
|
||||||
- ✅ Multi-payload support (mixed content in single message)
|
|
||||||
- ✅ Automatic transport selection based on payload size
|
|
||||||
- ✅ File server integration for large payloads
|
|
||||||
- ✅ Exponential backoff for URL fetching
|
|
||||||
- ✅ Correlation ID tracking
|
|
||||||
- ✅ Reply-to support for request-response pattern
|
|
||||||
|
|
||||||
## Supported Payload Types
|
|
||||||
|
|
||||||
| Type | Description |
|
|
||||||
|------|-------------|
|
|
||||||
| `text` | Plain text strings |
|
|
||||||
| `dictionary` | JSON-serializable dictionaries |
|
|
||||||
| `table` | Tabular data (Arrow IPC format) |
|
|
||||||
| `image` | Image data (PNG, JPG bytes) |
|
|
||||||
| `audio` | Audio data (WAV, MP3 bytes) |
|
|
||||||
| `video` | Video data (MP4, AVI bytes) |
|
|
||||||
| `binary` | Generic binary data |
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
1. Copy `nats_bridge.py` to your Micropython device
|
|
||||||
2. Ensure you have the following dependencies:
|
|
||||||
- `urequests` for HTTP requests
|
|
||||||
- `ubinascii` for base64 encoding
|
|
||||||
- `ujson` for JSON handling
|
|
||||||
- `usocket` for networking
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
### Basic Text Message
|
|
||||||
|
|
||||||
```python
|
|
||||||
from nats_bridge import smartsend, smartreceive
|
|
||||||
|
|
||||||
# Sender
|
|
||||||
data = [("message", "Hello World", "text")]
|
|
||||||
env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222")
|
|
||||||
|
|
||||||
# Receiver
|
|
||||||
payloads = smartreceive(msg)
|
|
||||||
for dataname, data, type in payloads:
|
|
||||||
print("Received {}: {}".format(dataname, data))
|
|
||||||
```
|
|
||||||
|
|
||||||
### Sending JSON Configuration
|
|
||||||
|
|
||||||
```python
|
|
||||||
from nats_bridge import smartsend
|
|
||||||
|
|
||||||
config = {
|
|
||||||
"wifi_ssid": "MyNetwork",
|
|
||||||
"wifi_password": "password123",
|
|
||||||
"update_interval": 60
|
|
||||||
}
|
|
||||||
|
|
||||||
data = [("config", config, "dictionary")]
|
|
||||||
env = smartsend("/device/config", data, nats_url="nats://localhost:4222")
|
|
||||||
```
|
|
||||||
|
|
||||||
### Mixed Content (Chat with Text + Image)
|
|
||||||
|
|
||||||
```python
|
|
||||||
from nats_bridge import smartsend
|
|
||||||
|
|
||||||
image_data = b"\x89PNG..." # PNG bytes
|
|
||||||
|
|
||||||
data = [
|
|
||||||
("message_text", "Hello with image!", "text"),
|
|
||||||
("user_avatar", image_data, "binary")
|
|
||||||
]
|
|
||||||
|
|
||||||
env = smartsend("/chat/mixed", data, nats_url="nats://localhost:4222")
|
|
||||||
```
|
|
||||||
|
|
||||||
### Request-Response Pattern
|
|
||||||
|
|
||||||
```python
|
|
||||||
from nats_bridge import smartsend
|
|
||||||
|
|
||||||
# Send command with reply-to
|
|
||||||
data = [("command", {"action": "read_sensor"}, "dictionary")]
|
|
||||||
env = smartsend(
|
|
||||||
"/device/command",
|
|
||||||
data,
|
|
||||||
nats_url="nats://localhost:4222",
|
|
||||||
reply_to="/device/response",
|
|
||||||
reply_to_msg_id="cmd-001"
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Large Payloads (File Server)
|
|
||||||
|
|
||||||
```python
|
|
||||||
from nats_bridge import smartsend
|
|
||||||
|
|
||||||
# Large data (> 1MB)
|
|
||||||
large_data = b"A" * 2000000 # 2MB
|
|
||||||
|
|
||||||
env = smartsend(
|
|
||||||
"/data/large",
|
|
||||||
[("large_file", large_data, "binary")],
|
|
||||||
nats_url="nats://localhost:4222",
|
|
||||||
fileserver_url="http://localhost:8080",
|
|
||||||
size_threshold=1000000 # 1MB threshold
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
## API Reference
|
|
||||||
|
|
||||||
### `smartsend(subject, data, ...)`
|
|
||||||
|
|
||||||
Send data via NATS with automatic transport selection.
|
|
||||||
|
|
||||||
**Arguments:**
|
|
||||||
- `subject` (str): NATS subject to publish to
|
|
||||||
- `data` (list): List of `(dataname, data, type)` tuples
|
|
||||||
- `nats_url` (str): NATS server URL (default: `nats://localhost:4222`)
|
|
||||||
- `fileserver_url` (str): HTTP file server URL (default: `http://localhost:8080`)
|
|
||||||
- `size_threshold` (int): Threshold in bytes (default: 1,000,000)
|
|
||||||
- `correlation_id` (str): Optional correlation ID for tracing
|
|
||||||
- `msg_purpose` (str): Message purpose (default: `"chat"`)
|
|
||||||
- `sender_name` (str): Sender name (default: `"NATSBridge"`)
|
|
||||||
- `receiver_name` (str): Receiver name (default: `""`)
|
|
||||||
- `receiver_id` (str): Receiver ID (default: `""`)
|
|
||||||
- `reply_to` (str): Reply topic (default: `""`)
|
|
||||||
- `reply_to_msg_id` (str): Reply message ID (default: `""`)
|
|
||||||
|
|
||||||
**Returns:** `MessageEnvelope` object
|
|
||||||
|
|
||||||
### `smartreceive(msg, ...)`
|
|
||||||
|
|
||||||
Receive and process NATS messages.
|
|
||||||
|
|
||||||
**Arguments:**
|
|
||||||
- `msg`: NATS message (dict or JSON string)
|
|
||||||
- `fileserver_download_handler` (function): Function to fetch data from URLs
|
|
||||||
- `max_retries` (int): Maximum retry attempts (default: 5)
|
|
||||||
- `base_delay` (int): Initial delay in ms (default: 100)
|
|
||||||
- `max_delay` (int): Maximum delay in ms (default: 5000)
|
|
||||||
|
|
||||||
**Returns:** List of `(dataname, data, type)` tuples
|
|
||||||
|
|
||||||
### `MessageEnvelope`
|
|
||||||
|
|
||||||
Represents a complete NATS message envelope.
|
|
||||||
|
|
||||||
**Attributes:**
|
|
||||||
- `correlation_id`: Unique identifier for tracing
|
|
||||||
- `msg_id`: Unique message identifier
|
|
||||||
- `timestamp`: Message publication timestamp
|
|
||||||
- `send_to`: NATS subject
|
|
||||||
- `msg_purpose`: Message purpose
|
|
||||||
- `sender_name`: Sender name
|
|
||||||
- `sender_id`: Sender UUID
|
|
||||||
- `receiver_name`: Receiver name
|
|
||||||
- `receiver_id`: Receiver UUID
|
|
||||||
- `reply_to`: Reply topic
|
|
||||||
- `reply_to_msg_id`: Reply message ID
|
|
||||||
- `broker_url`: NATS broker URL
|
|
||||||
- `metadata`: Message-level metadata
|
|
||||||
- `payloads`: List of MessagePayload objects
|
|
||||||
|
|
||||||
### `MessagePayload`
|
|
||||||
|
|
||||||
Represents a single payload within a message envelope.
|
|
||||||
|
|
||||||
**Attributes:**
|
|
||||||
- `id`: Unique payload identifier
|
|
||||||
- `dataname`: Name of the payload
|
|
||||||
- `type`: Payload type ("text", "dictionary", etc.)
|
|
||||||
- `transport`: Transport method ("direct" or "link")
|
|
||||||
- `encoding`: Encoding method ("none", "base64", etc.)
|
|
||||||
- `size`: Payload size in bytes
|
|
||||||
- `data`: Payload data (bytes for direct, URL for link)
|
|
||||||
- `metadata`: Payload-level metadata
|
|
||||||
|
|
||||||
## Examples
|
|
||||||
|
|
||||||
See `examples/micropython_example.py` for more detailed examples.
|
|
||||||
|
|
||||||
## Testing
|
|
||||||
|
|
||||||
Run the test suite:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python test/test_micropython_basic.py
|
|
||||||
```
|
|
||||||
|
|
||||||
## Requirements
|
|
||||||
|
|
||||||
- Micropython with networking support
|
|
||||||
- NATS server (nats.io)
|
|
||||||
- HTTP file server (optional, for large payloads)
|
|
||||||
|
|
||||||
## License
|
|
||||||
|
|
||||||
MIT
|
|
||||||
@@ -1,45 +1,60 @@
|
|||||||
"""
|
"""
|
||||||
Micropython NATS Bridge - Bi-Directional Data Bridge for Micropython
|
Python NATS Bridge - Bi-Directional Data Bridge
|
||||||
|
|
||||||
This module provides functionality for sending and receiving data over NATS
|
This module provides functionality for sending and receiving data over NATS
|
||||||
using the Claim-Check pattern for large payloads.
|
using the Claim-Check pattern for large payloads.
|
||||||
|
|
||||||
Supported types: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
Supported types: "text", "dictionary", "table", "image", "audio", "video", "binary"
|
||||||
|
|
||||||
|
Multi-Payload Support (Standard API):
|
||||||
|
The system uses a standardized list-of-tuples format for all payload operations.
|
||||||
|
Even when sending a single payload, the user must wrap it in a list.
|
||||||
|
|
||||||
|
API Standard:
|
||||||
|
# Input format for smartsend (always a list of tuples with type info)
|
||||||
|
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||||
|
|
||||||
|
# Output format for smartreceive (always returns a list of tuples)
|
||||||
|
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import random
|
|
||||||
import time
|
import time
|
||||||
import usocket
|
|
||||||
import uselect
|
|
||||||
import ustruct
|
|
||||||
import uuid
|
import uuid
|
||||||
|
|
||||||
try:
|
|
||||||
import ussl
|
|
||||||
HAS_SSL = True
|
|
||||||
except ImportError:
|
|
||||||
HAS_SSL = False
|
|
||||||
|
|
||||||
# Constants
|
# Constants
|
||||||
DEFAULT_SIZE_THRESHOLD = 1000000 # 1MB - threshold for switching from direct to link transport
|
DEFAULT_SIZE_THRESHOLD = 1000000 # 1MB - threshold for switching from direct to link transport
|
||||||
DEFAULT_NATS_URL = "nats://localhost:4222"
|
DEFAULT_BROKER_URL = "nats://localhost:4222"
|
||||||
DEFAULT_FILESERVER_URL = "http://localhost:8080"
|
DEFAULT_FILESERVER_URL = "http://localhost:8080"
|
||||||
|
|
||||||
# ============================================= 100 ============================================== #
|
# ============================================= 100 ============================================== #
|
||||||
|
|
||||||
|
|
||||||
class MessagePayload:
|
class MessagePayload:
|
||||||
"""Internal message payload structure representing a single payload within a NATS message envelope."""
|
"""Internal message payload structure representing a single payload within a NATS message envelope.
|
||||||
|
|
||||||
def __init__(self, data, msg_type, id="", dataname="", transport="direct",
|
This structure represents a single payload within a NATS message envelope.
|
||||||
|
It supports both direct transport (base64-encoded data) and link transport (URL-based).
|
||||||
|
|
||||||
|
Attributes:
|
||||||
|
id: Unique identifier for this payload (e.g., "uuid4")
|
||||||
|
dataname: Name of the payload (e.g., "login_image")
|
||||||
|
payload_type: Payload type ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
||||||
|
transport: Transport method ("direct" or "link")
|
||||||
|
encoding: Encoding method ("none", "json", "base64", "arrow-ipc")
|
||||||
|
size: Size of the payload in bytes
|
||||||
|
data: Payload data (bytes for direct, URL for link)
|
||||||
|
metadata: Optional metadata dictionary
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, data, payload_type, id="", dataname="", transport="direct",
|
||||||
encoding="none", size=0, metadata=None):
|
encoding="none", size=0, metadata=None):
|
||||||
"""
|
"""
|
||||||
Initialize a MessagePayload.
|
Initialize a MessagePayload.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
data: Payload data (bytes for direct, URL string for link)
|
data: Payload data (base64 string for direct, URL string for link)
|
||||||
msg_type: Payload type ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
payload_type: Payload type ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
||||||
id: Unique identifier for this payload (auto-generated if empty)
|
id: Unique identifier for this payload (auto-generated if empty)
|
||||||
dataname: Name of the payload (auto-generated UUID if empty)
|
dataname: Name of the payload (auto-generated UUID if empty)
|
||||||
transport: Transport method ("direct" or "link")
|
transport: Transport method ("direct" or "link")
|
||||||
@@ -49,7 +64,7 @@ class MessagePayload:
|
|||||||
"""
|
"""
|
||||||
self.id = id if id else self._generate_uuid()
|
self.id = id if id else self._generate_uuid()
|
||||||
self.dataname = dataname if dataname else self._generate_uuid()
|
self.dataname = dataname if dataname else self._generate_uuid()
|
||||||
self.type = msg_type
|
self.payload_type = payload_type
|
||||||
self.transport = transport
|
self.transport = transport
|
||||||
self.encoding = encoding
|
self.encoding = encoding
|
||||||
self.size = size
|
self.size = size
|
||||||
@@ -65,7 +80,7 @@ class MessagePayload:
|
|||||||
payload_dict = {
|
payload_dict = {
|
||||||
"id": self.id,
|
"id": self.id,
|
||||||
"dataname": self.dataname,
|
"dataname": self.dataname,
|
||||||
"type": self.type,
|
"payload_type": self.payload_type,
|
||||||
"transport": self.transport,
|
"transport": self.transport,
|
||||||
"encoding": self.encoding,
|
"encoding": self.encoding,
|
||||||
"size": self.size,
|
"size": self.size,
|
||||||
@@ -152,20 +167,24 @@ class MessageEnvelope:
|
|||||||
return "2026-02-21T" + time.strftime("%H:%M:%S", time.localtime())
|
return "2026-02-21T" + time.strftime("%H:%M:%S", time.localtime())
|
||||||
|
|
||||||
def to_json(self):
|
def to_json(self):
|
||||||
"""Convert envelope to JSON string."""
|
"""Convert envelope to JSON string.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
str: JSON string representation of the envelope using snake_case field names
|
||||||
|
"""
|
||||||
obj = {
|
obj = {
|
||||||
"correlationId": self.correlation_id,
|
"correlation_id": self.correlation_id,
|
||||||
"msgId": self.msg_id,
|
"msg_id": self.msg_id,
|
||||||
"timestamp": self.timestamp,
|
"timestamp": self.timestamp,
|
||||||
"sendTo": self.send_to,
|
"send_to": self.send_to,
|
||||||
"msgPurpose": self.msg_purpose,
|
"msg_purpose": self.msg_purpose,
|
||||||
"senderName": self.sender_name,
|
"sender_name": self.sender_name,
|
||||||
"senderId": self.sender_id,
|
"sender_id": self.sender_id,
|
||||||
"receiverName": self.receiver_name,
|
"receiver_name": self.receiver_name,
|
||||||
"receiverId": self.receiver_id,
|
"receiver_id": self.receiver_id,
|
||||||
"replyTo": self.reply_to,
|
"reply_to": self.reply_to,
|
||||||
"replyToMsgId": self.reply_to_msg_id,
|
"reply_to_msg_id": self.reply_to_msg_id,
|
||||||
"brokerURL": self.broker_url
|
"broker_url": self.broker_url
|
||||||
}
|
}
|
||||||
|
|
||||||
# Include metadata if not empty
|
# Include metadata if not empty
|
||||||
@@ -188,68 +207,126 @@ def log_trace(correlation_id, message):
|
|||||||
print("[{}] [Correlation: {}] {}".format(timestamp, correlation_id, message))
|
print("[{}] [Correlation: {}] {}".format(timestamp, correlation_id, message))
|
||||||
|
|
||||||
|
|
||||||
def _serialize_data(data, msg_type):
|
def _serialize_data(data, payload_type):
|
||||||
"""Serialize data according to specified format.
|
"""Serialize data according to specified format.
|
||||||
|
|
||||||
|
This function serializes arbitrary data into a binary representation based on the specified type.
|
||||||
|
It supports multiple serialization formats for different data types.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
data: Data to serialize
|
data: Data to serialize
|
||||||
msg_type: Target format ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
- "text": String
|
||||||
|
- "dictionary": JSON-serializable dict
|
||||||
|
- "table": Tabular data (pandas DataFrame or list of dicts)
|
||||||
|
- "image", "audio", "video", "binary": bytes
|
||||||
|
payload_type: Target format ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
bytes: Binary representation of the serialized data
|
bytes: Binary representation of the serialized data
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> text_bytes = _serialize_data("Hello World", "text")
|
||||||
|
>>> json_bytes = _serialize_data({"key": "value"}, "dictionary")
|
||||||
|
>>> table_bytes = _serialize_data([{"id": 1, "name": "Alice"}], "table")
|
||||||
"""
|
"""
|
||||||
if msg_type == "text":
|
if payload_type == "text":
|
||||||
if isinstance(data, str):
|
if isinstance(data, str):
|
||||||
return data.encode('utf-8')
|
return data.encode('utf-8')
|
||||||
else:
|
else:
|
||||||
raise ValueError("Text data must be a string")
|
raise ValueError("Text data must be a string")
|
||||||
|
|
||||||
elif msg_type == "dictionary":
|
elif payload_type == "dictionary":
|
||||||
if isinstance(data, dict):
|
if isinstance(data, dict):
|
||||||
json_str = json.dumps(data)
|
json_str = json.dumps(data)
|
||||||
return json_str.encode('utf-8')
|
return json_str.encode('utf-8')
|
||||||
else:
|
else:
|
||||||
raise ValueError("Dictionary data must be a dict")
|
raise ValueError("Dictionary data must be a dict")
|
||||||
|
|
||||||
elif msg_type in ("image", "audio", "video", "binary"):
|
elif payload_type == "table":
|
||||||
|
# Support pandas DataFrame or list of dicts
|
||||||
|
try:
|
||||||
|
import pandas as pd
|
||||||
|
if isinstance(data, pd.DataFrame):
|
||||||
|
# Convert DataFrame to JSON and then to bytes
|
||||||
|
json_str = data.to_json(orient='records', force_ascii=False)
|
||||||
|
return json_str.encode('utf-8')
|
||||||
|
elif isinstance(data, list) and len(data) > 0 and isinstance(data[0], dict):
|
||||||
|
# List of dicts
|
||||||
|
json_str = json.dumps(data)
|
||||||
|
return json_str.encode('utf-8')
|
||||||
|
else:
|
||||||
|
raise ValueError("Table data must be a pandas DataFrame or list of dicts")
|
||||||
|
except ImportError:
|
||||||
|
# Fallback: if pandas not available, treat as list of dicts
|
||||||
|
if isinstance(data, list):
|
||||||
|
json_str = json.dumps(data)
|
||||||
|
return json_str.encode('utf-8')
|
||||||
|
else:
|
||||||
|
raise ValueError("Table data requires pandas DataFrame or list of dicts (pandas not available)")
|
||||||
|
|
||||||
|
elif payload_type in ("image", "audio", "video", "binary"):
|
||||||
if isinstance(data, bytes):
|
if isinstance(data, bytes):
|
||||||
return data
|
return data
|
||||||
else:
|
else:
|
||||||
raise ValueError("{} data must be bytes".format(msg_type.capitalize()))
|
raise ValueError("{} data must be bytes".format(payload_type.capitalize()))
|
||||||
|
|
||||||
else:
|
else:
|
||||||
raise ValueError("Unknown type: {}".format(msg_type))
|
raise ValueError("Unknown payload_type: {}".format(payload_type))
|
||||||
|
|
||||||
|
|
||||||
def _deserialize_data(data_bytes, msg_type, correlation_id):
|
def _deserialize_data(data_bytes, payload_type, correlation_id):
|
||||||
"""Deserialize bytes to data based on type.
|
"""Deserialize bytes to data based on type.
|
||||||
|
|
||||||
|
This function converts serialized bytes back to Python data based on type.
|
||||||
|
It handles "text" (string), "dictionary" (JSON deserialization), "table" (JSON deserialization),
|
||||||
|
"image" (binary data), "audio" (binary data), "video" (binary data), and "binary" (binary data).
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
data_bytes: Serialized data as bytes
|
data_bytes: Serialized data as bytes
|
||||||
msg_type: Data type ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
payload_type: Data type ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
||||||
correlation_id: Correlation ID for logging
|
correlation_id: Correlation ID for logging
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Deserialized data
|
Deserialized data:
|
||||||
|
- "text": str
|
||||||
|
- "dictionary": dict
|
||||||
|
- "table": list of dicts (or pandas DataFrame if available)
|
||||||
|
- "image", "audio", "video", "binary": bytes
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> text_data = _deserialize_data(b"Hello", "text", "corr_id")
|
||||||
|
>>> json_data = _deserialize_data(b'{"key": "value"}', "dictionary", "corr_id")
|
||||||
|
>>> table_data = _deserialize_data(b'[{"id": 1}]', "table", "corr_id")
|
||||||
"""
|
"""
|
||||||
if msg_type == "text":
|
if payload_type == "text":
|
||||||
return data_bytes.decode('utf-8')
|
return data_bytes.decode('utf-8')
|
||||||
|
|
||||||
elif msg_type == "dictionary":
|
elif payload_type == "dictionary":
|
||||||
json_str = data_bytes.decode('utf-8')
|
json_str = data_bytes.decode('utf-8')
|
||||||
return json.loads(json_str)
|
return json.loads(json_str)
|
||||||
|
|
||||||
elif msg_type in ("image", "audio", "video", "binary"):
|
elif payload_type == "table":
|
||||||
|
# Deserialize table data (JSON format)
|
||||||
|
json_str = data_bytes.decode('utf-8')
|
||||||
|
table_data = json.loads(json_str)
|
||||||
|
# If pandas is available, try to convert to DataFrame
|
||||||
|
try:
|
||||||
|
import pandas as pd
|
||||||
|
return pd.DataFrame(table_data)
|
||||||
|
except ImportError:
|
||||||
|
return table_data
|
||||||
|
|
||||||
|
elif payload_type in ("image", "audio", "video", "binary"):
|
||||||
return data_bytes
|
return data_bytes
|
||||||
|
|
||||||
else:
|
else:
|
||||||
raise ValueError("Unknown type: {}".format(msg_type))
|
raise ValueError("Unknown payload_type: {}".format(payload_type))
|
||||||
|
|
||||||
|
|
||||||
class NATSConnection:
|
class NATSConnection:
|
||||||
"""Simple NATS connection for Micropython."""
|
"""Simple NATS connection for Python and Micropython."""
|
||||||
|
|
||||||
def __init__(self, url=DEFAULT_NATS_URL):
|
def __init__(self, url=DEFAULT_BROKER_URL):
|
||||||
"""Initialize NATS connection.
|
"""Initialize NATS connection.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
@@ -276,9 +353,19 @@ class NATSConnection:
|
|||||||
|
|
||||||
def connect(self):
|
def connect(self):
|
||||||
"""Connect to NATS server."""
|
"""Connect to NATS server."""
|
||||||
|
# Use socket for both Python and Micropython
|
||||||
|
try:
|
||||||
|
import socket
|
||||||
|
addr = socket.getaddrinfo(self.host, self.port)[0][-1]
|
||||||
|
self.conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||||
|
self.conn.connect(addr)
|
||||||
|
except NameError:
|
||||||
|
# Micropython fallback
|
||||||
|
import usocket
|
||||||
addr = usocket.getaddrinfo(self.host, self.port)[0][-1]
|
addr = usocket.getaddrinfo(self.host, self.port)[0][-1]
|
||||||
self.conn = usocket.socket()
|
self.conn = usocket.socket()
|
||||||
self.conn.connect(addr)
|
self.conn.connect(addr)
|
||||||
|
|
||||||
log_trace("", "Connected to NATS server at {}:{}".format(self.host, self.port))
|
log_trace("", "Connected to NATS server at {}:{}".format(self.host, self.port))
|
||||||
|
|
||||||
def publish(self, subject, message):
|
def publish(self, subject, message):
|
||||||
@@ -294,7 +381,15 @@ class NATSConnection:
|
|||||||
# Simple NATS protocol implementation
|
# Simple NATS protocol implementation
|
||||||
msg = "PUB {} {}\r\n".format(subject, len(message))
|
msg = "PUB {} {}\r\n".format(subject, len(message))
|
||||||
msg = msg.encode('utf-8') + message + b"\r\n"
|
msg = msg.encode('utf-8') + message + b"\r\n"
|
||||||
|
|
||||||
|
try:
|
||||||
|
import socket
|
||||||
self.conn.send(msg)
|
self.conn.send(msg)
|
||||||
|
except NameError:
|
||||||
|
# Micropython fallback
|
||||||
|
import usocket
|
||||||
|
self.conn.send(msg)
|
||||||
|
|
||||||
log_trace("", "Message published to {}".format(subject))
|
log_trace("", "Message published to {}".format(subject))
|
||||||
|
|
||||||
def subscribe(self, subject, callback):
|
def subscribe(self, subject, callback):
|
||||||
@@ -335,11 +430,14 @@ class NATSConnection:
|
|||||||
def _fetch_with_backoff(url, max_retries=5, base_delay=100, max_delay=5000, correlation_id=""):
|
def _fetch_with_backoff(url, max_retries=5, base_delay=100, max_delay=5000, correlation_id=""):
|
||||||
"""Fetch data from URL with exponential backoff.
|
"""Fetch data from URL with exponential backoff.
|
||||||
|
|
||||||
|
This function retrieves data from a URL with retry logic using
|
||||||
|
exponential backoff to handle transient failures.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
url: URL to fetch from
|
url: URL to fetch from
|
||||||
max_retries: Maximum number of retry attempts
|
max_retries: Maximum number of retry attempts (default: 5)
|
||||||
base_delay: Initial delay in milliseconds
|
base_delay: Initial delay in milliseconds (default: 100)
|
||||||
max_delay: Maximum delay in milliseconds
|
max_delay: Maximum delay in milliseconds (default: 5000)
|
||||||
correlation_id: Correlation ID for logging
|
correlation_id: Correlation ID for logging
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
@@ -347,33 +445,54 @@ def _fetch_with_backoff(url, max_retries=5, base_delay=100, max_delay=5000, corr
|
|||||||
|
|
||||||
Raises:
|
Raises:
|
||||||
Exception: If all retry attempts fail
|
Exception: If all retry attempts fail
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> data = _fetch_with_backoff("http://example.com/file.zip", 5, 100, 5000, "corr_id")
|
||||||
"""
|
"""
|
||||||
delay = base_delay
|
delay = base_delay
|
||||||
for attempt in range(1, max_retries + 1):
|
for attempt in range(1, max_retries + 1):
|
||||||
try:
|
try:
|
||||||
# Simple HTTP GET request
|
# Simple HTTP GET request
|
||||||
# This is a simplified implementation
|
# Try urequests for Micropython first, then requests for Python
|
||||||
# For production, you'd want a proper HTTP client
|
try:
|
||||||
import urequests
|
import urequests
|
||||||
response = urequests.get(url)
|
response = urequests.get(url)
|
||||||
if response.status_code == 200:
|
status_code = response.status_code
|
||||||
|
content = response.content
|
||||||
|
except ImportError:
|
||||||
|
try:
|
||||||
|
import requests
|
||||||
|
response = requests.get(url)
|
||||||
|
response.raise_for_status()
|
||||||
|
status_code = response.status_code
|
||||||
|
content = response.content
|
||||||
|
except ImportError:
|
||||||
|
raise Exception("No HTTP library available (urequests or requests)")
|
||||||
|
|
||||||
|
if status_code == 200:
|
||||||
log_trace(correlation_id, "Successfully fetched data from {} on attempt {}".format(url, attempt))
|
log_trace(correlation_id, "Successfully fetched data from {} on attempt {}".format(url, attempt))
|
||||||
return response.content
|
return content
|
||||||
else:
|
else:
|
||||||
raise Exception("Failed to fetch: {}".format(response.status_code))
|
raise Exception("Failed to fetch: {}".format(status_code))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
log_trace(correlation_id, "Attempt {} failed: {}".format(attempt, str(e)))
|
log_trace(correlation_id, "Attempt {} failed: {}".format(attempt, str(e)))
|
||||||
if attempt < max_retries:
|
if attempt < max_retries:
|
||||||
time.sleep(delay / 1000.0)
|
time.sleep(delay / 1000.0)
|
||||||
delay = min(delay * 2, max_delay)
|
delay = min(delay * 2, max_delay)
|
||||||
|
|
||||||
|
raise Exception("Failed to fetch data after {} attempts".format(max_retries))
|
||||||
|
|
||||||
def plik_oneshot_upload(file_server_url, filename, data):
|
|
||||||
|
def plik_oneshot_upload(fileserver_url, dataname, data):
|
||||||
"""Upload a single file to a plik server using one-shot mode.
|
"""Upload a single file to a plik server using one-shot mode.
|
||||||
|
|
||||||
|
This function uploads raw byte data to a plik server in one-shot mode (no upload session).
|
||||||
|
It first creates a one-shot upload session by sending a POST request with {"OneShot": true},
|
||||||
|
retrieves an upload ID and token, then uploads the file data as multipart form data using the token.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
file_server_url: Base URL of the plik server
|
fileserver_url: Base URL of the plik server (e.g., "http://localhost:8080")
|
||||||
filename: Name of the file being uploaded
|
dataname: Name of the file being uploaded
|
||||||
data: Raw byte data of the file content
|
data: Raw byte data of the file content
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
@@ -382,23 +501,31 @@ def plik_oneshot_upload(file_server_url, filename, data):
|
|||||||
- "uploadid": ID of the one-shot upload session
|
- "uploadid": ID of the one-shot upload session
|
||||||
- "fileid": ID of the uploaded file within the session
|
- "fileid": ID of the uploaded file within the session
|
||||||
- "url": Full URL to download the uploaded file
|
- "url": Full URL to download the uploaded file
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> result = plik_oneshot_upload("http://localhost:8080", "test.txt", b"hello world")
|
||||||
|
>>> result["status"], result["uploadid"], result["fileid"], result["url"]
|
||||||
"""
|
"""
|
||||||
import urequests
|
|
||||||
import json
|
import json
|
||||||
|
|
||||||
|
try:
|
||||||
|
import urequests
|
||||||
|
except ImportError:
|
||||||
|
import requests as urequests
|
||||||
|
|
||||||
# Get upload ID
|
# Get upload ID
|
||||||
url_get_upload_id = "{}/upload".format(file_server_url)
|
url_get_upload_id = "{}/upload".format(fileserver_url)
|
||||||
headers = {"Content-Type": "application/json"}
|
headers = {"Content-Type": "application/json"}
|
||||||
body = json.dumps({"OneShot": True})
|
body = json.dumps({"OneShot": True})
|
||||||
|
|
||||||
response = urequests.post(url_get_upload_id, headers=headers, data=body)
|
response = urequests.post(url_get_upload_id, headers=headers, data=body)
|
||||||
response_json = json.loads(response.content)
|
response_json = json.loads(response.text if hasattr(response, 'text') else response.content)
|
||||||
|
|
||||||
uploadid = response_json.get("id")
|
uploadid = response_json.get("id")
|
||||||
uploadtoken = response_json.get("uploadToken")
|
uploadtoken = response_json.get("uploadToken")
|
||||||
|
|
||||||
# Upload file
|
# Upload file
|
||||||
url_upload = "{}/file/{}".format(file_server_url, uploadid)
|
url_upload = "{}/file/{}".format(fileserver_url, uploadid)
|
||||||
headers = {"X-UploadToken": uploadtoken}
|
headers = {"X-UploadToken": uploadtoken}
|
||||||
|
|
||||||
# For Micropython, we need to construct the multipart form data manually
|
# For Micropython, we need to construct the multipart form data manually
|
||||||
@@ -407,7 +534,7 @@ def plik_oneshot_upload(file_server_url, filename, data):
|
|||||||
|
|
||||||
# Create multipart body
|
# Create multipart body
|
||||||
part1 = "--{}\r\n".format(boundary)
|
part1 = "--{}\r\n".format(boundary)
|
||||||
part1 += "Content-Disposition: form-data; name=\"file\"; filename=\"{}\"\r\n".format(filename)
|
part1 += "Content-Disposition: form-data; name=\"file\"; filename=\"{}\"\r\n".format(dataname)
|
||||||
part1 += "Content-Type: application/octet-stream\r\n\r\n"
|
part1 += "Content-Type: application/octet-stream\r\n\r\n"
|
||||||
part1_bytes = part1.encode('utf-8')
|
part1_bytes = part1.encode('utf-8')
|
||||||
|
|
||||||
@@ -421,10 +548,10 @@ def plik_oneshot_upload(file_server_url, filename, data):
|
|||||||
content_type = "multipart/form-data; boundary={}".format(boundary)
|
content_type = "multipart/form-data; boundary={}".format(boundary)
|
||||||
|
|
||||||
response = urequests.post(url_upload, headers={"Content-Type": content_type}, data=full_body)
|
response = urequests.post(url_upload, headers={"Content-Type": content_type}, data=full_body)
|
||||||
response_json = json.loads(response.content)
|
response_json = json.loads(response.text if hasattr(response, 'text') else response.content)
|
||||||
|
|
||||||
fileid = response_json.get("id")
|
fileid = response_json.get("id")
|
||||||
url = "{}/file/{}/{}".format(file_server_url, uploadid, filename)
|
url = "{}/file/{}/{}".format(fileserver_url, uploadid, dataname)
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"status": response.status_code,
|
"status": response.status_code,
|
||||||
@@ -434,10 +561,10 @@ def plik_oneshot_upload(file_server_url, filename, data):
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def smartsend(subject, data, nats_url=DEFAULT_NATS_URL, fileserver_url=DEFAULT_FILESERVER_URL,
|
def smartsend(subject, data, broker_url=DEFAULT_BROKER_URL, fileserver_url=DEFAULT_FILESERVER_URL,
|
||||||
fileserver_upload_handler=plik_oneshot_upload, size_threshold=DEFAULT_SIZE_THRESHOLD,
|
fileserver_upload_handler=plik_oneshot_upload, size_threshold=DEFAULT_SIZE_THRESHOLD,
|
||||||
correlation_id=None, msg_purpose="chat", sender_name="NATSBridge",
|
correlation_id=None, msg_purpose="chat", sender_name="NATSBridge",
|
||||||
receiver_name="", receiver_id="", reply_to="", reply_to_msg_id=""):
|
receiver_name="", receiver_id="", reply_to="", reply_to_msg_id="", is_publish=True):
|
||||||
"""Send data either directly via NATS or via a fileserver URL, depending on payload size.
|
"""Send data either directly via NATS or via a fileserver URL, depending on payload size.
|
||||||
|
|
||||||
This function intelligently routes data delivery based on payload size relative to a threshold.
|
This function intelligently routes data delivery based on payload size relative to a threshold.
|
||||||
@@ -447,24 +574,38 @@ def smartsend(subject, data, nats_url=DEFAULT_NATS_URL, fileserver_url=DEFAULT_F
|
|||||||
|
|
||||||
Args:
|
Args:
|
||||||
subject: NATS subject to publish the message to
|
subject: NATS subject to publish the message to
|
||||||
data: List of (dataname, data, type) tuples to send
|
data: List of (dataname, data, payload_type) tuples to send
|
||||||
nats_url: URL of the NATS server
|
- dataname: Name of the payload
|
||||||
|
- data: The actual data to send
|
||||||
|
- payload_type: Payload type ("text", "dictionary", "table", "image", "audio", "video", "binary")
|
||||||
|
broker_url: URL of the NATS server
|
||||||
fileserver_url: URL of the HTTP file server
|
fileserver_url: URL of the HTTP file server
|
||||||
fileserver_upload_handler: Function to handle fileserver uploads
|
fileserver_upload_handler: Function to handle fileserver uploads (must return dict with "status", "uploadid", "fileid", "url" keys)
|
||||||
size_threshold: Threshold in bytes separating direct vs link transport
|
size_threshold: Threshold in bytes separating direct vs link transport (default: 1MB)
|
||||||
correlation_id: Optional correlation ID for tracing
|
correlation_id: Optional correlation ID for tracing; if None, a UUID is generated
|
||||||
msg_purpose: Purpose of the message
|
msg_purpose: Purpose of the message ("ACK", "NACK", "updateStatus", "shutdown", "chat", etc.)
|
||||||
sender_name: Name of the sender
|
sender_name: Name of the sender
|
||||||
receiver_name: Name of the receiver
|
receiver_name: Name of the receiver (empty string means broadcast)
|
||||||
receiver_id: UUID of the receiver
|
receiver_id: UUID of the receiver (empty string means broadcast)
|
||||||
reply_to: Topic to reply to
|
reply_to: Topic to reply to (empty string if no reply expected)
|
||||||
reply_to_msg_id: Message ID this message is replying to
|
reply_to_msg_id: Message ID this message is replying to
|
||||||
|
is_publish: Whether to automatically publish the message to NATS (default: True)
|
||||||
|
- When True: message is published to NATS
|
||||||
|
- When False: returns envelope and JSON string without publishing
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
MessageEnvelope: The envelope object for tracking
|
tuple: (env, env_json_str) where:
|
||||||
|
- env: MessageEnvelope object with all metadata and payloads
|
||||||
|
- env_json_str: JSON string representation of the envelope for publishing
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> data = [("message", "Hello World!", "text")]
|
||||||
|
>>> env, env_json_str = smartsend("/test", data)
|
||||||
|
>>> # env: MessageEnvelope with all metadata and payloads
|
||||||
|
>>> # env_json_str: JSON string for publishing
|
||||||
"""
|
"""
|
||||||
# Generate correlation ID if not provided
|
# Generate correlation ID if not provided
|
||||||
cid = correlation_id if correlation_id else str(uuid.uuid4())
|
cid = correlation_id if correlation_id is not None else str(uuid.uuid4())
|
||||||
|
|
||||||
log_trace(cid, "Starting smartsend for subject: {}".format(subject))
|
log_trace(cid, "Starting smartsend for subject: {}".format(subject))
|
||||||
|
|
||||||
@@ -479,16 +620,19 @@ def smartsend(subject, data, nats_url=DEFAULT_NATS_URL, fileserver_url=DEFAULT_F
|
|||||||
payload_bytes = _serialize_data(payload_data, payload_type)
|
payload_bytes = _serialize_data(payload_data, payload_type)
|
||||||
|
|
||||||
payload_size = len(payload_bytes)
|
payload_size = len(payload_bytes)
|
||||||
log_trace(cid, "Serialized payload '{}' (type: {}) size: {} bytes".format(
|
log_trace(cid, "Serialized payload '{}' (payload_type: {}) size: {} bytes".format(
|
||||||
dataname, payload_type, payload_size))
|
dataname, payload_type, payload_size))
|
||||||
|
|
||||||
# Decision: Direct vs Link
|
# Decision: Direct vs Link
|
||||||
if payload_size < size_threshold:
|
if payload_size < size_threshold:
|
||||||
# Direct path - Base64 encode and send via NATS
|
# Direct path - Base64 encode and send via NATS
|
||||||
payload_b64 = _serialize_data(payload_bytes, "binary") # Already bytes
|
|
||||||
# Convert to base64 string for JSON
|
# Convert to base64 string for JSON
|
||||||
|
try:
|
||||||
import ubinascii
|
import ubinascii
|
||||||
payload_b64_str = ubinascii.b2a_base64(payload_bytes).decode('utf-8').strip()
|
payload_b64_str = ubinascii.b2a_base64(payload_bytes).decode('utf-8').strip()
|
||||||
|
except ImportError:
|
||||||
|
import base64
|
||||||
|
payload_b64_str = base64.b64encode(payload_bytes).decode('utf-8')
|
||||||
|
|
||||||
log_trace(cid, "Using direct transport for {} bytes".format(payload_size))
|
log_trace(cid, "Using direct transport for {} bytes".format(payload_size))
|
||||||
|
|
||||||
@@ -511,10 +655,10 @@ def smartsend(subject, data, nats_url=DEFAULT_NATS_URL, fileserver_url=DEFAULT_F
|
|||||||
# Upload to HTTP server
|
# Upload to HTTP server
|
||||||
response = fileserver_upload_handler(fileserver_url, dataname, payload_bytes)
|
response = fileserver_upload_handler(fileserver_url, dataname, payload_bytes)
|
||||||
|
|
||||||
if response["status"] != 200:
|
if response.get("status") != 200:
|
||||||
raise Exception("Failed to upload data to fileserver: {}".format(response["status"]))
|
raise Exception("Failed to upload data to fileserver: {}".format(response.get("status")))
|
||||||
|
|
||||||
url = response["url"]
|
url = response.get("url")
|
||||||
log_trace(cid, "Uploaded to URL: {}".format(url))
|
log_trace(cid, "Uploaded to URL: {}".format(url))
|
||||||
|
|
||||||
# Create MessagePayload for link transport
|
# Create MessagePayload for link transport
|
||||||
@@ -543,19 +687,21 @@ def smartsend(subject, data, nats_url=DEFAULT_NATS_URL, fileserver_url=DEFAULT_F
|
|||||||
receiver_id=receiver_id,
|
receiver_id=receiver_id,
|
||||||
reply_to=reply_to,
|
reply_to=reply_to,
|
||||||
reply_to_msg_id=reply_to_msg_id,
|
reply_to_msg_id=reply_to_msg_id,
|
||||||
broker_url=nats_url,
|
broker_url=broker_url,
|
||||||
metadata={}
|
metadata={}
|
||||||
)
|
)
|
||||||
|
|
||||||
msg_json = env.to_json()
|
msg_json = env.to_json()
|
||||||
|
|
||||||
# Publish to NATS
|
# Publish to NATS if is_publish is True
|
||||||
nats_conn = NATSConnection(nats_url)
|
if is_publish:
|
||||||
|
nats_conn = NATSConnection(broker_url)
|
||||||
nats_conn.connect()
|
nats_conn.connect()
|
||||||
nats_conn.publish(subject, msg_json)
|
nats_conn.publish(subject, msg_json)
|
||||||
nats_conn.close()
|
nats_conn.close()
|
||||||
|
|
||||||
return env
|
# Return tuple of (envelope, json_string) for both direct and link transport
|
||||||
|
return (env, msg_json)
|
||||||
|
|
||||||
|
|
||||||
def smartreceive(msg, fileserver_download_handler=_fetch_with_backoff, max_retries=5,
|
def smartreceive(msg, fileserver_download_handler=_fetch_with_backoff, max_retries=5,
|
||||||
@@ -566,18 +712,29 @@ def smartreceive(msg, fileserver_download_handler=_fetch_with_backoff, max_retri
|
|||||||
(base64 decoded payloads) and link transport (URL-based payloads).
|
(base64 decoded payloads) and link transport (URL-based payloads).
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
msg: NATS message to process (dict with payload data)
|
msg: NATS message to process (dict or JSON string with envelope data)
|
||||||
fileserver_download_handler: Function to handle downloading data from file server URLs
|
fileserver_download_handler: Function to handle downloading data from file server URLs
|
||||||
max_retries: Maximum retry attempts for fetching URL
|
Receives: (url, max_retries, base_delay, max_delay, correlation_id)
|
||||||
base_delay: Initial delay for exponential backoff in ms
|
Returns: bytes (the downloaded data)
|
||||||
max_delay: Maximum delay for exponential backoff in ms
|
max_retries: Maximum retry attempts for fetching URL (default: 5)
|
||||||
|
base_delay: Initial delay for exponential backoff in ms (default: 100)
|
||||||
|
max_delay: Maximum delay for exponential backoff in ms (default: 5000)
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
list: List of (dataname, data, type) tuples
|
dict: Envelope dictionary with metadata and 'payloads' field containing list of
|
||||||
|
(dataname, data, payload_type) tuples
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> env = smartreceive(msg)
|
||||||
|
>>> # env contains envelope metadata and payloads field
|
||||||
|
>>> # env["payloads"] = [(dataname1, data1, payload_type1), ...]
|
||||||
|
>>> for dataname, data, payload_type in env["payloads"]:
|
||||||
|
... print("Received {} of type {}: {}".format(dataname, payload_type, data))
|
||||||
"""
|
"""
|
||||||
# Parse the JSON envelope
|
# Parse the JSON envelope
|
||||||
json_data = msg if isinstance(msg, dict) else json.loads(msg)
|
json_data = msg if isinstance(msg, dict) else json.loads(msg)
|
||||||
log_trace(json_data.get("correlationId", ""), "Processing received message")
|
correlation_id = json_data.get("correlation_id", "")
|
||||||
|
log_trace(correlation_id, "Processing received message")
|
||||||
|
|
||||||
# Process all payloads in the envelope
|
# Process all payloads in the envelope
|
||||||
payloads_list = []
|
payloads_list = []
|
||||||
@@ -591,43 +748,50 @@ def smartreceive(msg, fileserver_download_handler=_fetch_with_backoff, max_retri
|
|||||||
dataname = payload.get("dataname", "")
|
dataname = payload.get("dataname", "")
|
||||||
|
|
||||||
if transport == "direct":
|
if transport == "direct":
|
||||||
log_trace(json_data.get("correlationId", ""),
|
log_trace(correlation_id,
|
||||||
"Direct transport - decoding payload '{}'".format(dataname))
|
"Direct transport - decoding payload '{}'".format(dataname))
|
||||||
|
|
||||||
# Extract base64 payload from the payload
|
# Extract base64 payload from the payload
|
||||||
payload_b64 = payload.get("data", "")
|
payload_b64 = payload.get("data", "")
|
||||||
|
|
||||||
# Decode Base64 payload
|
# Decode Base64 payload
|
||||||
|
try:
|
||||||
import ubinascii
|
import ubinascii
|
||||||
payload_bytes = ubinascii.a2b_base64(payload_b64.encode('utf-8'))
|
payload_bytes = ubinascii.a2b_base64(payload_b64.encode('utf-8'))
|
||||||
|
except ImportError:
|
||||||
|
import base64
|
||||||
|
payload_bytes = base64.b64decode(payload_b64)
|
||||||
|
|
||||||
# Deserialize based on type
|
# Deserialize based on type
|
||||||
data_type = payload.get("type", "")
|
payload_type = payload.get("payload_type", "")
|
||||||
data = _deserialize_data(payload_bytes, data_type, json_data.get("correlationId", ""))
|
data = _deserialize_data(payload_bytes, payload_type, correlation_id)
|
||||||
|
|
||||||
payloads_list.append((dataname, data, data_type))
|
payloads_list.append((dataname, data, payload_type))
|
||||||
|
|
||||||
elif transport == "link":
|
elif transport == "link":
|
||||||
# Extract download URL from the payload
|
# Extract download URL from the payload
|
||||||
url = payload.get("data", "")
|
url = payload.get("data", "")
|
||||||
log_trace(json_data.get("correlationId", ""),
|
log_trace(correlation_id,
|
||||||
"Link transport - fetching '{}' from URL: {}".format(dataname, url))
|
"Link transport - fetching '{}' from URL: {}".format(dataname, url))
|
||||||
|
|
||||||
# Fetch with exponential backoff
|
# Fetch with exponential backoff
|
||||||
downloaded_data = fileserver_download_handler(
|
downloaded_data = fileserver_download_handler(
|
||||||
url, max_retries, base_delay, max_delay, json_data.get("correlationId", "")
|
url, max_retries, base_delay, max_delay, correlation_id
|
||||||
)
|
)
|
||||||
|
|
||||||
# Deserialize based on type
|
# Deserialize based on type
|
||||||
data_type = payload.get("type", "")
|
payload_type = payload.get("payload_type", "")
|
||||||
data = _deserialize_data(downloaded_data, data_type, json_data.get("correlationId", ""))
|
data = _deserialize_data(downloaded_data, payload_type, correlation_id)
|
||||||
|
|
||||||
payloads_list.append((dataname, data, data_type))
|
payloads_list.append((dataname, data, payload_type))
|
||||||
|
|
||||||
else:
|
else:
|
||||||
raise ValueError("Unknown transport type for payload '{}': {}".format(dataname, transport))
|
raise ValueError("Unknown transport type for payload '{}': {}".format(dataname, transport))
|
||||||
|
|
||||||
return payloads_list
|
# Replace payloads field with the processed list of (dataname, data, payload_type) tuples
|
||||||
|
json_data["payloads"] = payloads_list
|
||||||
|
|
||||||
|
return json_data
|
||||||
|
|
||||||
|
|
||||||
# Utility functions
|
# Utility functions
|
||||||
@@ -643,11 +807,11 @@ def get_timestamp():
|
|||||||
|
|
||||||
# Example usage
|
# Example usage
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
print("NATSBridge for Micropython")
|
print("NATSBridge - Bi-Directional Data Bridge")
|
||||||
print("=========================")
|
print("=======================================")
|
||||||
print("This module provides:")
|
print("This module provides:")
|
||||||
print(" - MessageEnvelope: Message envelope structure")
|
print(" - MessageEnvelope: Message envelope structure with snake_case fields")
|
||||||
print(" - MessagePayload: Payload structure")
|
print(" - MessagePayload: Payload structure with payload_type field")
|
||||||
print(" - smartsend: Send data via NATS with automatic transport selection")
|
print(" - smartsend: Send data via NATS with automatic transport selection")
|
||||||
print(" - smartreceive: Receive and process messages from NATS")
|
print(" - smartreceive: Receive and process messages from NATS")
|
||||||
print(" - plik_oneshot_upload: Upload files to HTTP file server")
|
print(" - plik_oneshot_upload: Upload files to HTTP file server")
|
||||||
@@ -655,10 +819,12 @@ if __name__ == "__main__":
|
|||||||
print()
|
print()
|
||||||
print("Usage:")
|
print("Usage:")
|
||||||
print(" from nats_bridge import smartsend, smartreceive")
|
print(" from nats_bridge import smartsend, smartreceive")
|
||||||
print(" data = [(\"message\", \"Hello World\", \"text\")]")
|
print()
|
||||||
print(" env = smartsend(\"my.subject\", data)")
|
print(" # Send data (list of (dataname, data, payload_type) tuples)")
|
||||||
|
print(" data = [(\"message\", \"Hello World!\", \"text\")]")
|
||||||
|
print(" env, env_json_str = smartsend(\"my.subject\", data)")
|
||||||
print()
|
print()
|
||||||
print(" # On receiver:")
|
print(" # On receiver:")
|
||||||
print(" payloads = smartreceive(msg)")
|
print(" env = smartreceive(msg)")
|
||||||
print(" for dataname, data, type in payloads:")
|
print(" for dataname, data, payload_type in env[\"payloads\"]:")
|
||||||
print(" print(f\"Received {dataname} of type {type}: {data}\")")
|
print(" print(\"Received {} of type {}: {}\".format(dataname, payload_type, data))")
|
||||||
@@ -37,8 +37,9 @@ async function test_dict_receive() {
|
|||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
// Result is a list of {dataname, data, type} objects
|
// Result is an envelope dictionary with payloads field
|
||||||
for (const { dataname, data, type } of result) {
|
// Access payloads with result.payloads
|
||||||
|
for (const { dataname, data, type } of result.payloads) {
|
||||||
if (typeof data === 'object' && data !== null && !Array.isArray(data)) {
|
if (typeof data === 'object' && data !== null && !Array.isArray(data)) {
|
||||||
log_trace(`Received Dictionary '${dataname}' of type ${type}`);
|
log_trace(`Received Dictionary '${dataname}' of type ${type}`);
|
||||||
|
|
||||||
@@ -118,7 +118,7 @@ async function test_dict_send() {
|
|||||||
// Use smartsend with dictionary type
|
// Use smartsend with dictionary type
|
||||||
// For small Dictionary: will use direct transport (JSON encoded)
|
// For small Dictionary: will use direct transport (JSON encoded)
|
||||||
// For large Dictionary: will use link transport (uploaded to fileserver)
|
// For large Dictionary: will use link transport (uploaded to fileserver)
|
||||||
const env = await smartsend(
|
const { env, env_json_str } = await smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2],
|
[data1, data2],
|
||||||
{
|
{
|
||||||
@@ -132,7 +132,8 @@ async function test_dict_send() {
|
|||||||
receiverName: "",
|
receiverName: "",
|
||||||
receiverId: "",
|
receiverId: "",
|
||||||
replyTo: "",
|
replyTo: "",
|
||||||
replyToMsgId: ""
|
replyToMsgId: "",
|
||||||
|
isPublish: true // Publish the message to NATS
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -36,8 +36,9 @@ async function test_large_binary_receive() {
|
|||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
// Result is a list of {dataname, data, type} objects
|
// Result is an envelope dictionary with payloads field
|
||||||
for (const { dataname, data, type } of result) {
|
// Access payloads with result.payloads
|
||||||
|
for (const { dataname, data, type } of result.payloads) {
|
||||||
if (data instanceof Uint8Array || Array.isArray(data)) {
|
if (data instanceof Uint8Array || Array.isArray(data)) {
|
||||||
const file_size = data.length;
|
const file_size = data.length;
|
||||||
log_trace(`Received ${file_size} bytes of binary data for '${dataname}' of type ${type}`);
|
log_trace(`Received ${file_size} bytes of binary data for '${dataname}' of type ${type}`);
|
||||||
@@ -98,7 +98,7 @@ async function test_large_binary_send() {
|
|||||||
|
|
||||||
// Use smartsend with binary type - will automatically use link transport
|
// Use smartsend with binary type - will automatically use link transport
|
||||||
// if file size exceeds the threshold (1MB by default)
|
// if file size exceeds the threshold (1MB by default)
|
||||||
const env = await smartsend(
|
const { env, env_json_str } = await smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2],
|
[data1, data2],
|
||||||
{
|
{
|
||||||
@@ -112,7 +112,8 @@ async function test_large_binary_send() {
|
|||||||
receiverName: "",
|
receiverName: "",
|
||||||
receiverId: "",
|
receiverId: "",
|
||||||
replyTo: "",
|
replyTo: "",
|
||||||
replyToMsgId: ""
|
replyToMsgId: "",
|
||||||
|
isPublish: true // Publish the message to NATS
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -222,7 +222,7 @@ async function test_mix_send() {
|
|||||||
];
|
];
|
||||||
|
|
||||||
// Use smartsend with mixed content
|
// Use smartsend with mixed content
|
||||||
const env = await smartsend(
|
const { env, env_json_str } = await smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
payloads,
|
payloads,
|
||||||
{
|
{
|
||||||
@@ -236,7 +236,8 @@ async function test_mix_send() {
|
|||||||
receiverName: "",
|
receiverName: "",
|
||||||
receiverId: "",
|
receiverId: "",
|
||||||
replyTo: "",
|
replyTo: "",
|
||||||
replyToMsgId: ""
|
replyToMsgId: "",
|
||||||
|
isPublish: true // Publish the message to NATS
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -40,10 +40,11 @@ async function test_mix_receive() {
|
|||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
log_trace(`Received ${result.length} payloads`);
|
log_trace(`Received ${result.payloads.length} payloads`);
|
||||||
|
|
||||||
// Result is a list of {dataname, data, type} objects
|
// Result is an envelope dictionary with payloads field
|
||||||
for (const { dataname, data, type } of result) {
|
// Access payloads with result.payloads
|
||||||
|
for (const { dataname, data, type } of result.payloads) {
|
||||||
log_trace(`\n=== Payload: ${dataname} (type: ${type}) ===`);
|
log_trace(`\n=== Payload: ${dataname} (type: ${type}) ===`);
|
||||||
|
|
||||||
// Handle different data types
|
// Handle different data types
|
||||||
@@ -122,13 +123,13 @@ async function test_mix_receive() {
|
|||||||
|
|
||||||
// Summary
|
// Summary
|
||||||
console.log("\n=== Verification Summary ===");
|
console.log("\n=== Verification Summary ===");
|
||||||
const text_count = result.filter(x => x.type === "text").length;
|
const text_count = result.payloads.filter(x => x.type === "text").length;
|
||||||
const dict_count = result.filter(x => x.type === "dictionary").length;
|
const dict_count = result.payloads.filter(x => x.type === "dictionary").length;
|
||||||
const table_count = result.filter(x => x.type === "table").length;
|
const table_count = result.payloads.filter(x => x.type === "table").length;
|
||||||
const image_count = result.filter(x => x.type === "image").length;
|
const image_count = result.payloads.filter(x => x.type === "image").length;
|
||||||
const audio_count = result.filter(x => x.type === "audio").length;
|
const audio_count = result.payloads.filter(x => x.type === "audio").length;
|
||||||
const video_count = result.filter(x => x.type === "video").length;
|
const video_count = result.payloads.filter(x => x.type === "video").length;
|
||||||
const binary_count = result.filter(x => x.type === "binary").length;
|
const binary_count = result.payloads.filter(x => x.type === "binary").length;
|
||||||
|
|
||||||
log_trace(`Text payloads: ${text_count}`);
|
log_trace(`Text payloads: ${text_count}`);
|
||||||
log_trace(`Dictionary payloads: ${dict_count}`);
|
log_trace(`Dictionary payloads: ${dict_count}`);
|
||||||
@@ -140,7 +141,7 @@ async function test_mix_receive() {
|
|||||||
|
|
||||||
// Print transport type info for each payload if available
|
// Print transport type info for each payload if available
|
||||||
console.log("\n=== Payload Details ===");
|
console.log("\n=== Payload Details ===");
|
||||||
for (const { dataname, data, type } of result) {
|
for (const { dataname, data, type } of result.payloads) {
|
||||||
if (["image", "audio", "video", "binary"].includes(type)) {
|
if (["image", "audio", "video", "binary"].includes(type)) {
|
||||||
log_trace(`${dataname}: ${data.length} bytes (binary)`);
|
log_trace(`${dataname}: ${data.length} bytes (binary)`);
|
||||||
} else if (type === "table") {
|
} else if (type === "table") {
|
||||||
@@ -40,8 +40,9 @@ async function test_table_receive() {
|
|||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
// Result is a list of {dataname, data, type} objects
|
// Result is an envelope dictionary with payloads field
|
||||||
for (const { dataname, data, type } of result) {
|
// Access payloads with result.payloads
|
||||||
|
for (const { dataname, data, type } of result.payloads) {
|
||||||
if (Array.isArray(data)) {
|
if (Array.isArray(data)) {
|
||||||
log_trace(`Received Table '${dataname}' of type ${type}`);
|
log_trace(`Received Table '${dataname}' of type ${type}`);
|
||||||
|
|
||||||
@@ -118,7 +118,7 @@ async function test_table_send() {
|
|||||||
// Use smartsend with table type
|
// Use smartsend with table type
|
||||||
// For small Table: will use direct transport (Arrow IPC encoded)
|
// For small Table: will use direct transport (Arrow IPC encoded)
|
||||||
// For large Table: will use link transport (uploaded to fileserver)
|
// For large Table: will use link transport (uploaded to fileserver)
|
||||||
const env = await smartsend(
|
const { env, env_json_str } = await smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2],
|
[data1, data2],
|
||||||
{
|
{
|
||||||
@@ -132,7 +132,8 @@ async function test_table_send() {
|
|||||||
receiverName: "",
|
receiverName: "",
|
||||||
receiverId: "",
|
receiverId: "",
|
||||||
replyTo: "",
|
replyTo: "",
|
||||||
replyToMsgId: ""
|
replyToMsgId: "",
|
||||||
|
isPublish: true // Publish the message to NATS
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -37,8 +37,9 @@ async function test_text_receive() {
|
|||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
// Result is a list of {dataname, data, type} objects
|
// Result is an envelope dictionary with payloads field
|
||||||
for (const { dataname, data, type } of result) {
|
// Access payloads with result.payloads
|
||||||
|
for (const { dataname, data, type } of result.payloads) {
|
||||||
if (typeof data === 'string') {
|
if (typeof data === 'string') {
|
||||||
log_trace(`Received text '${dataname}' of type ${type}`);
|
log_trace(`Received text '${dataname}' of type ${type}`);
|
||||||
log_trace(` Length: ${data.length} characters`);
|
log_trace(` Length: ${data.length} characters`);
|
||||||
@@ -94,7 +94,7 @@ async function test_text_send() {
|
|||||||
// Use smartsend with text type
|
// Use smartsend with text type
|
||||||
// For small text: will use direct transport (Base64 encoded UTF-8)
|
// For small text: will use direct transport (Base64 encoded UTF-8)
|
||||||
// For large text: will use link transport (uploaded to fileserver)
|
// For large text: will use link transport (uploaded to fileserver)
|
||||||
const env = await smartsend(
|
const { env, env_json_str } = await smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2],
|
[data1, data2],
|
||||||
{
|
{
|
||||||
@@ -108,7 +108,8 @@ async function test_text_send() {
|
|||||||
receiverName: "",
|
receiverName: "",
|
||||||
receiverId: "",
|
receiverId: "",
|
||||||
replyTo: "",
|
replyTo: "",
|
||||||
replyToMsgId: ""
|
replyToMsgId: "",
|
||||||
|
isPublish: true // Publish the message to NATS
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -42,8 +42,8 @@ function test_dict_receive()
|
|||||||
max_delay = 5000
|
max_delay = 5000
|
||||||
)
|
)
|
||||||
|
|
||||||
# Result is a list of (dataname, data, data_type) tuples
|
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
|
||||||
for (dataname, data, data_type) in result
|
for (dataname, data, data_type) in result["payloads"]
|
||||||
if isa(data, JSON.Object{String, Any})
|
if isa(data, JSON.Object{String, Any})
|
||||||
log_trace("Received Dictionary '$dataname' of type $data_type")
|
log_trace("Received Dictionary '$dataname' of type $data_type")
|
||||||
|
|
||||||
@@ -92,12 +92,12 @@ function test_dict_send()
|
|||||||
# Use smartsend with dictionary type
|
# Use smartsend with dictionary type
|
||||||
# For small Dictionary: will use direct transport (JSON encoded)
|
# For small Dictionary: will use direct transport (JSON encoded)
|
||||||
# For large Dictionary: will use link transport (uploaded to fileserver)
|
# For large Dictionary: will use link transport (uploaded to fileserver)
|
||||||
env = NATSBridge.smartsend(
|
env, env_json_str = NATSBridge.smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2], # List of (dataname, data, type) tuples
|
[data1, data2]; # List of (dataname, data, type) tuples
|
||||||
nats_url = NATS_URL,
|
broker_url = NATS_URL,
|
||||||
fileserver_url = FILESERVER_URL,
|
fileserver_url = FILESERVER_URL,
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
fileserver_upload_handler = plik_upload_handler,
|
||||||
size_threshold = 1_000_000, # 1MB threshold
|
size_threshold = 1_000_000, # 1MB threshold
|
||||||
correlation_id = correlation_id,
|
correlation_id = correlation_id,
|
||||||
msg_purpose = "chat",
|
msg_purpose = "chat",
|
||||||
@@ -105,7 +105,8 @@ function test_dict_send()
|
|||||||
receiver_name = "",
|
receiver_name = "",
|
||||||
receiver_id = "",
|
receiver_id = "",
|
||||||
reply_to = "",
|
reply_to = "",
|
||||||
reply_to_msg_id = ""
|
reply_to_msg_id = "",
|
||||||
|
is_publish = true # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace("Sent message with $(length(env.payloads)) payloads")
|
log_trace("Sent message with $(length(env.payloads)) payloads")
|
||||||
@@ -114,7 +115,7 @@ function test_dict_send()
|
|||||||
for (i, payload) in enumerate(env.payloads)
|
for (i, payload) in enumerate(env.payloads)
|
||||||
log_trace("Payload $i ('$payload.dataname'):")
|
log_trace("Payload $i ('$payload.dataname'):")
|
||||||
log_trace(" Transport: $(payload.transport)")
|
log_trace(" Transport: $(payload.transport)")
|
||||||
log_trace(" Type: $(payload.type)")
|
log_trace(" Type: $(payload.payload_type)")
|
||||||
log_trace(" Size: $(payload.size) bytes")
|
log_trace(" Size: $(payload.size) bytes")
|
||||||
log_trace(" Encoding: $(payload.encoding)")
|
log_trace(" Encoding: $(payload.encoding)")
|
||||||
|
|
||||||
@@ -44,8 +44,8 @@ function test_large_binary_receive()
|
|||||||
max_delay = 5000
|
max_delay = 5000
|
||||||
)
|
)
|
||||||
|
|
||||||
# Result is a list of (dataname, data) tuples
|
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
|
||||||
for (dataname, data, data_type) in result
|
for (dataname, data, data_type) in result["payloads"]
|
||||||
# Check transport type from the envelope
|
# Check transport type from the envelope
|
||||||
# For link transport, data is the URL string
|
# For link transport, data is the URL string
|
||||||
# For direct transport, data is the actual payload bytes
|
# For direct transport, data is the actual payload bytes
|
||||||
@@ -79,12 +79,12 @@ function test_large_binary_send()
|
|||||||
# Use smartsend with binary type - will automatically use link transport
|
# Use smartsend with binary type - will automatically use link transport
|
||||||
# if file size exceeds the threshold (1MB by default)
|
# if file size exceeds the threshold (1MB by default)
|
||||||
# API: smartsend(subject, [(dataname, data, type), ...]; keywords...)
|
# API: smartsend(subject, [(dataname, data, type), ...]; keywords...)
|
||||||
env = NATSBridge.smartsend(
|
env, env_json_str = NATSBridge.smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2], # List of (dataname, data, type) tuples
|
[data1, data2]; # List of (dataname, data, type) tuples
|
||||||
nats_url = NATS_URL;
|
broker_url = NATS_URL;
|
||||||
fileserver_url = FILESERVER_URL,
|
fileserver_url = FILESERVER_URL,
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
fileserver_upload_handler = plik_upload_handler,
|
||||||
size_threshold = 1_000_000,
|
size_threshold = 1_000_000,
|
||||||
correlation_id = correlation_id,
|
correlation_id = correlation_id,
|
||||||
msg_purpose = "chat",
|
msg_purpose = "chat",
|
||||||
@@ -92,11 +92,12 @@ function test_large_binary_send()
|
|||||||
receiver_name = "",
|
receiver_name = "",
|
||||||
receiver_id = "",
|
receiver_id = "",
|
||||||
reply_to = "",
|
reply_to = "",
|
||||||
reply_to_msg_id = ""
|
reply_to_msg_id = "",
|
||||||
|
is_publish = true # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace("Sent message with transport: $(env.payloads[1].transport)")
|
log_trace("Sent message with transport: $(env.payloads[1].transport)")
|
||||||
log_trace("Envelope type: $(env.payloads[1].type)")
|
log_trace("Envelope type: $(env.payloads[1].payload_type)")
|
||||||
|
|
||||||
# Check if link transport was used
|
# Check if link transport was used
|
||||||
if env.payloads[1].transport == "link"
|
if env.payloads[1].transport == "link"
|
||||||
@@ -45,10 +45,10 @@ function test_mix_receive()
|
|||||||
max_delay = 5000
|
max_delay = 5000
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace("Received $(length(result)) payloads")
|
log_trace("Received $(length(result["payloads"])) payloads")
|
||||||
|
|
||||||
# Result is a list of (dataname, data, data_type) tuples
|
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
|
||||||
for (dataname, data, data_type) in result
|
for (dataname, data, data_type) in result["payloads"]
|
||||||
log_trace("\n=== Payload: $dataname (type: $data_type) ===")
|
log_trace("\n=== Payload: $dataname (type: $data_type) ===")
|
||||||
|
|
||||||
# Handle different data types
|
# Handle different data types
|
||||||
@@ -178,13 +178,13 @@ function test_mix_receive()
|
|||||||
|
|
||||||
# Summary
|
# Summary
|
||||||
println("\n=== Verification Summary ===")
|
println("\n=== Verification Summary ===")
|
||||||
text_count = count(x -> x[3] == "text", result)
|
text_count = count(x -> x[3] == "text", result["payloads"])
|
||||||
dict_count = count(x -> x[3] == "dictionary", result)
|
dict_count = count(x -> x[3] == "dictionary", result["payloads"])
|
||||||
table_count = count(x -> x[3] == "table", result)
|
table_count = count(x -> x[3] == "table", result["payloads"])
|
||||||
image_count = count(x -> x[3] == "image", result)
|
image_count = count(x -> x[3] == "image", result["payloads"])
|
||||||
audio_count = count(x -> x[3] == "audio", result)
|
audio_count = count(x -> x[3] == "audio", result["payloads"])
|
||||||
video_count = count(x -> x[3] == "video", result)
|
video_count = count(x -> x[3] == "video", result["payloads"])
|
||||||
binary_count = count(x -> x[3] == "binary", result)
|
binary_count = count(x -> x[3] == "binary", result["payloads"])
|
||||||
|
|
||||||
log_trace("Text payloads: $text_count")
|
log_trace("Text payloads: $text_count")
|
||||||
log_trace("Dictionary payloads: $dict_count")
|
log_trace("Dictionary payloads: $dict_count")
|
||||||
@@ -196,7 +196,7 @@ function test_mix_receive()
|
|||||||
|
|
||||||
# Print transport type info for each payload if available
|
# Print transport type info for each payload if available
|
||||||
println("\n=== Payload Details ===")
|
println("\n=== Payload Details ===")
|
||||||
for (dataname, data, data_type) in result
|
for (dataname, data, data_type) in result["payloads"]
|
||||||
if data_type in ["image", "audio", "video", "binary"]
|
if data_type in ["image", "audio", "video", "binary"]
|
||||||
log_trace("$dataname: $(length(data)) bytes (binary)")
|
log_trace("$dataname: $(length(data)) bytes (binary)")
|
||||||
elseif data_type == "table"
|
elseif data_type == "table"
|
||||||
@@ -186,12 +186,12 @@ function test_mix_send()
|
|||||||
]
|
]
|
||||||
|
|
||||||
# Use smartsend with mixed content
|
# Use smartsend with mixed content
|
||||||
env = NATSBridge.smartsend(
|
env, env_json_str = NATSBridge.smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
payloads, # List of (dataname, data, type) tuples
|
payloads; # List of (dataname, data, type) tuples
|
||||||
nats_url = NATS_URL,
|
broker_url = NATS_URL,
|
||||||
fileserver_url = FILESERVER_URL,
|
fileserver_url = FILESERVER_URL,
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
fileserver_upload_handler = plik_upload_handler,
|
||||||
size_threshold = 1_000_000, # 1MB threshold
|
size_threshold = 1_000_000, # 1MB threshold
|
||||||
correlation_id = correlation_id,
|
correlation_id = correlation_id,
|
||||||
msg_purpose = "chat",
|
msg_purpose = "chat",
|
||||||
@@ -199,7 +199,8 @@ function test_mix_send()
|
|||||||
receiver_name = "",
|
receiver_name = "",
|
||||||
receiver_id = "",
|
receiver_id = "",
|
||||||
reply_to = "",
|
reply_to = "",
|
||||||
reply_to_msg_id = ""
|
reply_to_msg_id = "",
|
||||||
|
is_publish = true # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace("Sent message with $(length(env.payloads)) payloads")
|
log_trace("Sent message with $(length(env.payloads)) payloads")
|
||||||
@@ -208,7 +209,7 @@ function test_mix_send()
|
|||||||
for (i, payload) in enumerate(env.payloads)
|
for (i, payload) in enumerate(env.payloads)
|
||||||
log_trace("Payload $i ('$payload.dataname'):")
|
log_trace("Payload $i ('$payload.dataname'):")
|
||||||
log_trace(" Transport: $(payload.transport)")
|
log_trace(" Transport: $(payload.transport)")
|
||||||
log_trace(" Type: $(payload.type)")
|
log_trace(" Type: $(payload.payload_type)")
|
||||||
log_trace(" Size: $(payload.size) bytes")
|
log_trace(" Size: $(payload.size) bytes")
|
||||||
log_trace(" Encoding: $(payload.encoding)")
|
log_trace(" Encoding: $(payload.encoding)")
|
||||||
|
|
||||||
@@ -42,8 +42,8 @@ function test_table_receive()
|
|||||||
max_delay = 5000
|
max_delay = 5000
|
||||||
)
|
)
|
||||||
|
|
||||||
# Result is a list of (dataname, data, data_type) tuples
|
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
|
||||||
for (dataname, data, data_type) in result
|
for (dataname, data, data_type) in result["payloads"]
|
||||||
data = DataFrame(data)
|
data = DataFrame(data)
|
||||||
if isa(data, DataFrame)
|
if isa(data, DataFrame)
|
||||||
log_trace("Received DataFrame '$dataname' of type $data_type")
|
log_trace("Received DataFrame '$dataname' of type $data_type")
|
||||||
@@ -90,12 +90,12 @@ function test_table_send()
|
|||||||
# Use smartsend with table type
|
# Use smartsend with table type
|
||||||
# For small DataFrame: will use direct transport (Base64 encoded Arrow IPC)
|
# For small DataFrame: will use direct transport (Base64 encoded Arrow IPC)
|
||||||
# For large DataFrame: will use link transport (uploaded to fileserver)
|
# For large DataFrame: will use link transport (uploaded to fileserver)
|
||||||
env = NATSBridge.smartsend(
|
env, env_json_str = NATSBridge.smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2], # List of (dataname, data, type) tuples
|
[data1, data2]; # List of (dataname, data, type) tuples
|
||||||
nats_url = NATS_URL,
|
broker_url = NATS_URL,
|
||||||
fileserver_url = FILESERVER_URL,
|
fileserver_url = FILESERVER_URL,
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
fileserver_upload_handler = plik_upload_handler,
|
||||||
size_threshold = 1_000_000, # 1MB threshold
|
size_threshold = 1_000_000, # 1MB threshold
|
||||||
correlation_id = correlation_id,
|
correlation_id = correlation_id,
|
||||||
msg_purpose = "chat",
|
msg_purpose = "chat",
|
||||||
@@ -103,7 +103,8 @@ function test_table_send()
|
|||||||
receiver_name = "",
|
receiver_name = "",
|
||||||
receiver_id = "",
|
receiver_id = "",
|
||||||
reply_to = "",
|
reply_to = "",
|
||||||
reply_to_msg_id = ""
|
reply_to_msg_id = "",
|
||||||
|
is_publish = true # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace("Sent message with $(length(env.payloads)) payloads")
|
log_trace("Sent message with $(length(env.payloads)) payloads")
|
||||||
@@ -112,7 +113,7 @@ function test_table_send()
|
|||||||
for (i, payload) in enumerate(env.payloads)
|
for (i, payload) in enumerate(env.payloads)
|
||||||
log_trace("Payload $i ('$payload.dataname'):")
|
log_trace("Payload $i ('$payload.dataname'):")
|
||||||
log_trace(" Transport: $(payload.transport)")
|
log_trace(" Transport: $(payload.transport)")
|
||||||
log_trace(" Type: $(payload.type)")
|
log_trace(" Type: $(payload.payload_type)")
|
||||||
log_trace(" Size: $(payload.size) bytes")
|
log_trace(" Size: $(payload.size) bytes")
|
||||||
log_trace(" Encoding: $(payload.encoding)")
|
log_trace(" Encoding: $(payload.encoding)")
|
||||||
|
|
||||||
@@ -42,8 +42,8 @@ function test_text_receive()
|
|||||||
max_delay = 5000
|
max_delay = 5000
|
||||||
)
|
)
|
||||||
|
|
||||||
# Result is a list of (dataname, data, data_type) tuples
|
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
|
||||||
for (dataname, data, data_type) in result
|
for (dataname, data, data_type) in result["payloads"]
|
||||||
if isa(data, String)
|
if isa(data, String)
|
||||||
log_trace("Received text '$dataname' of type $data_type")
|
log_trace("Received text '$dataname' of type $data_type")
|
||||||
log_trace(" Length: $(length(data)) characters")
|
log_trace(" Length: $(length(data)) characters")
|
||||||
@@ -75,12 +75,12 @@ function test_text_send()
|
|||||||
# Use smartsend with text type
|
# Use smartsend with text type
|
||||||
# For small text: will use direct transport (Base64 encoded UTF-8)
|
# For small text: will use direct transport (Base64 encoded UTF-8)
|
||||||
# For large text: will use link transport (uploaded to fileserver)
|
# For large text: will use link transport (uploaded to fileserver)
|
||||||
env = NATSBridge.smartsend(
|
env, env_json_str = NATSBridge.smartsend(
|
||||||
SUBJECT,
|
SUBJECT,
|
||||||
[data1, data2], # List of (dataname, data, type) tuples
|
[data1, data2]; # List of (dataname, data, type) tuples
|
||||||
nats_url = NATS_URL,
|
broker_url = NATS_URL,
|
||||||
fileserver_url = FILESERVER_URL,
|
fileserver_url = FILESERVER_URL,
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
fileserver_upload_handler = plik_upload_handler,
|
||||||
size_threshold = 1_000_000, # 1MB threshold
|
size_threshold = 1_000_000, # 1MB threshold
|
||||||
correlation_id = correlation_id,
|
correlation_id = correlation_id,
|
||||||
msg_purpose = "chat",
|
msg_purpose = "chat",
|
||||||
@@ -88,7 +88,8 @@ function test_text_send()
|
|||||||
receiver_name = "",
|
receiver_name = "",
|
||||||
receiver_id = "",
|
receiver_id = "",
|
||||||
reply_to = "",
|
reply_to = "",
|
||||||
reply_to_msg_id = ""
|
reply_to_msg_id = "",
|
||||||
|
is_publish = true # Publish the message to NATS
|
||||||
)
|
)
|
||||||
|
|
||||||
log_trace("Sent message with $(length(env.payloads)) payloads")
|
log_trace("Sent message with $(length(env.payloads)) payloads")
|
||||||
@@ -97,7 +98,7 @@ function test_text_send()
|
|||||||
for (i, payload) in enumerate(env.payloads)
|
for (i, payload) in enumerate(env.payloads)
|
||||||
log_trace("Payload $i ('$payload.dataname'):")
|
log_trace("Payload $i ('$payload.dataname'):")
|
||||||
log_trace(" Transport: $(payload.transport)")
|
log_trace(" Transport: $(payload.transport)")
|
||||||
log_trace(" Type: $(payload.type)")
|
log_trace(" Type: $(payload.payload_type)")
|
||||||
log_trace(" Size: $(payload.size) bytes")
|
log_trace(" Size: $(payload.size) bytes")
|
||||||
log_trace(" Encoding: $(payload.encoding)")
|
log_trace(" Encoding: $(payload.encoding)")
|
||||||
|
|
||||||
@@ -1,220 +1,207 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
"""
|
"""
|
||||||
Micropython NATS Bridge - Basic Test Examples
|
Basic functionality test for nats_bridge.py
|
||||||
|
Tests the core classes and functions without NATS connection
|
||||||
This module demonstrates basic usage of the NATSBridge for Micropython.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import sys
|
import sys
|
||||||
sys.path.insert(0, "../src")
|
import os
|
||||||
|
|
||||||
from nats_bridge import MessageEnvelope, MessagePayload, smartsend, smartreceive, log_trace
|
# Add src to path for import
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||||
|
|
||||||
|
from nats_bridge import (
|
||||||
|
MessagePayload,
|
||||||
|
MessageEnvelope,
|
||||||
|
smartsend,
|
||||||
|
smartreceive,
|
||||||
|
log_trace,
|
||||||
|
generate_uuid,
|
||||||
|
get_timestamp,
|
||||||
|
_serialize_data,
|
||||||
|
_deserialize_data
|
||||||
|
)
|
||||||
import json
|
import json
|
||||||
|
|
||||||
# ============================================= 100 ============================================== #
|
|
||||||
|
|
||||||
|
def test_message_payload():
|
||||||
|
"""Test MessagePayload class"""
|
||||||
|
print("\n=== Testing MessagePayload ===")
|
||||||
|
|
||||||
def test_text_message():
|
# Test direct transport with text
|
||||||
"""Test sending and receiving text messages."""
|
payload1 = MessagePayload(
|
||||||
print("\n=== Test 1: Text Message ===")
|
data="Hello World",
|
||||||
|
msg_type="text",
|
||||||
# Send text message
|
id="test-id-1",
|
||||||
data = [
|
dataname="message",
|
||||||
("message", "Hello World", "text"),
|
transport="direct",
|
||||||
("greeting", "Good morning!", "text")
|
encoding="base64",
|
||||||
]
|
size=11
|
||||||
|
|
||||||
env = smartsend(
|
|
||||||
"/test/text",
|
|
||||||
data,
|
|
||||||
nats_url="nats://localhost:4222",
|
|
||||||
size_threshold=1000000
|
|
||||||
)
|
)
|
||||||
|
|
||||||
print("Sent envelope:")
|
assert payload1.id == "test-id-1"
|
||||||
print(" Subject: {}".format(env.send_to))
|
assert payload1.dataname == "message"
|
||||||
print(" Correlation ID: {}".format(env.correlation_id))
|
assert payload1.type == "text"
|
||||||
print(" Payloads: {}".format(len(env.payloads)))
|
assert payload1.transport == "direct"
|
||||||
|
assert payload1.encoding == "base64"
|
||||||
|
assert payload1.size == 11
|
||||||
|
print(" [PASS] MessagePayload with text data")
|
||||||
|
|
||||||
# Expected output on receiver:
|
# Test link transport with URL
|
||||||
# payloads = smartreceive(msg)
|
payload2 = MessagePayload(
|
||||||
# for dataname, data, type in payloads:
|
data="http://example.com/file.txt",
|
||||||
# print("Received {}: {}".format(dataname, data))
|
msg_type="binary",
|
||||||
|
id="test-id-2",
|
||||||
|
dataname="file",
|
||||||
def test_dictionary_message():
|
transport="link",
|
||||||
"""Test sending and receiving dictionary messages."""
|
encoding="none",
|
||||||
print("\n=== Test 2: Dictionary Message ===")
|
size=1000
|
||||||
|
|
||||||
# Send dictionary message
|
|
||||||
config = {
|
|
||||||
"step_size": 0.01,
|
|
||||||
"iterations": 1000,
|
|
||||||
"threshold": 0.5
|
|
||||||
}
|
|
||||||
|
|
||||||
data = [
|
|
||||||
("config", config, "dictionary")
|
|
||||||
]
|
|
||||||
|
|
||||||
env = smartsend(
|
|
||||||
"/test/dictionary",
|
|
||||||
data,
|
|
||||||
nats_url="nats://localhost:4222",
|
|
||||||
size_threshold=1000000
|
|
||||||
)
|
)
|
||||||
|
|
||||||
print("Sent envelope:")
|
assert payload2.transport == "link"
|
||||||
print(" Subject: {}".format(env.send_to))
|
assert payload2.data == "http://example.com/file.txt"
|
||||||
print(" Payloads: {}".format(len(env.payloads)))
|
print(" [PASS] MessagePayload with link transport")
|
||||||
|
|
||||||
# Expected output on receiver:
|
# Test to_dict method
|
||||||
# payloads = smartreceive(msg)
|
payload_dict = payload1.to_dict()
|
||||||
# for dataname, data, type in payloads:
|
assert "id" in payload_dict
|
||||||
# if type == "dictionary":
|
assert "dataname" in payload_dict
|
||||||
# print("Config: {}".format(data))
|
assert "type" in payload_dict
|
||||||
|
assert "transport" in payload_dict
|
||||||
|
assert "data" in payload_dict
|
||||||
|
print(" [PASS] MessagePayload.to_dict() method")
|
||||||
|
|
||||||
|
|
||||||
def test_mixed_payloads():
|
def test_message_envelope():
|
||||||
"""Test sending mixed payload types in a single message."""
|
"""Test MessageEnvelope class"""
|
||||||
print("\n=== Test 3: Mixed Payloads ===")
|
print("\n=== Testing MessageEnvelope ===")
|
||||||
|
|
||||||
# Mixed content: text, dictionary, and binary
|
# Create payloads
|
||||||
image_data = b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR" # PNG header (example)
|
payload1 = MessagePayload("Hello", "text", id="p1", dataname="msg1")
|
||||||
|
payload2 = MessagePayload("http://example.com/file", "binary", id="p2", dataname="file", transport="link")
|
||||||
|
|
||||||
data = [
|
# Create envelope
|
||||||
("message_text", "Hello!", "text"),
|
env = MessageEnvelope(
|
||||||
("user_config", {"theme": "dark", "volume": 80}, "dictionary"),
|
send_to="/test/subject",
|
||||||
("user_image", image_data, "binary")
|
payloads=[payload1, payload2],
|
||||||
]
|
correlation_id="test-correlation-id",
|
||||||
|
msg_id="test-msg-id",
|
||||||
env = smartsend(
|
msg_purpose="chat",
|
||||||
"/test/mixed",
|
sender_name="test_sender",
|
||||||
data,
|
receiver_name="test_receiver",
|
||||||
nats_url="nats://localhost:4222",
|
reply_to="/test/reply"
|
||||||
size_threshold=1000000
|
|
||||||
)
|
)
|
||||||
|
|
||||||
print("Sent envelope:")
|
assert env.send_to == "/test/subject"
|
||||||
print(" Subject: {}".format(env.send_to))
|
assert env.correlation_id == "test-correlation-id"
|
||||||
print(" Payloads: {}".format(len(env.payloads)))
|
assert env.msg_id == "test-msg-id"
|
||||||
|
assert env.msg_purpose == "chat"
|
||||||
|
assert len(env.payloads) == 2
|
||||||
|
print(" [PASS] MessageEnvelope creation")
|
||||||
|
|
||||||
# Expected output on receiver:
|
# Test to_json method
|
||||||
# payloads = smartreceive(msg)
|
json_str = env.to_json()
|
||||||
# for dataname, data, type in payloads:
|
json_data = json.loads(json_str)
|
||||||
# print("Received {}: {} (type: {})".format(dataname, data if type != "binary" else len(data), type))
|
assert json_data["sendTo"] == "/test/subject"
|
||||||
|
assert json_data["correlationId"] == "test-correlation-id"
|
||||||
|
assert json_data["msgPurpose"] == "chat"
|
||||||
|
assert len(json_data["payloads"]) == 2
|
||||||
|
print(" [PASS] MessageEnvelope.to_json() method")
|
||||||
|
|
||||||
|
|
||||||
def test_large_payload():
|
def test_serialize_data():
|
||||||
"""Test sending large payloads that require fileserver upload."""
|
"""Test _serialize_data function"""
|
||||||
print("\n=== Test 4: Large Payload (Link Transport) ===")
|
print("\n=== Testing _serialize_data ===")
|
||||||
|
|
||||||
# Create large data (> 1MB would trigger link transport)
|
# Test text serialization
|
||||||
# For testing, we'll use a smaller size but configure threshold lower
|
text_bytes = _serialize_data("Hello", "text")
|
||||||
large_data = b"A" * 100000 # 100KB
|
assert isinstance(text_bytes, bytes)
|
||||||
|
assert text_bytes == b"Hello"
|
||||||
|
print(" [PASS] Text serialization")
|
||||||
|
|
||||||
data = [
|
# Test dictionary serialization
|
||||||
("large_data", large_data, "binary")
|
dict_data = {"key": "value", "number": 42}
|
||||||
]
|
dict_bytes = _serialize_data(dict_data, "dictionary")
|
||||||
|
assert isinstance(dict_bytes, bytes)
|
||||||
|
parsed = json.loads(dict_bytes.decode('utf-8'))
|
||||||
|
assert parsed["key"] == "value"
|
||||||
|
print(" [PASS] Dictionary serialization")
|
||||||
|
|
||||||
# Use a lower threshold for testing
|
# Test binary serialization
|
||||||
env = smartsend(
|
binary_data = b"\x00\x01\x02"
|
||||||
"/test/large",
|
binary_bytes = _serialize_data(binary_data, "binary")
|
||||||
data,
|
assert binary_bytes == b"\x00\x01\x02"
|
||||||
nats_url="nats://localhost:4222",
|
print(" [PASS] Binary serialization")
|
||||||
fileserver_url="http://localhost:8080",
|
|
||||||
size_threshold=50000 # 50KB threshold for testing
|
|
||||||
)
|
|
||||||
|
|
||||||
print("Sent envelope:")
|
# Test image serialization
|
||||||
print(" Subject: {}".format(env.send_to))
|
image_data = bytes([1, 2, 3, 4, 5])
|
||||||
print(" Payloads: {}".format(len(env.payloads)))
|
image_bytes = _serialize_data(image_data, "image")
|
||||||
for p in env.payloads:
|
assert image_bytes == image_data
|
||||||
print(" - Transport: {}, Type: {}".format(p.transport, p.type))
|
print(" [PASS] Image serialization")
|
||||||
|
|
||||||
|
|
||||||
def test_reply_to():
|
def test_deserialize_data():
|
||||||
"""Test sending messages with reply-to functionality."""
|
"""Test _deserialize_data function"""
|
||||||
print("\n=== Test 5: Reply To ===")
|
print("\n=== Testing _deserialize_data ===")
|
||||||
|
|
||||||
data = [
|
# Test text deserialization
|
||||||
("command", {"action": "start"}, "dictionary")
|
text_bytes = b"Hello"
|
||||||
]
|
text_data = _deserialize_data(text_bytes, "text", "test-correlation-id")
|
||||||
|
assert text_data == "Hello"
|
||||||
|
print(" [PASS] Text deserialization")
|
||||||
|
|
||||||
env = smartsend(
|
# Test dictionary deserialization
|
||||||
"/test/command",
|
dict_bytes = b'{"key": "value"}'
|
||||||
data,
|
dict_data = _deserialize_data(dict_bytes, "dictionary", "test-correlation-id")
|
||||||
nats_url="nats://localhost:4222",
|
assert dict_data == {"key": "value"}
|
||||||
reply_to="/test/response",
|
print(" [PASS] Dictionary deserialization")
|
||||||
reply_to_msg_id="reply-123",
|
|
||||||
msg_purpose="command"
|
|
||||||
)
|
|
||||||
|
|
||||||
print("Sent envelope:")
|
# Test binary deserialization
|
||||||
print(" Subject: {}".format(env.send_to))
|
binary_data = b"\x00\x01\x02"
|
||||||
print(" Reply To: {}".format(env.reply_to))
|
binary_result = _deserialize_data(binary_data, "binary", "test-correlation-id")
|
||||||
print(" Reply To Msg ID: {}".format(env.reply_to_msg_id))
|
assert binary_result == b"\x00\x01\x02"
|
||||||
|
print(" [PASS] Binary deserialization")
|
||||||
|
|
||||||
|
|
||||||
def test_correlation_id():
|
def test_utilities():
|
||||||
"""Test using custom correlation IDs for tracing."""
|
"""Test utility functions"""
|
||||||
print("\n=== Test 6: Custom Correlation ID ===")
|
print("\n=== Testing Utility Functions ===")
|
||||||
|
|
||||||
custom_cid = "trace-abc123"
|
# Test generate_uuid
|
||||||
data = [
|
uuid1 = generate_uuid()
|
||||||
("message", "Test with correlation ID", "text")
|
uuid2 = generate_uuid()
|
||||||
]
|
assert uuid1 != uuid2
|
||||||
|
print(f" [PASS] generate_uuid() - generated: {uuid1}")
|
||||||
|
|
||||||
env = smartsend(
|
# Test get_timestamp
|
||||||
"/test/correlation",
|
timestamp = get_timestamp()
|
||||||
data,
|
assert "T" in timestamp
|
||||||
nats_url="nats://localhost:4222",
|
print(f" [PASS] get_timestamp() - generated: {timestamp}")
|
||||||
correlation_id=custom_cid
|
|
||||||
)
|
|
||||||
|
|
||||||
print("Sent envelope with correlation ID: {}".format(env.correlation_id))
|
|
||||||
print("This ID can be used to trace the message flow.")
|
|
||||||
|
|
||||||
|
|
||||||
def test_multiple_payloads():
|
def main():
|
||||||
"""Test sending multiple payloads in one message."""
|
"""Run all tests"""
|
||||||
print("\n=== Test 7: Multiple Payloads ===")
|
print("=" * 60)
|
||||||
|
print("NATSBridge Python/Micropython - Basic Functionality Tests")
|
||||||
|
print("=" * 60)
|
||||||
|
|
||||||
data = [
|
try:
|
||||||
("text_message", "Hello", "text"),
|
test_message_payload()
|
||||||
("json_data", {"key": "value", "number": 42}, "dictionary"),
|
test_message_envelope()
|
||||||
("table_data", b"\x01\x02\x03\x04", "binary"),
|
test_serialize_data()
|
||||||
("audio_data", b"\x00\x01\x02\x03", "binary")
|
test_deserialize_data()
|
||||||
]
|
test_utilities()
|
||||||
|
|
||||||
env = smartsend(
|
print("\n" + "=" * 60)
|
||||||
"/test/multiple",
|
print("ALL TESTS PASSED!")
|
||||||
data,
|
print("=" * 60)
|
||||||
nats_url="nats://localhost:4222",
|
|
||||||
size_threshold=1000000
|
|
||||||
)
|
|
||||||
|
|
||||||
print("Sent {} payloads in one message".format(len(env.payloads)))
|
except Exception as e:
|
||||||
|
print(f"\n[FAIL] Test failed with error: {e}")
|
||||||
|
import traceback
|
||||||
|
traceback.print_exc()
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
print("Micropython NATS Bridge Test Suite")
|
main()
|
||||||
print("==================================")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Run tests
|
|
||||||
test_text_message()
|
|
||||||
test_dictionary_message()
|
|
||||||
test_mixed_payloads()
|
|
||||||
test_large_payload()
|
|
||||||
test_reply_to()
|
|
||||||
test_correlation_id()
|
|
||||||
test_multiple_payloads()
|
|
||||||
|
|
||||||
print("\n=== All tests completed ===")
|
|
||||||
print()
|
|
||||||
print("Note: These tests require:")
|
|
||||||
print(" 1. A running NATS server at nats://localhost:4222")
|
|
||||||
print(" 2. An HTTP file server at http://localhost:8080 (for large payloads)")
|
|
||||||
print()
|
|
||||||
print("To run the tests:")
|
|
||||||
print(" python test_micropython_basic.py")
|
|
||||||
70
test/test_micropython_dict_receiver.py
Normal file
70
test/test_micropython_dict_receiver.py
Normal file
@@ -0,0 +1,70 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Test script for dictionary transport testing - Receiver
|
||||||
|
Tests receiving dictionary messages via NATS using nats_bridge.py smartreceive
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
import json
|
||||||
|
|
||||||
|
# Add src to path for import
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||||
|
|
||||||
|
from nats_bridge import smartreceive, log_trace
|
||||||
|
import nats
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
SUBJECT = "/NATSBridge_dict_test"
|
||||||
|
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
log_trace("", f"Starting dictionary transport receiver test...")
|
||||||
|
log_trace("", f"Note: This receiver will wait for messages from the sender.")
|
||||||
|
log_trace("", f"Run test_micropython_dict_sender.py first to send test data.")
|
||||||
|
|
||||||
|
# Connect to NATS
|
||||||
|
nc = await nats.connect(NATS_URL)
|
||||||
|
log_trace("", f"Connected to NATS at {NATS_URL}")
|
||||||
|
|
||||||
|
# Subscribe to the subject
|
||||||
|
async def message_handler(msg):
|
||||||
|
log_trace("", f"Received message on {msg.subject}")
|
||||||
|
|
||||||
|
# Use smartreceive to handle the data
|
||||||
|
result = smartreceive(msg.data)
|
||||||
|
|
||||||
|
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
|
||||||
|
for dataname, data, data_type in result["payloads"]:
|
||||||
|
if isinstance(data, dict):
|
||||||
|
log_trace(result.get("correlationId", ""), f"Received dictionary '{dataname}' of type {data_type}")
|
||||||
|
log_trace(result.get("correlationId", ""), f" Keys: {list(data.keys())}")
|
||||||
|
|
||||||
|
# Display first few items for small dicts
|
||||||
|
if isinstance(data, dict) and len(data) <= 10:
|
||||||
|
log_trace(result.get("correlationId", ""), f" Content: {json.dumps(data, indent=2)}")
|
||||||
|
else:
|
||||||
|
# For large dicts, show summary
|
||||||
|
log_trace(result.get("correlationId", ""), f" Summary: {json.dumps(data, default=str)[:200]}...")
|
||||||
|
|
||||||
|
# Save to file
|
||||||
|
output_path = f"./received_{dataname}.json"
|
||||||
|
with open(output_path, 'w') as f:
|
||||||
|
json.dump(data, f, indent=2)
|
||||||
|
log_trace(result.get("correlationId", ""), f"Saved dictionary to {output_path}")
|
||||||
|
else:
|
||||||
|
log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}")
|
||||||
|
|
||||||
|
sid = await nc.subscribe(SUBJECT, cb=message_handler)
|
||||||
|
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
|
||||||
|
|
||||||
|
# Keep listening for 120 seconds
|
||||||
|
await asyncio.sleep(120)
|
||||||
|
await nc.close()
|
||||||
|
log_trace("", "Test completed.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(main())
|
||||||
100
test/test_micropython_dict_sender.py
Normal file
100
test/test_micropython_dict_sender.py
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Test script for dictionary transport testing - Micropython
|
||||||
|
Tests sending dictionary messages via NATS using nats_bridge.py smartsend
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
|
||||||
|
# Add src to path for import
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||||
|
|
||||||
|
from nats_bridge import smartsend, log_trace
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
SUBJECT = "/NATSBridge_dict_test"
|
||||||
|
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||||
|
FILESERVER_URL = "http://192.168.88.104:8080"
|
||||||
|
SIZE_THRESHOLD = 1_000_000 # 1MB
|
||||||
|
|
||||||
|
# Create correlation ID for tracing
|
||||||
|
correlation_id = str(uuid.uuid4())
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
# Create a small dictionary (will use direct transport)
|
||||||
|
small_dict = {
|
||||||
|
"name": "test",
|
||||||
|
"value": 42,
|
||||||
|
"enabled": True,
|
||||||
|
"metadata": {
|
||||||
|
"version": "1.0.0",
|
||||||
|
"timestamp": "2026-02-22T12:00:00Z"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Create a large dictionary (will use link transport if > 1MB)
|
||||||
|
# Generate a larger dictionary (~2MB to ensure link transport)
|
||||||
|
large_dict = {
|
||||||
|
"id": str(uuid.uuid4()),
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"index": i,
|
||||||
|
"name": f"item_{i}",
|
||||||
|
"value": i * 1.5,
|
||||||
|
"data": "x" * 10000 # Large string per item
|
||||||
|
}
|
||||||
|
for i in range(200)
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"count": 200,
|
||||||
|
"created": "2026-02-22T12:00:00Z"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Test data 1: small dictionary
|
||||||
|
data1 = ("small_dict", small_dict, "dictionary")
|
||||||
|
|
||||||
|
# Test data 2: large dictionary
|
||||||
|
data2 = ("large_dict", large_dict, "dictionary")
|
||||||
|
|
||||||
|
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
|
||||||
|
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
||||||
|
|
||||||
|
# Use smartsend with dictionary type
|
||||||
|
env, env_json_str = smartsend(
|
||||||
|
SUBJECT,
|
||||||
|
[data1, data2], # List of (dataname, data, type) tuples
|
||||||
|
nats_url=NATS_URL,
|
||||||
|
fileserver_url=FILESERVER_URL,
|
||||||
|
size_threshold=SIZE_THRESHOLD,
|
||||||
|
correlation_id=correlation_id,
|
||||||
|
msg_purpose="chat",
|
||||||
|
sender_name="dict_sender",
|
||||||
|
receiver_name="",
|
||||||
|
receiver_id="",
|
||||||
|
reply_to="",
|
||||||
|
reply_to_msg_id="",
|
||||||
|
is_publish=True # Publish the message to NATS
|
||||||
|
)
|
||||||
|
|
||||||
|
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
||||||
|
|
||||||
|
# Log transport type for each payload
|
||||||
|
for i, payload in enumerate(env.payloads):
|
||||||
|
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
|
||||||
|
log_trace(correlation_id, f" Transport: {payload.transport}")
|
||||||
|
log_trace(correlation_id, f" Type: {payload.type}")
|
||||||
|
log_trace(correlation_id, f" Size: {payload.size} bytes")
|
||||||
|
log_trace(correlation_id, f" Encoding: {payload.encoding}")
|
||||||
|
|
||||||
|
if payload.transport == "link":
|
||||||
|
log_trace(correlation_id, f" URL: {payload.data}")
|
||||||
|
|
||||||
|
print(f"Test completed. Correlation ID: {correlation_id}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
65
test/test_micropython_file_receiver.py
Normal file
65
test/test_micropython_file_receiver.py
Normal file
@@ -0,0 +1,65 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Test script for file transport testing - Receiver
|
||||||
|
Tests receiving binary files via NATS using nats_bridge.py smartreceive
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
|
||||||
|
# Add src to path for import
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||||
|
|
||||||
|
from nats_bridge import smartreceive, log_trace
|
||||||
|
import nats
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
SUBJECT = "/NATSBridge_file_test"
|
||||||
|
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
log_trace("", f"Starting file transport receiver test...")
|
||||||
|
log_trace("", f"Note: This receiver will wait for messages from the sender.")
|
||||||
|
log_trace("", f"Run test_micropython_file_sender.py first to send test data.")
|
||||||
|
|
||||||
|
# Connect to NATS
|
||||||
|
nc = await nats.connect(NATS_URL)
|
||||||
|
log_trace("", f"Connected to NATS at {NATS_URL}")
|
||||||
|
|
||||||
|
# Subscribe to the subject
|
||||||
|
async def message_handler(msg):
|
||||||
|
log_trace("", f"Received message on {msg.subject}")
|
||||||
|
|
||||||
|
# Use smartreceive to handle the data
|
||||||
|
result = smartreceive(msg.data)
|
||||||
|
|
||||||
|
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
|
||||||
|
for dataname, data, data_type in result["payloads"]:
|
||||||
|
if isinstance(data, bytes):
|
||||||
|
log_trace(result.get("correlationId", ""), f"Received binary '{dataname}' of type {data_type}")
|
||||||
|
log_trace(result.get("correlationId", ""), f" Size: {len(data)} bytes")
|
||||||
|
|
||||||
|
# Display first 100 bytes as hex
|
||||||
|
log_trace(result.get("correlationId", ""), f" First 100 bytes (hex): {data[:100].hex()}")
|
||||||
|
|
||||||
|
# Save to file
|
||||||
|
output_path = f"./received_{dataname}.bin"
|
||||||
|
with open(output_path, 'wb') as f:
|
||||||
|
f.write(data)
|
||||||
|
log_trace(result.get("correlationId", ""), f"Saved binary to {output_path}")
|
||||||
|
else:
|
||||||
|
log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}")
|
||||||
|
|
||||||
|
sid = await nc.subscribe(SUBJECT, cb=message_handler)
|
||||||
|
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
|
||||||
|
|
||||||
|
# Keep listening for 120 seconds
|
||||||
|
await asyncio.sleep(120)
|
||||||
|
await nc.close()
|
||||||
|
log_trace("", "Test completed.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(main())
|
||||||
80
test/test_micropython_file_sender.py
Normal file
80
test/test_micropython_file_sender.py
Normal file
@@ -0,0 +1,80 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Test script for file transport testing - Micropython
|
||||||
|
Tests sending binary files via NATS using nats_bridge.py smartsend
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
|
||||||
|
# Add src to path for import
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||||
|
|
||||||
|
from nats_bridge import smartsend, log_trace
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
SUBJECT = "/NATSBridge_file_test"
|
||||||
|
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||||
|
FILESERVER_URL = "http://192.168.88.104:8080"
|
||||||
|
SIZE_THRESHOLD = 1_000_000 # 1MB
|
||||||
|
|
||||||
|
# Create correlation ID for tracing
|
||||||
|
correlation_id = str(uuid.uuid4())
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
# Create small binary data (will use direct transport)
|
||||||
|
small_binary = b"This is small binary data for testing direct transport."
|
||||||
|
small_binary += b"\x00" * 100 # Add some null bytes
|
||||||
|
|
||||||
|
# Create large binary data (will use link transport if > 1MB)
|
||||||
|
# Generate a larger binary (~2MB to ensure link transport)
|
||||||
|
large_binary = bytes([
|
||||||
|
(i * 7) % 256 for i in range(2_000_000)
|
||||||
|
])
|
||||||
|
|
||||||
|
# Test data 1: small binary (direct transport)
|
||||||
|
data1 = ("small_binary", small_binary, "binary")
|
||||||
|
|
||||||
|
# Test data 2: large binary (link transport)
|
||||||
|
data2 = ("large_binary", large_binary, "binary")
|
||||||
|
|
||||||
|
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
|
||||||
|
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
||||||
|
|
||||||
|
# Use smartsend with binary type
|
||||||
|
env, env_json_str = smartsend(
|
||||||
|
SUBJECT,
|
||||||
|
[data1, data2], # List of (dataname, data, type) tuples
|
||||||
|
nats_url=NATS_URL,
|
||||||
|
fileserver_url=FILESERVER_URL,
|
||||||
|
size_threshold=SIZE_THRESHOLD,
|
||||||
|
correlation_id=correlation_id,
|
||||||
|
msg_purpose="chat",
|
||||||
|
sender_name="file_sender",
|
||||||
|
receiver_name="",
|
||||||
|
receiver_id="",
|
||||||
|
reply_to="",
|
||||||
|
reply_to_msg_id="",
|
||||||
|
is_publish=True # Publish the message to NATS
|
||||||
|
)
|
||||||
|
|
||||||
|
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
||||||
|
|
||||||
|
# Log transport type for each payload
|
||||||
|
for i, payload in enumerate(env.payloads):
|
||||||
|
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
|
||||||
|
log_trace(correlation_id, f" Transport: {payload.transport}")
|
||||||
|
log_trace(correlation_id, f" Type: {payload.type}")
|
||||||
|
log_trace(correlation_id, f" Size: {payload.size} bytes")
|
||||||
|
log_trace(correlation_id, f" Encoding: {payload.encoding}")
|
||||||
|
|
||||||
|
if payload.transport == "link":
|
||||||
|
log_trace(correlation_id, f" URL: {payload.data}")
|
||||||
|
|
||||||
|
print(f"Test completed. Correlation ID: {correlation_id}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
97
test/test_micropython_mixed_receiver.py
Normal file
97
test/test_micropython_mixed_receiver.py
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Test script for mixed payload testing - Receiver
|
||||||
|
Tests receiving mixed payload types via NATS using nats_bridge.py smartreceive
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
import json
|
||||||
|
|
||||||
|
# Add src to path for import
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||||
|
|
||||||
|
from nats_bridge import smartreceive, log_trace
|
||||||
|
import nats
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
SUBJECT = "/NATSBridge_mixed_test"
|
||||||
|
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
log_trace("", f"Starting mixed payload receiver test...")
|
||||||
|
log_trace("", f"Note: This receiver will wait for messages from the sender.")
|
||||||
|
log_trace("", f"Run test_micropython_mixed_sender.py first to send test data.")
|
||||||
|
|
||||||
|
# Connect to NATS
|
||||||
|
nc = await nats.connect(NATS_URL)
|
||||||
|
log_trace("", f"Connected to NATS at {NATS_URL}")
|
||||||
|
|
||||||
|
# Subscribe to the subject
|
||||||
|
async def message_handler(msg):
|
||||||
|
log_trace("", f"Received message on {msg.subject}")
|
||||||
|
|
||||||
|
# Use smartreceive to handle the data
|
||||||
|
result = smartreceive(msg.data)
|
||||||
|
|
||||||
|
log_trace(result.get("correlationId", ""), f"Received envelope with {len(result['payloads'])} payloads")
|
||||||
|
|
||||||
|
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
|
||||||
|
for dataname, data, data_type in result["payloads"]:
|
||||||
|
log_trace(result.get("correlationId", ""), f"\n--- Payload: {dataname} (type: {data_type}) ---")
|
||||||
|
|
||||||
|
if isinstance(data, str):
|
||||||
|
log_trace(result.get("correlationId", ""), f" Type: text/string")
|
||||||
|
log_trace(result.get("correlationId", ""), f" Length: {len(data)} characters")
|
||||||
|
if len(data) <= 100:
|
||||||
|
log_trace(result.get("correlationId", ""), f" Content: {data}")
|
||||||
|
else:
|
||||||
|
log_trace(result.get("correlationId", ""), f" First 100 chars: {data[:100]}...")
|
||||||
|
# Save to file
|
||||||
|
output_path = f"./received_{dataname}.txt"
|
||||||
|
with open(output_path, 'w') as f:
|
||||||
|
f.write(data)
|
||||||
|
log_trace(result.get("correlationId", ""), f" Saved to: {output_path}")
|
||||||
|
|
||||||
|
elif isinstance(data, dict):
|
||||||
|
log_trace(result.get("correlationId", ""), f" Type: dictionary")
|
||||||
|
log_trace(result.get("correlationId", ""), f" Keys: {list(data.keys())}")
|
||||||
|
log_trace(result.get("correlationId", ""), f" Content: {json.dumps(data, indent=2)}")
|
||||||
|
# Save to file
|
||||||
|
output_path = f"./received_{dataname}.json"
|
||||||
|
with open(output_path, 'w') as f:
|
||||||
|
json.dump(data, f, indent=2)
|
||||||
|
log_trace(result.get("correlationId", ""), f" Saved to: {output_path}")
|
||||||
|
|
||||||
|
elif isinstance(data, bytes):
|
||||||
|
log_trace(result.get("correlationId", ""), f" Type: binary")
|
||||||
|
log_trace(result.get("correlationId", ""), f" Size: {len(data)} bytes")
|
||||||
|
log_trace(result.get("correlationId", ""), f" First 100 bytes (hex): {data[:100].hex()}")
|
||||||
|
# Save to file
|
||||||
|
output_path = f"./received_{dataname}.bin"
|
||||||
|
with open(output_path, 'wb') as f:
|
||||||
|
f.write(data)
|
||||||
|
log_trace(result.get("correlationId", ""), f" Saved to: {output_path}")
|
||||||
|
else:
|
||||||
|
log_trace(result.get("correlationId", ""), f" Received unexpected data type: {type(data)}")
|
||||||
|
|
||||||
|
# Log envelope metadata
|
||||||
|
log_trace(result.get("correlationId", ""), f"\n--- Envelope Metadata ---")
|
||||||
|
log_trace(result.get("correlationId", ""), f" Correlation ID: {result.get('correlationId', 'N/A')}")
|
||||||
|
log_trace(result.get("correlationId", ""), f" Message ID: {result.get('msgId', 'N/A')}")
|
||||||
|
log_trace(result.get("correlationId", ""), f" Sender: {result.get('senderName', 'N/A')}")
|
||||||
|
log_trace(result.get("correlationId", ""), f" Purpose: {result.get('msgPurpose', 'N/A')}")
|
||||||
|
|
||||||
|
sid = await nc.subscribe(SUBJECT, cb=message_handler)
|
||||||
|
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
|
||||||
|
|
||||||
|
# Keep listening for 120 seconds
|
||||||
|
await asyncio.sleep(120)
|
||||||
|
await nc.close()
|
||||||
|
log_trace("", "Test completed.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(main())
|
||||||
94
test/test_micropython_mixed_sender.py
Normal file
94
test/test_micropython_mixed_sender.py
Normal file
@@ -0,0 +1,94 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Test script for mixed payload testing - Micropython
|
||||||
|
Tests sending mixed payload types via NATS using nats_bridge.py smartsend
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
|
||||||
|
# Add src to path for import
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||||
|
|
||||||
|
from nats_bridge import smartsend, log_trace
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
SUBJECT = "/NATSBridge_mixed_test"
|
||||||
|
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||||
|
FILESERVER_URL = "http://192.168.88.104:8080"
|
||||||
|
SIZE_THRESHOLD = 1_000_000 # 1MB
|
||||||
|
|
||||||
|
# Create correlation ID for tracing
|
||||||
|
correlation_id = str(uuid.uuid4())
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
# Create payloads for mixed content test
|
||||||
|
|
||||||
|
# 1. Small text (direct transport)
|
||||||
|
text_data = "Hello, this is a text message for testing mixed payloads!"
|
||||||
|
|
||||||
|
# 2. Small dictionary (direct transport)
|
||||||
|
dict_data = {
|
||||||
|
"status": "ok",
|
||||||
|
"code": 200,
|
||||||
|
"message": "Test successful",
|
||||||
|
"items": [1, 2, 3]
|
||||||
|
}
|
||||||
|
|
||||||
|
# 3. Small binary (direct transport)
|
||||||
|
binary_data = b"\x00\x01\x02\x03\x04\x05" + b"\xff" * 100
|
||||||
|
|
||||||
|
# 4. Large text (link transport - will use fileserver)
|
||||||
|
large_text = "\n".join([
|
||||||
|
f"Line {i}: This is a large text payload for link transport testing. " * 50
|
||||||
|
for i in range(100)
|
||||||
|
])
|
||||||
|
|
||||||
|
# Test data list - mixed payload types
|
||||||
|
data = [
|
||||||
|
("message_text", text_data, "text"),
|
||||||
|
("config_dict", dict_data, "dictionary"),
|
||||||
|
("small_binary", binary_data, "binary"),
|
||||||
|
("large_text", large_text, "text"),
|
||||||
|
]
|
||||||
|
|
||||||
|
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
|
||||||
|
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
||||||
|
|
||||||
|
# Use smartsend with mixed types
|
||||||
|
env, env_json_str = smartsend(
|
||||||
|
SUBJECT,
|
||||||
|
data, # List of (dataname, data, type) tuples
|
||||||
|
nats_url=NATS_URL,
|
||||||
|
fileserver_url=FILESERVER_URL,
|
||||||
|
size_threshold=SIZE_THRESHOLD,
|
||||||
|
correlation_id=correlation_id,
|
||||||
|
msg_purpose="chat",
|
||||||
|
sender_name="mixed_sender",
|
||||||
|
receiver_name="",
|
||||||
|
receiver_id="",
|
||||||
|
reply_to="",
|
||||||
|
reply_to_msg_id="",
|
||||||
|
is_publish=True # Publish the message to NATS
|
||||||
|
)
|
||||||
|
|
||||||
|
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
||||||
|
|
||||||
|
# Log transport type for each payload
|
||||||
|
for i, payload in enumerate(env.payloads):
|
||||||
|
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
|
||||||
|
log_trace(correlation_id, f" Transport: {payload.transport}")
|
||||||
|
log_trace(correlation_id, f" Type: {payload.type}")
|
||||||
|
log_trace(correlation_id, f" Size: {payload.size} bytes")
|
||||||
|
log_trace(correlation_id, f" Encoding: {payload.encoding}")
|
||||||
|
|
||||||
|
if payload.transport == "link":
|
||||||
|
log_trace(correlation_id, f" URL: {payload.data}")
|
||||||
|
|
||||||
|
print(f"Test completed. Correlation ID: {correlation_id}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
69
test/test_micropython_text_receiver.py
Normal file
69
test/test_micropython_text_receiver.py
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Test script for text transport testing - Receiver
|
||||||
|
Tests receiving text messages via NATS using nats_bridge.py smartreceive
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
import json
|
||||||
|
|
||||||
|
# Add src to path for import
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||||
|
|
||||||
|
from nats_bridge import smartreceive, log_trace
|
||||||
|
import nats
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
SUBJECT = "/NATSBridge_text_test"
|
||||||
|
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
log_trace("", f"Starting text transport receiver test...")
|
||||||
|
log_trace("", f"Note: This receiver will wait for messages from the sender.")
|
||||||
|
log_trace("", f"Run test_micropython_text_sender.py first to send test data.")
|
||||||
|
|
||||||
|
# Connect to NATS
|
||||||
|
nc = await nats.connect(NATS_URL)
|
||||||
|
log_trace("", f"Connected to NATS at {NATS_URL}")
|
||||||
|
|
||||||
|
# Subscribe to the subject
|
||||||
|
async def message_handler(msg):
|
||||||
|
log_trace("", f"Received message on {msg.subject}")
|
||||||
|
|
||||||
|
# Use smartreceive to handle the data
|
||||||
|
result = smartreceive(msg.data)
|
||||||
|
|
||||||
|
# Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples
|
||||||
|
for dataname, data, data_type in result["payloads"]:
|
||||||
|
if isinstance(data, str):
|
||||||
|
log_trace(result.get("correlationId", ""), f"Received text '{dataname}' of type {data_type}")
|
||||||
|
log_trace(result.get("correlationId", ""), f" Length: {len(data)} characters")
|
||||||
|
|
||||||
|
# Display first 100 characters
|
||||||
|
if len(data) > 100:
|
||||||
|
log_trace(result.get("correlationId", ""), f" First 100 characters: {data[:100]}...")
|
||||||
|
else:
|
||||||
|
log_trace(result.get("correlationId", ""), f" Content: {data}")
|
||||||
|
|
||||||
|
# Save to file
|
||||||
|
output_path = f"./received_{dataname}.txt"
|
||||||
|
with open(output_path, 'w') as f:
|
||||||
|
f.write(data)
|
||||||
|
log_trace(result.get("correlationId", ""), f"Saved text to {output_path}")
|
||||||
|
else:
|
||||||
|
log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}")
|
||||||
|
|
||||||
|
sid = await nc.subscribe(SUBJECT, cb=message_handler)
|
||||||
|
log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}")
|
||||||
|
|
||||||
|
# Keep listening for 120 seconds
|
||||||
|
await asyncio.sleep(120)
|
||||||
|
await nc.close()
|
||||||
|
log_trace("", "Test completed.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(main())
|
||||||
82
test/test_micropython_text_sender.py
Normal file
82
test/test_micropython_text_sender.py
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Test script for text transport testing - Micropython
|
||||||
|
Tests sending text messages via NATS using nats_bridge.py smartsend
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
|
||||||
|
# Add src to path for import
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
|
||||||
|
|
||||||
|
from nats_bridge import smartsend, log_trace
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
SUBJECT = "/NATSBridge_text_test"
|
||||||
|
NATS_URL = "nats://nats.yiem.cc:4222"
|
||||||
|
FILESERVER_URL = "http://192.168.88.104:8080"
|
||||||
|
SIZE_THRESHOLD = 1_000_000 # 1MB
|
||||||
|
|
||||||
|
# Create correlation ID for tracing
|
||||||
|
correlation_id = str(uuid.uuid4())
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
# Create a small text (will use direct transport)
|
||||||
|
small_text = "Hello, this is a small text message. Testing direct transport via NATS."
|
||||||
|
|
||||||
|
# Create a large text (will use link transport if > 1MB)
|
||||||
|
# Generate a larger text (~2MB to ensure link transport)
|
||||||
|
large_text = "\n".join([
|
||||||
|
f"Line {i}: This is a sample text line with some content to pad the size. " * 100
|
||||||
|
for i in range(500)
|
||||||
|
])
|
||||||
|
|
||||||
|
# Test data 1: small text
|
||||||
|
data1 = ("small_text", small_text, "text")
|
||||||
|
|
||||||
|
# Test data 2: large text
|
||||||
|
data2 = ("large_text", large_text, "text")
|
||||||
|
|
||||||
|
log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}")
|
||||||
|
log_trace(correlation_id, f"Correlation ID: {correlation_id}")
|
||||||
|
|
||||||
|
# Use smartsend with text type
|
||||||
|
# For small text: will use direct transport (Base64 encoded UTF-8)
|
||||||
|
# For large text: will use link transport (uploaded to fileserver)
|
||||||
|
env, env_json_str = smartsend(
|
||||||
|
SUBJECT,
|
||||||
|
[data1, data2], # List of (dataname, data, type) tuples
|
||||||
|
nats_url=NATS_URL,
|
||||||
|
fileserver_url=FILESERVER_URL,
|
||||||
|
size_threshold=SIZE_THRESHOLD,
|
||||||
|
correlation_id=correlation_id,
|
||||||
|
msg_purpose="chat",
|
||||||
|
sender_name="text_sender",
|
||||||
|
receiver_name="",
|
||||||
|
receiver_id="",
|
||||||
|
reply_to="",
|
||||||
|
reply_to_msg_id="",
|
||||||
|
is_publish=True # Publish the message to NATS
|
||||||
|
)
|
||||||
|
|
||||||
|
log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads")
|
||||||
|
|
||||||
|
# Log transport type for each payload
|
||||||
|
for i, payload in enumerate(env.payloads):
|
||||||
|
log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):")
|
||||||
|
log_trace(correlation_id, f" Transport: {payload.transport}")
|
||||||
|
log_trace(correlation_id, f" Type: {payload.type}")
|
||||||
|
log_trace(correlation_id, f" Size: {payload.size} bytes")
|
||||||
|
log_trace(correlation_id, f" Encoding: {payload.encoding}")
|
||||||
|
|
||||||
|
if payload.transport == "link":
|
||||||
|
log_trace(correlation_id, f" URL: {payload.data}")
|
||||||
|
|
||||||
|
print(f"Test completed. Correlation ID: {correlation_id}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -1,634 +0,0 @@
|
|||||||
# NATSBridge.jl Tutorial
|
|
||||||
|
|
||||||
A comprehensive tutorial for learning how to use NATSBridge.jl for bi-directional communication between Julia and JavaScript services using NATS.
|
|
||||||
|
|
||||||
## Table of Contents
|
|
||||||
|
|
||||||
1. [What is NATSBridge.jl?](#what-is-natsbridgejl)
|
|
||||||
2. [Key Concepts](#key-concepts)
|
|
||||||
3. [Installation](#installation)
|
|
||||||
4. [Basic Usage](#basic-usage)
|
|
||||||
5. [Payload Types](#payload-types)
|
|
||||||
6. [Transport Strategies](#transport-strategies)
|
|
||||||
7. [Advanced Features](#advanced-features)
|
|
||||||
8. [Complete Examples](#complete-examples)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## What is NATSBridge.jl?
|
|
||||||
|
|
||||||
NATSBridge.jl is a Julia module that provides a high-level API for sending and receiving data across network boundaries using NATS as the message bus. It implements the **Claim-Check pattern** for handling large payloads efficiently.
|
|
||||||
|
|
||||||
### Core Features
|
|
||||||
|
|
||||||
- **Bi-directional communication**: Julia ↔ JavaScript
|
|
||||||
- **Smart transport selection**: Automatic direct vs link transport based on payload size
|
|
||||||
- **Multi-payload support**: Send multiple payloads of different types in a single message
|
|
||||||
- **Claim-check pattern**: Upload large files to HTTP server, send only URLs via NATS
|
|
||||||
- **Type-aware serialization**: Different serialization strategies for different data types
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Key Concepts
|
|
||||||
|
|
||||||
### 1. msgEnvelope_v1 (Message Envelope)
|
|
||||||
|
|
||||||
The `msgEnvelope_v1` structure provides a comprehensive message format for bidirectional communication:
|
|
||||||
|
|
||||||
```julia
|
|
||||||
struct msgEnvelope_v1
|
|
||||||
correlationId::String # Unique identifier to track messages
|
|
||||||
msgId::String # This message id
|
|
||||||
timestamp::String # Message published timestamp
|
|
||||||
|
|
||||||
sendTo::String # Topic/subject the sender sends to
|
|
||||||
msgPurpose::String # Purpose (ACK | NACK | updateStatus | shutdown | chat)
|
|
||||||
senderName::String # Sender name (e.g., "agent-wine-web-frontend")
|
|
||||||
senderId::String # Sender id (uuid4)
|
|
||||||
receiverName::String # Message receiver name (e.g., "agent-backend")
|
|
||||||
receiverId::String # Message receiver id (uuid4 or nothing for broadcast)
|
|
||||||
replyTo::String # Topic to reply to
|
|
||||||
replyToMsgId::String # Message id this message is replying to
|
|
||||||
brokerURL::String # NATS server address
|
|
||||||
|
|
||||||
metadata::Dict{String, Any}
|
|
||||||
payloads::AbstractArray{msgPayload_v1} # Multiple payloads stored here
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. msgPayload_v1 (Payload Structure)
|
|
||||||
|
|
||||||
The `msgPayload_v1` structure provides flexible payload handling:
|
|
||||||
|
|
||||||
```julia
|
|
||||||
struct msgPayload_v1
|
|
||||||
id::String # Id of this payload (e.g., "uuid4")
|
|
||||||
dataname::String # Name of this payload (e.g., "login_image")
|
|
||||||
type::String # "text | dictionary | table | image | audio | video | binary"
|
|
||||||
transport::String # "direct | link"
|
|
||||||
encoding::String # "none | json | base64 | arrow-ipc"
|
|
||||||
size::Integer # Data size in bytes
|
|
||||||
data::Any # Payload data in case of direct transport or a URL in case of link
|
|
||||||
metadata::Dict{String, Any} # Dict("checksum" => "sha256_hash", ...)
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Standard API Format
|
|
||||||
|
|
||||||
The system uses a **standardized list-of-tuples format** for all payload operations:
|
|
||||||
|
|
||||||
```julia
|
|
||||||
# Input format for smartsend (always a list of tuples with type info)
|
|
||||||
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
|
||||||
|
|
||||||
# Output format for smartreceive (always returns a list of tuples)
|
|
||||||
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
|
||||||
```
|
|
||||||
|
|
||||||
**Important**: Even when sending a single payload, you must wrap it in a list.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using Pkg
|
|
||||||
Pkg.add("NATS")
|
|
||||||
Pkg.add("JSON")
|
|
||||||
Pkg.add("Arrow")
|
|
||||||
Pkg.add("HTTP")
|
|
||||||
Pkg.add("UUIDs")
|
|
||||||
Pkg.add("Dates")
|
|
||||||
Pkg.add("Base64")
|
|
||||||
Pkg.add("PrettyPrinting")
|
|
||||||
Pkg.add("DataFrames")
|
|
||||||
```
|
|
||||||
|
|
||||||
Then include the NATSBridge module:
|
|
||||||
|
|
||||||
```julia
|
|
||||||
include("NATSBridge.jl")
|
|
||||||
using .NATSBridge
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Basic Usage
|
|
||||||
|
|
||||||
### Sending Data (smartsend)
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
|
|
||||||
# Send a simple dictionary
|
|
||||||
data = Dict("key" => "value")
|
|
||||||
env = NATSBridge.smartsend("my.subject", [("dataname1", data, "dictionary")])
|
|
||||||
```
|
|
||||||
|
|
||||||
### Receiving Data (smartreceive)
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
|
|
||||||
# Subscribe to a NATS subject
|
|
||||||
NATS.subscribe("my.subject") do msg
|
|
||||||
# Process the message
|
|
||||||
result = NATSBridge.smartreceive(
|
|
||||||
msg,
|
|
||||||
max_retries = 5,
|
|
||||||
base_delay = 100,
|
|
||||||
max_delay = 5000
|
|
||||||
)
|
|
||||||
|
|
||||||
# result is a list of (dataname, data, type) tuples
|
|
||||||
for (dataname, data, type) in result
|
|
||||||
println("Received $dataname of type $type")
|
|
||||||
println("Data: $data")
|
|
||||||
end
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Payload Types
|
|
||||||
|
|
||||||
NATSBridge.jl supports the following payload types:
|
|
||||||
|
|
||||||
| Type | Description | Serialization |
|
|
||||||
|------|-------------|---------------|
|
|
||||||
| `text` | Plain text | UTF-8 encoding |
|
|
||||||
| `dictionary` | JSON-serializable data (Dict, NamedTuple) | JSON |
|
|
||||||
| `table` | Tabular data (DataFrame, array of structs) | Apache Arrow IPC |
|
|
||||||
| `image` | Image data (Bitmap, PNG/JPG bytes) | Binary |
|
|
||||||
| `audio` | Audio data (WAV, MP3 bytes) | Binary |
|
|
||||||
| `video` | Video data (MP4, AVI bytes) | Binary |
|
|
||||||
| `binary` | Generic binary data | Binary |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Transport Strategies
|
|
||||||
|
|
||||||
NATSBridge.jl automatically selects the appropriate transport strategy based on payload size:
|
|
||||||
|
|
||||||
### Direct Transport (< 1MB)
|
|
||||||
|
|
||||||
Small payloads are encoded as Base64 and sent directly over NATS.
|
|
||||||
|
|
||||||
```julia
|
|
||||||
# Small data (< 1MB) - uses direct transport
|
|
||||||
small_data = rand(1000) # ~8KB
|
|
||||||
env = NATSBridge.smartsend("small", [("data", small_data, "table")])
|
|
||||||
```
|
|
||||||
|
|
||||||
### Link Transport (≥ 1MB)
|
|
||||||
|
|
||||||
Large payloads are uploaded to an HTTP file server, and only the URL is sent via NATS.
|
|
||||||
|
|
||||||
```julia
|
|
||||||
# Large data (≥ 1MB) - uses link transport
|
|
||||||
large_data = rand(10_000_000) # ~80MB
|
|
||||||
env = NATSBridge.smartsend("large", [("data", large_data, "table")])
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Complete Examples
|
|
||||||
|
|
||||||
### Example 1: Text Message
|
|
||||||
|
|
||||||
**Sender:**
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
using UUIDs
|
|
||||||
|
|
||||||
const SUBJECT = "/NATSBridge_text_test"
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const FILESERVER_URL = "http://localhost:8080"
|
|
||||||
|
|
||||||
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
|
||||||
url_getUploadID = "$fileserver_url/upload"
|
|
||||||
headers = ["Content-Type" => "application/json"]
|
|
||||||
body = """{ "OneShot" : true }"""
|
|
||||||
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
uploadid = responseJson["id"]
|
|
||||||
uploadtoken = responseJson["uploadToken"]
|
|
||||||
|
|
||||||
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
|
|
||||||
url_upload = "$fileserver_url/file/$uploadid"
|
|
||||||
headers = ["X-UploadToken" => uploadtoken]
|
|
||||||
|
|
||||||
form = HTTP.Form(Dict("file" => file_multipart))
|
|
||||||
httpResponse = HTTP.post(url_upload, headers, form)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
|
|
||||||
fileid = responseJson["id"]
|
|
||||||
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
|
|
||||||
|
|
||||||
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
|
||||||
end
|
|
||||||
|
|
||||||
function test_text_send()
|
|
||||||
small_text = "Hello, this is a small text message."
|
|
||||||
large_text = join(["Line $i: " for i in 1:50000], "")
|
|
||||||
|
|
||||||
data1 = ("small_text", small_text, "text")
|
|
||||||
data2 = ("large_text", large_text, "text")
|
|
||||||
|
|
||||||
env = NATSBridge.smartsend(
|
|
||||||
SUBJECT,
|
|
||||||
[data1, data2],
|
|
||||||
nats_url = NATS_URL,
|
|
||||||
fileserver_url = FILESERVER_URL,
|
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
|
||||||
size_threshold = 1_000_000,
|
|
||||||
correlation_id = string(uuid4()),
|
|
||||||
msg_purpose = "chat",
|
|
||||||
sender_name = "text_sender"
|
|
||||||
)
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
**Receiver:**
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
|
|
||||||
const SUBJECT = "/NATSBridge_text_test"
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
|
|
||||||
function test_text_receive()
|
|
||||||
conn = NATS.connect(NATS_URL)
|
|
||||||
NATS.subscribe(conn, SUBJECT) do msg
|
|
||||||
result = NATSBridge.smartreceive(
|
|
||||||
msg,
|
|
||||||
max_retries = 5,
|
|
||||||
base_delay = 100,
|
|
||||||
max_delay = 5000
|
|
||||||
)
|
|
||||||
|
|
||||||
for (dataname, data, data_type) in result
|
|
||||||
if data_type == "text"
|
|
||||||
println("Received text: $data")
|
|
||||||
write("./received_$dataname.txt", data)
|
|
||||||
end
|
|
||||||
end
|
|
||||||
end
|
|
||||||
sleep(120)
|
|
||||||
NATS.drain(conn)
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
### Example 2: Dictionary (JSON) Message
|
|
||||||
|
|
||||||
**Sender:**
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
using UUIDs
|
|
||||||
|
|
||||||
const SUBJECT = "/NATSBridge_dict_test"
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const FILESERVER_URL = "http://localhost:8080"
|
|
||||||
|
|
||||||
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
|
||||||
url_getUploadID = "$fileserver_url/upload"
|
|
||||||
headers = ["Content-Type" => "application/json"]
|
|
||||||
body = """{ "OneShot" : true }"""
|
|
||||||
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
uploadid = responseJson["id"]
|
|
||||||
uploadtoken = responseJson["uploadToken"]
|
|
||||||
|
|
||||||
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
|
|
||||||
url_upload = "$fileserver_url/file/$uploadid"
|
|
||||||
headers = ["X-UploadToken" => uploadtoken]
|
|
||||||
|
|
||||||
form = HTTP.Form(Dict("file" => file_multipart))
|
|
||||||
httpResponse = HTTP.post(url_upload, headers, form)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
|
|
||||||
fileid = responseJson["id"]
|
|
||||||
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
|
|
||||||
|
|
||||||
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
|
||||||
end
|
|
||||||
|
|
||||||
function test_dict_send()
|
|
||||||
small_dict = Dict("name" => "Alice", "age" => 30)
|
|
||||||
large_dict = Dict("ids" => collect(1:50000), "names" => ["User_$i" for i in 1:50000])
|
|
||||||
|
|
||||||
data1 = ("small_dict", small_dict, "dictionary")
|
|
||||||
data2 = ("large_dict", large_dict, "dictionary")
|
|
||||||
|
|
||||||
env = NATSBridge.smartsend(
|
|
||||||
SUBJECT,
|
|
||||||
[data1, data2],
|
|
||||||
nats_url = NATS_URL,
|
|
||||||
fileserver_url = FILESERVER_URL,
|
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
|
||||||
size_threshold = 1_000_000,
|
|
||||||
correlation_id = string(uuid4()),
|
|
||||||
msg_purpose = "chat"
|
|
||||||
)
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
**Receiver:**
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
|
|
||||||
const SUBJECT = "/NATSBridge_dict_test"
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
|
|
||||||
function test_dict_receive()
|
|
||||||
conn = NATS.connect(NATS_URL)
|
|
||||||
NATS.subscribe(conn, SUBJECT) do msg
|
|
||||||
result = NATSBridge.smartreceive(
|
|
||||||
msg,
|
|
||||||
max_retries = 5,
|
|
||||||
base_delay = 100,
|
|
||||||
max_delay = 5000
|
|
||||||
)
|
|
||||||
|
|
||||||
for (dataname, data, data_type) in result
|
|
||||||
if data_type == "dictionary"
|
|
||||||
println("Received dictionary: $data")
|
|
||||||
write("./received_$dataname.json", JSON.json(data, 2))
|
|
||||||
end
|
|
||||||
end
|
|
||||||
end
|
|
||||||
sleep(120)
|
|
||||||
NATS.drain(conn)
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
### Example 3: DataFrame (Table) Message
|
|
||||||
|
|
||||||
**Sender:**
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
using DataFrames
|
|
||||||
using UUIDs
|
|
||||||
|
|
||||||
const SUBJECT = "/NATSBridge_table_test"
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const FILESERVER_URL = "http://localhost:8080"
|
|
||||||
|
|
||||||
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
|
||||||
url_getUploadID = "$fileserver_url/upload"
|
|
||||||
headers = ["Content-Type" => "application/json"]
|
|
||||||
body = """{ "OneShot" : true }"""
|
|
||||||
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
uploadid = responseJson["id"]
|
|
||||||
uploadtoken = responseJson["uploadToken"]
|
|
||||||
|
|
||||||
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
|
|
||||||
url_upload = "$fileserver_url/file/$uploadid"
|
|
||||||
headers = ["X-UploadToken" => uploadtoken]
|
|
||||||
|
|
||||||
form = HTTP.Form(Dict("file" => file_multipart))
|
|
||||||
httpResponse = HTTP.post(url_upload, headers, form)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
|
|
||||||
fileid = responseJson["id"]
|
|
||||||
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
|
|
||||||
|
|
||||||
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
|
||||||
end
|
|
||||||
|
|
||||||
function test_table_send()
|
|
||||||
small_df = DataFrame(id = 1:10, name = ["Alice", "Bob", "Charlie"], score = [95, 88, 92])
|
|
||||||
large_df = DataFrame(id = 1:50000, name = ["User_$i" for i in 1:50000], score = rand(1:100, 50000))
|
|
||||||
|
|
||||||
data1 = ("small_table", small_df, "table")
|
|
||||||
data2 = ("large_table", large_df, "table")
|
|
||||||
|
|
||||||
env = NATSBridge.smartsend(
|
|
||||||
SUBJECT,
|
|
||||||
[data1, data2],
|
|
||||||
nats_url = NATS_URL,
|
|
||||||
fileserver_url = FILESERVER_URL,
|
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
|
||||||
size_threshold = 1_000_000,
|
|
||||||
correlation_id = string(uuid4()),
|
|
||||||
msg_purpose = "chat"
|
|
||||||
)
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
**Receiver:**
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
using DataFrames
|
|
||||||
|
|
||||||
const SUBJECT = "/NATSBridge_table_test"
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
|
|
||||||
function test_table_receive()
|
|
||||||
conn = NATS.connect(NATS_URL)
|
|
||||||
NATS.subscribe(conn, SUBJECT) do msg
|
|
||||||
result = NATSBridge.smartreceive(
|
|
||||||
msg,
|
|
||||||
max_retries = 5,
|
|
||||||
base_delay = 100,
|
|
||||||
max_delay = 5000
|
|
||||||
)
|
|
||||||
|
|
||||||
for (dataname, data, data_type) in result
|
|
||||||
if data_type == "table"
|
|
||||||
data = DataFrame(data)
|
|
||||||
println("Received DataFrame with $(size(data, 1)) rows")
|
|
||||||
display(data[1:min(5, size(data, 1)), :])
|
|
||||||
end
|
|
||||||
end
|
|
||||||
end
|
|
||||||
sleep(120)
|
|
||||||
NATS.drain(conn)
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
### Example 4: Mixed Content (Chat with Text, Image, Audio)
|
|
||||||
|
|
||||||
**Sender:**
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
using DataFrames
|
|
||||||
using UUIDs
|
|
||||||
|
|
||||||
const SUBJECT = "/NATSBridge_mix_test"
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const FILESERVER_URL = "http://localhost:8080"
|
|
||||||
|
|
||||||
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
|
||||||
url_getUploadID = "$fileserver_url/upload"
|
|
||||||
headers = ["Content-Type" => "application/json"]
|
|
||||||
body = """{ "OneShot" : true }"""
|
|
||||||
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
uploadid = responseJson["id"]
|
|
||||||
uploadtoken = responseJson["uploadToken"]
|
|
||||||
|
|
||||||
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
|
|
||||||
url_upload = "$fileserver_url/file/$uploadid"
|
|
||||||
headers = ["X-UploadToken" => uploadtoken]
|
|
||||||
|
|
||||||
form = HTTP.Form(Dict("file" => file_multipart))
|
|
||||||
httpResponse = HTTP.post(url_upload, headers, form)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
|
|
||||||
fileid = responseJson["id"]
|
|
||||||
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
|
|
||||||
|
|
||||||
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
|
||||||
end
|
|
||||||
|
|
||||||
function test_mix_send()
|
|
||||||
# Text data
|
|
||||||
text_data = "Hello! This is a test chat message. 🎉"
|
|
||||||
|
|
||||||
# Dictionary data
|
|
||||||
dict_data = Dict("type" => "chat", "sender" => "serviceA")
|
|
||||||
|
|
||||||
# Small table data
|
|
||||||
table_data_small = DataFrame(id = 1:10, name = ["msg_$i" for i in 1:10])
|
|
||||||
|
|
||||||
# Large table data (link transport)
|
|
||||||
table_data_large = DataFrame(id = 1:150_000, name = ["msg_$i" for i in 1:150_000])
|
|
||||||
|
|
||||||
# Small image data (direct transport)
|
|
||||||
image_data = UInt8[rand(1:255) for _ in 1:100]
|
|
||||||
|
|
||||||
# Large image data (link transport)
|
|
||||||
large_image_data = UInt8[rand(1:255) for _ in 1:1_500_000]
|
|
||||||
|
|
||||||
# Small audio data (direct transport)
|
|
||||||
audio_data = UInt8[rand(1:255) for _ in 1:100]
|
|
||||||
|
|
||||||
# Large audio data (link transport)
|
|
||||||
large_audio_data = UInt8[rand(1:255) for _ in 1:1_500_000]
|
|
||||||
|
|
||||||
# Small video data (direct transport)
|
|
||||||
video_data = UInt8[rand(1:255) for _ in 1:150]
|
|
||||||
|
|
||||||
# Large video data (link transport)
|
|
||||||
large_video_data = UInt8[rand(1:255) for _ in 1:1_500_000]
|
|
||||||
|
|
||||||
# Small binary data (direct transport)
|
|
||||||
binary_data = UInt8[rand(1:255) for _ in 1:200]
|
|
||||||
|
|
||||||
# Large binary data (link transport)
|
|
||||||
large_binary_data = UInt8[rand(1:255) for _ in 1:1_500_000]
|
|
||||||
|
|
||||||
# Create payloads list - mixed content
|
|
||||||
payloads = [
|
|
||||||
# Small data (direct transport)
|
|
||||||
("chat_text", text_data, "text"),
|
|
||||||
("chat_json", dict_data, "dictionary"),
|
|
||||||
("chat_table_small", table_data_small, "table"),
|
|
||||||
|
|
||||||
# Large data (link transport)
|
|
||||||
("chat_table_large", table_data_large, "table"),
|
|
||||||
("user_image_large", large_image_data, "image"),
|
|
||||||
("audio_clip_large", large_audio_data, "audio"),
|
|
||||||
("video_clip_large", large_video_data, "video"),
|
|
||||||
("binary_file_large", large_binary_data, "binary")
|
|
||||||
]
|
|
||||||
|
|
||||||
env = NATSBridge.smartsend(
|
|
||||||
SUBJECT,
|
|
||||||
payloads,
|
|
||||||
nats_url = NATS_URL,
|
|
||||||
fileserver_url = FILESERVER_URL,
|
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
|
||||||
size_threshold = 1_000_000,
|
|
||||||
correlation_id = string(uuid4()),
|
|
||||||
msg_purpose = "chat",
|
|
||||||
sender_name = "mix_sender"
|
|
||||||
)
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
**Receiver:**
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
using DataFrames
|
|
||||||
|
|
||||||
const SUBJECT = "/NATSBridge_mix_test"
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
|
|
||||||
function test_mix_receive()
|
|
||||||
conn = NATS.connect(NATS_URL)
|
|
||||||
NATS.subscribe(conn, SUBJECT) do msg
|
|
||||||
result = NATSBridge.smartreceive(
|
|
||||||
msg,
|
|
||||||
max_retries = 5,
|
|
||||||
base_delay = 100,
|
|
||||||
max_delay = 5000
|
|
||||||
)
|
|
||||||
|
|
||||||
println("Received $(length(result)) payloads")
|
|
||||||
|
|
||||||
for (dataname, data, data_type) in result
|
|
||||||
println("\n=== Payload: $dataname (type: $data_type) ===")
|
|
||||||
|
|
||||||
if data_type == "text"
|
|
||||||
println(" Type: String")
|
|
||||||
println(" Length: $(length(data)) characters")
|
|
||||||
|
|
||||||
elseif data_type == "dictionary"
|
|
||||||
println(" Type: JSON Object")
|
|
||||||
println(" Keys: $(keys(data))")
|
|
||||||
|
|
||||||
elseif data_type == "table"
|
|
||||||
data = DataFrame(data)
|
|
||||||
println(" Type: DataFrame")
|
|
||||||
println(" Dimensions: $(size(data, 1)) rows x $(size(data, 2)) columns")
|
|
||||||
|
|
||||||
elseif data_type == "image"
|
|
||||||
println(" Type: Vector{UInt8}")
|
|
||||||
println(" Size: $(length(data)) bytes")
|
|
||||||
write("./received_$dataname.bin", data)
|
|
||||||
|
|
||||||
elseif data_type == "audio"
|
|
||||||
println(" Type: Vector{UInt8}")
|
|
||||||
println(" Size: $(length(data)) bytes")
|
|
||||||
write("./received_$dataname.bin", data)
|
|
||||||
|
|
||||||
elseif data_type == "video"
|
|
||||||
println(" Type: Vector{UInt8}")
|
|
||||||
println(" Size: $(length(data)) bytes")
|
|
||||||
write("./received_$dataname.bin", data)
|
|
||||||
|
|
||||||
elseif data_type == "binary"
|
|
||||||
println(" Type: Vector{UInt8}")
|
|
||||||
println(" Size: $(length(data)) bytes")
|
|
||||||
write("./received_$dataname.bin", data)
|
|
||||||
end
|
|
||||||
end
|
|
||||||
end
|
|
||||||
sleep(120)
|
|
||||||
NATS.drain(conn)
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Best Practices
|
|
||||||
|
|
||||||
1. **Always wrap payloads in a list** - Even for single payloads: `[("dataname", data, "type")]`
|
|
||||||
2. **Use appropriate transport** - Let NATSBridge handle size-based routing (default 1MB threshold)
|
|
||||||
3. **Customize size threshold** - Use `size_threshold` parameter to adjust the direct/link split
|
|
||||||
4. **Provide fileserver handler** - Implement `fileserverUploadHandler` for link transport
|
|
||||||
5. **Include correlation IDs** - Track messages across distributed systems
|
|
||||||
6. **Handle errors** - Implement proper error handling for network failures
|
|
||||||
7. **Close connections** - Ensure NATS connections are properly closed using `NATS.drain()`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
NATSBridge.jl provides a powerful abstraction for bi-directional communication between Julia and JavaScript services. By understanding the key concepts and following the best practices, you can build robust, scalable applications that leverage the full power of NATS messaging.
|
|
||||||
|
|
||||||
For more information, see:
|
|
||||||
- [`docs/architecture.md`](./architecture.md) - Detailed architecture documentation
|
|
||||||
- [`docs/implementation.md`](./implementation.md) - Implementation details
|
|
||||||
@@ -1,939 +0,0 @@
|
|||||||
# NATSBridge.jl Walkthrough: Building a Chat System
|
|
||||||
|
|
||||||
A step-by-step guided walkthrough for building a real-time chat system using NATSBridge.jl with mixed content support (text, images, audio, video, and files).
|
|
||||||
|
|
||||||
## Prerequisites
|
|
||||||
|
|
||||||
- Julia 1.7+
|
|
||||||
- NATS server running
|
|
||||||
- HTTP file server (Plik) running
|
|
||||||
|
|
||||||
## Step 1: Understanding the Chat System Architecture
|
|
||||||
|
|
||||||
### System Components
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
||||||
│ Chat System │
|
|
||||||
├─────────────────────────────────────────────────────────────────────────────┤
|
|
||||||
│ │
|
|
||||||
│ ┌──────────────┐ NATS ┌──────────────┐ │
|
|
||||||
│ │ Julia │◄───────┬───────► │ JavaScript │ │
|
|
||||||
│ │ Service │ │ │ Client │ │
|
|
||||||
│ │ │ │ │ │ │
|
|
||||||
│ │ - Text │ │ │ - Text │ │
|
|
||||||
│ │ - Images │ │ │ - Images │ │
|
|
||||||
│ │ - Audio │ ▼ │ - Audio │ │
|
|
||||||
│ │ - Video │ NATSBridge.jl │ - Files │ │
|
|
||||||
│ │ - Files │ │ │ - Tables │ │
|
|
||||||
│ └──────────────┘ │ └──────────────┘ │
|
|
||||||
│ │ │
|
|
||||||
│ ┌───────┴───────┐ │
|
|
||||||
│ │ NATS │ │
|
|
||||||
│ │ Server │ │
|
|
||||||
│ └─────────────┘ │
|
|
||||||
│ │
|
|
||||||
└─────────────────────────────────────────────────────────────────────────────┘
|
|
||||||
|
|
||||||
For large payloads (> 1MB):
|
|
||||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
||||||
│ File Server (Plik) │
|
|
||||||
│ │
|
|
||||||
│ Julia Service ──► Upload ──► File Server ──► Download ◄── JavaScript Client│
|
|
||||||
│ │
|
|
||||||
└─────────────────────────────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
### Message Format
|
|
||||||
|
|
||||||
Each chat message is an envelope containing multiple payloads:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"correlationId": "uuid4",
|
|
||||||
"msgId": "uuid4",
|
|
||||||
"timestamp": "2024-01-15T10:30:00Z",
|
|
||||||
"sendTo": "/chat/room1",
|
|
||||||
"msgPurpose": "chat",
|
|
||||||
"senderName": "user-1",
|
|
||||||
"senderId": "uuid4",
|
|
||||||
"receiverName": "user-2",
|
|
||||||
"receiverId": "uuid4",
|
|
||||||
"brokerURL": "nats://localhost:4222",
|
|
||||||
"payloads": [
|
|
||||||
{
|
|
||||||
"id": "uuid4",
|
|
||||||
"dataname": "message_text",
|
|
||||||
"type": "text",
|
|
||||||
"transport": "direct",
|
|
||||||
"encoding": "base64",
|
|
||||||
"size": 256,
|
|
||||||
"data": "SGVsbG8gV29ybGQh",
|
|
||||||
"metadata": {}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "uuid4",
|
|
||||||
"dataname": "user_image",
|
|
||||||
"type": "image",
|
|
||||||
"transport": "link",
|
|
||||||
"encoding": "none",
|
|
||||||
"size": 15433,
|
|
||||||
"data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/image.jpg",
|
|
||||||
"metadata": {}
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Step 2: Setting Up the Environment
|
|
||||||
|
|
||||||
### 1. Start NATS Server
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Using Docker
|
|
||||||
docker run -d -p 4222:4222 -p 8222:8222 --name nats-server nats:latest
|
|
||||||
|
|
||||||
# Or download from https://github.com/nats-io/nats-server/releases
|
|
||||||
./nats-server
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Start HTTP File Server (Plik)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Using Docker
|
|
||||||
docker run -d -p 8080:8080 --name plik plik/plik:latest
|
|
||||||
|
|
||||||
# Or download from https://github.com/arnaud-lb/plik/releases
|
|
||||||
./plikd -d
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Install Julia Dependencies
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using Pkg
|
|
||||||
Pkg.add("NATS")
|
|
||||||
Pkg.add("JSON")
|
|
||||||
Pkg.add("Arrow")
|
|
||||||
Pkg.add("HTTP")
|
|
||||||
Pkg.add("UUIDs")
|
|
||||||
Pkg.add("Dates")
|
|
||||||
Pkg.add("Base64")
|
|
||||||
Pkg.add("PrettyPrinting")
|
|
||||||
Pkg.add("DataFrames")
|
|
||||||
```
|
|
||||||
|
|
||||||
## Step 3: Basic Text-Only Chat
|
|
||||||
|
|
||||||
### Sender (User 1)
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATS
|
|
||||||
using JSON
|
|
||||||
using UUIDs
|
|
||||||
using Dates
|
|
||||||
using PrettyPrinting
|
|
||||||
using DataFrames
|
|
||||||
using Arrow
|
|
||||||
using HTTP
|
|
||||||
using Base64
|
|
||||||
|
|
||||||
# Include the bridge module
|
|
||||||
include("NATSBridge.jl")
|
|
||||||
using .NATSBridge
|
|
||||||
|
|
||||||
# Configuration
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const FILESERVER_URL = "http://localhost:8080"
|
|
||||||
const SUBJECT = "/chat/room1"
|
|
||||||
|
|
||||||
# File upload handler for plik server
|
|
||||||
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
|
||||||
url_getUploadID = "$fileserver_url/upload"
|
|
||||||
headers = ["Content-Type" => "application/json"]
|
|
||||||
body = """{ "OneShot" : true }"""
|
|
||||||
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
uploadid = responseJson["id"]
|
|
||||||
uploadtoken = responseJson["uploadToken"]
|
|
||||||
|
|
||||||
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
|
|
||||||
url_upload = "$fileserver_url/file/$uploadid"
|
|
||||||
headers = ["X-UploadToken" => uploadtoken]
|
|
||||||
|
|
||||||
form = HTTP.Form(Dict("file" => file_multipart))
|
|
||||||
httpResponse = HTTP.post(url_upload, headers, form)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
|
|
||||||
fileid = responseJson["id"]
|
|
||||||
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
|
|
||||||
|
|
||||||
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
|
||||||
end
|
|
||||||
|
|
||||||
# Send a simple text message
|
|
||||||
function send_text_message()
|
|
||||||
message_text = "Hello, how are you today?"
|
|
||||||
|
|
||||||
env = NATSBridge.smartsend(
|
|
||||||
SUBJECT,
|
|
||||||
[("message", message_text, "text")],
|
|
||||||
nats_url = NATS_URL,
|
|
||||||
fileserver_url = FILESERVER_URL,
|
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
|
||||||
size_threshold = 1_000_000,
|
|
||||||
correlation_id = string(uuid4()),
|
|
||||||
msg_purpose = "chat",
|
|
||||||
sender_name = "user-1"
|
|
||||||
)
|
|
||||||
|
|
||||||
println("Sent text message with correlation ID: $(env.correlationId)")
|
|
||||||
end
|
|
||||||
|
|
||||||
send_text_message()
|
|
||||||
```
|
|
||||||
|
|
||||||
### Receiver (User 2)
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATS
|
|
||||||
using JSON
|
|
||||||
using UUIDs
|
|
||||||
using Dates
|
|
||||||
using PrettyPrinting
|
|
||||||
using DataFrames
|
|
||||||
using Arrow
|
|
||||||
using HTTP
|
|
||||||
using Base64
|
|
||||||
|
|
||||||
# Include the bridge module
|
|
||||||
include("NATSBridge.jl")
|
|
||||||
using .NATSBridge
|
|
||||||
|
|
||||||
# Configuration
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const SUBJECT = "/chat/room1"
|
|
||||||
|
|
||||||
# Message handler
|
|
||||||
function message_handler(msg::NATS.Msg)
|
|
||||||
payloads = NATSBridge.smartreceive(
|
|
||||||
msg,
|
|
||||||
max_retries = 5,
|
|
||||||
base_delay = 100,
|
|
||||||
max_delay = 5000
|
|
||||||
)
|
|
||||||
|
|
||||||
# Extract the text message
|
|
||||||
for (dataname, data, data_type) in payloads
|
|
||||||
if data_type == "text"
|
|
||||||
println("Received message: $data")
|
|
||||||
# Save to file
|
|
||||||
write("./received_$dataname.txt", data)
|
|
||||||
end
|
|
||||||
end
|
|
||||||
end
|
|
||||||
|
|
||||||
# Subscribe to the chat room
|
|
||||||
NATS.subscribe(SUBJECT) do msg
|
|
||||||
message_handler(msg)
|
|
||||||
end
|
|
||||||
|
|
||||||
# Keep the program running
|
|
||||||
while true
|
|
||||||
sleep(1)
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
## Step 4: Adding Image Support
|
|
||||||
|
|
||||||
### Sending an Image
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATS
|
|
||||||
using JSON
|
|
||||||
using UUIDs
|
|
||||||
using Dates
|
|
||||||
using PrettyPrinting
|
|
||||||
using DataFrames
|
|
||||||
using Arrow
|
|
||||||
using HTTP
|
|
||||||
using Base64
|
|
||||||
|
|
||||||
include("NATSBridge.jl")
|
|
||||||
using .NATSBridge
|
|
||||||
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const FILESERVER_URL = "http://localhost:8080"
|
|
||||||
const SUBJECT = "/chat/room1"
|
|
||||||
|
|
||||||
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
|
||||||
url_getUploadID = "$fileserver_url/upload"
|
|
||||||
headers = ["Content-Type" => "application/json"]
|
|
||||||
body = """{ "OneShot" : true }"""
|
|
||||||
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
uploadid = responseJson["id"]
|
|
||||||
uploadtoken = responseJson["uploadToken"]
|
|
||||||
|
|
||||||
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
|
|
||||||
url_upload = "$fileserver_url/file/$uploadid"
|
|
||||||
headers = ["X-UploadToken" => uploadtoken]
|
|
||||||
|
|
||||||
form = HTTP.Form(Dict("file" => file_multipart))
|
|
||||||
httpResponse = HTTP.post(url_upload, headers, form)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
|
|
||||||
fileid = responseJson["id"]
|
|
||||||
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
|
|
||||||
|
|
||||||
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
|
||||||
end
|
|
||||||
|
|
||||||
function send_image()
|
|
||||||
# Read image file
|
|
||||||
image_data = read("screenshot.png", Vector{UInt8})
|
|
||||||
|
|
||||||
# Send with text message
|
|
||||||
env = NATSBridge.smartsend(
|
|
||||||
SUBJECT,
|
|
||||||
[
|
|
||||||
("text", "Check out this screenshot!", "text"),
|
|
||||||
("screenshot", image_data, "image")
|
|
||||||
],
|
|
||||||
nats_url = NATS_URL,
|
|
||||||
fileserver_url = FILESERVER_URL,
|
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
|
||||||
size_threshold = 1_000_000,
|
|
||||||
correlation_id = string(uuid4()),
|
|
||||||
msg_purpose = "chat",
|
|
||||||
sender_name = "user-1"
|
|
||||||
)
|
|
||||||
|
|
||||||
println("Sent image with correlation ID: $(env.correlationId)")
|
|
||||||
end
|
|
||||||
|
|
||||||
send_image()
|
|
||||||
```
|
|
||||||
|
|
||||||
### Receiving an Image
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATS
|
|
||||||
using JSON
|
|
||||||
using UUIDs
|
|
||||||
using Dates
|
|
||||||
using PrettyPrinting
|
|
||||||
using DataFrames
|
|
||||||
using Arrow
|
|
||||||
using HTTP
|
|
||||||
using Base64
|
|
||||||
|
|
||||||
include("NATSBridge.jl")
|
|
||||||
using .NATSBridge
|
|
||||||
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const SUBJECT = "/chat/room1"
|
|
||||||
|
|
||||||
function message_handler(msg::NATS.Msg)
|
|
||||||
payloads = NATSBridge.smartreceive(
|
|
||||||
msg,
|
|
||||||
max_retries = 5,
|
|
||||||
base_delay = 100,
|
|
||||||
max_delay = 5000
|
|
||||||
)
|
|
||||||
|
|
||||||
for (dataname, data, data_type) in payloads
|
|
||||||
if data_type == "text"
|
|
||||||
println("Text: $data")
|
|
||||||
elseif data_type == "image"
|
|
||||||
# Save image to file
|
|
||||||
filename = "received_$dataname.bin"
|
|
||||||
write(filename, data)
|
|
||||||
println("Saved image: $filename")
|
|
||||||
end
|
|
||||||
end
|
|
||||||
end
|
|
||||||
|
|
||||||
NATS.subscribe(SUBJECT) do msg
|
|
||||||
message_handler(msg)
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
## Step 5: Handling Large Files with Link Transport
|
|
||||||
|
|
||||||
### Automatic Transport Selection
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATS
|
|
||||||
using JSON
|
|
||||||
using UUIDs
|
|
||||||
using Dates
|
|
||||||
using PrettyPrinting
|
|
||||||
using DataFrames
|
|
||||||
using Arrow
|
|
||||||
using HTTP
|
|
||||||
using Base64
|
|
||||||
|
|
||||||
include("NATSBridge.jl")
|
|
||||||
using .NATSBridge
|
|
||||||
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const FILESERVER_URL = "http://localhost:8080"
|
|
||||||
const SUBJECT = "/chat/room1"
|
|
||||||
|
|
||||||
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
|
||||||
url_getUploadID = "$fileserver_url/upload"
|
|
||||||
headers = ["Content-Type" => "application/json"]
|
|
||||||
body = """{ "OneShot" : true }"""
|
|
||||||
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
uploadid = responseJson["id"]
|
|
||||||
uploadtoken = responseJson["uploadToken"]
|
|
||||||
|
|
||||||
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
|
|
||||||
url_upload = "$fileserver_url/file/$uploadid"
|
|
||||||
headers = ["X-UploadToken" => uploadtoken]
|
|
||||||
|
|
||||||
form = HTTP.Form(Dict("file" => file_multipart))
|
|
||||||
httpResponse = HTTP.post(url_upload, headers, form)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
|
|
||||||
fileid = responseJson["id"]
|
|
||||||
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
|
|
||||||
|
|
||||||
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
|
||||||
end
|
|
||||||
|
|
||||||
function send_large_file()
|
|
||||||
# Create a large file (> 1MB triggers link transport)
|
|
||||||
large_data = rand(10_000_000) # ~80MB
|
|
||||||
|
|
||||||
env = NATSBridge.smartsend(
|
|
||||||
SUBJECT,
|
|
||||||
[("large_file", large_data, "binary")],
|
|
||||||
nats_url = NATS_URL,
|
|
||||||
fileserver_url = FILESERVER_URL,
|
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
|
||||||
size_threshold = 1_000_000,
|
|
||||||
correlation_id = string(uuid4()),
|
|
||||||
msg_purpose = "chat",
|
|
||||||
sender_name = "user-1"
|
|
||||||
)
|
|
||||||
|
|
||||||
println("Uploaded large file to: $(env.payloads[1].data)")
|
|
||||||
println("Correlation ID: $(env.correlationId)")
|
|
||||||
end
|
|
||||||
|
|
||||||
send_large_file()
|
|
||||||
```
|
|
||||||
|
|
||||||
## Step 6: Audio and Video Support
|
|
||||||
|
|
||||||
### Sending Audio
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATS
|
|
||||||
using JSON
|
|
||||||
using UUIDs
|
|
||||||
using Dates
|
|
||||||
using PrettyPrinting
|
|
||||||
using DataFrames
|
|
||||||
using Arrow
|
|
||||||
using HTTP
|
|
||||||
using Base64
|
|
||||||
|
|
||||||
include("NATSBridge.jl")
|
|
||||||
using .NATSBridge
|
|
||||||
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const FILESERVER_URL = "http://localhost:8080"
|
|
||||||
const SUBJECT = "/chat/room1"
|
|
||||||
|
|
||||||
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
|
||||||
url_getUploadID = "$fileserver_url/upload"
|
|
||||||
headers = ["Content-Type" => "application/json"]
|
|
||||||
body = """{ "OneShot" : true }"""
|
|
||||||
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
uploadid = responseJson["id"]
|
|
||||||
uploadtoken = responseJson["uploadToken"]
|
|
||||||
|
|
||||||
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
|
|
||||||
url_upload = "$fileserver_url/file/$uploadid"
|
|
||||||
headers = ["X-UploadToken" => uploadtoken]
|
|
||||||
|
|
||||||
form = HTTP.Form(Dict("file" => file_multipart))
|
|
||||||
httpResponse = HTTP.post(url_upload, headers, form)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
|
|
||||||
fileid = responseJson["id"]
|
|
||||||
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
|
|
||||||
|
|
||||||
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
|
||||||
end
|
|
||||||
|
|
||||||
function send_audio()
|
|
||||||
# Read audio file (WAV, MP3, etc.)
|
|
||||||
audio_data = read("voice_message.mp3", Vector{UInt8})
|
|
||||||
|
|
||||||
env = NATSBridge.smartsend(
|
|
||||||
SUBJECT,
|
|
||||||
[("voice_message", audio_data, "audio")],
|
|
||||||
nats_url = NATS_URL,
|
|
||||||
fileserver_url = FILESERVER_URL,
|
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
|
||||||
size_threshold = 1_000_000,
|
|
||||||
correlation_id = string(uuid4()),
|
|
||||||
msg_purpose = "chat",
|
|
||||||
sender_name = "user-1"
|
|
||||||
)
|
|
||||||
|
|
||||||
println("Sent audio message: $(env.correlationId)")
|
|
||||||
end
|
|
||||||
|
|
||||||
send_audio()
|
|
||||||
```
|
|
||||||
|
|
||||||
### Sending Video
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATS
|
|
||||||
using JSON
|
|
||||||
using UUIDs
|
|
||||||
using Dates
|
|
||||||
using PrettyPrinting
|
|
||||||
using DataFrames
|
|
||||||
using Arrow
|
|
||||||
using HTTP
|
|
||||||
using Base64
|
|
||||||
|
|
||||||
include("NATSBridge.jl")
|
|
||||||
using .NATSBridge
|
|
||||||
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const FILESERVER_URL = "http://localhost:8080"
|
|
||||||
const SUBJECT = "/chat/room1"
|
|
||||||
|
|
||||||
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
|
||||||
url_getUploadID = "$fileserver_url/upload"
|
|
||||||
headers = ["Content-Type" => "application/json"]
|
|
||||||
body = """{ "OneShot" : true }"""
|
|
||||||
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
uploadid = responseJson["id"]
|
|
||||||
uploadtoken = responseJson["uploadToken"]
|
|
||||||
|
|
||||||
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
|
|
||||||
url_upload = "$fileserver_url/file/$uploadid"
|
|
||||||
headers = ["X-UploadToken" => uploadtoken]
|
|
||||||
|
|
||||||
form = HTTP.Form(Dict("file" => file_multipart))
|
|
||||||
httpResponse = HTTP.post(url_upload, headers, form)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
|
|
||||||
fileid = responseJson["id"]
|
|
||||||
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
|
|
||||||
|
|
||||||
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
|
||||||
end
|
|
||||||
|
|
||||||
function send_video()
|
|
||||||
# Read video file (MP4, AVI, etc.)
|
|
||||||
video_data = read("video_message.mp4", Vector{UInt8})
|
|
||||||
|
|
||||||
env = NATSBridge.smartsend(
|
|
||||||
SUBJECT,
|
|
||||||
[("video_message", video_data, "video")],
|
|
||||||
nats_url = NATS_URL,
|
|
||||||
fileserver_url = FILESERVER_URL,
|
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
|
||||||
size_threshold = 1_000_000,
|
|
||||||
correlation_id = string(uuid4()),
|
|
||||||
msg_purpose = "chat",
|
|
||||||
sender_name = "user-1"
|
|
||||||
)
|
|
||||||
|
|
||||||
println("Sent video message: $(env.correlationId)")
|
|
||||||
end
|
|
||||||
|
|
||||||
send_video()
|
|
||||||
```
|
|
||||||
|
|
||||||
## Step 7: Table/Data Exchange
|
|
||||||
|
|
||||||
### Sending Tabular Data
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATS
|
|
||||||
using JSON
|
|
||||||
using UUIDs
|
|
||||||
using Dates
|
|
||||||
using PrettyPrinting
|
|
||||||
using DataFrames
|
|
||||||
using Arrow
|
|
||||||
using HTTP
|
|
||||||
using Base64
|
|
||||||
|
|
||||||
include("NATSBridge.jl")
|
|
||||||
using .NATSBridge
|
|
||||||
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const FILESERVER_URL = "http://localhost:8080"
|
|
||||||
const SUBJECT = "/chat/room1"
|
|
||||||
|
|
||||||
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
|
||||||
url_getUploadID = "$fileserver_url/upload"
|
|
||||||
headers = ["Content-Type" => "application/json"]
|
|
||||||
body = """{ "OneShot" : true }"""
|
|
||||||
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
uploadid = responseJson["id"]
|
|
||||||
uploadtoken = responseJson["uploadToken"]
|
|
||||||
|
|
||||||
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
|
|
||||||
url_upload = "$fileserver_url/file/$uploadid"
|
|
||||||
headers = ["X-UploadToken" => uploadtoken]
|
|
||||||
|
|
||||||
form = HTTP.Form(Dict("file" => file_multipart))
|
|
||||||
httpResponse = HTTP.post(url_upload, headers, form)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
|
|
||||||
fileid = responseJson["id"]
|
|
||||||
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
|
|
||||||
|
|
||||||
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
|
||||||
end
|
|
||||||
|
|
||||||
function send_table()
|
|
||||||
# Create a DataFrame
|
|
||||||
df = DataFrame(
|
|
||||||
id = 1:5,
|
|
||||||
name = ["Alice", "Bob", "Charlie", "Diana", "Eve"],
|
|
||||||
score = [95, 88, 92, 98, 85],
|
|
||||||
grade = ['A', 'B', 'A', 'B', 'B']
|
|
||||||
)
|
|
||||||
|
|
||||||
env = NATSBridge.smartsend(
|
|
||||||
SUBJECT,
|
|
||||||
[("student_scores", df, "table")],
|
|
||||||
nats_url = NATS_URL,
|
|
||||||
fileserver_url = FILESERVER_URL,
|
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
|
||||||
size_threshold = 1_000_000,
|
|
||||||
correlation_id = string(uuid4()),
|
|
||||||
msg_purpose = "chat",
|
|
||||||
sender_name = "user-1"
|
|
||||||
)
|
|
||||||
|
|
||||||
println("Sent table with $(nrow(df)) rows")
|
|
||||||
end
|
|
||||||
|
|
||||||
send_table()
|
|
||||||
```
|
|
||||||
|
|
||||||
### Receiving and Using Tables
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATS
|
|
||||||
using JSON
|
|
||||||
using UUIDs
|
|
||||||
using Dates
|
|
||||||
using PrettyPrinting
|
|
||||||
using DataFrames
|
|
||||||
using Arrow
|
|
||||||
using HTTP
|
|
||||||
using Base64
|
|
||||||
|
|
||||||
include("NATSBridge.jl")
|
|
||||||
using .NATSBridge
|
|
||||||
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const SUBJECT = "/chat/room1"
|
|
||||||
|
|
||||||
function message_handler(msg::NATS.Msg)
|
|
||||||
payloads = NATSBridge.smartreceive(
|
|
||||||
msg,
|
|
||||||
max_retries = 5,
|
|
||||||
base_delay = 100,
|
|
||||||
max_delay = 5000
|
|
||||||
)
|
|
||||||
|
|
||||||
for (dataname, data, data_type) in payloads
|
|
||||||
if data_type == "table"
|
|
||||||
data = DataFrame(data)
|
|
||||||
println("Received table:")
|
|
||||||
show(data)
|
|
||||||
println("\nAverage score: $(mean(data.score))")
|
|
||||||
end
|
|
||||||
end
|
|
||||||
end
|
|
||||||
|
|
||||||
NATS.subscribe(SUBJECT) do msg
|
|
||||||
message_handler(msg)
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
## Step 8: Bidirectional Communication
|
|
||||||
|
|
||||||
### Request-Response Pattern
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATS
|
|
||||||
using JSON
|
|
||||||
using UUIDs
|
|
||||||
using Dates
|
|
||||||
using PrettyPrinting
|
|
||||||
using DataFrames
|
|
||||||
using Arrow
|
|
||||||
using HTTP
|
|
||||||
using Base64
|
|
||||||
|
|
||||||
include("NATSBridge.jl")
|
|
||||||
using .NATSBridge
|
|
||||||
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const SUBJECT = "/api/query"
|
|
||||||
const REPLY_SUBJECT = "/api/response"
|
|
||||||
|
|
||||||
# Request
|
|
||||||
function send_request()
|
|
||||||
query_data = Dict("query" => "SELECT * FROM users")
|
|
||||||
|
|
||||||
env = NATSBridge.smartsend(
|
|
||||||
SUBJECT,
|
|
||||||
[("sql_query", query_data, "dictionary")],
|
|
||||||
nats_url = NATS_URL,
|
|
||||||
fileserver_url = "http://localhost:8080",
|
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
|
||||||
size_threshold = 1_000_000,
|
|
||||||
correlation_id = string(uuid4()),
|
|
||||||
msg_purpose = "request",
|
|
||||||
sender_name = "frontend",
|
|
||||||
receiver_name = "backend",
|
|
||||||
reply_to = REPLY_SUBJECT,
|
|
||||||
reply_to_msg_id = string(uuid4())
|
|
||||||
)
|
|
||||||
|
|
||||||
println("Request sent: $(env.correlationId)")
|
|
||||||
end
|
|
||||||
|
|
||||||
# Response handler
|
|
||||||
function response_handler(msg::NATS.Msg)
|
|
||||||
payloads = NATSBridge.smartreceive(
|
|
||||||
msg,
|
|
||||||
max_retries = 5,
|
|
||||||
base_delay = 100,
|
|
||||||
max_delay = 5000
|
|
||||||
)
|
|
||||||
|
|
||||||
for (dataname, data, data_type) in payloads
|
|
||||||
if data_type == "table"
|
|
||||||
data = DataFrame(data)
|
|
||||||
println("Query results:")
|
|
||||||
show(data)
|
|
||||||
end
|
|
||||||
end
|
|
||||||
end
|
|
||||||
|
|
||||||
NATS.subscribe(REPLY_SUBJECT) do msg
|
|
||||||
response_handler(msg)
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
## Step 9: Complete Chat Application
|
|
||||||
|
|
||||||
### Full Chat System
|
|
||||||
|
|
||||||
```julia
|
|
||||||
module ChatApp
|
|
||||||
using NATS
|
|
||||||
using JSON
|
|
||||||
using UUIDs
|
|
||||||
using Dates
|
|
||||||
using PrettyPrinting
|
|
||||||
using DataFrames
|
|
||||||
using Arrow
|
|
||||||
using HTTP
|
|
||||||
using Base64
|
|
||||||
|
|
||||||
# Include the bridge module
|
|
||||||
include("../src/NATSBridge.jl")
|
|
||||||
using .NATSBridge
|
|
||||||
|
|
||||||
# Configuration
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
const FILESERVER_URL = "http://localhost:8080"
|
|
||||||
const SUBJECT = "/chat/room1"
|
|
||||||
|
|
||||||
# File upload handler for plik server
|
|
||||||
function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any}
|
|
||||||
url_getUploadID = "$fileserver_url/upload"
|
|
||||||
headers = ["Content-Type" => "application/json"]
|
|
||||||
body = """{ "OneShot" : true }"""
|
|
||||||
httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
uploadid = responseJson["id"]
|
|
||||||
uploadtoken = responseJson["uploadToken"]
|
|
||||||
|
|
||||||
file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream")
|
|
||||||
url_upload = "$fileserver_url/file/$uploadid"
|
|
||||||
headers = ["X-UploadToken" => uploadtoken]
|
|
||||||
|
|
||||||
form = HTTP.Form(Dict("file" => file_multipart))
|
|
||||||
httpResponse = HTTP.post(url_upload, headers, form)
|
|
||||||
responseJson = JSON.parse(String(httpResponse.body))
|
|
||||||
|
|
||||||
fileid = responseJson["id"]
|
|
||||||
url = "$fileserver_url/file/$uploadid/$fileid/$dataname"
|
|
||||||
|
|
||||||
return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url)
|
|
||||||
end
|
|
||||||
|
|
||||||
function send_chat_message(
|
|
||||||
text::String,
|
|
||||||
image_path::Union{String, Nothing}=nothing,
|
|
||||||
audio_path::Union{String, Nothing}=nothing
|
|
||||||
)
|
|
||||||
# Build payloads list
|
|
||||||
payloads = [("message_text", text, "text")]
|
|
||||||
|
|
||||||
if image_path !== nothing
|
|
||||||
image_data = read(image_path, Vector{UInt8})
|
|
||||||
push!(payloads, ("user_image", image_data, "image"))
|
|
||||||
end
|
|
||||||
|
|
||||||
if audio_path !== nothing
|
|
||||||
audio_data = read(audio_path, Vector{UInt8})
|
|
||||||
push!(payloads, ("user_audio", audio_data, "audio"))
|
|
||||||
end
|
|
||||||
|
|
||||||
env = NATSBridge.smartsend(
|
|
||||||
SUBJECT,
|
|
||||||
payloads,
|
|
||||||
nats_url = NATS_URL,
|
|
||||||
fileserver_url = FILESERVER_URL,
|
|
||||||
fileserverUploadHandler = plik_upload_handler,
|
|
||||||
size_threshold = 1_000_000,
|
|
||||||
correlation_id = string(uuid4()),
|
|
||||||
msg_purpose = "chat",
|
|
||||||
sender_name = "user-1"
|
|
||||||
)
|
|
||||||
|
|
||||||
println("Message sent with correlation ID: $(env.correlationId)")
|
|
||||||
end
|
|
||||||
|
|
||||||
function receive_chat_messages()
|
|
||||||
function message_handler(msg::NATS.Msg)
|
|
||||||
payloads = NATSBridge.smartreceive(
|
|
||||||
msg,
|
|
||||||
max_retries = 5,
|
|
||||||
base_delay = 100,
|
|
||||||
max_delay = 5000
|
|
||||||
)
|
|
||||||
|
|
||||||
println("\n--- New Message ---")
|
|
||||||
for (dataname, data, data_type) in payloads
|
|
||||||
if data_type == "text"
|
|
||||||
println("Text: $data")
|
|
||||||
elseif data_type == "image"
|
|
||||||
filename = "received_$dataname.bin"
|
|
||||||
write(filename, data)
|
|
||||||
println("Image saved: $filename")
|
|
||||||
elseif data_type == "audio"
|
|
||||||
filename = "received_$dataname.bin"
|
|
||||||
write(filename, data)
|
|
||||||
println("Audio saved: $filename")
|
|
||||||
elseif data_type == "table"
|
|
||||||
println("Table received:")
|
|
||||||
data = DataFrame(data)
|
|
||||||
show(data)
|
|
||||||
end
|
|
||||||
end
|
|
||||||
end
|
|
||||||
|
|
||||||
NATS.subscribe(SUBJECT) do msg
|
|
||||||
message_handler(msg)
|
|
||||||
end
|
|
||||||
println("Subscribed to: $SUBJECT")
|
|
||||||
end
|
|
||||||
|
|
||||||
function run_interactive_chat()
|
|
||||||
println("\n=== Interactive Chat ===")
|
|
||||||
println("1. Send a message")
|
|
||||||
println("2. Join a chat room")
|
|
||||||
println("3. Exit")
|
|
||||||
|
|
||||||
while true
|
|
||||||
print("\nSelect option (1-3): ")
|
|
||||||
choice = readline()
|
|
||||||
|
|
||||||
if choice == "1"
|
|
||||||
print("Enter message text: ")
|
|
||||||
text = readline()
|
|
||||||
send_chat_message(text)
|
|
||||||
elseif choice == "2"
|
|
||||||
receive_chat_messages()
|
|
||||||
elseif choice == "3"
|
|
||||||
break
|
|
||||||
end
|
|
||||||
end
|
|
||||||
end
|
|
||||||
|
|
||||||
end # module
|
|
||||||
|
|
||||||
# Run the chat app
|
|
||||||
using .ChatApp
|
|
||||||
ChatApp.run_interactive_chat()
|
|
||||||
```
|
|
||||||
|
|
||||||
## Step 10: Testing the Chat System
|
|
||||||
|
|
||||||
### Test Scenario 1: Text-Only Chat
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Terminal 1: Start the chat receiver
|
|
||||||
julia test_julia_to_julia_text_receiver.jl
|
|
||||||
|
|
||||||
# Terminal 2: Send a message
|
|
||||||
julia test_julia_to_julia_text_sender.jl
|
|
||||||
```
|
|
||||||
|
|
||||||
### Test Scenario 2: Image Chat
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Terminal 1: Receive messages
|
|
||||||
julia test_julia_to_julia_mix_payloads_receiver.jl
|
|
||||||
|
|
||||||
# Terminal 2: Send image
|
|
||||||
julia test_julia_to_julia_mix_payload_sender.jl
|
|
||||||
```
|
|
||||||
|
|
||||||
### Test Scenario 3: Large File Transfer
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Terminal 2: Send large file
|
|
||||||
julia test_julia_to_julia_mix_payload_sender.jl
|
|
||||||
```
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
This walkthrough demonstrated how to build a chat system using NATSBridge.jl with support for:
|
|
||||||
|
|
||||||
- Text messages
|
|
||||||
- Images (direct transport for small, link transport for large)
|
|
||||||
- Audio files
|
|
||||||
- Video files
|
|
||||||
- Tabular data (DataFrames)
|
|
||||||
- Bidirectional communication
|
|
||||||
- Mixed-content messages
|
|
||||||
|
|
||||||
The key takeaways are:
|
|
||||||
|
|
||||||
1. **Always wrap payloads in a list** - Even for single payloads: `[("dataname", data, "type")]`
|
|
||||||
2. **Use appropriate transport** - NATSBridge automatically handles size-based routing
|
|
||||||
3. **Support mixed content** - Multiple payloads of different types in one message
|
|
||||||
4. **Handle errors** - Implement proper error handling for network failures
|
|
||||||
5. **Use correlation IDs** - Track messages across distributed systems
|
|
||||||
|
|
||||||
For more information, see:
|
|
||||||
- [`docs/architecture.md`](./docs/architecture.md) - Detailed architecture documentation
|
|
||||||
- [`docs/implementation.md`](./docs/implementation.md) - Implementation details
|
|
||||||
Reference in New Issue
Block a user