diff --git a/Project.toml b/Project.toml index d8fdbc8..e362d04 100644 --- a/Project.toml +++ b/Project.toml @@ -1,6 +1,6 @@ name = "NATSBridge" uuid = "f2724d33-f338-4a57-b9f8-1be882570d10" -version = "0.4.1" +version = "0.4.2" authors = ["narawat "] [deps] diff --git a/docs/architecture.md b/docs/architecture.md index 65ad9fd..38129df 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,8 +1,13 @@ -# Architecture Documentation: Bi-Directional Data Bridge (Julia ↔ JavaScript) +# Architecture Documentation: Bi-Directional Data Bridge ## Overview -This document describes the architecture for a high-performance, bi-directional data bridge between a Julia service and a JavaScript (Node.js) service using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads. +This document describes the architecture for a high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads. + +The system enables seamless communication across all three platforms: +- **Julia ↔ JavaScript** bi-directional messaging +- **JavaScript ↔ Python/Micropython** bi-directional messaging +- **Julia ↔ Python/Micropython** bi-directional messaging (via JSON serialization) ### File Server Handler Architecture @@ -35,8 +40,24 @@ The system uses a **standardized list-of-tuples format** for all payload operati # Input format for smartsend (always a list of tuples with type info) [(dataname1, data1, type1), (dataname2, data2, type2), ...] -# Output format for smartreceive (always returns a list of tuples) -[(dataname1, data1, type1), (dataname2, data2, type2), ...] +# Output format for smartreceive (returns envelope dictionary with payloads field) +# Returns: Dict with envelope metadata and payloads field containing list of tuples +# { +# "correlationId": "...", +# "msgId": "...", +# "timestamp": "...", +# "sendTo": "...", +# "msgPurpose": "...", +# "senderName": "...", +# "senderId": "...", +# "receiverName": "...", +# "receiverId": "...", +# "replyTo": "...", +# "replyToMsgId": "...", +# "brokerURL": "...", +# "metadata": {...}, +# "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...] +# } ``` **Supported Types:** @@ -81,9 +102,10 @@ smartsend( nats_url="nats://localhost:4222" ) -# Receive always returns a list -payloads = smartreceive(msg, fileserverDownloadHandler, max_retries, base_delay, max_delay) -# payloads = [("dataname1", data1, type1), ("dataname2", data2, type2), ...] +# Receive returns a dictionary envelope with all metadata and deserialized payloads +envelope = smartreceive(msg, fileserverDownloadHandler, max_retries, base_delay, max_delay) +# envelope["payloads"] = [("dataname1", data1, type1), ("dataname2", data2, type2), ...] +# envelope["correlationId"], envelope["msgId"], etc. ``` ## Architecture Diagram @@ -118,7 +140,7 @@ flowchart TD ### 1. msgEnvelope_v1 - Message Envelope -The `msgEnvelope_v1` structure provides a comprehensive message format for bidirectional communication between Julia and JavaScript services. +The `msgEnvelope_v1` structure provides a comprehensive message format for bidirectional communication between Julia, JavaScript, and Python/Micropython applications. **Julia Structure:** ```julia @@ -194,7 +216,7 @@ end ### 2. msgPayload_v1 - Payload Structure -The `msgPayload_v1` structure provides flexible payload handling for various data types. +The `msgPayload_v1` structure provides flexible payload handling for various data types across all supported platforms. **Julia Structure:** ```julia @@ -222,15 +244,15 @@ end ┌─────────────────────────────────────────────────────────────┐ │ smartsend Function │ │ Accepts: [(dataname1, data1, type1), ...] │ -│ (No standalone type parameter - type per payload) │ +│ (Type is per payload, not standalone) │ └─────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ For each payload: │ -│ 1. Extract type from tuple │ -│ 2. Serialize based on type │ -│ 3. Check payload size │ +│ 1. Extract type from tuple │ +│ 2. Serialize based on type │ +│ 3. Check payload size │ └─────────────────────────────────────────────────────────────┘ │ ┌────────────────┴─-────────────────┐ @@ -249,19 +271,77 @@ end └─────────────────┘ └─────────────────┘ ``` -### 4. Julia Module Architecture +### 4. Cross-Platform Architecture + +```mermaid +flowchart TD + subgraph PythonMicropython + Py[Python/Micropython] + PySmartSend[smartsend] + PySmartReceive[smartreceive] + end + + subgraph JavaScript + JS[JavaScript] + JSSmartSend[smartsend] + JSSmartReceive[smartreceive] + end + + subgraph Julia + Julia[Julia] + JuliaSmartSend[smartsend] + JuliaSmartReceive[smartreceive] + end + + subgraph NATS + NATSServer[NATS Server] + end + + PySmartSend --> NATSServer + JSSmartSend --> NATSServer + JuliaSmartSend --> NATSServer + NATSServer --> PySmartReceive + NATSServer --> JSSmartReceive + NATSServer --> JuliaSmartReceive + + style PythonMicropython fill:#e1f5fe + style JavaScript fill:#f3e5f5 + style Julia fill:#e8f5e9 +``` + +### 5. Python/Micropython Module Architecture ```mermaid graph TD - subgraph JuliaModule - smartsendJulia[smartsend Julia] + subgraph PyModule + PySmartSend[smartsend] SizeCheck[Size Check] DirectPath[Direct Path] LinkPath[Link Path] HTTPClient[HTTP Client] end - smartsendJulia --> SizeCheck + PySmartSend --> SizeCheck + SizeCheck -->|< 1MB| DirectPath + SizeCheck -->|>= 1MB| LinkPath + LinkPath --> HTTPClient + + style PyModule fill:#b3e5fc +``` + +### 6. Julia Module Architecture + +```mermaid +graph TD + subgraph JuliaModule + JuliaSmartSend[smartsend] + SizeCheck[Size Check] + DirectPath[Direct Path] + LinkPath[Link Path] + HTTPClient[HTTP Client] + end + + JuliaSmartSend --> SizeCheck SizeCheck -->|< 1MB| DirectPath SizeCheck -->|>= 1MB| LinkPath LinkPath --> HTTPClient @@ -269,19 +349,19 @@ graph TD style JuliaModule fill:#c5e1a5 ``` -### 5. JavaScript Module Architecture +### 7. JavaScript Module Architecture ```mermaid graph TD subgraph JSModule - smartsendJS[smartsend JS] - smartreceiveJS[smartreceive JS] + JSSmartSend[smartsend] + JSSmartReceive[smartreceive] JetStreamConsumer[JetStream Pull Consumer] ApacheArrow[Apache Arrow] end - smartsendJS --> NATS - smartreceiveJS --> JetStreamConsumer + JSSmartSend --> NATS + JSSmartReceive --> JetStreamConsumer JetStreamConsumer --> ApacheArrow style JSModule fill:#f3e5f5 @@ -338,23 +418,25 @@ function smartreceive( # If direct: decode Base64 payload # If link: fetch from URL with exponential backoff using fileserverDownloadHandler # Deserialize payload based on type - # Return list of (dataname, data, type) tuples + # Return envelope dictionary with all metadata and deserialized payloads end ``` **Output Format:** -- Always returns a list of tuples: `[(dataname1, data1, type1), (dataname2, data2, type2), ...]` -- Even for single payloads: `[(dataname1, data1, type1)]` +- Returns a dictionary (key-value map) containing all envelope fields: + - `correlationId`, `msgId`, `timestamp`, `sendTo`, `msgPurpose`, `senderName`, `senderId`, `receiverName`, `receiverId`, `replyTo`, `replyToMsgId`, `brokerURL` + - `metadata` - Message-level metadata dictionary + - `payloads` - List of dictionaries, each containing deserialized payload data **Process Flow:** -1. Parse the JSON envelope to extract the `payloads` array +1. Parse the JSON envelope to extract all fields 2. Iterate through each payload in `payloads` 3. For each payload: - Determine transport type (`direct` or `link`) - If `direct`: decode Base64 data from the message - If `link`: fetch data from URL using exponential backoff (via `fileserverDownloadHandler`) - Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.) -4. Return list of `(dataname, data, type)` tuples +4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples **Note:** The `fileserverDownloadHandler` receives `(url::String, max_retries::Int, base_delay::Int, max_delay::Int, correlation_id::String)` and returns `Vector{UInt8}`. @@ -401,21 +483,27 @@ async function smartreceive(msg, options = {}) // - correlationId: optional correlation ID for tracing ``` +**Output Format:** +- Returns a dictionary (key-value map) containing all envelope fields: + - `correlationId`, `msgId`, `timestamp`, `sendTo`, `msgPurpose`, `senderName`, `senderId`, `receiverName`, `receiverId`, `replyTo`, `replyToMsgId`, `brokerURL` + - `metadata` - Message-level metadata dictionary + - `payloads` - List of dictionaries, each containing deserialized payload data + **Process Flow:** -1. Parse the JSON envelope to extract the `payloads` array +1. Parse the JSON envelope to extract all fields 2. Iterate through each payload in `payloads` 3. For each payload: - Determine transport type (`direct` or `link`) - If `direct`: decode Base64 data from the message - If `link`: fetch data from URL using exponential backoff - Deserialize based on payload type (`dictionary`, `table`, `binary`, etc.) -4. Return list of `(dataname, data, type)` tuples +4. Return envelope dictionary with `payloads` field containing list of `(dataname, data, type)` tuples ## Scenario Implementations ### Scenario 1: Command & Control (Small Dictionary) -**Julia (Receiver):** +**Julia (Sender/Receiver):** ```julia # Subscribe to control subject # Parse JSON envelope @@ -423,15 +511,21 @@ async function smartreceive(msg, options = {}) # Send acknowledgment ``` -**JavaScript (Sender):** +**JavaScript (Sender/Receiver):** ```javascript // Create small dictionary config // Send via smartsend with type="dictionary" ``` +**Python/Micropython (Sender/Receiver):** +```python +# Create small dictionary config +# Send via smartsend with type="dictionary" +``` + ### Scenario 2: Deep Dive Analysis (Large Arrow Table) -**Julia (Sender):** +**Julia (Sender/Receiver):** ```julia # Create large DataFrame # Convert to Arrow IPC stream @@ -440,7 +534,7 @@ async function smartreceive(msg, options = {}) # Publish NATS with URL ``` -**JavaScript (Receiver):** +**JavaScript (Sender/Receiver):** ```javascript // Receive NATS message with URL // Fetch data from HTTP server @@ -448,42 +542,64 @@ async function smartreceive(msg, options = {}) // Load into Perspective.js or D3 ``` +**Python/Micropython (Sender/Receiver):** +```python +# Create large DataFrame +# Convert to Arrow IPC stream +# Check size (> 1MB) +# Upload to HTTP server +# Publish NATS with URL +``` + ### Scenario 3: Live Audio Processing -**JavaScript (Sender):** +**JavaScript (Sender/Receiver):** ```javascript // Capture audio chunk // Send as binary with metadata headers // Use smartsend with type="audio" ``` -**Julia (Receiver):** +**Julia (Sender/Receiver):** ```julia -// Receive audio data -// Perform FFT or AI transcription -// Send results back (JSON + Arrow table) +# Receive audio data +# Perform FFT or AI transcription +# Send results back (JSON + Arrow table) +``` + +**Python/Micropython (Sender/Receiver):** +```python +# Capture audio chunk +# Send as binary with metadata headers +# Use smartsend with type="audio" ``` ### Scenario 4: Catch-Up (JetStream) -**Julia (Producer):** +**Julia (Producer/Consumer):** ```julia # Publish to JetStream # Include metadata for temporal tracking ``` -**JavaScript (Consumer):** +**JavaScript (Producer/Consumer):** ```javascript // Connect to JetStream // Request replay from last 10 minutes // Process historical and real-time messages ``` +**Python/Micropython (Producer/Consumer):** +```python +# Publish to JetStream +# Include metadata for temporal tracking +``` + ### Scenario 5: Selection (Low Bandwidth) -**Focus:** Small Arrow tables, Julia to JavaScript. The Action: Julia wants to send a small DataFrame to show on a JavaScript dashboard for the user to choose. +**Focus:** Small Arrow tables, cross-platform communication. The Action: Any platform wants to send a small DataFrame to show on any receiving application for the user to choose. -**Julia (Sender):** +**Julia (Sender/Receiver):** ```julia # Create small DataFrame (e.g., 50KB - 500KB) # Convert to Arrow IPC stream @@ -492,7 +608,7 @@ async function smartreceive(msg, options = {}) # Include metadata for dashboard selection context ``` -**JavaScript (Receiver):** +**JavaScript (Sender/Receiver):** ```javascript // Receive NATS message with direct transport // Decode Base64 payload @@ -502,11 +618,20 @@ async function smartreceive(msg, options = {}) // Send selection back to Julia ``` -**Use Case:** Julia server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to JavaScript dashboard for user selection. The selection is then sent back to Julia for processing. +**Python/Micropython (Sender/Receiver):** +```python +# Create small DataFrame (e.g., 50KB - 500KB) +# Convert to Arrow IPC stream +# Check payload size (< 1MB threshold) +# Publish directly to NATS with Base64-encoded payload +# Include metadata for dashboard selection context +``` + +**Use Case:** Any server generates a list of available options (e.g., file selections, configuration presets) as a small DataFrame and sends to any receiving application for user selection. The selection is then sent back to the sender for processing. ### Scenario 6: Chat System -**Focus:** Every conversational message is composed of any number and any combination of components, spanning the full spectrum from small to large. This includes text, images, audio, video, tables, and files—specifically accommodating everything from brief snippets to high-resolution images, large audio files, extensive tables, and massive documents. Support for claim-check delivery and full bi-directional messaging. +**Focus:** Every conversational message is composed of any number and any combination of components, spanning the full spectrum from small to large. This includes text, images, audio, video, tables, and files—specifically accommodating everything from brief snippets to high-resolution images, large audio files, extensive tables, and massive documents. Support for claim-check delivery and full bi-directional messaging across all platforms. **Multi-Payload Support:** The system supports mixed-payload messages where a single message can contain multiple payloads with different transport strategies. The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. @@ -545,7 +670,25 @@ async function smartreceive(msg, options = {}) // Support bidirectional reply with claim-check delivery confirmation ``` -**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components. +**Python/Micropython (Sender/Receiver):** +```python +# Build chat message with mixed payloads: +# - Text: direct transport (Base64) +# - Small images: direct transport (Base64) +# - Large images: link transport (HTTP URL) +# - Audio/video: link transport (HTTP URL) +# - Tables: direct or link depending on size +# - Files: link transport (HTTP URL) +# +# Each payload uses appropriate transport strategy: +# - Size < 1MB → direct (NATS + Base64) +# - Size >= 1MB → link (HTTP upload + NATS URL) +# +# Include claim-check metadata for delivery tracking +# Support bidirectional messaging with replyTo fields +``` + +**Use Case:** Full-featured chat system supporting rich media. User can send text, small images directly, or upload large files that get uploaded to HTTP server and referenced via URLs. Claim-check pattern ensures reliable delivery tracking for all message components across all platforms. **Implementation Note:** The `smartreceive` function iterates through all payloads in the envelope and processes each according to its transport type. See the standard API format in Section 1: `msgEnvelope_v1` supports `AbstractArray{msgPayload_v1}` for multiple payloads. diff --git a/docs/implementation.md b/docs/implementation.md index 109de2e..6cb1265 100644 --- a/docs/implementation.md +++ b/docs/implementation.md @@ -2,7 +2,22 @@ ## Overview -This document describes the implementation of the high-performance, bi-directional data bridge between Julia and JavaScript services using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads. +This document describes the implementation of the high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads. + +The system enables seamless communication across all three platforms: +- **Julia ↔ JavaScript** bi-directional messaging +- **JavaScript ↔ Python/Micropython** bi-directional messaging +- **Julia ↔ Python/Micropython** bi-directional messaging (via JSON serialization) + +### Implementation Files + +NATSBridge is implemented in three languages, each providing the same API: + +| Language | Implementation File | Description | +|----------|---------------------|-------------| +| **Julia** | [`src/NATSBridge.jl`](../src/NATSBridge.jl) | Full Julia implementation with Arrow IPC support | +| **JavaScript** | [`src/NATSBridge.js`](../src/NATSBridge.js) | JavaScript implementation for Node.js and browsers | +| **Python/Micropython** | [`src/nats_bridge.py`](../src/nats_bridge.py) | Python implementation for desktop and microcontrollers | ### Multi-Payload Support @@ -13,8 +28,24 @@ The implementation uses a **standardized list-of-tuples format** for all payload # Input format for smartsend (always a list of tuples with type info) [(dataname1, data1, type1), (dataname2, data2, type2), ...] -# Output format for smartreceive (always returns a list of tuples with type info) -[(dataname1, data1, type1), (dataname2, data2, type2), ...] +# Output format for smartreceive (returns envelope dictionary with payloads field) +# Returns: Dict with envelope metadata and payloads field containing list of tuples +# { +# "correlationId": "...", +# "msgId": "...", +# "timestamp": "...", +# "sendTo": "...", +# "msgPurpose": "...", +# "senderName": "...", +# "senderId": "...", +# "receiverName": "...", +# "receiverId": "...", +# "replyTo": "...", +# "replyToMsgId": "...", +# "brokerURL": "...", +# "metadata": {...}, +# "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...] +# } ``` Where `type` can be: `"text"`, `"dictionary"`, `"table"`, `"image"`, `"audio"`, `"video"`, `"binary"` @@ -27,49 +58,103 @@ smartsend("/test", [(dataname1, data1, "text")], ...) # Multiple payloads in one message (each payload has its own type) smartsend("/test", [(dataname1, data1, "dictionary"), (dataname2, data2, "table")], ...) -# Receive always returns a list with type info -payloads = smartreceive(msg, ...) -# payloads = [(dataname1, data1, "text"), (dataname2, data2, "table"), ...] +# Receive returns a dictionary envelope with all metadata and deserialized payloads +envelope = smartreceive(msg, ...) +# envelope["payloads"] = [(dataname1, data1, "text"), (dataname2, data2, "table"), ...] +# envelope["correlationId"], envelope["msgId"], etc. ``` +## Cross-Platform Interoperability + +NATSBridge is designed for seamless communication between Julia, JavaScript, and Python/Micropython applications. All three implementations share the same interface and data format, ensuring compatibility across platforms. + +### Platform-Specific Features + +| Feature | Julia | JavaScript | Python/Micropython | +|---------|-------|------------|-------------------| +| Direct NATS transport | ✅ | ✅ | ✅ | +| HTTP file server (Claim-Check) | ✅ | ✅ | ✅ | +| Arrow IPC tables | ✅ | ✅ | ✅ | +| Base64 encoding | ✅ | ✅ | ✅ | +| Exponential backoff | ✅ | ✅ | ✅ | +| Correlation ID tracking | ✅ | ✅ | ✅ | +| Reply-to support | ✅ | ✅ | ✅ | + +### Data Type Mapping + +| Type | Julia | JavaScript | Python/Micropython | +|------|-------|------------|-------------------| +| `text` | `String` | `String` | `str` | +| `dictionary` | `Dict` | `Object` | `dict` | +| `table` | `DataFrame` | `Array` | `DataFrame` / `list` | +| `image` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` | +| `audio` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` | +| `video` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` | +| `binary` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` | + +### Example: Julia ↔ Python ↔ JavaScript + +```julia +# Julia sender +using NATSBridge +data = [("message", "Hello from Julia!", "text")] +smartsend("/cross_platform", data, nats_url="nats://localhost:4222") +``` + +```javascript +// JavaScript receiver +const { smartreceive } = require('./src/NATSBridge'); +const envelope = await smartreceive(msg); +// envelope.payloads[0].data === "Hello from Julia!" +``` + +```python +# Python sender +from nats_bridge import smartsend +data = [("response", "Hello from Python!", "text")] +smartsend("/cross_platform", data, nats_url="nats://localhost:4222") +``` + +All three platforms can communicate seamlessly using the same NATS subjects and data format. + ## Architecture -The implementation follows the Claim-Check pattern: +All three implementations (Julia, JavaScript, Python/Micropython) follow the same Claim-Check pattern: ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ SmartSend Function │ └─────────────────────────────────────────────────────────────────────────┘ - │ - ▼ + │ + ▼ ┌─────────────────────────────────────────────────────────────────────────┐ │ Is payload size < 1MB? │ └─────────────────────────────────────────────────────────────────────────┘ - │ - ┌─────────────────┴─────────────────┐ - ▼ ▼ - ┌─────────────────┐ ┌─────────────────┐ - │ Direct Path │ │ Link Path │ - │ (< 1MB) │ │ (> 1MB) │ - │ │ │ │ - │ • Serialize to │ │ • Serialize to │ - │ IOBuffer │ │ IOBuffer │ - │ • Base64 encode │ │ • Upload to │ - │ • Publish to │ │ HTTP Server │ - │ NATS │ │ • Publish to │ - │ │ │ NATS with URL │ - └─────────────────┘ └─────────────────┘ + │ + ┌─────────────────┴─────────────────┐ + ▼ ▼ + ┌─────────────────┐ ┌─────────────────┐ + │ Direct Path │ │ Link Path │ + │ (< 1MB) │ │ (> 1MB) │ + │ │ │ │ + │ • Serialize to │ │ • Serialize to │ + │ Buffer │ │ Buffer │ + │ • Base64 encode │ │ • Upload to │ + │ • Publish to │ │ HTTP Server │ + │ NATS │ │ • Publish to │ + │ │ │ NATS with URL │ + └─────────────────┘ └─────────────────┘ ``` ## Files -### Julia Module: [`src/julia_bridge.jl`](../src/julia_bridge.jl) +### Julia Module: [`src/NATSBridge.jl`](../src/NATSBridge.jl) The Julia implementation provides: -- **[`MessageEnvelope`](../src/julia_bridge.jl)**: Struct for the unified JSON envelope -- **[`SmartSend()`](../src/julia_bridge.jl)**: Handles transport selection based on payload size -- **[`SmartReceive()`](../src/julia_bridge.jl)**: Handles both direct and link transport +- **[`MessageEnvelope`](src/NATSBridge.jl)**: Struct for the unified JSON envelope +- **[`SmartSend()`](src/NATSBridge.jl)**: Handles transport selection based on payload size +- **[`SmartReceive()`](src/NATSBridge.jl)**: Handles both direct and link transport ### JavaScript Module: [`src/NATSBridge.js`](../src/NATSBridge.js) @@ -77,8 +162,17 @@ The JavaScript implementation provides: - **`MessageEnvelope` class**: For the unified JSON envelope - **`MessagePayload` class**: For individual payload representation -- **[`smartsend()`](../src/NATSBridge.js)**: Handles transport selection based on payload size -- **[`smartreceive()`](../src/NATSBridge.js)**: Handles both direct and link transport +- **[`smartsend()`](src/NATSBridge.js)**: Handles transport selection based on payload size +- **[`smartreceive()`](src/NATSBridge.js)**: Handles both direct and link transport + +### Python/Micropython Module: [`src/nats_bridge.py`](../src/nats_bridge.py) + +The Python/Micropython implementation provides: + +- **`MessageEnvelope` class**: For the unified JSON envelope +- **`MessagePayload` class**: For individual payload representation +- **[`smartsend()`](src/nats_bridge.py)**: Handles transport selection based on payload size +- **[`smartreceive()`](src/nats_bridge.py)**: Handles both direct and link transport ## Installation @@ -100,6 +194,23 @@ Pkg.add("Dates") npm install nats.js apache-arrow uuid base64-url ``` +### Python/Micropython Dependencies + +1. Copy [`src/nats_bridge.py`](../src/nats_bridge.py) to your device +2. Ensure you have the following dependencies: + + **For Python (desktop):** + ```bash + pip install nats-py + ``` + + **For Micropython:** + - `urequests` for HTTP requests + - `base64` for base64 encoding (built-in) + - `json` for JSON handling (built-in) + - `socket` for networking (built-in) + - `uuid` for UUID generation (built-in) + ## Usage Tutorial ### Step 1: Start NATS Server @@ -138,34 +249,31 @@ node test/scenario3_julia_to_julia.js ### Scenario 0: Basic Multi-Payload Example -#### Julia (Sender) -```julia -using NATSBridge +#### Python/Micropython (Sender) +```python +from nats_bridge import smartsend # Send multiple payloads in one message (type is required per payload) smartsend( "/test", [("dataname1", data1, "dictionary"), ("dataname2", data2, "table")], nats_url="nats://localhost:4222", - fileserver_url="http://localhost:8080", - metadata=Dict("custom_key" => "custom_value") + fileserver_url="http://localhost:8080" ) # Even single payload must be wrapped in a list with type smartsend("/test", [("single_data", mydata, "dictionary")]) ``` -#### Julia (Receiver) -```julia -using NATSBridge +#### Python/Micropython (Receiver) +```python +from nats_bridge import smartreceive -# Receive returns a list of payloads with type info -payloads = smartreceive(msg, "http://localhost:8080") +# Receive returns a list of (dataname, data, type) tuples +payloads = smartreceive(msg) # payloads = [(dataname1, data1, "dictionary"), (dataname2, data2, "table"), ...] ``` -### Scenario 1: Command & Control (Small JSON) - #### JavaScript (Sender) ```javascript const { smartsend } = require('./src/NATSBridge'); @@ -198,27 +306,7 @@ const configs = [ await smartsend("control", configs); ``` -#### Julia (Receiver) -```julia -using NATS -using JSON3 - -# Subscribe to control subject -subscribe(nats, "control") do msg - env = MessageEnvelope(String(msg.data)) - config = JSON3.read(env.payload) - - # Execute simulation with parameters - step_size = config.step_size - iterations = config.iterations - - # Send acknowledgment - response = Dict("status" => "Running", "correlation_id" => env.correlation_id) - publish(nats, "control_response", JSON3.stringify(response)) -end -``` - -### JavaScript (Receiver) +#### JavaScript (Receiver) ```javascript const { smartreceive } = require('./src/NATSBridge'); @@ -227,13 +315,18 @@ const nc = await connect({ servers: ['nats://localhost:4222'] }); const sub = nc.subscribe("control"); for await (const msg of sub) { - const result = await smartreceive(msg); + const envelope = await smartreceive(msg); - // Process the result - for (const { dataname, data, type } of result) { + // Process the payloads from the envelope + for (const payload of envelope.payloads) { + const { dataname, data, type } = payload; console.log(`Received ${dataname} of type ${type}`); console.log(`Data: ${JSON.stringify(data)}`); } + + // Also access envelope metadata + console.log(`Correlation ID: ${envelope.correlationId}`); + console.log(`Message ID: ${envelope.msgId}`); } ``` @@ -259,15 +352,35 @@ await SmartSend("analysis_results", [("table_data", df, "table")]); ```javascript const { smartreceive } = require('./src/NATSBridge'); -const result = await smartreceive(msg); +const envelope = await smartreceive(msg); -// Use table data for visualization with Perspective.js or D3 +// Use table data from the payloads field // Note: Tables are sent as arrays of objects in JavaScript -const table = result; +const table = envelope.payloads; ``` ### Scenario 3: Live Binary Processing +#### Python/Micropython (Sender) +```python +from nats_bridge import smartsend + +# Binary data wrapped in a list +binary_data = [ + ("audio_chunk", binary_buffer, "binary") +] + +smartsend( + "binary_input", + binary_data, + nats_url="nats://localhost:4222", + metadata={ + "sample_rate": 44100, + "channels": 1 + } +) +``` + #### JavaScript (Sender) ```javascript const { smartsend } = require('./src/NATSBridge'); @@ -287,35 +400,35 @@ await smartsend("binary_input", binaryData, { }); ``` -#### Julia (Receiver) -```julia -using WAV -using DSP +#### Python/Micropython (Receiver) +```python +from nats_bridge import smartreceive # Receive binary data -function process_binary(data) - # Perform FFT or AI transcription - spectrum = fft(data) +def process_binary(msg): + envelope = smartreceive(msg) - # Send results back (JSON + Arrow table) - results = Dict("transcription" => "sample text", "spectrum" => spectrum) - await SmartSend("binary_output", results, "json") -end + # Process the binary data from envelope.payloads + for dataname, data, type in envelope["payloads"]: + if type == "binary": + # data is bytes + print(f"Received binary data: {dataname}, size: {len(data)}") + # Perform FFT or AI transcription here ``` -### JavaScript (Receiver) +#### JavaScript (Receiver) ```javascript const { smartreceive } = require('./src/NATSBridge'); // Receive binary data function process_binary(msg) { - const result = await smartreceive(msg); + const envelope = await smartreceive(msg); - // Process the binary data - for (const { dataname, data, type } of result) { - if (type === "binary") { + // Process the binary data from envelope.payloads + for (const payload of envelope.payloads) { + if (payload.type === "binary") { // data is an ArrayBuffer or Uint8Array - console.log(`Received binary data: ${dataname}, size: ${data.length}`); + console.log(`Received binary data: ${payload.dataname}, size: ${payload.data.length}`); // Perform FFT or AI transcription here } } @@ -353,13 +466,71 @@ const consumer = await js.pullSubscribe("health", { // Process historical and real-time messages for await (const msg of consumer) { - const result = await smartreceive(msg); - // result contains the list of payloads + const envelope = await smartreceive(msg); + // envelope.payloads contains the list of payloads // Each payload has: dataname, data, type msg.ack(); } ``` +### Scenario 4: Micropython Device Control + +**Focus:** Sending configuration to a Micropython device over NATS. This demonstrates the lightweight nature of the Python implementation suitable for microcontrollers. + +**Python/Micropython (Receiver/Device):** +```python +from nats_bridge import smartsend, smartreceive +import json + +# Device configuration handler +def handle_device_config(msg): + envelope = smartreceive(msg) + + # Process configuration from payloads + for dataname, data, type in envelope["payloads"]: + if type == "dictionary": + print(f"Received configuration: {data}") + # Apply configuration to device + if "wifi_ssid" in data: + wifi_ssid = data["wifi_ssid"] + wifi_password = data["wifi_password"] + update_wifi_config(wifi_ssid, wifi_password) + + # Send confirmation back + config = { + "status": "configured", + "wifi_ssid": "MyNetwork", + "ip": get_device_ip() + } + smartsend( + "device/response", + [("config", config, "dictionary")], + nats_url="nats://localhost:4222", + reply_to=envelope.get("replyTo") + ) +``` + +**JavaScript (Sender/Controller):** +```javascript +const { smartsend } = require('./src/NATSBridge'); + +// Send configuration to Micropython device +await smartsend("device/config", [ + { + dataname: "config", + data: { + wifi_ssid: "MyNetwork", + wifi_password: "password123", + update_interval: 60, + temperature_threshold: 30.0 + }, + type: "dictionary" + } +]); +``` + +**Use Case:** A controller sends WiFi and operational configuration to a Micropython device (e.g., ESP32). The device receives the configuration, applies it, and sends back a confirmation with its current status. + ### Scenario 5: Selection (Low Bandwidth) **Focus:** Small Arrow tables, Julia to JavaScript. The Action: Julia wants to send a small DataFrame to show on a JavaScript dashboard for the user to choose. @@ -395,11 +566,11 @@ smartsend( const { smartreceive, smartsend } = require('./src/NATSBridge'); // Receive NATS message with direct transport -const result = await smartreceive(msg); +const envelope = await smartreceive(msg); // Decode Base64 payload (for direct transport) -// For tables, data is an array of objects -const table = result; // Array of objects +// For tables, data is in envelope.payloads +const table = envelope.payloads; // Array of objects // User makes selection const selection = uiComponent.getSelectedOption(); @@ -558,7 +729,7 @@ await smartsend("chat.room123", message); ### Exponential Backoff - Maximum retry count: 5 - Base delay: 100ms, max delay: 5000ms -- Implemented in both Julia and JavaScript implementations +- Implemented in all three implementations (Julia, JavaScript, Python/Micropython) ### Correlation ID Logging - Log correlation_id at every stage @@ -567,14 +738,73 @@ await smartsend("chat.room123", message); ## Testing -Run the test scripts: +Run the test scripts for each platform: + +### Python/Micropython Tests ```bash -# Scenario 1: Command & Control (JavaScript sender) -node test/scenario1_command_control.js +# Basic functionality test +python test/test_micropython_basic.py +``` -# Scenario 2: Large Arrow Table (JavaScript sender) -node test/scenario2_large_table.js +### JavaScript Tests + +```bash +# Text message exchange +node test/test_js_to_js_text_sender.js +node test/test_js_to_js_text_receiver.js + +# Dictionary exchange +node test/test_js_to_js_dict_sender.js +node test/test_js_to_js_dict_receiver.js + +# File transfer (direct transport) +node test/test_js_to_js_file_sender.js +node test/test_js_to_js_file_receiver.js + +# Mixed payload types +node test/test_js_to_js_mix_payloads_sender.js +node test/test_js_to_js_mix_payloads_receiver.js + +# Table (Arrow IPC) exchange +node test/test_js_to_js_table_sender.js +node test/test_js_to_js_table_receiver.js +``` + +### Julia Tests + +```bash +# Text message exchange +julia test/test_julia_to_julia_text_sender.jl +julia test/test_julia_to_julia_text_receiver.jl + +# Dictionary exchange +julia test/test_julia_to_julia_dict_sender.jl +julia test/test_julia_to_julia_dict_receiver.jl + +# File transfer +julia test/test_julia_to_julia_file_sender.jl +julia test/test_julia_to_julia_file_receiver.jl + +# Mixed payload types +julia test/test_julia_to_julia_mix_payloads_sender.jl +julia test/test_julia_to_julia_mix_payloads_receiver.jl + +# Table exchange +julia test/test_julia_to_julia_table_sender.jl +julia test/test_julia_to_julia_table_receiver.jl +``` + +### Cross-Platform Tests + +```bash +# Julia ↔ JavaScript communication +julia test/test_julia_to_julia_text_sender.jl +node test/test_js_to_js_text_receiver.js + +# Python ↔ JavaScript communication +python test/test_micropython_basic.py +node test/test_js_to_js_text_receiver.js ``` ## Troubleshooting @@ -582,18 +812,24 @@ node test/scenario2_large_table.js ### Common Issues 1. **NATS Connection Failed** - - Ensure NATS server is running - - Check NATS_URL configuration + - **Julia/JavaScript/Python**: Ensure NATS server is running + - **Python/Micropython**: Check `nats_url` parameter and network connectivity 2. **HTTP Upload Failed** - Ensure file server is running - - Check FILESERVER_URL configuration + - Check `fileserver_url` configuration - Verify upload permissions + - **Micropython**: Ensure `urequests` is available and network is connected 3. **Arrow IPC Deserialization Error** - Ensure data is properly serialized to Arrow format - Check Arrow version compatibility +4. **Python/Micropython Specific Issues** + - **Import Error**: Ensure `nats_bridge.py` is in the correct path + - **Memory Error (Micropython)**: Reduce payload size or use link transport for large payloads + - **Unicode Error**: Ensure proper encoding when sending text data + ## License MIT \ No newline at end of file diff --git a/etc.jl b/etc.jl index 85c5745..e69de29 100644 --- a/etc.jl +++ b/etc.jl @@ -1,21 +0,0 @@ -Check architecture.jl, NATSBridge.jl and its test files: -- test_julia_to_julia_table_receiver.jl -- test_julia_to_julia_table_sender.jl. - -Now I want to test sending a mix-content message from Julia serviceA to Julia serviceB, for example, a chat system. -The test message must show that any combination and any number and any data size of text | json | table | image | audio | video | binary can be send and receive. - -Can you write me the following test files: -- test_julia_to_julia_mix_receiver.jl -- test_julia_to_julia_mix_sender.jl - - - - -1. create a tutorial file "tutorial_julia.md" for NATSBridge.jl -2. create a walkthrough file "walkthrough_julia.md" for NATSBridge.jl - -You may consult architecture.md for more info. - - - diff --git a/examples/micropython_example.py b/examples/micropython_example.py deleted file mode 100644 index 59db61f..0000000 --- a/examples/micropython_example.py +++ /dev/null @@ -1,221 +0,0 @@ -""" -Micropython NATS Bridge - Simple Example - -This example demonstrates the basic usage of the NATSBridge for Micropython. -""" - -import sys -sys.path.insert(0, "../src") - -from nats_bridge import smartsend, smartreceive, log_trace -import json - - -def example_simple_chat(): - """ - Simple chat example: Send text messages via NATS. - - Sender (this script): - - Sends a text message to NATS - - Uses direct transport (no fileserver needed) - - Receiver (separate script): - - Listens to NATS - - Receives and processes the message - """ - print("=== Simple Chat Example ===") - print() - - # Define the message data as list of (dataname, data, type) tuples - data = [ - ("message", "Hello from Micropython!", "text") - ] - - # Send the message - env = smartsend( - "/chat/room1", - data, - nats_url="nats://localhost:4222", - msg_purpose="chat", - sender_name="micropython-client" - ) - - print("Message sent!") - print(" Subject: {}".format(env.send_to)) - print(" Correlation ID: {}".format(env.correlation_id)) - print(" Payloads: {}".format(len(env.payloads))) - print() - - # Expected receiver output: - print("Expected receiver output:") - print(" [timestamp] [Correlation: ...] Starting smartsend for subject: /chat/room1") - print(" [timestamp] [Correlation: ...] Serialized payload 'message' (type: text) size: 22 bytes") - print(" [timestamp] [Correlation: ...] Using direct transport for 22 bytes") - print(" [timestamp] [Correlation: ...] Message published to /chat/room1") - print() - - return env - - -def example_send_json(): - """ - Example: Send JSON configuration to a Micropython device. - - This demonstrates sending structured data (dictionary type). - """ - print("\n=== Send JSON Configuration ===") - print() - - # Define configuration as dictionary - config = { - "wifi_ssid": "MyNetwork", - "wifi_password": "password123", - "server_host": "mqtt.example.com", - "server_port": 1883, - "update_interval": 60 - } - - # Send configuration - data = [ - ("device_config", config, "dictionary") - ] - - env = smartsend( - "/device/config", - data, - nats_url="nats://localhost:4222", - msg_purpose="updateStatus", - sender_name="server" - ) - - print("Configuration sent!") - print(" Subject: {}".format(env.send_to)) - print(" Payloads: {}".format(len(env.payloads))) - print() - - return env - - -def example_receive_message(msg): - """ - Example: Receive and process a NATS message. - - Args: - msg: The NATS message received (should be dict or JSON string) - - Returns: - list: List of (dataname, data, type) tuples - """ - print("\n=== Receive Message ===") - print() - - # Process the message - payloads = smartreceive( - msg, - fileserver_download_handler=None, # Not needed for direct transport - max_retries=3, - base_delay=100, - max_delay=1000 - ) - - print("Received {} payload(s):".format(len(payloads))) - for dataname, data, type in payloads: - print(" - {}: {} (type: {})".format(dataname, data, type)) - - return payloads - - -def example_mixed_content(): - """ - Example: Send mixed content (text + dictionary + binary). - - This demonstrates the multi-payload capability. - """ - print("\n=== Mixed Content Example ===") - print() - - # Create mixed content - image_data = b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR" # Example PNG header - - data = [ - ("message_text", "Hello with image!", "text"), - ("user_config", {"theme": "dark", "notifications": True}, "dictionary"), - ("user_avatar", image_data, "binary") - ] - - env = smartsend( - "/chat/mixed", - data, - nats_url="nats://localhost:4222", - msg_purpose="chat", - sender_name="micropython-client" - ) - - print("Mixed content sent!") - print(" Subject: {}".format(env.send_to)) - print(" Payloads:") - for p in env.payloads: - print(" - {} (transport: {}, type: {}, size: {} bytes)".format( - p.dataname, p.transport, p.type, p.size)) - - return env - - -def example_reply(): - """ - Example: Send a message with reply-to functionality. - - This demonstrates request-response pattern. - """ - print("\n=== Request-Response Example ===") - print() - - # Send command - data = [ - ("command", {"action": "read_sensor", "sensor_id": "temp1"}, "dictionary") - ] - - env = smartsend( - "/device/command", - data, - nats_url="nats://localhost:4222", - msg_purpose="command", - sender_name="server", - reply_to="/device/response", - reply_to_msg_id="cmd-001" - ) - - print("Command sent!") - print(" Subject: {}".format(env.send_to)) - print(" Reply To: {}".format(env.reply_to)) - print(" Reply To Msg ID: {}".format(env.reply_to_msg_id)) - print() - - print("Expected receiver behavior:") - print(" 1. Receive command on /device/command") - print(" 2. Process command") - print(" 3. Send response to /device/response") - print(" 4. Include replyToMsgId in response") - - return env - - -if __name__ == "__main__": - print("Micropython NATS Bridge Examples") - print("================================") - print() - - # Run examples - example_simple_chat() - example_send_json() - example_mixed_content() - example_reply() - - print("\n=== Examples Completed ===") - print() - print("To run these examples, you need:") - print(" 1. A running NATS server at nats://localhost:4222") - print(" 2. Import the nats_bridge module") - print(" 3. Call the desired example function") - print() - print("For more examples, see test/test_micropython_basic.py") \ No newline at end of file diff --git a/examples/tutorial.md b/examples/tutorial.md new file mode 100644 index 0000000..fcc71dc --- /dev/null +++ b/examples/tutorial.md @@ -0,0 +1,604 @@ +# NATSBridge Tutorial + +A step-by-step guide to get started with NATSBridge - a high-performance, bi-directional data bridge for **Julia**, **JavaScript**, and **Python/Micropython**. + +## Table of Contents + +1. [Overview](#overview) +2. [Prerequisites](#prerequisites) +3. [Installation](#installation) +4. [Quick Start](#quick-start) +5. [Basic Examples](#basic-examples) +6. [Advanced Usage](#advanced-usage) +7. [Cross-Platform Communication](#cross-platform-communication) + +--- + +## Overview + +NATSBridge enables seamless communication between Julia, JavaScript, and Python/Micropython applications through NATS, with automatic transport selection based on payload size: + +- **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded) +- **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL + +### Supported Payload Types + +| Type | Description | +|------|-------------| +| `text` | Plain text strings | +| `dictionary` | JSON-serializable dictionaries | +| `table` | Tabular data (Arrow IPC format) | +| `image` | Image data (PNG, JPG bytes) | +| `audio` | Audio data (WAV, MP3 bytes) | +| `video` | Video data (MP4, AVI bytes) | +| `binary` | Generic binary data | + +--- + +## Prerequisites + +Before you begin, ensure you have: + +1. **NATS Server** running (or accessible) +2. **HTTP File Server** (optional, for large payloads > 1MB) +3. **One of the supported platforms**: Julia, JavaScript (Node.js), or Python/Micropython + +--- + +## Installation + +### Julia + +```julia +using Pkg +Pkg.add("NATS") +Pkg.add("Arrow") +Pkg.add("JSON3") +Pkg.add("HTTP") +Pkg.add("UUIDs") +Pkg.add("Dates") +``` + +### JavaScript + +```bash +npm install nats.js apache-arrow uuid base64-url +``` + +### Python/Micropython + +1. Copy `src/nats_bridge.py` to your device +2. Install dependencies: + +**For Python (desktop):** +```bash +pip install nats-py +``` + +**For Micropython:** +- `urequests` for HTTP requests +- `base64` for base64 encoding (built-in) +- `json` for JSON handling (built-in) + +--- + +## Quick Start + +### Step 1: Start NATS Server + +```bash +docker run -p 4222:4222 nats:latest +``` + +### Step 2: Start HTTP File Server (Optional) + +```bash +# Create a directory for file uploads +mkdir -p /tmp/fileserver + +# Use Python's built-in server +python3 -m http.server 8080 --directory /tmp/fileserver +``` + +### Step 3: Send Your First Message + +#### Python/Micropython + +```python +from nats_bridge import smartsend + +# Send a text message +data = [("message", "Hello World", "text")] +env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222") +print("Message sent!") +``` + +#### JavaScript + +```javascript +const { smartsend } = require('./src/NATSBridge'); + +// Send a text message +await smartsend("/chat/room1", [ + { dataname: "message", data: "Hello World", type: "text" } +], { natsUrl: "nats://localhost:4222" }); + +console.log("Message sent!"); +``` + +#### Julia + +```julia +using NATSBridge + +# Send a text message +data = [("message", "Hello World", "text")] +env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222") +println("Message sent!") +``` + +### Step 4: Receive Messages + +#### Python/Micropython + +```python +from nats_bridge import smartreceive + +# Receive and process message +envelope = smartreceive(msg) +for dataname, data, type in envelope["payloads"]: + print(f"Received {dataname}: {data}") +``` + +#### JavaScript + +```javascript +const { smartreceive } = require('./src/NATSBridge'); + +// Receive and process message +const envelope = await smartreceive(msg); +for (const payload of envelope.payloads) { + console.log(`Received ${payload.dataname}: ${payload.data}`); +} +``` + +#### Julia + +```julia +using NATSBridge + +# Receive and process message +envelope = smartreceive(msg, fileserverDownloadHandler) +for (dataname, data, type) in envelope["payloads"] + println("Received $dataname: $data") +end +``` + +--- + +## Basic Examples + +### Example 1: Sending a Dictionary + +#### Python/Micropython + +```python +from nats_bridge import smartsend + +# Create configuration dictionary +config = { + "wifi_ssid": "MyNetwork", + "wifi_password": "password123", + "update_interval": 60 +} + +# Send as dictionary type +data = [("config", config, "dictionary")] +env = smartsend("/device/config", data, nats_url="nats://localhost:4222") +``` + +#### JavaScript + +```javascript +const { smartsend } = require('./src/NATSBridge'); + +const config = { + wifi_ssid: "MyNetwork", + wifi_password: "password123", + update_interval: 60 +}; + +await smartsend("/device/config", [ + { dataname: "config", data: config, type: "dictionary" } +]); +``` + +#### Julia + +```julia +using NATSBridge + +config = Dict( + "wifi_ssid" => "MyNetwork", + "wifi_password" => "password123", + "update_interval" => 60 +) + +data = [("config", config, "dictionary")] +smartsend("/device/config", data) +``` + +### Example 2: Sending Binary Data (Image) + +#### Python/Micropython + +```python +from nats_bridge import smartsend + +# Read image file +with open("image.png", "rb") as f: + image_data = f.read() + +# Send as binary type +data = [("user_image", image_data, "binary")] +env = smartsend("/chat/image", data, nats_url="nats://localhost:4222") +``` + +#### JavaScript + +```javascript +const { smartsend } = require('./src/NATSBridge'); + +// Read image file (Node.js) +const fs = require('fs'); +const image_data = fs.readFileSync('image.png'); + +await smartsend("/chat/image", [ + { dataname: "user_image", data: image_data, type: "binary" } +]); +``` + +#### Julia + +```julia +using NATSBridge + +# Read image file +image_data = read("image.png") + +data = [("user_image", image_data, "binary")] +smartsend("/chat/image", data) +``` + +### Example 3: Request-Response Pattern + +#### Python/Micropython (Requester) + +```python +from nats_bridge import smartsend + +# Send command with reply-to +data = [("command", {"action": "read_sensor"}, "dictionary")] +env = smartsend( + "/device/command", + data, + nats_url="nats://localhost:4222", + reply_to="/device/response", + reply_to_msg_id="cmd-001" +) +``` + +#### JavaScript (Responder) + +```javascript +const { smartreceive, smartsend } = require('./src/NATSBridge'); + +// Subscribe to command topic +const sub = nc.subscribe("/device/command"); + +for await (const msg of sub) { + const envelope = await smartreceive(msg); + + // Process command + for (const payload of envelope.payloads) { + if (payload.dataname === "command") { + const command = payload.data; + + if (command.action === "read_sensor") { + // Read sensor and send response + const response = { + sensor_id: "sensor-001", + value: 42.5, + timestamp: new Date().toISOString() + }; + + await smartsend("/device/response", [ + { dataname: "sensor_data", data: response, type: "dictionary" } + ], { + reply_to: envelope.replyTo, + reply_to_msg_id: envelope.msgId + }); + } + } + } +} +``` + +--- + +## Advanced Usage + +### Example 4: Large Payloads (File Server) + +For payloads larger than 1MB, NATSBridge automatically uses the file server: + +#### Python/Micropython + +```python +from nats_bridge import smartsend +import os + +# Create large data (> 1MB) +large_data = os.urandom(2_000_000) # 2MB of random data + +# Send with file server URL +env = smartsend( + "/data/large", + [("large_file", large_data, "binary")], + nats_url="nats://localhost:4222", + fileserver_url="http://localhost:8080", + size_threshold=1_000_000 +) + +# The envelope will contain the download URL +print(f"File uploaded to: {env.payloads[0].data}") +``` + +#### JavaScript + +```javascript +const { smartsend } = require('./src/NATSBridge'); + +// Create large data (> 1MB) +const largeData = new ArrayBuffer(2_000_000); +const view = new Uint8Array(largeData); +view.fill(42); // Fill with some data + +await smartsend("/data/large", [ + { dataname: "large_file", data: largeData, type: "binary" } +], { + fileserverUrl: "http://localhost:8080", + sizeThreshold: 1_000_000 +}); +``` + +#### Julia + +```julia +using NATSBridge + +# Create large data (> 1MB) +large_data = rand(UInt8, 2_000_000) + +env = smartsend( + "/data/large", + [("large_file", large_data, "binary")], + fileserver_url="http://localhost:8080" +) + +# The envelope will contain the download URL +println("File uploaded to: $(env.payloads[1].data)") +``` + +### Example 5: Mixed Content (Chat with Text + Image) + +NATSBridge supports sending multiple payloads with different types in a single message: + +#### Python/Micropython + +```python +from nats_bridge import smartsend + +# Read image file +with open("avatar.png", "rb") as f: + image_data = f.read() + +# Send mixed content +data = [ + ("message_text", "Hello with image!", "text"), + ("user_avatar", image_data, "image") +] + +env = smartsend("/chat/mixed", data, nats_url="nats://localhost:4222") +``` + +#### JavaScript + +```javascript +const { smartsend } = require('./src/NATSBridge'); + +const fs = require('fs'); + +await smartsend("/chat/mixed", [ + { + dataname: "message_text", + data: "Hello with image!", + type: "text" + }, + { + dataname: "user_avatar", + data: fs.readFileSync("avatar.png"), + type: "image" + } +]); +``` + +#### Julia + +```julia +using NATSBridge + +image_data = read("avatar.png") + +data = [ + ("message_text", "Hello with image!", "text"), + ("user_avatar", image_data, "image") +] + +smartsend("/chat/mixed", data) +``` + +### Example 6: Table Data (Arrow IPC) + +For tabular data, NATSBridge uses Apache Arrow IPC format: + +#### Python/Micropython + +```python +from nats_bridge import smartsend +import pandas as pd + +# Create DataFrame +df = pd.DataFrame({ + "id": [1, 2, 3], + "name": ["Alice", "Bob", "Charlie"], + "score": [95, 88, 92] +}) + +# Send as table type +data = [("students", df, "table")] +env = smartsend("/data/students", data, nats_url="nats://localhost:4222") +``` + +#### Julia + +```julia +using NATSBridge +using DataFrames + +# Create DataFrame +df = DataFrame( + id = [1, 2, 3], + name = ["Alice", "Bob", "Charlie"], + score = [95, 88, 92] +) + +data = [("students", df, "table")] +smartsend("/data/students", data) +``` + +--- + +## Cross-Platform Communication + +NATSBridge enables seamless communication between different platforms: + +### Julia ↔ JavaScript + +#### Julia Sender + +```julia +using NATSBridge + +# Send dictionary from Julia to JavaScript +config = Dict("step_size" => 0.01, "iterations" => 1000) +data = [("config", config, "dictionary")] +smartsend("/analysis/config", data, nats_url="nats://localhost:4222") +``` + +#### JavaScript Receiver + +```javascript +const { smartreceive } = require('./src/NATSBridge'); + +// Receive dictionary from Julia +const envelope = await smartreceive(msg); +for (const payload of envelope.payloads) { + if (payload.type === "dictionary") { + console.log("Received config:", payload.data); + // payload.data = { step_size: 0.01, iterations: 1000 } + } +} +``` + +### JavaScript ↔ Python + +#### JavaScript Sender + +```javascript +const { smartsend } = require('./src/NATSBridge'); + +await smartsend("/data/transfer", [ + { dataname: "message", data: "Hello from JS!", type: "text" } +]); +``` + +#### Python Receiver + +```python +from nats_bridge import smartreceive + +envelope = smartreceive(msg) +for dataname, data, type in envelope["payloads"]: + if type == "text": + print(f"Received from JS: {data}") +``` + +### Python ↔ Julia + +#### Python Sender + +```python +from nats_bridge import smartsend + +data = [("message", "Hello from Python!", "text")] +smartsend("/chat/python", data) +``` + +#### Julia Receiver + +```julia +using NATSBridge + +envelope = smartreceive(msg, fileserverDownloadHandler) +for (dataname, data, type) in envelope["payloads"] + if type == "text" + println("Received from Python: $data") + end +end +``` + +--- + +## Next Steps + +1. **Explore the test directory** for more examples +2. **Check the documentation** for advanced configuration options +3. **Join the community** to share your use cases + +--- + +## Troubleshooting + +### Connection Issues + +- Ensure NATS server is running: `docker ps | grep nats` +- Check firewall settings +- Verify NATS URL configuration + +### File Server Issues + +- Ensure file server is running and accessible +- Check upload permissions +- Verify file server URL configuration + +### Serialization Errors + +- Verify data type matches the specified type +- Check that binary data is in the correct format (bytes/Vector{UInt8}) + +--- + +## License + +MIT \ No newline at end of file diff --git a/examples/walkthrough.md b/examples/walkthrough.md new file mode 100644 index 0000000..0e42bbb --- /dev/null +++ b/examples/walkthrough.md @@ -0,0 +1,1045 @@ +# NATSBridge Walkthrough + +A comprehensive guide to building real-world applications with NATSBridge. + +## Table of Contents + +1. [Introduction](#introduction) +2. [Architecture Overview](#architecture-overview) +3. [Building a Chat Application](#building-a-chat-application) +4. [Building a File Transfer System](#building-a-file-transfer-system) +5. [Building a Streaming Data Pipeline](#building-a-streaming-data-pipeline) +6. [Building a Micropython IoT Device](#building-a-micropython-iot-device) +7. [Building a Cross-Platform Dashboard](#building-a-cross-platform-dashboard) +8. [Performance Optimization](#performance-optimimization) +9. [Best Practices](#best-practices) + +--- + +## Introduction + +This walkthrough will guide you through building several real-world applications using NATSBridge. We'll cover: + +- Chat applications with rich media support +- File transfer systems with claim-check pattern +- Streaming data pipelines +- Micropython IoT devices +- Cross-platform dashboards + +Each section builds on the previous one, gradually increasing in complexity. + +--- + +## Architecture Overview + +### System Components + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ NATSBridge Architecture │ +├─────────────────────────────────────────────────────────────────┤ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ Julia │ │ JavaScript │ │ Python/Micr │ │ +│ │ (NATS.jl) │◄──►│ (nats.js) │◄──►│ opython │ │ +│ └──────────────┘ └──────────────┘ └──────────────┘ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌──────────────────────────────────────────────┐ │ +│ │ NATS │ │ +│ │ (Message Broker) │ │ +│ └──────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌──────────────────┐ │ +│ │ File Server │ │ +│ │ (HTTP Upload) │ │ +│ └──────────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Message Flow + +1. **Sender** creates a message envelope with payloads +2. **NATSBridge** serializes and encodes payloads +3. **Transport Decision**: Small payloads go directly to NATS, large payloads are uploaded to file server +4. **NATS** routes messages to subscribers +5. **Receiver** fetches payloads (from NATS or file server) +6. **NATSBridge** deserializes and decodes payloads + +--- + +## Building a Chat Application + +Let's build a full-featured chat application that supports text, images, and file attachments. + +### Step 1: Set Up the Project + +```bash +# Create project directory +mkdir -p chat-app/src +cd chat-app + +# Create configuration file +cat > config.json << 'EOF' +{ + "nats_url": "nats://localhost:4222", + "fileserver_url": "http://localhost:8080", + "size_threshold": 1048576 +} +EOF +``` + +### Step 2: Create the Chat Interface (JavaScript) + +```javascript +// src/chat-ui.js +class ChatUI { + constructor() { + this.messages = []; + this.currentRoom = null; + this.setupEventListeners(); + } + + setupEventListeners() { + document.getElementById('send-btn').addEventListener('click', () => this.sendMessage()); + document.getElementById('file-input').addEventListener('change', (e) => this.handleFileSelect(e)); + } + + async sendMessage() { + const messageInput = document.getElementById('message-input'); + const text = messageInput.value.trim(); + + if (!text && !this.selectedFile) return; + + const data = []; + + // Add text message + if (text) { + data.push({ + dataname: "text", + data: text, + type: "text" + }); + } + + // Add file if selected + if (this.selectedFile) { + const fileData = await this.readFile(this.selectedFile); + data.push({ + dataname: "attachment", + data: fileData, + type: this.getFileType(this.selectedFile.type) + }); + } + + await smartsend( + `/chat/${this.currentRoom}`, + data, + { + fileserverUrl: window.config.fileserver_url, + sizeThreshold: window.config.size_threshold + } + ); + + messageInput.value = ''; + this.selectedFile = null; + document.getElementById('file-name').textContent = ''; + } + + handleFileSelect(event) { + this.selectedFile = event.target.files[0]; + document.getElementById('file-name').textContent = `Selected: ${this.selectedFile.name}`; + } + + readFile(file) { + return new Promise((resolve, reject) => { + const reader = new FileReader(); + reader.onload = () => resolve(reader.result); + reader.onerror = reject; + reader.readAsArrayBuffer(file); + }); + } + + getFileType(mimeType) { + if (mimeType.startsWith('image/')) return 'image'; + if (mimeType.startsWith('audio/')) return 'audio'; + if (mimeType.startsWith('video/')) return 'video'; + return 'binary'; + } + + addMessage(user, text, attachment = null) { + const messagesContainer = document.getElementById('messages'); + const messageDiv = document.createElement('div'); + messageDiv.className = 'message'; + + let content = `
${user}
`; + content += `
${text}
`; + + if (attachment) { + if (attachment.type === 'image') { + content += ``; + } else { + content += `Download attachment`; + } + } + + messageDiv.innerHTML = content; + messagesContainer.appendChild(messageDiv); + messagesContainer.scrollTop = messagesContainer.scrollHeight; + } +} +``` + +### Step 3: Create the Message Handler + +```javascript +// src/chat-handler.js +const { smartreceive } = require('./NATSBridge'); + +class ChatHandler { + constructor(natsConnection) { + this.nats = natsConnection; + this.ui = new ChatUI(); + } + + async start() { + // Subscribe to chat rooms + const rooms = ['general', 'tech', 'random']; + + for (const room of rooms) { + await this.nats.subscribe(`/chat/${room}`, async (msg) => { + await this.handleMessage(msg); + }); + } + + console.log('Chat handler started'); + } + + async handleMessage(msg) { + const envelope = await smartreceive(msg, { + fileserverDownloadHandler: this.downloadFile.bind(this) + }); + + // Extract sender info from envelope + const sender = envelope.senderName || 'Anonymous'; + + // Process each payload + for (const payload of envelope.payloads) { + if (payload.type === 'text') { + this.ui.addMessage(sender, payload.data); + } else if (payload.type === 'image') { + // Convert to data URL for display + const base64 = this.arrayBufferToBase64(payload.data); + this.ui.addMessage(sender, null, { + type: 'image', + data: `data:image/png;base64,${base64}` + }); + } else { + // For other types, use file server URL + this.ui.addMessage(sender, null, { + type: payload.type, + data: payload.data + }); + } + } + } + + downloadFile(url, max_retries, base_delay, max_delay, correlation_id) { + return fetch(url) + .then(response => response.arrayBuffer()); + } + + arrayBufferToBase64(buffer) { + const bytes = new Uint8Array(buffer); + let binary = ''; + for (let i = 0; i < bytes.length; i++) { + binary += String.fromCharCode(bytes[i]); + } + return btoa(binary); + } +} +``` + +### Step 4: Run the Application + +```bash +# Start NATS +docker run -p 4222:4222 nats:latest + +# Start file server +mkdir -p /tmp/fileserver +python3 -m http.server 8080 --directory /tmp/fileserver + +# Run chat app +node src/chat-ui.js +node src/chat-handler.js +``` + +--- + +## Building a File Transfer System + +Let's build a file transfer system that handles large files efficiently. + +### Step 1: File Upload Service + +```javascript +// src/file-upload-service.js +const { smartsend } = require('./NATSBridge'); + +class FileUploadService { + constructor(natsUrl, fileserverUrl) { + this.natsUrl = natsUrl; + this.fileserverUrl = fileserverUrl; + } + + async uploadFile(filePath, recipient) { + const fs = require('fs'); + const fileData = fs.readFileSync(filePath); + const fileName = require('path').basename(filePath); + + const data = [{ + dataname: fileName, + data: fileData, + type: 'binary' + }]; + + const envelope = await smartsend( + `/files/${recipient}`, + data, + { + natsUrl: this.natsUrl, + fileserverUrl: this.fileserverUrl, + sizeThreshold: 1048576 + } + ); + + return envelope; + } + + async uploadLargeFile(filePath, recipient) { + // For very large files, stream upload + const fs = require('fs'); + const fileSize = fs.statSync(filePath).size; + + if (fileSize > 100 * 1024 * 1024) { // > 100MB + console.log('File too large for direct upload, using streaming...'); + return await this.streamUpload(filePath, recipient); + } + + return await this.uploadFile(filePath, recipient); + } + + async streamUpload(filePath, recipient) { + // Implement streaming upload to file server + // This would require a more sophisticated file server + // For now, we'll use the standard upload + return await this.uploadFile(filePath, recipient); + } +} + +module.exports = FileUploadService; +``` + +### Step 2: File Download Service + +```javascript +// src/file-download-service.js +const { smartreceive } = require('./NATSBridge'); +const fs = require('fs'); + +class FileDownloadService { + constructor(natsUrl) { + this.natsUrl = natsUrl; + this.downloads = new Map(); + } + + async downloadFile(sender, downloadId) { + // Subscribe to sender's file channel + const envelope = await smartreceive(msg, { + fileserverDownloadHandler: this.fetchFromUrl.bind(this) + }); + + // Process each payload + for (const payload of envelope.payloads) { + if (payload.type === 'binary') { + const filePath = `/downloads/${payload.dataname}`; + fs.writeFileSync(filePath, payload.data); + console.log(`File saved to ${filePath}`); + } + } + } + + async fetchFromUrl(url, max_retries, base_delay, max_delay, correlation_id) { + const response = await fetch(url); + if (!response.ok) { + throw new Error(`Failed to fetch: ${response.status}`); + } + return await response.arrayBuffer(); + } +} + +module.exports = FileDownloadService; +``` + +### Step 3: File Transfer CLI + +```javascript +// src/cli.js +const { smartsend, smartreceive } = require('./NATSBridge'); +const fs = require('fs'); +const readline = require('readline'); + +const rl = readline.createInterface({ + input: process.stdin, + output: process.stdout +}); + +async function main() { + const config = JSON.parse(fs.readFileSync('config.json', 'utf8')); + + console.log('File Transfer System'); + console.log('===================='); + console.log('1. Upload file'); + console.log('2. Download file'); + console.log('3. List pending downloads'); + + const choice = await rl.question('Enter choice: '); + + if (choice === '1') { + await uploadFile(config); + } else if (choice === '2') { + await downloadFile(config); + } + + rl.close(); +} + +async function uploadFile(config) { + const filePath = await rl.question('Enter file path: '); + const recipient = await rl.question('Enter recipient: '); + + const fileService = new FileUploadService(config.nats_url, config.fileserver_url); + + try { + const envelope = await fileService.uploadFile(filePath, recipient); + console.log('Upload successful!'); + console.log(`File ID: ${envelope.payloads[0].id}`); + } catch (error) { + console.error('Upload failed:', error.message); + } +} + +async function downloadFile(config) { + const sender = await rl.question('Enter sender: '); + const fileService = new FileDownloadService(config.nats_url); + + try { + await fileService.downloadFile(sender); + console.log('Download complete!'); + } catch (error) { + console.error('Download failed:', error.message); + } +} + +main(); +``` + +--- + +## Building a Streaming Data Pipeline + +Let's build a data pipeline that processes streaming data from sensors. + +### Step 1: Sensor Data Model + +```python +# src/sensor_data.py +from dataclasses import dataclass +from datetime import datetime +from typing import List, Dict, Any +import json + +@dataclass +class SensorReading: + sensor_id: str + timestamp: str + value: float + unit: str + metadata: Dict[str, Any] = None + + def to_dict(self) -> Dict[str, Any]: + return { + "sensor_id": self.sensor_id, + "timestamp": self.timestamp, + "value": self.value, + "unit": self.unit, + "metadata": self.metadata or {} + } + +class SensorBatch: + def __init__(self): + self.readings: List[SensorReading] = [] + + def add_reading(self, reading: SensorReading): + self.readings.append(reading) + + def to_dataframe(self): + import pandas as pd + data = [r.to_dict() for r in self.readings] + return pd.DataFrame(data) +``` + +### Step 2: Sensor Sender + +```python +# src/sensor-sender.py +from nats_bridge import smartsend +from sensor_data import SensorReading, SensorBatch +import time +import random + +class SensorSender: + def __init__(self, nats_url: str, fileserver_url: str): + self.nats_url = nats_url + self.fileserver_url = fileserver_url + + def send_reading(self, sensor_id: str, value: float, unit: str): + reading = SensorReading( + sensor_id=sensor_id, + timestamp=datetime.now().isoformat(), + value=value, + unit=unit + ) + + data = [("reading", reading.to_dict(), "dictionary")] + + smartsend( + f"/sensors/{sensor_id}", + data, + nats_url=self.nats_url, + fileserver_url=self.fileserver_url + ) + + def send_batch(self, readings: List[SensorReading]): + batch = SensorBatch() + for reading in readings: + batch.add_reading(reading) + + df = batch.to_dataframe() + + # Convert to Arrow IPC format + import pyarrow as pa + table = pa.Table.from_pandas(df) + + # Serialize to Arrow IPC + import io + buf = io.BytesIO() + with pa.ipc.new_stream(buf, table.schema) as writer: + writer.write_table(table) + + arrow_data = buf.getvalue() + + # Send based on size + if len(arrow_data) < 1048576: # < 1MB + data = [("batch", arrow_data, "table")] + smartsend( + f"/sensors/batch", + data, + nats_url=self.nats_url, + fileserver_url=self.fileserver_url + ) + else: + # Upload to file server + from nats_bridge import plik_oneshot_upload + response = plik_oneshot_upload(self.fileserver_url, "batch.arrow", arrow_data) + print(f"Uploaded batch to {response['url']}") +``` + +### Step 3: Sensor Receiver + +```python +# src/sensor-receiver.py +from nats_bridge import smartreceive +from sensor_data import SensorReading +import pandas as pd +import pyarrow as pa +import io + +class SensorReceiver: + def __init__(self, fileserver_download_handler): + self.fileserver_download_handler = fileserver_download_handler + + def process_reading(self, msg): + envelope = smartreceive(msg, self.fileserver_download_handler) + + for dataname, data, data_type in envelope["payloads"]: + if data_type == "dictionary": + reading = SensorReading( + sensor_id=data["sensor_id"], + timestamp=data["timestamp"], + value=data["value"], + unit=data["unit"], + metadata=data.get("metadata", {}) + ) + print(f"Received: {reading}") + + elif data_type == "table": + # Deserialize Arrow IPC + table = pa.ipc.open_stream(io.BytesIO(data)).read_all() + df = table.to_pandas() + print(f"Received batch with {len(df)} readings") + print(df) +``` + +--- + +## Building a Micropython IoT Device + +Let's build an IoT device using Micropython that connects to NATS. + +### Step 1: Device Configuration + +```python +# device_config.py +import json + +class DeviceConfig: + def __init__(self, ssid, password, nats_url, device_id): + self.ssid = ssid + self.password = password + self.nats_url = nats_url + self.device_id = device_id + + def to_dict(self): + return { + "ssid": self.ssid, + "password": self.password, + "nats_url": self.nats_url, + "device_id": self.device_id + } +``` + +### Step 2: Device Bridge + +```python +# device_bridge.py +from nats_bridge import smartsend, smartreceive +import json + +class DeviceBridge: + def __init__(self, config): + self.config = config + self.nats_url = config.nats_url + + def connect(self): + # Connect to WiFi + import network + wlan = network.WLAN(network.STA_IF) + wlan.active(True) + wlan.connect(self.config.ssid, self.config.password) + + while not wlan.isconnected(): + pass + + print('Connected to WiFi') + print('Network config:', wlan.ifconfig()) + + def send_status(self, status): + data = [("status", status, "dictionary")] + smartsend( + f"/devices/{self.config.device_id}/status", + data, + nats_url=self.nats_url + ) + + def send_sensor_data(self, sensor_id, value, unit): + data = [ + ("sensor_id", sensor_id, "text"), + ("value", value, "dictionary") + ] + smartsend( + f"/devices/{self.config.device_id}/sensors/{sensor_id}", + data, + nats_url=self.nats_url + ) + + def receive_commands(self, callback): + # Subscribe to commands + import socket + # Simplified subscription - in production, use proper NATS client + + while True: + # Poll for messages + msg = self._poll_for_message() + if msg: + envelope = smartreceive(msg) + + # Process payloads + for dataname, data, data_type in envelope["payloads"]: + if dataname == "command": + callback(data) + + def _poll_for_message(self): + # Simplified message polling + # In production, implement proper NATS subscription + return None +``` + +### Step 3: Device Application + +```python +# device_app.py +from device_config import DeviceConfig +from device_bridge import DeviceBridge +import time +import random + +# Load configuration +config = DeviceConfig( + ssid="MyNetwork", + password="password123", + nats_url="nats://localhost:4222", + device_id="device-001" +) + +bridge = DeviceBridge(config) +bridge.connect() + +# Send initial status +bridge.send_status({ + "status": "online", + "version": "1.0.0", + "uptime": 0 +}) + +# Main loop +while True: + # Read sensor data + temperature = random.uniform(20, 30) + humidity = random.uniform(40, 60) + + bridge.send_sensor_data("temperature", temperature, "celsius") + bridge.send_sensor_data("humidity", humidity, "percent") + + # Send status update + bridge.send_status({ + "status": "online", + "temperature": temperature, + "humidity": humidity + }) + + time.sleep(60) # Every minute +``` + +--- + +## Building a Cross-Platform Dashboard + +Let's build a dashboard that displays data from multiple platforms. + +### Step 1: Dashboard Server (Python) + +```python +# src/dashboard-server.py +from nats_bridge import smartsend, smartreceive +import pandas as pd +import pyarrow as pa +import io + +class DashboardServer: + def __init__(self, nats_url, fileserver_url): + self.nats_url = nats_url + self.fileserver_url = fileserver_url + + def broadcast_data(self, df): + # Convert to Arrow IPC + table = pa.Table.from_pandas(df) + buf = io.BytesIO() + with pa.ipc.new_stream(buf, table.schema) as writer: + writer.write_table(table) + + arrow_data = buf.getvalue() + + # Broadcast to all subscribers + data = [("data", arrow_data, "table")] + smartsend( + "/dashboard/data", + data, + nats_url=self.nats_url, + fileserver_url=self.fileserver_url + ) + + def receive_selection(self, callback): + def handler(msg): + envelope = smartreceive(msg) + + for dataname, data, data_type in envelope["payloads"]: + if data_type == "dictionary": + callback(data) + + # Subscribe to selections + import threading + thread = threading.Thread(target=self._listen_for_selections, args=(handler,)) + thread.daemon = True + thread.start() + + def _listen_for_selections(self, handler): + # Simplified subscription + # In production, implement proper NATS subscription + pass +``` + +### Step 2: Dashboard UI (JavaScript) + +```javascript +// src/dashboard-ui.js +class DashboardUI { + constructor() { + this.data = null; + this.setupEventListeners(); + } + + setupEventListeners() { + document.getElementById('refresh-btn').addEventListener('click', () => this.refreshData()); + document.getElementById('export-btn').addEventListener('click', () => this.exportData()); + } + + async refreshData() { + // Request fresh data + await smartsend("/dashboard/request", [ + { dataname: "request", data: { type: "refresh" }, type: "dictionary" } + ], { + fileserverUrl: window.config.fileserver_url + }); + } + + async fetchData() { + // Subscribe to data updates + const envelope = await smartreceive(msg, { + fileserverDownloadHandler: this.fetchFromUrl.bind(this) + }); + + // Process table data + for (const payload of envelope.payloads) { + if (payload.type === 'table') { + // Deserialize Arrow IPC + this.data = this.deserializeArrow(payload.data); + this.renderTable(); + } + } + } + + deserializeArrow(data) { + // Deserialize Arrow IPC to JavaScript array + // This would require the apache-arrow library + return JSON.parse(JSON.stringify(data)); // Simplified + } + + renderTable() { + const tableContainer = document.getElementById('data-table'); + tableContainer.innerHTML = ''; + + if (!this.data) return; + + // Render table headers + const headers = Object.keys(this.data[0]); + const thead = document.createElement('thead'); + const headerRow = document.createElement('tr'); + + headers.forEach(header => { + const th = document.createElement('th'); + th.textContent = header; + headerRow.appendChild(th); + }); + + thead.appendChild(headerRow); + tableContainer.appendChild(thead); + + // Render table rows + const tbody = document.createElement('tbody'); + + this.data.forEach(row => { + const tr = document.createElement('tr'); + headers.forEach(header => { + const td = document.createElement('td'); + td.textContent = row[header]; + tr.appendChild(td); + }); + tbody.appendChild(tr); + }); + + tableContainer.appendChild(tbody); + } + + exportData() { + const csv = this.toCSV(); + const blob = new Blob([csv], { type: 'text/csv' }); + const url = URL.createObjectURL(blob); + + const a = document.createElement('a'); + a.href = url; + a.download = 'dashboard_data.csv'; + a.click(); + } + + toCSV() { + if (!this.data) return ''; + + const headers = Object.keys(this.data[0]); + const rows = this.data.map(row => + headers.map(h => row[h]).join(',') + ); + + return [headers.join(','), ...rows].join('\n'); + } + + fetchFromUrl(url) { + return fetch(url) + .then(response => response.arrayBuffer()); + } +} +``` + +--- + +## Performance Optimization + +### 1. Batch Processing + +```python +# Batch multiple readings into a single message +def send_batch_readings(self, readings): + batch = SensorBatch() + for reading in readings: + batch.add_reading(reading) + + df = batch.to_dataframe() + + # Convert to Arrow IPC + table = pa.Table.from_pandas(df) + buf = io.BytesIO() + with pa.ipc.new_stream(buf, table.schema) as writer: + writer.write_table(table) + + arrow_data = buf.getvalue() + + # Send as single message + smartsend( + "/sensors/batch", + [("batch", arrow_data, "table")], + nats_url=self.nats_url + ) +``` + +### 2. Connection Pooling + +```javascript +// Reuse NATS connections +const nats = require('nats'); + +class ConnectionPool { + constructor() { + this.connections = new Map(); + } + + getConnection(natsUrl) { + if (!this.connections.has(natsUrl)) { + this.connections.set(natsUrl, nats.connect({ servers: [natsUrl] })); + } + return this.connections.get(natsUrl); + } + + closeAll() { + this.connections.forEach(conn => conn.close()); + this.connections.clear(); + } +} +``` + +### 3. Caching + +```python +# Cache file server responses +from functools import lru_cache + +@lru_cache(maxsize=100) +def fetch_with_caching(url, max_retries=5, base_delay=100, max_delay=5000, correlation_id=""): + return _fetch_with_backoff(url, max_retries, base_delay, max_delay, correlation_id) +``` + +--- + +## Best Practices + +### 1. Error Handling + +```python +def safe_smartsend(subject, data, **kwargs): + try: + return smartsend(subject, data, **kwargs) + except Exception as e: + print(f"Failed to send message: {e}") + return None +``` + +### 2. Logging + +```python +import logging + +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + +def log_send(subject, data, correlation_id): + logger.info(f"Sending to {subject}: {len(data)} payloads, correlation_id={correlation_id}") + +def log_receive(correlation_id, num_payloads): + logger.info(f"Received message: {num_payloads} payloads, correlation_id={correlation_id}") +``` + +### 3. Rate Limiting + +```python +import time +from collections import deque + +class RateLimiter: + def __init__(self, max_requests, time_window): + self.max_requests = max_requests + self.time_window = time_window + self.requests = deque() + + def allow(self): + now = time.time() + + # Remove old requests + while self.requests and self.requests[0] < now - self.time_window: + self.requests.popleft() + + if len(self.requests) >= self.max_requests: + return False + + self.requests.append(now) + return True +``` + +--- + +## Conclusion + +This walkthrough covered: + +- Building a chat application with rich media support +- Building a file transfer system with claim-check pattern +- Building a streaming data pipeline for sensor data +- Building a Micropython IoT device +- Building a cross-platform dashboard + +For more information, check the [API documentation](../src/README.md) and [test examples](../test/). + +--- + +## License + +MIT \ No newline at end of file diff --git a/src/NATSBridge.jl b/src/NATSBridge.jl index 1ed7258..9e224ec 100644 --- a/src/NATSBridge.jl +++ b/src/NATSBridge.jl @@ -759,8 +759,8 @@ function smartreceive( error("Unknown transport type for payload '$dataname': $(transport)") # Throw error for unknown transport end end - - return payloads_list # Return list of (dataname, data, data_type) tuples + json_data["payloads"] = payloads_list + return json_data # Return envelope with list of (dataname, data, data_type) tuples in payloads field end diff --git a/src/NATSBridge.js b/src/NATSBridge.js index e794c4f..f939433 100644 --- a/src/NATSBridge.js +++ b/src/NATSBridge.js @@ -603,7 +603,7 @@ async function smartreceive(msg, options = {}) { * @param {number} options.baseDelay - Initial delay for exponential backoff in ms (default: 100) * @param {number} options.maxDelay - Maximum delay for exponential backoff in ms (default: 5000) * - * @returns {Promise} - List of {dataname, data, type} objects + * @returns {Promise} - Envelope dictionary with metadata and payloads field containing list of {dataname, data, type} objects */ const { fileserverDownloadHandler = _fetch_with_backoff, @@ -664,7 +664,10 @@ async function smartreceive(msg, options = {}) { } } - return payloads_list; + // Replace payloads array with the processed list of {dataname, data, type} tuples + json_data.payloads = payloads_list; + + return json_data; } // Export for Node.js diff --git a/src/README.md b/src/README.md index ed394a2..0b3daa9 100644 --- a/src/README.md +++ b/src/README.md @@ -1,17 +1,17 @@ -# NATSBridge for Micropython +# NATSBridge -A high-performance, bi-directional data bridge for Micropython devices using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads. +A high-performance, bi-directional data bridge for **Julia**, **JavaScript**, and **Python/Micropython** using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads. ## Overview -This module provides functionality for sending and receiving data over NATS with automatic transport selection based on payload size: +NATSBridge enables seamless communication between Julia, JavaScript, and Python/Micropython applications through NATS, with automatic transport selection based on payload size: - **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded) - **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL ## Features -- ✅ Bi-directional NATS communication +- ✅ Bi-directional NATS communication across Julia ↔ JavaScript ↔ Python/Micropython - ✅ Multi-payload support (mixed content in single message) - ✅ Automatic transport selection based on payload size - ✅ File server integration for large payloads @@ -31,19 +31,56 @@ This module provides functionality for sending and receiving data over NATS with | `video` | Video data (MP4, AVI bytes) | | `binary` | Generic binary data | +## Implementation Guides + +### [Julia Implementation](../tutorial_julia.md) + +See the [Julia tutorial](../tutorial_julia.md) for getting started with Julia. + +### [JavaScript Implementation](#javascript-implementation) + +See [`NATSBridge.js`](NATSBridge.js) for the JavaScript implementation. + +### [Python/Micropython Implementation](#pythonmicropython-implementation) + +See [`nats_bridge.py`](nats_bridge.py) for the Python/Micropython implementation. + ## Installation -1. Copy `nats_bridge.py` to your Micropython device +### Julia + +```julia +using Pkg +Pkg.add("NATS") +Pkg.add("Arrow") +Pkg.add("JSON3") +Pkg.add("HTTP") +Pkg.add("UUIDs") +Pkg.add("Dates") +``` + +### JavaScript + +```bash +npm install nats.js apache-arrow uuid base64-url +``` + +### Python/Micropython + +1. Copy `nats_bridge.py` to your device 2. Ensure you have the following dependencies: - - `urequests` for HTTP requests - - `ubinascii` for base64 encoding - - `ujson` for JSON handling - - `usocket` for networking + - `urequests` for HTTP requests (Micropython) + - `requests` for HTTP requests (Python) + - `base64` for base64 encoding + - `json` for JSON handling + - `socket` for networking (Micropython) ## Usage ### Basic Text Message +#### Python/Micropython + ```python from nats_bridge import smartsend, smartreceive @@ -57,8 +94,39 @@ for dataname, data, type in payloads: print("Received {}: {}".format(dataname, data)) ``` +#### Julia + +```julia +using NATSBridge + +# Sender +data = [("message", "Hello World", "text")] +env = smartsend("/chat/room1", data, nats_url="nats://localhost:4222") + +# Receiver +envelope = smartreceive(msg, fileserverDownloadHandler) +# envelope["payloads"] = [("message", "Hello World", "text"), ...] +``` + +#### JavaScript + +```javascript +const { smartsend, smartreceive } = require('./src/NATSBridge'); + +// Sender +await smartsend("/chat/room1", [ + { dataname: "message", data: "Hello World", type: "text" } +], { natsUrl: "nats://localhost:4222" }); + +// Receiver +const envelope = await smartreceive(msg); +// envelope.payloads = [{ dataname: "message", data: "Hello World", type: "text" }, ...] +``` + ### Sending JSON Configuration +#### Python/Micropython + ```python from nats_bridge import smartsend @@ -74,6 +142,8 @@ env = smartsend("/device/config", data, nats_url="nats://localhost:4222") ### Mixed Content (Chat with Text + Image) +#### Python/Micropython + ```python from nats_bridge import smartsend @@ -89,6 +159,8 @@ env = smartsend("/chat/mixed", data, nats_url="nats://localhost:4222") ### Request-Response Pattern +#### Python/Micropython + ```python from nats_bridge import smartsend @@ -105,6 +177,8 @@ env = smartsend( ### Large Payloads (File Server) +#### Python/Micropython + ```python from nats_bridge import smartsend @@ -191,21 +265,30 @@ Represents a single payload within a message envelope. ## Examples -See `examples/micropython_example.py` for more detailed examples. +See [`examples/micropython_example.py`](../examples/micropython_example.py) for more detailed examples. ## Testing Run the test suite: ```bash +# Python/Micropython python test/test_micropython_basic.py + +# JavaScript +node test/test_js_to_js_text_sender.js +node test/test_js_to_js_text_receiver.js + +# Julia +julia test/test_julia_to_julia_text_sender.jl +julia test/test_julia_to_julia_text_receiver.jl ``` ## Requirements -- Micropython with networking support -- NATS server (nats.io) -- HTTP file server (optional, for large payloads) +- **Julia**: NATS server (nats.io), HTTP file server (optional) +- **JavaScript**: NATS server (nats.io), HTTP file server (optional) +- **Python/Micropython**: NATS server (nats.io), HTTP file server (optional) ## License diff --git a/src/nats_bridge.py b/src/nats_bridge.py index a5497af..a0fdcc3 100644 --- a/src/nats_bridge.py +++ b/src/nats_bridge.py @@ -559,7 +559,7 @@ def smartsend(subject, data, nats_url=DEFAULT_NATS_URL, fileserver_url=DEFAULT_F def smartreceive(msg, fileserver_download_handler=_fetch_with_backoff, max_retries=5, - base_delay=100, max_delay=5000): + base_delay=100, max_delay=5000): """Receive and process messages from NATS. This function processes incoming NATS messages, handling both direct transport @@ -573,7 +573,7 @@ def smartreceive(msg, fileserver_download_handler=_fetch_with_backoff, max_retri max_delay: Maximum delay for exponential backoff in ms Returns: - list: List of (dataname, data, type) tuples + dict: Envelope dictionary with metadata and 'payloads' field containing list of (dataname, data, type) tuples """ # Parse the JSON envelope json_data = msg if isinstance(msg, dict) else json.loads(msg) @@ -611,7 +611,7 @@ def smartreceive(msg, fileserver_download_handler=_fetch_with_backoff, max_retri # Extract download URL from the payload url = payload.get("data", "") log_trace(json_data.get("correlationId", ""), - "Link transport - fetching '{}' from URL: {}".format(dataname, url)) + "Link transport - fetching '{}' from URL: {}".format(dataname, url)) # Fetch with exponential backoff downloaded_data = fileserver_download_handler( @@ -627,7 +627,10 @@ def smartreceive(msg, fileserver_download_handler=_fetch_with_backoff, max_retri else: raise ValueError("Unknown transport type for payload '{}': {}".format(dataname, transport)) - return payloads_list + # Replace payloads field with the processed list of (dataname, data, type) tuples + json_data["payloads"] = payloads_list + + return json_data # Utility functions diff --git a/test/test_js_to_js_dict_receiver.js b/test/test_js_to_js_dict_receiver.js index de1db08..7463b9b 100644 --- a/test/test_js_to_js_dict_receiver.js +++ b/test/test_js_to_js_dict_receiver.js @@ -37,8 +37,9 @@ async function test_dict_receive() { } ); - // Result is a list of {dataname, data, type} objects - for (const { dataname, data, type } of result) { + // Result is an envelope dictionary with payloads field + // Access payloads with result.payloads + for (const { dataname, data, type } of result.payloads) { if (typeof data === 'object' && data !== null && !Array.isArray(data)) { log_trace(`Received Dictionary '${dataname}' of type ${type}`); diff --git a/test/test_js_to_js_file_receiver.js b/test/test_js_to_js_file_receiver.js index e8018d1..4943791 100644 --- a/test/test_js_to_js_file_receiver.js +++ b/test/test_js_to_js_file_receiver.js @@ -36,8 +36,9 @@ async function test_large_binary_receive() { } ); - // Result is a list of {dataname, data, type} objects - for (const { dataname, data, type } of result) { + // Result is an envelope dictionary with payloads field + // Access payloads with result.payloads + for (const { dataname, data, type } of result.payloads) { if (data instanceof Uint8Array || Array.isArray(data)) { const file_size = data.length; log_trace(`Received ${file_size} bytes of binary data for '${dataname}' of type ${type}`); diff --git a/test/test_js_to_js_mix_payloads_receiver.js b/test/test_js_to_js_mix_payloads_receiver.js index 5d9c758..9ebf93b 100644 --- a/test/test_js_to_js_mix_payloads_receiver.js +++ b/test/test_js_to_js_mix_payloads_receiver.js @@ -40,10 +40,11 @@ async function test_mix_receive() { } ); - log_trace(`Received ${result.length} payloads`); + log_trace(`Received ${result.payloads.length} payloads`); - // Result is a list of {dataname, data, type} objects - for (const { dataname, data, type } of result) { + // Result is an envelope dictionary with payloads field + // Access payloads with result.payloads + for (const { dataname, data, type } of result.payloads) { log_trace(`\n=== Payload: ${dataname} (type: ${type}) ===`); // Handle different data types @@ -122,13 +123,13 @@ async function test_mix_receive() { // Summary console.log("\n=== Verification Summary ==="); - const text_count = result.filter(x => x.type === "text").length; - const dict_count = result.filter(x => x.type === "dictionary").length; - const table_count = result.filter(x => x.type === "table").length; - const image_count = result.filter(x => x.type === "image").length; - const audio_count = result.filter(x => x.type === "audio").length; - const video_count = result.filter(x => x.type === "video").length; - const binary_count = result.filter(x => x.type === "binary").length; + const text_count = result.payloads.filter(x => x.type === "text").length; + const dict_count = result.payloads.filter(x => x.type === "dictionary").length; + const table_count = result.payloads.filter(x => x.type === "table").length; + const image_count = result.payloads.filter(x => x.type === "image").length; + const audio_count = result.payloads.filter(x => x.type === "audio").length; + const video_count = result.payloads.filter(x => x.type === "video").length; + const binary_count = result.payloads.filter(x => x.type === "binary").length; log_trace(`Text payloads: ${text_count}`); log_trace(`Dictionary payloads: ${dict_count}`); @@ -140,7 +141,7 @@ async function test_mix_receive() { // Print transport type info for each payload if available console.log("\n=== Payload Details ==="); - for (const { dataname, data, type } of result) { + for (const { dataname, data, type } of result.payloads) { if (["image", "audio", "video", "binary"].includes(type)) { log_trace(`${dataname}: ${data.length} bytes (binary)`); } else if (type === "table") { diff --git a/test/test_js_to_js_table_receiver.js b/test/test_js_to_js_table_receiver.js index 9ecadd5..f5672f0 100644 --- a/test/test_js_to_js_table_receiver.js +++ b/test/test_js_to_js_table_receiver.js @@ -40,8 +40,9 @@ async function test_table_receive() { } ); - // Result is a list of {dataname, data, type} objects - for (const { dataname, data, type } of result) { + // Result is an envelope dictionary with payloads field + // Access payloads with result.payloads + for (const { dataname, data, type } of result.payloads) { if (Array.isArray(data)) { log_trace(`Received Table '${dataname}' of type ${type}`); diff --git a/test/test_js_to_js_text_receiver.js b/test/test_js_to_js_text_receiver.js index 0bf8c4f..ef46822 100644 --- a/test/test_js_to_js_text_receiver.js +++ b/test/test_js_to_js_text_receiver.js @@ -37,8 +37,9 @@ async function test_text_receive() { } ); - // Result is a list of {dataname, data, type} objects - for (const { dataname, data, type } of result) { + // Result is an envelope dictionary with payloads field + // Access payloads with result.payloads + for (const { dataname, data, type } of result.payloads) { if (typeof data === 'string') { log_trace(`Received text '${dataname}' of type ${type}`); log_trace(` Length: ${data.length} characters`); diff --git a/test/test_julia_to_julia_dict_receiver.jl b/test/test_julia_to_julia_dict_receiver.jl index 1a0b4ed..6fb98a8 100644 --- a/test/test_julia_to_julia_dict_receiver.jl +++ b/test/test_julia_to_julia_dict_receiver.jl @@ -42,8 +42,8 @@ function test_dict_receive() max_delay = 5000 ) - # Result is a list of (dataname, data, data_type) tuples - for (dataname, data, data_type) in result + # Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples + for (dataname, data, data_type) in result["payloads"] if isa(data, JSON.Object{String, Any}) log_trace("Received Dictionary '$dataname' of type $data_type") diff --git a/test/test_julia_to_julia_dict_sender.jl b/test/test_julia_to_julia_dict_sender.jl index 4371e9d..9c28d5c 100644 --- a/test/test_julia_to_julia_dict_sender.jl +++ b/test/test_julia_to_julia_dict_sender.jl @@ -94,7 +94,7 @@ function test_dict_send() # For large Dictionary: will use link transport (uploaded to fileserver) env = NATSBridge.smartsend( SUBJECT, - [data1, data2], # List of (dataname, data, type) tuples + [data1, data2]; # List of (dataname, data, type) tuples nats_url = NATS_URL, fileserver_url = FILESERVER_URL, fileserverUploadHandler = plik_upload_handler, diff --git a/test/test_julia_to_julia_file_receiver.jl b/test/test_julia_to_julia_file_receiver.jl index d072812..bc2632e 100644 --- a/test/test_julia_to_julia_file_receiver.jl +++ b/test/test_julia_to_julia_file_receiver.jl @@ -44,8 +44,8 @@ function test_large_binary_receive() max_delay = 5000 ) - # Result is a list of (dataname, data) tuples - for (dataname, data, data_type) in result + # Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples + for (dataname, data, data_type) in result["payloads"] # Check transport type from the envelope # For link transport, data is the URL string # For direct transport, data is the actual payload bytes diff --git a/test/test_julia_to_julia_file_sender.jl b/test/test_julia_to_julia_file_sender.jl index 6f5834a..64b83ff 100644 --- a/test/test_julia_to_julia_file_sender.jl +++ b/test/test_julia_to_julia_file_sender.jl @@ -81,7 +81,7 @@ function test_large_binary_send() # API: smartsend(subject, [(dataname, data, type), ...]; keywords...) env = NATSBridge.smartsend( SUBJECT, - [data1, data2], # List of (dataname, data, type) tuples + [data1, data2]; # List of (dataname, data, type) tuples nats_url = NATS_URL; fileserver_url = FILESERVER_URL, fileserverUploadHandler = plik_upload_handler, diff --git a/test/test_julia_to_julia_mix_payloads_receiver.jl b/test/test_julia_to_julia_mix_payloads_receiver.jl index e5c693a..1c33866 100644 --- a/test/test_julia_to_julia_mix_payloads_receiver.jl +++ b/test/test_julia_to_julia_mix_payloads_receiver.jl @@ -45,10 +45,10 @@ function test_mix_receive() max_delay = 5000 ) - log_trace("Received $(length(result)) payloads") + log_trace("Received $(length(result["payloads"])) payloads") - # Result is a list of (dataname, data, data_type) tuples - for (dataname, data, data_type) in result + # Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples + for (dataname, data, data_type) in result["payloads"] log_trace("\n=== Payload: $dataname (type: $data_type) ===") # Handle different data types @@ -178,13 +178,13 @@ function test_mix_receive() # Summary println("\n=== Verification Summary ===") - text_count = count(x -> x[3] == "text", result) - dict_count = count(x -> x[3] == "dictionary", result) - table_count = count(x -> x[3] == "table", result) - image_count = count(x -> x[3] == "image", result) - audio_count = count(x -> x[3] == "audio", result) - video_count = count(x -> x[3] == "video", result) - binary_count = count(x -> x[3] == "binary", result) + text_count = count(x -> x[3] == "text", result["payloads"]) + dict_count = count(x -> x[3] == "dictionary", result["payloads"]) + table_count = count(x -> x[3] == "table", result["payloads"]) + image_count = count(x -> x[3] == "image", result["payloads"]) + audio_count = count(x -> x[3] == "audio", result["payloads"]) + video_count = count(x -> x[3] == "video", result["payloads"]) + binary_count = count(x -> x[3] == "binary", result["payloads"]) log_trace("Text payloads: $text_count") log_trace("Dictionary payloads: $dict_count") @@ -196,7 +196,7 @@ function test_mix_receive() # Print transport type info for each payload if available println("\n=== Payload Details ===") - for (dataname, data, data_type) in result + for (dataname, data, data_type) in result["payloads"] if data_type in ["image", "audio", "video", "binary"] log_trace("$dataname: $(length(data)) bytes (binary)") elseif data_type == "table" diff --git a/test/test_julia_to_julia_mix_payload_sender.jl b/test/test_julia_to_julia_mix_payloads_sender.jl similarity index 99% rename from test/test_julia_to_julia_mix_payload_sender.jl rename to test/test_julia_to_julia_mix_payloads_sender.jl index 00a3951..c9fab9b 100644 --- a/test/test_julia_to_julia_mix_payload_sender.jl +++ b/test/test_julia_to_julia_mix_payloads_sender.jl @@ -188,7 +188,7 @@ function test_mix_send() # Use smartsend with mixed content env = NATSBridge.smartsend( SUBJECT, - payloads, # List of (dataname, data, type) tuples + payloads; # List of (dataname, data, type) tuples nats_url = NATS_URL, fileserver_url = FILESERVER_URL, fileserverUploadHandler = plik_upload_handler, diff --git a/test/test_julia_to_julia_table_receiver.jl b/test/test_julia_to_julia_table_receiver.jl index 10db547..2ea9866 100644 --- a/test/test_julia_to_julia_table_receiver.jl +++ b/test/test_julia_to_julia_table_receiver.jl @@ -42,8 +42,8 @@ function test_table_receive() max_delay = 5000 ) - # Result is a list of (dataname, data, data_type) tuples - for (dataname, data, data_type) in result + # Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples + for (dataname, data, data_type) in result["payloads"] data = DataFrame(data) if isa(data, DataFrame) log_trace("Received DataFrame '$dataname' of type $data_type") diff --git a/test/test_julia_to_julia_table_sender.jl b/test/test_julia_to_julia_table_sender.jl index 301ac64..ed8c4b7 100644 --- a/test/test_julia_to_julia_table_sender.jl +++ b/test/test_julia_to_julia_table_sender.jl @@ -92,7 +92,7 @@ function test_table_send() # For large DataFrame: will use link transport (uploaded to fileserver) env = NATSBridge.smartsend( SUBJECT, - [data1, data2], # List of (dataname, data, type) tuples + [data1, data2]; # List of (dataname, data, type) tuples nats_url = NATS_URL, fileserver_url = FILESERVER_URL, fileserverUploadHandler = plik_upload_handler, diff --git a/test/test_julia_to_julia_text_receiver.jl b/test/test_julia_to_julia_text_receiver.jl index 0d974b9..c7f7402 100644 --- a/test/test_julia_to_julia_text_receiver.jl +++ b/test/test_julia_to_julia_text_receiver.jl @@ -42,8 +42,8 @@ function test_text_receive() max_delay = 5000 ) - # Result is a list of (dataname, data, data_type) tuples - for (dataname, data, data_type) in result + # Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples + for (dataname, data, data_type) in result["payloads"] if isa(data, String) log_trace("Received text '$dataname' of type $data_type") log_trace(" Length: $(length(data)) characters") diff --git a/test/test_julia_to_julia_text_sender.jl b/test/test_julia_to_julia_text_sender.jl index 83265be..782a12b 100644 --- a/test/test_julia_to_julia_text_sender.jl +++ b/test/test_julia_to_julia_text_sender.jl @@ -77,7 +77,7 @@ function test_text_send() # For large text: will use link transport (uploaded to fileserver) env = NATSBridge.smartsend( SUBJECT, - [data1, data2], # List of (dataname, data, type) tuples + [data1, data2]; # List of (dataname, data, type) tuples nats_url = NATS_URL, fileserver_url = FILESERVER_URL, fileserverUploadHandler = plik_upload_handler, diff --git a/test/test_micropython_basic.py b/test/test_micropython_basic.py index 68d3439..31884cb 100644 --- a/test/test_micropython_basic.py +++ b/test/test_micropython_basic.py @@ -1,220 +1,207 @@ +#!/usr/bin/env python3 """ -Micropython NATS Bridge - Basic Test Examples - -This module demonstrates basic usage of the NATSBridge for Micropython. +Basic functionality test for nats_bridge.py +Tests the core classes and functions without NATS connection """ import sys -sys.path.insert(0, "../src") +import os -from nats_bridge import MessageEnvelope, MessagePayload, smartsend, smartreceive, log_trace +# Add src to path for import +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src')) + +from nats_bridge import ( + MessagePayload, + MessageEnvelope, + smartsend, + smartreceive, + log_trace, + generate_uuid, + get_timestamp, + _serialize_data, + _deserialize_data +) import json -# ============================================= 100 ============================================== # - -def test_text_message(): - """Test sending and receiving text messages.""" - print("\n=== Test 1: Text Message ===") +def test_message_payload(): + """Test MessagePayload class""" + print("\n=== Testing MessagePayload ===") - # Send text message - data = [ - ("message", "Hello World", "text"), - ("greeting", "Good morning!", "text") - ] - - env = smartsend( - "/test/text", - data, - nats_url="nats://localhost:4222", - size_threshold=1000000 + # Test direct transport with text + payload1 = MessagePayload( + data="Hello World", + msg_type="text", + id="test-id-1", + dataname="message", + transport="direct", + encoding="base64", + size=11 ) - print("Sent envelope:") - print(" Subject: {}".format(env.send_to)) - print(" Correlation ID: {}".format(env.correlation_id)) - print(" Payloads: {}".format(len(env.payloads))) + assert payload1.id == "test-id-1" + assert payload1.dataname == "message" + assert payload1.type == "text" + assert payload1.transport == "direct" + assert payload1.encoding == "base64" + assert payload1.size == 11 + print(" [PASS] MessagePayload with text data") - # Expected output on receiver: - # payloads = smartreceive(msg) - # for dataname, data, type in payloads: - # print("Received {}: {}".format(dataname, data)) - - -def test_dictionary_message(): - """Test sending and receiving dictionary messages.""" - print("\n=== Test 2: Dictionary Message ===") - - # Send dictionary message - config = { - "step_size": 0.01, - "iterations": 1000, - "threshold": 0.5 - } - - data = [ - ("config", config, "dictionary") - ] - - env = smartsend( - "/test/dictionary", - data, - nats_url="nats://localhost:4222", - size_threshold=1000000 + # Test link transport with URL + payload2 = MessagePayload( + data="http://example.com/file.txt", + msg_type="binary", + id="test-id-2", + dataname="file", + transport="link", + encoding="none", + size=1000 ) - print("Sent envelope:") - print(" Subject: {}".format(env.send_to)) - print(" Payloads: {}".format(len(env.payloads))) + assert payload2.transport == "link" + assert payload2.data == "http://example.com/file.txt" + print(" [PASS] MessagePayload with link transport") - # Expected output on receiver: - # payloads = smartreceive(msg) - # for dataname, data, type in payloads: - # if type == "dictionary": - # print("Config: {}".format(data)) + # Test to_dict method + payload_dict = payload1.to_dict() + assert "id" in payload_dict + assert "dataname" in payload_dict + assert "type" in payload_dict + assert "transport" in payload_dict + assert "data" in payload_dict + print(" [PASS] MessagePayload.to_dict() method") -def test_mixed_payloads(): - """Test sending mixed payload types in a single message.""" - print("\n=== Test 3: Mixed Payloads ===") +def test_message_envelope(): + """Test MessageEnvelope class""" + print("\n=== Testing MessageEnvelope ===") - # Mixed content: text, dictionary, and binary - image_data = b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR" # PNG header (example) + # Create payloads + payload1 = MessagePayload("Hello", "text", id="p1", dataname="msg1") + payload2 = MessagePayload("http://example.com/file", "binary", id="p2", dataname="file", transport="link") - data = [ - ("message_text", "Hello!", "text"), - ("user_config", {"theme": "dark", "volume": 80}, "dictionary"), - ("user_image", image_data, "binary") - ] - - env = smartsend( - "/test/mixed", - data, - nats_url="nats://localhost:4222", - size_threshold=1000000 + # Create envelope + env = MessageEnvelope( + send_to="/test/subject", + payloads=[payload1, payload2], + correlation_id="test-correlation-id", + msg_id="test-msg-id", + msg_purpose="chat", + sender_name="test_sender", + receiver_name="test_receiver", + reply_to="/test/reply" ) - print("Sent envelope:") - print(" Subject: {}".format(env.send_to)) - print(" Payloads: {}".format(len(env.payloads))) + assert env.send_to == "/test/subject" + assert env.correlation_id == "test-correlation-id" + assert env.msg_id == "test-msg-id" + assert env.msg_purpose == "chat" + assert len(env.payloads) == 2 + print(" [PASS] MessageEnvelope creation") - # Expected output on receiver: - # payloads = smartreceive(msg) - # for dataname, data, type in payloads: - # print("Received {}: {} (type: {})".format(dataname, data if type != "binary" else len(data), type)) + # Test to_json method + json_str = env.to_json() + json_data = json.loads(json_str) + assert json_data["sendTo"] == "/test/subject" + assert json_data["correlationId"] == "test-correlation-id" + assert json_data["msgPurpose"] == "chat" + assert len(json_data["payloads"]) == 2 + print(" [PASS] MessageEnvelope.to_json() method") -def test_large_payload(): - """Test sending large payloads that require fileserver upload.""" - print("\n=== Test 4: Large Payload (Link Transport) ===") +def test_serialize_data(): + """Test _serialize_data function""" + print("\n=== Testing _serialize_data ===") - # Create large data (> 1MB would trigger link transport) - # For testing, we'll use a smaller size but configure threshold lower - large_data = b"A" * 100000 # 100KB + # Test text serialization + text_bytes = _serialize_data("Hello", "text") + assert isinstance(text_bytes, bytes) + assert text_bytes == b"Hello" + print(" [PASS] Text serialization") - data = [ - ("large_data", large_data, "binary") - ] + # Test dictionary serialization + dict_data = {"key": "value", "number": 42} + dict_bytes = _serialize_data(dict_data, "dictionary") + assert isinstance(dict_bytes, bytes) + parsed = json.loads(dict_bytes.decode('utf-8')) + assert parsed["key"] == "value" + print(" [PASS] Dictionary serialization") - # Use a lower threshold for testing - env = smartsend( - "/test/large", - data, - nats_url="nats://localhost:4222", - fileserver_url="http://localhost:8080", - size_threshold=50000 # 50KB threshold for testing - ) + # Test binary serialization + binary_data = b"\x00\x01\x02" + binary_bytes = _serialize_data(binary_data, "binary") + assert binary_bytes == b"\x00\x01\x02" + print(" [PASS] Binary serialization") - print("Sent envelope:") - print(" Subject: {}".format(env.send_to)) - print(" Payloads: {}".format(len(env.payloads))) - for p in env.payloads: - print(" - Transport: {}, Type: {}".format(p.transport, p.type)) + # Test image serialization + image_data = bytes([1, 2, 3, 4, 5]) + image_bytes = _serialize_data(image_data, "image") + assert image_bytes == image_data + print(" [PASS] Image serialization") -def test_reply_to(): - """Test sending messages with reply-to functionality.""" - print("\n=== Test 5: Reply To ===") +def test_deserialize_data(): + """Test _deserialize_data function""" + print("\n=== Testing _deserialize_data ===") - data = [ - ("command", {"action": "start"}, "dictionary") - ] + # Test text deserialization + text_bytes = b"Hello" + text_data = _deserialize_data(text_bytes, "text", "test-correlation-id") + assert text_data == "Hello" + print(" [PASS] Text deserialization") - env = smartsend( - "/test/command", - data, - nats_url="nats://localhost:4222", - reply_to="/test/response", - reply_to_msg_id="reply-123", - msg_purpose="command" - ) + # Test dictionary deserialization + dict_bytes = b'{"key": "value"}' + dict_data = _deserialize_data(dict_bytes, "dictionary", "test-correlation-id") + assert dict_data == {"key": "value"} + print(" [PASS] Dictionary deserialization") - print("Sent envelope:") - print(" Subject: {}".format(env.send_to)) - print(" Reply To: {}".format(env.reply_to)) - print(" Reply To Msg ID: {}".format(env.reply_to_msg_id)) + # Test binary deserialization + binary_data = b"\x00\x01\x02" + binary_result = _deserialize_data(binary_data, "binary", "test-correlation-id") + assert binary_result == b"\x00\x01\x02" + print(" [PASS] Binary deserialization") -def test_correlation_id(): - """Test using custom correlation IDs for tracing.""" - print("\n=== Test 6: Custom Correlation ID ===") +def test_utilities(): + """Test utility functions""" + print("\n=== Testing Utility Functions ===") - custom_cid = "trace-abc123" - data = [ - ("message", "Test with correlation ID", "text") - ] + # Test generate_uuid + uuid1 = generate_uuid() + uuid2 = generate_uuid() + assert uuid1 != uuid2 + print(f" [PASS] generate_uuid() - generated: {uuid1}") - env = smartsend( - "/test/correlation", - data, - nats_url="nats://localhost:4222", - correlation_id=custom_cid - ) - - print("Sent envelope with correlation ID: {}".format(env.correlation_id)) - print("This ID can be used to trace the message flow.") + # Test get_timestamp + timestamp = get_timestamp() + assert "T" in timestamp + print(f" [PASS] get_timestamp() - generated: {timestamp}") -def test_multiple_payloads(): - """Test sending multiple payloads in one message.""" - print("\n=== Test 7: Multiple Payloads ===") +def main(): + """Run all tests""" + print("=" * 60) + print("NATSBridge Python/Micropython - Basic Functionality Tests") + print("=" * 60) - data = [ - ("text_message", "Hello", "text"), - ("json_data", {"key": "value", "number": 42}, "dictionary"), - ("table_data", b"\x01\x02\x03\x04", "binary"), - ("audio_data", b"\x00\x01\x02\x03", "binary") - ] - - env = smartsend( - "/test/multiple", - data, - nats_url="nats://localhost:4222", - size_threshold=1000000 - ) - - print("Sent {} payloads in one message".format(len(env.payloads))) + try: + test_message_payload() + test_message_envelope() + test_serialize_data() + test_deserialize_data() + test_utilities() + + print("\n" + "=" * 60) + print("ALL TESTS PASSED!") + print("=" * 60) + + except Exception as e: + print(f"\n[FAIL] Test failed with error: {e}") + import traceback + traceback.print_exc() + sys.exit(1) if __name__ == "__main__": - print("Micropython NATS Bridge Test Suite") - print("==================================") - print() - - # Run tests - test_text_message() - test_dictionary_message() - test_mixed_payloads() - test_large_payload() - test_reply_to() - test_correlation_id() - test_multiple_payloads() - - print("\n=== All tests completed ===") - print() - print("Note: These tests require:") - print(" 1. A running NATS server at nats://localhost:4222") - print(" 2. An HTTP file server at http://localhost:8080 (for large payloads)") - print() - print("To run the tests:") - print(" python test_micropython_basic.py") \ No newline at end of file + main() \ No newline at end of file diff --git a/test/test_micropython_dict_receiver.py b/test/test_micropython_dict_receiver.py new file mode 100644 index 0000000..ae6df9f --- /dev/null +++ b/test/test_micropython_dict_receiver.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +""" +Test script for dictionary transport testing - Receiver +Tests receiving dictionary messages via NATS using nats_bridge.py smartreceive +""" + +import sys +import os +import json + +# Add src to path for import +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src')) + +from nats_bridge import smartreceive, log_trace +import nats +import asyncio + +# Configuration +SUBJECT = "/NATSBridge_dict_test" +NATS_URL = "nats://nats.yiem.cc:4222" + + +async def main(): + log_trace("", f"Starting dictionary transport receiver test...") + log_trace("", f"Note: This receiver will wait for messages from the sender.") + log_trace("", f"Run test_micropython_dict_sender.py first to send test data.") + + # Connect to NATS + nc = await nats.connect(NATS_URL) + log_trace("", f"Connected to NATS at {NATS_URL}") + + # Subscribe to the subject + async def message_handler(msg): + log_trace("", f"Received message on {msg.subject}") + + # Use smartreceive to handle the data + result = smartreceive(msg.data) + + # Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples + for dataname, data, data_type in result["payloads"]: + if isinstance(data, dict): + log_trace(result.get("correlationId", ""), f"Received dictionary '{dataname}' of type {data_type}") + log_trace(result.get("correlationId", ""), f" Keys: {list(data.keys())}") + + # Display first few items for small dicts + if isinstance(data, dict) and len(data) <= 10: + log_trace(result.get("correlationId", ""), f" Content: {json.dumps(data, indent=2)}") + else: + # For large dicts, show summary + log_trace(result.get("correlationId", ""), f" Summary: {json.dumps(data, default=str)[:200]}...") + + # Save to file + output_path = f"./received_{dataname}.json" + with open(output_path, 'w') as f: + json.dump(data, f, indent=2) + log_trace(result.get("correlationId", ""), f"Saved dictionary to {output_path}") + else: + log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}") + + sid = await nc.subscribe(SUBJECT, cb=message_handler) + log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}") + + # Keep listening for 120 seconds + await asyncio.sleep(120) + await nc.close() + log_trace("", "Test completed.") + + +if __name__ == "__main__": + asyncio.run(main()) \ No newline at end of file diff --git a/test/test_micropython_dict_sender.py b/test/test_micropython_dict_sender.py new file mode 100644 index 0000000..3d63106 --- /dev/null +++ b/test/test_micropython_dict_sender.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python3 +""" +Test script for dictionary transport testing - Micropython +Tests sending dictionary messages via NATS using nats_bridge.py smartsend +""" + +import sys +import os + +# Add src to path for import +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src')) + +from nats_bridge import smartsend, log_trace +import uuid + +# Configuration +SUBJECT = "/NATSBridge_dict_test" +NATS_URL = "nats://nats.yiem.cc:4222" +FILESERVER_URL = "http://192.168.88.104:8080" +SIZE_THRESHOLD = 1_000_000 # 1MB + +# Create correlation ID for tracing +correlation_id = str(uuid.uuid4()) + + +def main(): + # Create a small dictionary (will use direct transport) + small_dict = { + "name": "test", + "value": 42, + "enabled": True, + "metadata": { + "version": "1.0.0", + "timestamp": "2026-02-22T12:00:00Z" + } + } + + # Create a large dictionary (will use link transport if > 1MB) + # Generate a larger dictionary (~2MB to ensure link transport) + large_dict = { + "id": str(uuid.uuid4()), + "items": [ + { + "index": i, + "name": f"item_{i}", + "value": i * 1.5, + "data": "x" * 10000 # Large string per item + } + for i in range(200) + ], + "metadata": { + "count": 200, + "created": "2026-02-22T12:00:00Z" + } + } + + # Test data 1: small dictionary + data1 = ("small_dict", small_dict, "dictionary") + + # Test data 2: large dictionary + data2 = ("large_dict", large_dict, "dictionary") + + log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}") + log_trace(correlation_id, f"Correlation ID: {correlation_id}") + + # Use smartsend with dictionary type + env = smartsend( + SUBJECT, + [data1, data2], # List of (dataname, data, type) tuples + nats_url=NATS_URL, + fileserver_url=FILESERVER_URL, + size_threshold=SIZE_THRESHOLD, + correlation_id=correlation_id, + msg_purpose="chat", + sender_name="dict_sender", + receiver_name="", + receiver_id="", + reply_to="", + reply_to_msg_id="" + ) + + log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads") + + # Log transport type for each payload + for i, payload in enumerate(env.payloads): + log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):") + log_trace(correlation_id, f" Transport: {payload.transport}") + log_trace(correlation_id, f" Type: {payload.type}") + log_trace(correlation_id, f" Size: {payload.size} bytes") + log_trace(correlation_id, f" Encoding: {payload.encoding}") + + if payload.transport == "link": + log_trace(correlation_id, f" URL: {payload.data}") + + print(f"Test completed. Correlation ID: {correlation_id}") + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/test/test_micropython_file_receiver.py b/test/test_micropython_file_receiver.py new file mode 100644 index 0000000..98a6ebc --- /dev/null +++ b/test/test_micropython_file_receiver.py @@ -0,0 +1,65 @@ +#!/usr/bin/env python3 +""" +Test script for file transport testing - Receiver +Tests receiving binary files via NATS using nats_bridge.py smartreceive +""" + +import sys +import os + +# Add src to path for import +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src')) + +from nats_bridge import smartreceive, log_trace +import nats +import asyncio + +# Configuration +SUBJECT = "/NATSBridge_file_test" +NATS_URL = "nats://nats.yiem.cc:4222" + + +async def main(): + log_trace("", f"Starting file transport receiver test...") + log_trace("", f"Note: This receiver will wait for messages from the sender.") + log_trace("", f"Run test_micropython_file_sender.py first to send test data.") + + # Connect to NATS + nc = await nats.connect(NATS_URL) + log_trace("", f"Connected to NATS at {NATS_URL}") + + # Subscribe to the subject + async def message_handler(msg): + log_trace("", f"Received message on {msg.subject}") + + # Use smartreceive to handle the data + result = smartreceive(msg.data) + + # Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples + for dataname, data, data_type in result["payloads"]: + if isinstance(data, bytes): + log_trace(result.get("correlationId", ""), f"Received binary '{dataname}' of type {data_type}") + log_trace(result.get("correlationId", ""), f" Size: {len(data)} bytes") + + # Display first 100 bytes as hex + log_trace(result.get("correlationId", ""), f" First 100 bytes (hex): {data[:100].hex()}") + + # Save to file + output_path = f"./received_{dataname}.bin" + with open(output_path, 'wb') as f: + f.write(data) + log_trace(result.get("correlationId", ""), f"Saved binary to {output_path}") + else: + log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}") + + sid = await nc.subscribe(SUBJECT, cb=message_handler) + log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}") + + # Keep listening for 120 seconds + await asyncio.sleep(120) + await nc.close() + log_trace("", "Test completed.") + + +if __name__ == "__main__": + asyncio.run(main()) \ No newline at end of file diff --git a/test/test_micropython_file_sender.py b/test/test_micropython_file_sender.py new file mode 100644 index 0000000..9219c0f --- /dev/null +++ b/test/test_micropython_file_sender.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +""" +Test script for file transport testing - Micropython +Tests sending binary files via NATS using nats_bridge.py smartsend +""" + +import sys +import os + +# Add src to path for import +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src')) + +from nats_bridge import smartsend, log_trace +import uuid + +# Configuration +SUBJECT = "/NATSBridge_file_test" +NATS_URL = "nats://nats.yiem.cc:4222" +FILESERVER_URL = "http://192.168.88.104:8080" +SIZE_THRESHOLD = 1_000_000 # 1MB + +# Create correlation ID for tracing +correlation_id = str(uuid.uuid4()) + + +def main(): + # Create small binary data (will use direct transport) + small_binary = b"This is small binary data for testing direct transport." + small_binary += b"\x00" * 100 # Add some null bytes + + # Create large binary data (will use link transport if > 1MB) + # Generate a larger binary (~2MB to ensure link transport) + large_binary = bytes([ + (i * 7) % 256 for i in range(2_000_000) + ]) + + # Test data 1: small binary (direct transport) + data1 = ("small_binary", small_binary, "binary") + + # Test data 2: large binary (link transport) + data2 = ("large_binary", large_binary, "binary") + + log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}") + log_trace(correlation_id, f"Correlation ID: {correlation_id}") + + # Use smartsend with binary type + env = smartsend( + SUBJECT, + [data1, data2], # List of (dataname, data, type) tuples + nats_url=NATS_URL, + fileserver_url=FILESERVER_URL, + size_threshold=SIZE_THRESHOLD, + correlation_id=correlation_id, + msg_purpose="chat", + sender_name="file_sender", + receiver_name="", + receiver_id="", + reply_to="", + reply_to_msg_id="" + ) + + log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads") + + # Log transport type for each payload + for i, payload in enumerate(env.payloads): + log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):") + log_trace(correlation_id, f" Transport: {payload.transport}") + log_trace(correlation_id, f" Type: {payload.type}") + log_trace(correlation_id, f" Size: {payload.size} bytes") + log_trace(correlation_id, f" Encoding: {payload.encoding}") + + if payload.transport == "link": + log_trace(correlation_id, f" URL: {payload.data}") + + print(f"Test completed. Correlation ID: {correlation_id}") + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/test/test_micropython_mixed_receiver.py b/test/test_micropython_mixed_receiver.py new file mode 100644 index 0000000..d28722d --- /dev/null +++ b/test/test_micropython_mixed_receiver.py @@ -0,0 +1,97 @@ +#!/usr/bin/env python3 +""" +Test script for mixed payload testing - Receiver +Tests receiving mixed payload types via NATS using nats_bridge.py smartreceive +""" + +import sys +import os +import json + +# Add src to path for import +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src')) + +from nats_bridge import smartreceive, log_trace +import nats +import asyncio + +# Configuration +SUBJECT = "/NATSBridge_mixed_test" +NATS_URL = "nats://nats.yiem.cc:4222" + + +async def main(): + log_trace("", f"Starting mixed payload receiver test...") + log_trace("", f"Note: This receiver will wait for messages from the sender.") + log_trace("", f"Run test_micropython_mixed_sender.py first to send test data.") + + # Connect to NATS + nc = await nats.connect(NATS_URL) + log_trace("", f"Connected to NATS at {NATS_URL}") + + # Subscribe to the subject + async def message_handler(msg): + log_trace("", f"Received message on {msg.subject}") + + # Use smartreceive to handle the data + result = smartreceive(msg.data) + + log_trace(result.get("correlationId", ""), f"Received envelope with {len(result['payloads'])} payloads") + + # Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples + for dataname, data, data_type in result["payloads"]: + log_trace(result.get("correlationId", ""), f"\n--- Payload: {dataname} (type: {data_type}) ---") + + if isinstance(data, str): + log_trace(result.get("correlationId", ""), f" Type: text/string") + log_trace(result.get("correlationId", ""), f" Length: {len(data)} characters") + if len(data) <= 100: + log_trace(result.get("correlationId", ""), f" Content: {data}") + else: + log_trace(result.get("correlationId", ""), f" First 100 chars: {data[:100]}...") + # Save to file + output_path = f"./received_{dataname}.txt" + with open(output_path, 'w') as f: + f.write(data) + log_trace(result.get("correlationId", ""), f" Saved to: {output_path}") + + elif isinstance(data, dict): + log_trace(result.get("correlationId", ""), f" Type: dictionary") + log_trace(result.get("correlationId", ""), f" Keys: {list(data.keys())}") + log_trace(result.get("correlationId", ""), f" Content: {json.dumps(data, indent=2)}") + # Save to file + output_path = f"./received_{dataname}.json" + with open(output_path, 'w') as f: + json.dump(data, f, indent=2) + log_trace(result.get("correlationId", ""), f" Saved to: {output_path}") + + elif isinstance(data, bytes): + log_trace(result.get("correlationId", ""), f" Type: binary") + log_trace(result.get("correlationId", ""), f" Size: {len(data)} bytes") + log_trace(result.get("correlationId", ""), f" First 100 bytes (hex): {data[:100].hex()}") + # Save to file + output_path = f"./received_{dataname}.bin" + with open(output_path, 'wb') as f: + f.write(data) + log_trace(result.get("correlationId", ""), f" Saved to: {output_path}") + else: + log_trace(result.get("correlationId", ""), f" Received unexpected data type: {type(data)}") + + # Log envelope metadata + log_trace(result.get("correlationId", ""), f"\n--- Envelope Metadata ---") + log_trace(result.get("correlationId", ""), f" Correlation ID: {result.get('correlationId', 'N/A')}") + log_trace(result.get("correlationId", ""), f" Message ID: {result.get('msgId', 'N/A')}") + log_trace(result.get("correlationId", ""), f" Sender: {result.get('senderName', 'N/A')}") + log_trace(result.get("correlationId", ""), f" Purpose: {result.get('msgPurpose', 'N/A')}") + + sid = await nc.subscribe(SUBJECT, cb=message_handler) + log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}") + + # Keep listening for 120 seconds + await asyncio.sleep(120) + await nc.close() + log_trace("", "Test completed.") + + +if __name__ == "__main__": + asyncio.run(main()) \ No newline at end of file diff --git a/test/test_micropython_mixed_sender.py b/test/test_micropython_mixed_sender.py new file mode 100644 index 0000000..00e24d3 --- /dev/null +++ b/test/test_micropython_mixed_sender.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python3 +""" +Test script for mixed payload testing - Micropython +Tests sending mixed payload types via NATS using nats_bridge.py smartsend +""" + +import sys +import os + +# Add src to path for import +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src')) + +from nats_bridge import smartsend, log_trace +import uuid + +# Configuration +SUBJECT = "/NATSBridge_mixed_test" +NATS_URL = "nats://nats.yiem.cc:4222" +FILESERVER_URL = "http://192.168.88.104:8080" +SIZE_THRESHOLD = 1_000_000 # 1MB + +# Create correlation ID for tracing +correlation_id = str(uuid.uuid4()) + + +def main(): + # Create payloads for mixed content test + + # 1. Small text (direct transport) + text_data = "Hello, this is a text message for testing mixed payloads!" + + # 2. Small dictionary (direct transport) + dict_data = { + "status": "ok", + "code": 200, + "message": "Test successful", + "items": [1, 2, 3] + } + + # 3. Small binary (direct transport) + binary_data = b"\x00\x01\x02\x03\x04\x05" + b"\xff" * 100 + + # 4. Large text (link transport - will use fileserver) + large_text = "\n".join([ + f"Line {i}: This is a large text payload for link transport testing. " * 50 + for i in range(100) + ]) + + # Test data list - mixed payload types + data = [ + ("message_text", text_data, "text"), + ("config_dict", dict_data, "dictionary"), + ("small_binary", binary_data, "binary"), + ("large_text", large_text, "text"), + ] + + log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}") + log_trace(correlation_id, f"Correlation ID: {correlation_id}") + + # Use smartsend with mixed types + env = smartsend( + SUBJECT, + data, # List of (dataname, data, type) tuples + nats_url=NATS_URL, + fileserver_url=FILESERVER_URL, + size_threshold=SIZE_THRESHOLD, + correlation_id=correlation_id, + msg_purpose="chat", + sender_name="mixed_sender", + receiver_name="", + receiver_id="", + reply_to="", + reply_to_msg_id="" + ) + + log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads") + + # Log transport type for each payload + for i, payload in enumerate(env.payloads): + log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):") + log_trace(correlation_id, f" Transport: {payload.transport}") + log_trace(correlation_id, f" Type: {payload.type}") + log_trace(correlation_id, f" Size: {payload.size} bytes") + log_trace(correlation_id, f" Encoding: {payload.encoding}") + + if payload.transport == "link": + log_trace(correlation_id, f" URL: {payload.data}") + + print(f"Test completed. Correlation ID: {correlation_id}") + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/test/test_micropython_text_receiver.py b/test/test_micropython_text_receiver.py new file mode 100644 index 0000000..ee585d3 --- /dev/null +++ b/test/test_micropython_text_receiver.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python3 +""" +Test script for text transport testing - Receiver +Tests receiving text messages via NATS using nats_bridge.py smartreceive +""" + +import sys +import os +import json + +# Add src to path for import +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src')) + +from nats_bridge import smartreceive, log_trace +import nats +import asyncio + +# Configuration +SUBJECT = "/NATSBridge_text_test" +NATS_URL = "nats://nats.yiem.cc:4222" + + +async def main(): + log_trace("", f"Starting text transport receiver test...") + log_trace("", f"Note: This receiver will wait for messages from the sender.") + log_trace("", f"Run test_micropython_text_sender.py first to send test data.") + + # Connect to NATS + nc = await nats.connect(NATS_URL) + log_trace("", f"Connected to NATS at {NATS_URL}") + + # Subscribe to the subject + async def message_handler(msg): + log_trace("", f"Received message on {msg.subject}") + + # Use smartreceive to handle the data + result = smartreceive(msg.data) + + # Result is an envelope dictionary with payloads field containing list of (dataname, data, data_type) tuples + for dataname, data, data_type in result["payloads"]: + if isinstance(data, str): + log_trace(result.get("correlationId", ""), f"Received text '{dataname}' of type {data_type}") + log_trace(result.get("correlationId", ""), f" Length: {len(data)} characters") + + # Display first 100 characters + if len(data) > 100: + log_trace(result.get("correlationId", ""), f" First 100 characters: {data[:100]}...") + else: + log_trace(result.get("correlationId", ""), f" Content: {data}") + + # Save to file + output_path = f"./received_{dataname}.txt" + with open(output_path, 'w') as f: + f.write(data) + log_trace(result.get("correlationId", ""), f"Saved text to {output_path}") + else: + log_trace(result.get("correlationId", ""), f"Received unexpected data type for '{dataname}': {type(data)}") + + sid = await nc.subscribe(SUBJECT, cb=message_handler) + log_trace("", f"Subscribed to {SUBJECT} with subscription ID: {sid}") + + # Keep listening for 120 seconds + await asyncio.sleep(120) + await nc.close() + log_trace("", "Test completed.") + + +if __name__ == "__main__": + asyncio.run(main()) \ No newline at end of file diff --git a/test/test_micropython_text_sender.py b/test/test_micropython_text_sender.py new file mode 100644 index 0000000..26200ed --- /dev/null +++ b/test/test_micropython_text_sender.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python3 +""" +Test script for text transport testing - Micropython +Tests sending text messages via NATS using nats_bridge.py smartsend +""" + +import sys +import os + +# Add src to path for import +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src')) + +from nats_bridge import smartsend, log_trace +import uuid + +# Configuration +SUBJECT = "/NATSBridge_text_test" +NATS_URL = "nats://nats.yiem.cc:4222" +FILESERVER_URL = "http://192.168.88.104:8080" +SIZE_THRESHOLD = 1_000_000 # 1MB + +# Create correlation ID for tracing +correlation_id = str(uuid.uuid4()) + + +def main(): + # Create a small text (will use direct transport) + small_text = "Hello, this is a small text message. Testing direct transport via NATS." + + # Create a large text (will use link transport if > 1MB) + # Generate a larger text (~2MB to ensure link transport) + large_text = "\n".join([ + f"Line {i}: This is a sample text line with some content to pad the size. " * 100 + for i in range(500) + ]) + + # Test data 1: small text + data1 = ("small_text", small_text, "text") + + # Test data 2: large text + data2 = ("large_text", large_text, "text") + + log_trace(correlation_id, f"Starting smartsend for subject: {SUBJECT}") + log_trace(correlation_id, f"Correlation ID: {correlation_id}") + + # Use smartsend with text type + # For small text: will use direct transport (Base64 encoded UTF-8) + # For large text: will use link transport (uploaded to fileserver) + env = smartsend( + SUBJECT, + [data1, data2], # List of (dataname, data, type) tuples + nats_url=NATS_URL, + fileserver_url=FILESERVER_URL, + size_threshold=SIZE_THRESHOLD, + correlation_id=correlation_id, + msg_purpose="chat", + sender_name="text_sender", + receiver_name="", + receiver_id="", + reply_to="", + reply_to_msg_id="" + ) + + log_trace(correlation_id, f"Sent message with {len(env.payloads)} payloads") + + # Log transport type for each payload + for i, payload in enumerate(env.payloads): + log_trace(correlation_id, f"Payload {i+1} ('{payload.dataname}'):") + log_trace(correlation_id, f" Transport: {payload.transport}") + log_trace(correlation_id, f" Type: {payload.type}") + log_trace(correlation_id, f" Size: {payload.size} bytes") + log_trace(correlation_id, f" Encoding: {payload.encoding}") + + if payload.transport == "link": + log_trace(correlation_id, f" URL: {payload.data}") + + print(f"Test completed. Correlation ID: {correlation_id}") + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/tutorial_julia.md b/tutorial_julia.md deleted file mode 100644 index 2e395df..0000000 --- a/tutorial_julia.md +++ /dev/null @@ -1,634 +0,0 @@ -# NATSBridge.jl Tutorial - -A comprehensive tutorial for learning how to use NATSBridge.jl for bi-directional communication between Julia and JavaScript services using NATS. - -## Table of Contents - -1. [What is NATSBridge.jl?](#what-is-natsbridgejl) -2. [Key Concepts](#key-concepts) -3. [Installation](#installation) -4. [Basic Usage](#basic-usage) -5. [Payload Types](#payload-types) -6. [Transport Strategies](#transport-strategies) -7. [Advanced Features](#advanced-features) -8. [Complete Examples](#complete-examples) - ---- - -## What is NATSBridge.jl? - -NATSBridge.jl is a Julia module that provides a high-level API for sending and receiving data across network boundaries using NATS as the message bus. It implements the **Claim-Check pattern** for handling large payloads efficiently. - -### Core Features - -- **Bi-directional communication**: Julia ↔ JavaScript -- **Smart transport selection**: Automatic direct vs link transport based on payload size -- **Multi-payload support**: Send multiple payloads of different types in a single message -- **Claim-check pattern**: Upload large files to HTTP server, send only URLs via NATS -- **Type-aware serialization**: Different serialization strategies for different data types - ---- - -## Key Concepts - -### 1. msgEnvelope_v1 (Message Envelope) - -The `msgEnvelope_v1` structure provides a comprehensive message format for bidirectional communication: - -```julia -struct msgEnvelope_v1 - correlationId::String # Unique identifier to track messages - msgId::String # This message id - timestamp::String # Message published timestamp - - sendTo::String # Topic/subject the sender sends to - msgPurpose::String # Purpose (ACK | NACK | updateStatus | shutdown | chat) - senderName::String # Sender name (e.g., "agent-wine-web-frontend") - senderId::String # Sender id (uuid4) - receiverName::String # Message receiver name (e.g., "agent-backend") - receiverId::String # Message receiver id (uuid4 or nothing for broadcast) - replyTo::String # Topic to reply to - replyToMsgId::String # Message id this message is replying to - brokerURL::String # NATS server address - - metadata::Dict{String, Any} - payloads::AbstractArray{msgPayload_v1} # Multiple payloads stored here -end -``` - -### 2. msgPayload_v1 (Payload Structure) - -The `msgPayload_v1` structure provides flexible payload handling: - -```julia -struct msgPayload_v1 - id::String # Id of this payload (e.g., "uuid4") - dataname::String # Name of this payload (e.g., "login_image") - type::String # "text | dictionary | table | image | audio | video | binary" - transport::String # "direct | link" - encoding::String # "none | json | base64 | arrow-ipc" - size::Integer # Data size in bytes - data::Any # Payload data in case of direct transport or a URL in case of link - metadata::Dict{String, Any} # Dict("checksum" => "sha256_hash", ...) -end -``` - -### 3. Standard API Format - -The system uses a **standardized list-of-tuples format** for all payload operations: - -```julia -# Input format for smartsend (always a list of tuples with type info) -[(dataname1, data1, type1), (dataname2, data2, type2), ...] - -# Output format for smartreceive (always returns a list of tuples) -[(dataname1, data1, type1), (dataname2, data2, type2), ...] -``` - -**Important**: Even when sending a single payload, you must wrap it in a list. - ---- - -## Installation - -```julia -using Pkg -Pkg.add("NATS") -Pkg.add("JSON") -Pkg.add("Arrow") -Pkg.add("HTTP") -Pkg.add("UUIDs") -Pkg.add("Dates") -Pkg.add("Base64") -Pkg.add("PrettyPrinting") -Pkg.add("DataFrames") -``` - -Then include the NATSBridge module: - -```julia -include("NATSBridge.jl") -using .NATSBridge -``` - ---- - -## Basic Usage - -### Sending Data (smartsend) - -```julia -using NATSBridge - -# Send a simple dictionary -data = Dict("key" => "value") -env = NATSBridge.smartsend("my.subject", [("dataname1", data, "dictionary")]) -``` - -### Receiving Data (smartreceive) - -```julia -using NATSBridge - -# Subscribe to a NATS subject -NATS.subscribe("my.subject") do msg - # Process the message - result = NATSBridge.smartreceive( - msg, - max_retries = 5, - base_delay = 100, - max_delay = 5000 - ) - - # result is a list of (dataname, data, type) tuples - for (dataname, data, type) in result - println("Received $dataname of type $type") - println("Data: $data") - end -end -``` - ---- - -## Payload Types - -NATSBridge.jl supports the following payload types: - -| Type | Description | Serialization | -|------|-------------|---------------| -| `text` | Plain text | UTF-8 encoding | -| `dictionary` | JSON-serializable data (Dict, NamedTuple) | JSON | -| `table` | Tabular data (DataFrame, array of structs) | Apache Arrow IPC | -| `image` | Image data (Bitmap, PNG/JPG bytes) | Binary | -| `audio` | Audio data (WAV, MP3 bytes) | Binary | -| `video` | Video data (MP4, AVI bytes) | Binary | -| `binary` | Generic binary data | Binary | - ---- - -## Transport Strategies - -NATSBridge.jl automatically selects the appropriate transport strategy based on payload size: - -### Direct Transport (< 1MB) - -Small payloads are encoded as Base64 and sent directly over NATS. - -```julia -# Small data (< 1MB) - uses direct transport -small_data = rand(1000) # ~8KB -env = NATSBridge.smartsend("small", [("data", small_data, "table")]) -``` - -### Link Transport (≥ 1MB) - -Large payloads are uploaded to an HTTP file server, and only the URL is sent via NATS. - -```julia -# Large data (≥ 1MB) - uses link transport -large_data = rand(10_000_000) # ~80MB -env = NATSBridge.smartsend("large", [("data", large_data, "table")]) -``` - ---- - -## Complete Examples - -### Example 1: Text Message - -**Sender:** -```julia -using NATSBridge -using UUIDs - -const SUBJECT = "/NATSBridge_text_test" -const NATS_URL = "nats://localhost:4222" -const FILESERVER_URL = "http://localhost:8080" - -function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any} - url_getUploadID = "$fileserver_url/upload" - headers = ["Content-Type" => "application/json"] - body = """{ "OneShot" : true }""" - httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false) - responseJson = JSON.parse(String(httpResponse.body)) - uploadid = responseJson["id"] - uploadtoken = responseJson["uploadToken"] - - file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream") - url_upload = "$fileserver_url/file/$uploadid" - headers = ["X-UploadToken" => uploadtoken] - - form = HTTP.Form(Dict("file" => file_multipart)) - httpResponse = HTTP.post(url_upload, headers, form) - responseJson = JSON.parse(String(httpResponse.body)) - - fileid = responseJson["id"] - url = "$fileserver_url/file/$uploadid/$fileid/$dataname" - - return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url) -end - -function test_text_send() - small_text = "Hello, this is a small text message." - large_text = join(["Line $i: " for i in 1:50000], "") - - data1 = ("small_text", small_text, "text") - data2 = ("large_text", large_text, "text") - - env = NATSBridge.smartsend( - SUBJECT, - [data1, data2], - nats_url = NATS_URL, - fileserver_url = FILESERVER_URL, - fileserverUploadHandler = plik_upload_handler, - size_threshold = 1_000_000, - correlation_id = string(uuid4()), - msg_purpose = "chat", - sender_name = "text_sender" - ) -end -``` - -**Receiver:** -```julia -using NATSBridge - -const SUBJECT = "/NATSBridge_text_test" -const NATS_URL = "nats://localhost:4222" - -function test_text_receive() - conn = NATS.connect(NATS_URL) - NATS.subscribe(conn, SUBJECT) do msg - result = NATSBridge.smartreceive( - msg, - max_retries = 5, - base_delay = 100, - max_delay = 5000 - ) - - for (dataname, data, data_type) in result - if data_type == "text" - println("Received text: $data") - write("./received_$dataname.txt", data) - end - end - end - sleep(120) - NATS.drain(conn) -end -``` - -### Example 2: Dictionary (JSON) Message - -**Sender:** -```julia -using NATSBridge -using UUIDs - -const SUBJECT = "/NATSBridge_dict_test" -const NATS_URL = "nats://localhost:4222" -const FILESERVER_URL = "http://localhost:8080" - -function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any} - url_getUploadID = "$fileserver_url/upload" - headers = ["Content-Type" => "application/json"] - body = """{ "OneShot" : true }""" - httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false) - responseJson = JSON.parse(String(httpResponse.body)) - uploadid = responseJson["id"] - uploadtoken = responseJson["uploadToken"] - - file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream") - url_upload = "$fileserver_url/file/$uploadid" - headers = ["X-UploadToken" => uploadtoken] - - form = HTTP.Form(Dict("file" => file_multipart)) - httpResponse = HTTP.post(url_upload, headers, form) - responseJson = JSON.parse(String(httpResponse.body)) - - fileid = responseJson["id"] - url = "$fileserver_url/file/$uploadid/$fileid/$dataname" - - return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url) -end - -function test_dict_send() - small_dict = Dict("name" => "Alice", "age" => 30) - large_dict = Dict("ids" => collect(1:50000), "names" => ["User_$i" for i in 1:50000]) - - data1 = ("small_dict", small_dict, "dictionary") - data2 = ("large_dict", large_dict, "dictionary") - - env = NATSBridge.smartsend( - SUBJECT, - [data1, data2], - nats_url = NATS_URL, - fileserver_url = FILESERVER_URL, - fileserverUploadHandler = plik_upload_handler, - size_threshold = 1_000_000, - correlation_id = string(uuid4()), - msg_purpose = "chat" - ) -end -``` - -**Receiver:** -```julia -using NATSBridge - -const SUBJECT = "/NATSBridge_dict_test" -const NATS_URL = "nats://localhost:4222" - -function test_dict_receive() - conn = NATS.connect(NATS_URL) - NATS.subscribe(conn, SUBJECT) do msg - result = NATSBridge.smartreceive( - msg, - max_retries = 5, - base_delay = 100, - max_delay = 5000 - ) - - for (dataname, data, data_type) in result - if data_type == "dictionary" - println("Received dictionary: $data") - write("./received_$dataname.json", JSON.json(data, 2)) - end - end - end - sleep(120) - NATS.drain(conn) -end -``` - -### Example 3: DataFrame (Table) Message - -**Sender:** -```julia -using NATSBridge -using DataFrames -using UUIDs - -const SUBJECT = "/NATSBridge_table_test" -const NATS_URL = "nats://localhost:4222" -const FILESERVER_URL = "http://localhost:8080" - -function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any} - url_getUploadID = "$fileserver_url/upload" - headers = ["Content-Type" => "application/json"] - body = """{ "OneShot" : true }""" - httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false) - responseJson = JSON.parse(String(httpResponse.body)) - uploadid = responseJson["id"] - uploadtoken = responseJson["uploadToken"] - - file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream") - url_upload = "$fileserver_url/file/$uploadid" - headers = ["X-UploadToken" => uploadtoken] - - form = HTTP.Form(Dict("file" => file_multipart)) - httpResponse = HTTP.post(url_upload, headers, form) - responseJson = JSON.parse(String(httpResponse.body)) - - fileid = responseJson["id"] - url = "$fileserver_url/file/$uploadid/$fileid/$dataname" - - return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url) -end - -function test_table_send() - small_df = DataFrame(id = 1:10, name = ["Alice", "Bob", "Charlie"], score = [95, 88, 92]) - large_df = DataFrame(id = 1:50000, name = ["User_$i" for i in 1:50000], score = rand(1:100, 50000)) - - data1 = ("small_table", small_df, "table") - data2 = ("large_table", large_df, "table") - - env = NATSBridge.smartsend( - SUBJECT, - [data1, data2], - nats_url = NATS_URL, - fileserver_url = FILESERVER_URL, - fileserverUploadHandler = plik_upload_handler, - size_threshold = 1_000_000, - correlation_id = string(uuid4()), - msg_purpose = "chat" - ) -end -``` - -**Receiver:** -```julia -using NATSBridge -using DataFrames - -const SUBJECT = "/NATSBridge_table_test" -const NATS_URL = "nats://localhost:4222" - -function test_table_receive() - conn = NATS.connect(NATS_URL) - NATS.subscribe(conn, SUBJECT) do msg - result = NATSBridge.smartreceive( - msg, - max_retries = 5, - base_delay = 100, - max_delay = 5000 - ) - - for (dataname, data, data_type) in result - if data_type == "table" - data = DataFrame(data) - println("Received DataFrame with $(size(data, 1)) rows") - display(data[1:min(5, size(data, 1)), :]) - end - end - end - sleep(120) - NATS.drain(conn) -end -``` - -### Example 4: Mixed Content (Chat with Text, Image, Audio) - -**Sender:** -```julia -using NATSBridge -using DataFrames -using UUIDs - -const SUBJECT = "/NATSBridge_mix_test" -const NATS_URL = "nats://localhost:4222" -const FILESERVER_URL = "http://localhost:8080" - -function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any} - url_getUploadID = "$fileserver_url/upload" - headers = ["Content-Type" => "application/json"] - body = """{ "OneShot" : true }""" - httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false) - responseJson = JSON.parse(String(httpResponse.body)) - uploadid = responseJson["id"] - uploadtoken = responseJson["uploadToken"] - - file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream") - url_upload = "$fileserver_url/file/$uploadid" - headers = ["X-UploadToken" => uploadtoken] - - form = HTTP.Form(Dict("file" => file_multipart)) - httpResponse = HTTP.post(url_upload, headers, form) - responseJson = JSON.parse(String(httpResponse.body)) - - fileid = responseJson["id"] - url = "$fileserver_url/file/$uploadid/$fileid/$dataname" - - return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url) -end - -function test_mix_send() - # Text data - text_data = "Hello! This is a test chat message. 🎉" - - # Dictionary data - dict_data = Dict("type" => "chat", "sender" => "serviceA") - - # Small table data - table_data_small = DataFrame(id = 1:10, name = ["msg_$i" for i in 1:10]) - - # Large table data (link transport) - table_data_large = DataFrame(id = 1:150_000, name = ["msg_$i" for i in 1:150_000]) - - # Small image data (direct transport) - image_data = UInt8[rand(1:255) for _ in 1:100] - - # Large image data (link transport) - large_image_data = UInt8[rand(1:255) for _ in 1:1_500_000] - - # Small audio data (direct transport) - audio_data = UInt8[rand(1:255) for _ in 1:100] - - # Large audio data (link transport) - large_audio_data = UInt8[rand(1:255) for _ in 1:1_500_000] - - # Small video data (direct transport) - video_data = UInt8[rand(1:255) for _ in 1:150] - - # Large video data (link transport) - large_video_data = UInt8[rand(1:255) for _ in 1:1_500_000] - - # Small binary data (direct transport) - binary_data = UInt8[rand(1:255) for _ in 1:200] - - # Large binary data (link transport) - large_binary_data = UInt8[rand(1:255) for _ in 1:1_500_000] - - # Create payloads list - mixed content - payloads = [ - # Small data (direct transport) - ("chat_text", text_data, "text"), - ("chat_json", dict_data, "dictionary"), - ("chat_table_small", table_data_small, "table"), - - # Large data (link transport) - ("chat_table_large", table_data_large, "table"), - ("user_image_large", large_image_data, "image"), - ("audio_clip_large", large_audio_data, "audio"), - ("video_clip_large", large_video_data, "video"), - ("binary_file_large", large_binary_data, "binary") - ] - - env = NATSBridge.smartsend( - SUBJECT, - payloads, - nats_url = NATS_URL, - fileserver_url = FILESERVER_URL, - fileserverUploadHandler = plik_upload_handler, - size_threshold = 1_000_000, - correlation_id = string(uuid4()), - msg_purpose = "chat", - sender_name = "mix_sender" - ) -end -``` - -**Receiver:** -```julia -using NATSBridge -using DataFrames - -const SUBJECT = "/NATSBridge_mix_test" -const NATS_URL = "nats://localhost:4222" - -function test_mix_receive() - conn = NATS.connect(NATS_URL) - NATS.subscribe(conn, SUBJECT) do msg - result = NATSBridge.smartreceive( - msg, - max_retries = 5, - base_delay = 100, - max_delay = 5000 - ) - - println("Received $(length(result)) payloads") - - for (dataname, data, data_type) in result - println("\n=== Payload: $dataname (type: $data_type) ===") - - if data_type == "text" - println(" Type: String") - println(" Length: $(length(data)) characters") - - elseif data_type == "dictionary" - println(" Type: JSON Object") - println(" Keys: $(keys(data))") - - elseif data_type == "table" - data = DataFrame(data) - println(" Type: DataFrame") - println(" Dimensions: $(size(data, 1)) rows x $(size(data, 2)) columns") - - elseif data_type == "image" - println(" Type: Vector{UInt8}") - println(" Size: $(length(data)) bytes") - write("./received_$dataname.bin", data) - - elseif data_type == "audio" - println(" Type: Vector{UInt8}") - println(" Size: $(length(data)) bytes") - write("./received_$dataname.bin", data) - - elseif data_type == "video" - println(" Type: Vector{UInt8}") - println(" Size: $(length(data)) bytes") - write("./received_$dataname.bin", data) - - elseif data_type == "binary" - println(" Type: Vector{UInt8}") - println(" Size: $(length(data)) bytes") - write("./received_$dataname.bin", data) - end - end - end - sleep(120) - NATS.drain(conn) -end -``` - ---- - -## Best Practices - -1. **Always wrap payloads in a list** - Even for single payloads: `[("dataname", data, "type")]` -2. **Use appropriate transport** - Let NATSBridge handle size-based routing (default 1MB threshold) -3. **Customize size threshold** - Use `size_threshold` parameter to adjust the direct/link split -4. **Provide fileserver handler** - Implement `fileserverUploadHandler` for link transport -5. **Include correlation IDs** - Track messages across distributed systems -6. **Handle errors** - Implement proper error handling for network failures -7. **Close connections** - Ensure NATS connections are properly closed using `NATS.drain()` - ---- - -## Conclusion - -NATSBridge.jl provides a powerful abstraction for bi-directional communication between Julia and JavaScript services. By understanding the key concepts and following the best practices, you can build robust, scalable applications that leverage the full power of NATS messaging. - -For more information, see: -- [`docs/architecture.md`](./architecture.md) - Detailed architecture documentation -- [`docs/implementation.md`](./implementation.md) - Implementation details \ No newline at end of file diff --git a/walkthrough_julia.md b/walkthrough_julia.md deleted file mode 100644 index d7081dc..0000000 --- a/walkthrough_julia.md +++ /dev/null @@ -1,939 +0,0 @@ -# NATSBridge.jl Walkthrough: Building a Chat System - -A step-by-step guided walkthrough for building a real-time chat system using NATSBridge.jl with mixed content support (text, images, audio, video, and files). - -## Prerequisites - -- Julia 1.7+ -- NATS server running -- HTTP file server (Plik) running - -## Step 1: Understanding the Chat System Architecture - -### System Components - -``` -┌─────────────────────────────────────────────────────────────────────────────┐ -│ Chat System │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌──────────────┐ NATS ┌──────────────┐ │ -│ │ Julia │◄───────┬───────► │ JavaScript │ │ -│ │ Service │ │ │ Client │ │ -│ │ │ │ │ │ │ -│ │ - Text │ │ │ - Text │ │ -│ │ - Images │ │ │ - Images │ │ -│ │ - Audio │ ▼ │ - Audio │ │ -│ │ - Video │ NATSBridge.jl │ - Files │ │ -│ │ - Files │ │ │ - Tables │ │ -│ └──────────────┘ │ └──────────────┘ │ -│ │ │ -│ ┌───────┴───────┐ │ -│ │ NATS │ │ -│ │ Server │ │ -│ └─────────────┘ │ -│ │ -└─────────────────────────────────────────────────────────────────────────────┘ - -For large payloads (> 1MB): -┌─────────────────────────────────────────────────────────────────────────────┐ -│ File Server (Plik) │ -│ │ -│ Julia Service ──► Upload ──► File Server ──► Download ◄── JavaScript Client│ -│ │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - -### Message Format - -Each chat message is an envelope containing multiple payloads: - -```json -{ - "correlationId": "uuid4", - "msgId": "uuid4", - "timestamp": "2024-01-15T10:30:00Z", - "sendTo": "/chat/room1", - "msgPurpose": "chat", - "senderName": "user-1", - "senderId": "uuid4", - "receiverName": "user-2", - "receiverId": "uuid4", - "brokerURL": "nats://localhost:4222", - "payloads": [ - { - "id": "uuid4", - "dataname": "message_text", - "type": "text", - "transport": "direct", - "encoding": "base64", - "size": 256, - "data": "SGVsbG8gV29ybGQh", - "metadata": {} - }, - { - "id": "uuid4", - "dataname": "user_image", - "type": "image", - "transport": "link", - "encoding": "none", - "size": 15433, - "data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/image.jpg", - "metadata": {} - } - ] -} -``` - -## Step 2: Setting Up the Environment - -### 1. Start NATS Server - -```bash -# Using Docker -docker run -d -p 4222:4222 -p 8222:8222 --name nats-server nats:latest - -# Or download from https://github.com/nats-io/nats-server/releases -./nats-server -``` - -### 2. Start HTTP File Server (Plik) - -```bash -# Using Docker -docker run -d -p 8080:8080 --name plik plik/plik:latest - -# Or download from https://github.com/arnaud-lb/plik/releases -./plikd -d -``` - -### 3. Install Julia Dependencies - -```julia -using Pkg -Pkg.add("NATS") -Pkg.add("JSON") -Pkg.add("Arrow") -Pkg.add("HTTP") -Pkg.add("UUIDs") -Pkg.add("Dates") -Pkg.add("Base64") -Pkg.add("PrettyPrinting") -Pkg.add("DataFrames") -``` - -## Step 3: Basic Text-Only Chat - -### Sender (User 1) - -```julia -using NATS -using JSON -using UUIDs -using Dates -using PrettyPrinting -using DataFrames -using Arrow -using HTTP -using Base64 - -# Include the bridge module -include("NATSBridge.jl") -using .NATSBridge - -# Configuration -const NATS_URL = "nats://localhost:4222" -const FILESERVER_URL = "http://localhost:8080" -const SUBJECT = "/chat/room1" - -# File upload handler for plik server -function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any} - url_getUploadID = "$fileserver_url/upload" - headers = ["Content-Type" => "application/json"] - body = """{ "OneShot" : true }""" - httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false) - responseJson = JSON.parse(String(httpResponse.body)) - uploadid = responseJson["id"] - uploadtoken = responseJson["uploadToken"] - - file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream") - url_upload = "$fileserver_url/file/$uploadid" - headers = ["X-UploadToken" => uploadtoken] - - form = HTTP.Form(Dict("file" => file_multipart)) - httpResponse = HTTP.post(url_upload, headers, form) - responseJson = JSON.parse(String(httpResponse.body)) - - fileid = responseJson["id"] - url = "$fileserver_url/file/$uploadid/$fileid/$dataname" - - return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url) -end - -# Send a simple text message -function send_text_message() - message_text = "Hello, how are you today?" - - env = NATSBridge.smartsend( - SUBJECT, - [("message", message_text, "text")], - nats_url = NATS_URL, - fileserver_url = FILESERVER_URL, - fileserverUploadHandler = plik_upload_handler, - size_threshold = 1_000_000, - correlation_id = string(uuid4()), - msg_purpose = "chat", - sender_name = "user-1" - ) - - println("Sent text message with correlation ID: $(env.correlationId)") -end - -send_text_message() -``` - -### Receiver (User 2) - -```julia -using NATS -using JSON -using UUIDs -using Dates -using PrettyPrinting -using DataFrames -using Arrow -using HTTP -using Base64 - -# Include the bridge module -include("NATSBridge.jl") -using .NATSBridge - -# Configuration -const NATS_URL = "nats://localhost:4222" -const SUBJECT = "/chat/room1" - -# Message handler -function message_handler(msg::NATS.Msg) - payloads = NATSBridge.smartreceive( - msg, - max_retries = 5, - base_delay = 100, - max_delay = 5000 - ) - - # Extract the text message - for (dataname, data, data_type) in payloads - if data_type == "text" - println("Received message: $data") - # Save to file - write("./received_$dataname.txt", data) - end - end -end - -# Subscribe to the chat room -NATS.subscribe(SUBJECT) do msg - message_handler(msg) -end - -# Keep the program running -while true - sleep(1) -end -``` - -## Step 4: Adding Image Support - -### Sending an Image - -```julia -using NATS -using JSON -using UUIDs -using Dates -using PrettyPrinting -using DataFrames -using Arrow -using HTTP -using Base64 - -include("NATSBridge.jl") -using .NATSBridge - -const NATS_URL = "nats://localhost:4222" -const FILESERVER_URL = "http://localhost:8080" -const SUBJECT = "/chat/room1" - -function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any} - url_getUploadID = "$fileserver_url/upload" - headers = ["Content-Type" => "application/json"] - body = """{ "OneShot" : true }""" - httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false) - responseJson = JSON.parse(String(httpResponse.body)) - uploadid = responseJson["id"] - uploadtoken = responseJson["uploadToken"] - - file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream") - url_upload = "$fileserver_url/file/$uploadid" - headers = ["X-UploadToken" => uploadtoken] - - form = HTTP.Form(Dict("file" => file_multipart)) - httpResponse = HTTP.post(url_upload, headers, form) - responseJson = JSON.parse(String(httpResponse.body)) - - fileid = responseJson["id"] - url = "$fileserver_url/file/$uploadid/$fileid/$dataname" - - return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url) -end - -function send_image() - # Read image file - image_data = read("screenshot.png", Vector{UInt8}) - - # Send with text message - env = NATSBridge.smartsend( - SUBJECT, - [ - ("text", "Check out this screenshot!", "text"), - ("screenshot", image_data, "image") - ], - nats_url = NATS_URL, - fileserver_url = FILESERVER_URL, - fileserverUploadHandler = plik_upload_handler, - size_threshold = 1_000_000, - correlation_id = string(uuid4()), - msg_purpose = "chat", - sender_name = "user-1" - ) - - println("Sent image with correlation ID: $(env.correlationId)") -end - -send_image() -``` - -### Receiving an Image - -```julia -using NATS -using JSON -using UUIDs -using Dates -using PrettyPrinting -using DataFrames -using Arrow -using HTTP -using Base64 - -include("NATSBridge.jl") -using .NATSBridge - -const NATS_URL = "nats://localhost:4222" -const SUBJECT = "/chat/room1" - -function message_handler(msg::NATS.Msg) - payloads = NATSBridge.smartreceive( - msg, - max_retries = 5, - base_delay = 100, - max_delay = 5000 - ) - - for (dataname, data, data_type) in payloads - if data_type == "text" - println("Text: $data") - elseif data_type == "image" - # Save image to file - filename = "received_$dataname.bin" - write(filename, data) - println("Saved image: $filename") - end - end -end - -NATS.subscribe(SUBJECT) do msg - message_handler(msg) -end -``` - -## Step 5: Handling Large Files with Link Transport - -### Automatic Transport Selection - -```julia -using NATS -using JSON -using UUIDs -using Dates -using PrettyPrinting -using DataFrames -using Arrow -using HTTP -using Base64 - -include("NATSBridge.jl") -using .NATSBridge - -const NATS_URL = "nats://localhost:4222" -const FILESERVER_URL = "http://localhost:8080" -const SUBJECT = "/chat/room1" - -function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any} - url_getUploadID = "$fileserver_url/upload" - headers = ["Content-Type" => "application/json"] - body = """{ "OneShot" : true }""" - httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false) - responseJson = JSON.parse(String(httpResponse.body)) - uploadid = responseJson["id"] - uploadtoken = responseJson["uploadToken"] - - file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream") - url_upload = "$fileserver_url/file/$uploadid" - headers = ["X-UploadToken" => uploadtoken] - - form = HTTP.Form(Dict("file" => file_multipart)) - httpResponse = HTTP.post(url_upload, headers, form) - responseJson = JSON.parse(String(httpResponse.body)) - - fileid = responseJson["id"] - url = "$fileserver_url/file/$uploadid/$fileid/$dataname" - - return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url) -end - -function send_large_file() - # Create a large file (> 1MB triggers link transport) - large_data = rand(10_000_000) # ~80MB - - env = NATSBridge.smartsend( - SUBJECT, - [("large_file", large_data, "binary")], - nats_url = NATS_URL, - fileserver_url = FILESERVER_URL, - fileserverUploadHandler = plik_upload_handler, - size_threshold = 1_000_000, - correlation_id = string(uuid4()), - msg_purpose = "chat", - sender_name = "user-1" - ) - - println("Uploaded large file to: $(env.payloads[1].data)") - println("Correlation ID: $(env.correlationId)") -end - -send_large_file() -``` - -## Step 6: Audio and Video Support - -### Sending Audio - -```julia -using NATS -using JSON -using UUIDs -using Dates -using PrettyPrinting -using DataFrames -using Arrow -using HTTP -using Base64 - -include("NATSBridge.jl") -using .NATSBridge - -const NATS_URL = "nats://localhost:4222" -const FILESERVER_URL = "http://localhost:8080" -const SUBJECT = "/chat/room1" - -function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any} - url_getUploadID = "$fileserver_url/upload" - headers = ["Content-Type" => "application/json"] - body = """{ "OneShot" : true }""" - httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false) - responseJson = JSON.parse(String(httpResponse.body)) - uploadid = responseJson["id"] - uploadtoken = responseJson["uploadToken"] - - file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream") - url_upload = "$fileserver_url/file/$uploadid" - headers = ["X-UploadToken" => uploadtoken] - - form = HTTP.Form(Dict("file" => file_multipart)) - httpResponse = HTTP.post(url_upload, headers, form) - responseJson = JSON.parse(String(httpResponse.body)) - - fileid = responseJson["id"] - url = "$fileserver_url/file/$uploadid/$fileid/$dataname" - - return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url) -end - -function send_audio() - # Read audio file (WAV, MP3, etc.) - audio_data = read("voice_message.mp3", Vector{UInt8}) - - env = NATSBridge.smartsend( - SUBJECT, - [("voice_message", audio_data, "audio")], - nats_url = NATS_URL, - fileserver_url = FILESERVER_URL, - fileserverUploadHandler = plik_upload_handler, - size_threshold = 1_000_000, - correlation_id = string(uuid4()), - msg_purpose = "chat", - sender_name = "user-1" - ) - - println("Sent audio message: $(env.correlationId)") -end - -send_audio() -``` - -### Sending Video - -```julia -using NATS -using JSON -using UUIDs -using Dates -using PrettyPrinting -using DataFrames -using Arrow -using HTTP -using Base64 - -include("NATSBridge.jl") -using .NATSBridge - -const NATS_URL = "nats://localhost:4222" -const FILESERVER_URL = "http://localhost:8080" -const SUBJECT = "/chat/room1" - -function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any} - url_getUploadID = "$fileserver_url/upload" - headers = ["Content-Type" => "application/json"] - body = """{ "OneShot" : true }""" - httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false) - responseJson = JSON.parse(String(httpResponse.body)) - uploadid = responseJson["id"] - uploadtoken = responseJson["uploadToken"] - - file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream") - url_upload = "$fileserver_url/file/$uploadid" - headers = ["X-UploadToken" => uploadtoken] - - form = HTTP.Form(Dict("file" => file_multipart)) - httpResponse = HTTP.post(url_upload, headers, form) - responseJson = JSON.parse(String(httpResponse.body)) - - fileid = responseJson["id"] - url = "$fileserver_url/file/$uploadid/$fileid/$dataname" - - return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url) -end - -function send_video() - # Read video file (MP4, AVI, etc.) - video_data = read("video_message.mp4", Vector{UInt8}) - - env = NATSBridge.smartsend( - SUBJECT, - [("video_message", video_data, "video")], - nats_url = NATS_URL, - fileserver_url = FILESERVER_URL, - fileserverUploadHandler = plik_upload_handler, - size_threshold = 1_000_000, - correlation_id = string(uuid4()), - msg_purpose = "chat", - sender_name = "user-1" - ) - - println("Sent video message: $(env.correlationId)") -end - -send_video() -``` - -## Step 7: Table/Data Exchange - -### Sending Tabular Data - -```julia -using NATS -using JSON -using UUIDs -using Dates -using PrettyPrinting -using DataFrames -using Arrow -using HTTP -using Base64 - -include("NATSBridge.jl") -using .NATSBridge - -const NATS_URL = "nats://localhost:4222" -const FILESERVER_URL = "http://localhost:8080" -const SUBJECT = "/chat/room1" - -function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any} - url_getUploadID = "$fileserver_url/upload" - headers = ["Content-Type" => "application/json"] - body = """{ "OneShot" : true }""" - httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false) - responseJson = JSON.parse(String(httpResponse.body)) - uploadid = responseJson["id"] - uploadtoken = responseJson["uploadToken"] - - file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream") - url_upload = "$fileserver_url/file/$uploadid" - headers = ["X-UploadToken" => uploadtoken] - - form = HTTP.Form(Dict("file" => file_multipart)) - httpResponse = HTTP.post(url_upload, headers, form) - responseJson = JSON.parse(String(httpResponse.body)) - - fileid = responseJson["id"] - url = "$fileserver_url/file/$uploadid/$fileid/$dataname" - - return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url) -end - -function send_table() - # Create a DataFrame - df = DataFrame( - id = 1:5, - name = ["Alice", "Bob", "Charlie", "Diana", "Eve"], - score = [95, 88, 92, 98, 85], - grade = ['A', 'B', 'A', 'B', 'B'] - ) - - env = NATSBridge.smartsend( - SUBJECT, - [("student_scores", df, "table")], - nats_url = NATS_URL, - fileserver_url = FILESERVER_URL, - fileserverUploadHandler = plik_upload_handler, - size_threshold = 1_000_000, - correlation_id = string(uuid4()), - msg_purpose = "chat", - sender_name = "user-1" - ) - - println("Sent table with $(nrow(df)) rows") -end - -send_table() -``` - -### Receiving and Using Tables - -```julia -using NATS -using JSON -using UUIDs -using Dates -using PrettyPrinting -using DataFrames -using Arrow -using HTTP -using Base64 - -include("NATSBridge.jl") -using .NATSBridge - -const NATS_URL = "nats://localhost:4222" -const SUBJECT = "/chat/room1" - -function message_handler(msg::NATS.Msg) - payloads = NATSBridge.smartreceive( - msg, - max_retries = 5, - base_delay = 100, - max_delay = 5000 - ) - - for (dataname, data, data_type) in payloads - if data_type == "table" - data = DataFrame(data) - println("Received table:") - show(data) - println("\nAverage score: $(mean(data.score))") - end - end -end - -NATS.subscribe(SUBJECT) do msg - message_handler(msg) -end -``` - -## Step 8: Bidirectional Communication - -### Request-Response Pattern - -```julia -using NATS -using JSON -using UUIDs -using Dates -using PrettyPrinting -using DataFrames -using Arrow -using HTTP -using Base64 - -include("NATSBridge.jl") -using .NATSBridge - -const NATS_URL = "nats://localhost:4222" -const SUBJECT = "/api/query" -const REPLY_SUBJECT = "/api/response" - -# Request -function send_request() - query_data = Dict("query" => "SELECT * FROM users") - - env = NATSBridge.smartsend( - SUBJECT, - [("sql_query", query_data, "dictionary")], - nats_url = NATS_URL, - fileserver_url = "http://localhost:8080", - fileserverUploadHandler = plik_upload_handler, - size_threshold = 1_000_000, - correlation_id = string(uuid4()), - msg_purpose = "request", - sender_name = "frontend", - receiver_name = "backend", - reply_to = REPLY_SUBJECT, - reply_to_msg_id = string(uuid4()) - ) - - println("Request sent: $(env.correlationId)") -end - -# Response handler -function response_handler(msg::NATS.Msg) - payloads = NATSBridge.smartreceive( - msg, - max_retries = 5, - base_delay = 100, - max_delay = 5000 - ) - - for (dataname, data, data_type) in payloads - if data_type == "table" - data = DataFrame(data) - println("Query results:") - show(data) - end - end -end - -NATS.subscribe(REPLY_SUBJECT) do msg - response_handler(msg) -end -``` - -## Step 9: Complete Chat Application - -### Full Chat System - -```julia -module ChatApp -using NATS -using JSON -using UUIDs -using Dates -using PrettyPrinting -using DataFrames -using Arrow -using HTTP -using Base64 - -# Include the bridge module -include("../src/NATSBridge.jl") -using .NATSBridge - -# Configuration -const NATS_URL = "nats://localhost:4222" -const FILESERVER_URL = "http://localhost:8080" -const SUBJECT = "/chat/room1" - -# File upload handler for plik server -function plik_upload_handler(fileserver_url::String, dataname::String, data::Vector{UInt8})::Dict{String, Any} - url_getUploadID = "$fileserver_url/upload" - headers = ["Content-Type" => "application/json"] - body = """{ "OneShot" : true }""" - httpResponse = HTTP.request("POST", url_getUploadID, headers, body; body_is_form=false) - responseJson = JSON.parse(String(httpResponse.body)) - uploadid = responseJson["id"] - uploadtoken = responseJson["uploadToken"] - - file_multipart = HTTP.Multipart(dataname, IOBuffer(data), "application/octet-stream") - url_upload = "$fileserver_url/file/$uploadid" - headers = ["X-UploadToken" => uploadtoken] - - form = HTTP.Form(Dict("file" => file_multipart)) - httpResponse = HTTP.post(url_upload, headers, form) - responseJson = JSON.parse(String(httpResponse.body)) - - fileid = responseJson["id"] - url = "$fileserver_url/file/$uploadid/$fileid/$dataname" - - return Dict("status" => httpResponse.status, "uploadid" => uploadid, "fileid" => fileid, "url" => url) -end - -function send_chat_message( - text::String, - image_path::Union{String, Nothing}=nothing, - audio_path::Union{String, Nothing}=nothing -) - # Build payloads list - payloads = [("message_text", text, "text")] - - if image_path !== nothing - image_data = read(image_path, Vector{UInt8}) - push!(payloads, ("user_image", image_data, "image")) - end - - if audio_path !== nothing - audio_data = read(audio_path, Vector{UInt8}) - push!(payloads, ("user_audio", audio_data, "audio")) - end - - env = NATSBridge.smartsend( - SUBJECT, - payloads, - nats_url = NATS_URL, - fileserver_url = FILESERVER_URL, - fileserverUploadHandler = plik_upload_handler, - size_threshold = 1_000_000, - correlation_id = string(uuid4()), - msg_purpose = "chat", - sender_name = "user-1" - ) - - println("Message sent with correlation ID: $(env.correlationId)") -end - -function receive_chat_messages() - function message_handler(msg::NATS.Msg) - payloads = NATSBridge.smartreceive( - msg, - max_retries = 5, - base_delay = 100, - max_delay = 5000 - ) - - println("\n--- New Message ---") - for (dataname, data, data_type) in payloads - if data_type == "text" - println("Text: $data") - elseif data_type == "image" - filename = "received_$dataname.bin" - write(filename, data) - println("Image saved: $filename") - elseif data_type == "audio" - filename = "received_$dataname.bin" - write(filename, data) - println("Audio saved: $filename") - elseif data_type == "table" - println("Table received:") - data = DataFrame(data) - show(data) - end - end - end - - NATS.subscribe(SUBJECT) do msg - message_handler(msg) - end - println("Subscribed to: $SUBJECT") -end - -function run_interactive_chat() - println("\n=== Interactive Chat ===") - println("1. Send a message") - println("2. Join a chat room") - println("3. Exit") - - while true - print("\nSelect option (1-3): ") - choice = readline() - - if choice == "1" - print("Enter message text: ") - text = readline() - send_chat_message(text) - elseif choice == "2" - receive_chat_messages() - elseif choice == "3" - break - end - end -end - -end # module - -# Run the chat app -using .ChatApp -ChatApp.run_interactive_chat() -``` - -## Step 10: Testing the Chat System - -### Test Scenario 1: Text-Only Chat - -```bash -# Terminal 1: Start the chat receiver -julia test_julia_to_julia_text_receiver.jl - -# Terminal 2: Send a message -julia test_julia_to_julia_text_sender.jl -``` - -### Test Scenario 2: Image Chat - -```bash -# Terminal 1: Receive messages -julia test_julia_to_julia_mix_payloads_receiver.jl - -# Terminal 2: Send image -julia test_julia_to_julia_mix_payload_sender.jl -``` - -### Test Scenario 3: Large File Transfer - -```bash -# Terminal 2: Send large file -julia test_julia_to_julia_mix_payload_sender.jl -``` - -## Conclusion - -This walkthrough demonstrated how to build a chat system using NATSBridge.jl with support for: - -- Text messages -- Images (direct transport for small, link transport for large) -- Audio files -- Video files -- Tabular data (DataFrames) -- Bidirectional communication -- Mixed-content messages - -The key takeaways are: - -1. **Always wrap payloads in a list** - Even for single payloads: `[("dataname", data, "type")]` -2. **Use appropriate transport** - NATSBridge automatically handles size-based routing -3. **Support mixed content** - Multiple payloads of different types in one message -4. **Handle errors** - Implement proper error handling for network failures -5. **Use correlation IDs** - Track messages across distributed systems - -For more information, see: -- [`docs/architecture.md`](./docs/architecture.md) - Detailed architecture documentation -- [`docs/implementation.md`](./docs/implementation.md) - Implementation details \ No newline at end of file