update

2026-02-22 21:55:18 +07:00
parent 69f2173f75
commit def75d8f86
5 changed files with 541 additions and 127 deletions
--- a/docs/implementation.md
+++ b/docs/implementation.md
@@ -2,7 +2,22 @@

 ## Overview

-This document describes the implementation of the high-performance, bi-directional data bridge between Julia and JavaScript services using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
+This document describes the implementation of the high-performance, bi-directional data bridge between **Julia**, **JavaScript**, and **Python/Micropython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
+
+The system enables seamless communication across all three platforms:
+- **Julia ↔ JavaScript** bi-directional messaging
+- **JavaScript ↔ Python/Micropython** bi-directional messaging
+- **Julia ↔ Python/Micropython** bi-directional messaging (via JSON serialization)
+
+### Implementation Files
+
+NATSBridge is implemented in three languages, each providing the same API:
+
+| Language | Implementation File | Description |
+|----------|---------------------|-------------|
+| **Julia** | [`src/NATSBridge.jl`](../src/NATSBridge.jl) | Full Julia implementation with Arrow IPC support |
+| **JavaScript** | [`src/NATSBridge.js`](../src/NATSBridge.js) | JavaScript implementation for Node.js and browsers |
+| **Python/Micropython** | [`src/nats_bridge.py`](../src/nats_bridge.py) | Python implementation for desktop and microcontrollers |

 ### Multi-Payload Support

@@ -49,44 +64,97 @@ envelope = smartreceive(msg, ...)
 # envelope["correlationId"], envelope["msgId"], etc.
 ```

+## Cross-Platform Interoperability
+
+NATSBridge is designed for seamless communication between Julia, JavaScript, and Python/Micropython applications. All three implementations share the same interface and data format, ensuring compatibility across platforms.
+
+### Platform-Specific Features
+
+| Feature | Julia | JavaScript | Python/Micropython |
+|---------|-------|------------|-------------------|
+| Direct NATS transport | ✅ | ✅ | ✅ |
+| HTTP file server (Claim-Check) | ✅ | ✅ | ✅ |
+| Arrow IPC tables | ✅ | ✅ | ✅ |
+| Base64 encoding | ✅ | ✅ | ✅ |
+| Exponential backoff | ✅ | ✅ | ✅ |
+| Correlation ID tracking | ✅ | ✅ | ✅ |
+| Reply-to support | ✅ | ✅ | ✅ |
+
+### Data Type Mapping
+
+| Type | Julia | JavaScript | Python/Micropython |
+|------|-------|------------|-------------------|
+| `text` | `String` | `String` | `str` |
+| `dictionary` | `Dict` | `Object` | `dict` |
+| `table` | `DataFrame` | `Array<Object>` | `DataFrame` / `list` |
+| `image` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
+| `audio` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
+| `video` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
+| `binary` | `Vector{UInt8}` | `ArrayBuffer/Uint8Array` | `bytes` |
+
+### Example: Julia ↔ Python ↔ JavaScript
+
+```julia
+# Julia sender
+using NATSBridge
+data = [("message", "Hello from Julia!", "text")]
+smartsend("/cross_platform", data, nats_url="nats://localhost:4222")
+```
+
+```javascript
+// JavaScript receiver
+const { smartreceive } = require('./src/NATSBridge');
+const envelope = await smartreceive(msg);
+// envelope.payloads[0].data === "Hello from Julia!"
+```
+
+```python
+# Python sender
+from nats_bridge import smartsend
+data = [("response", "Hello from Python!", "text")]
+smartsend("/cross_platform", data, nats_url="nats://localhost:4222")
+```
+
+All three platforms can communicate seamlessly using the same NATS subjects and data format.
+
 ## Architecture

-The implementation follows the Claim-Check pattern:
+All three implementations (Julia, JavaScript, Python/Micropython) follow the same Claim-Check pattern:

 ```
 ┌─────────────────────────────────────────────────────────────────────────┐
 │                         SmartSend Function                              │
 └─────────────────────────────────────────────────────────────────────────┘
-                             │
-                             ▼
+                              │
+                              ▼
 ┌─────────────────────────────────────────────────────────────────────────┐
 │  Is payload size < 1MB?                                                 │
 └─────────────────────────────────────────────────────────────────────────┘
-                             │
-           ┌─────────────────┴─────────────────┐
-           ▼                                   ▼
-     ┌─────────────────┐                 ┌─────────────────┐
-     │  Direct Path    │                 │  Link Path      │
-     │  (< 1MB)        │                 │  (> 1MB)        │
-     │                 │                 │                 │
-     │ • Serialize to  │                 │ • Serialize to  │
-     │   IOBuffer      │                 │   IOBuffer      │
-     │ • Base64 encode │                 │ • Upload to     │
-     │ • Publish to    │                 │   HTTP Server   │
-     │   NATS          │                 │ • Publish to    │
-     │                 │                 │   NATS with URL │
-     └─────────────────┘                 └─────────────────┘
+                              │
+            ┌─────────────────┴─────────────────┐
+            ▼                                   ▼
+      ┌─────────────────┐                 ┌─────────────────┐
+      │  Direct Path    │                 │  Link Path      │
+      │  (< 1MB)        │                 │  (> 1MB)        │
+      │                 │                 │                 │
+      │ • Serialize to  │                 │ • Serialize to  │
+      │   Buffer        │                 │   Buffer        │
+      │ • Base64 encode │                 │ • Upload to     │
+      │ • Publish to    │                 │   HTTP Server   │
+      │   NATS          │                 │ • Publish to    │
+      │                 │                 │   NATS with URL │
+      └─────────────────┘                 └─────────────────┘
 ```

 ## Files

-### Julia Module: [`src/julia_bridge.jl`](../src/julia_bridge.jl)
+### Julia Module: [`src/NATSBridge.jl`](../src/NATSBridge.jl)

 The Julia implementation provides:

- **[`MessageEnvelope`](../src/julia_bridge.jl)**: Struct for the unified JSON envelope
- **[`SmartSend()`](../src/julia_bridge.jl)**: Handles transport selection based on payload size
- **[`SmartReceive()`](../src/julia_bridge.jl)**: Handles both direct and link transport
+- **[`MessageEnvelope`](src/NATSBridge.jl)**: Struct for the unified JSON envelope
+- **[`SmartSend()`](src/NATSBridge.jl)**: Handles transport selection based on payload size
+- **[`SmartReceive()`](src/NATSBridge.jl)**: Handles both direct and link transport

 ### JavaScript Module: [`src/NATSBridge.js`](../src/NATSBridge.js)

@@ -94,8 +162,17 @@ The JavaScript implementation provides:

 - **`MessageEnvelope` class**: For the unified JSON envelope
 - **`MessagePayload` class**: For individual payload representation
- **[`smartsend()`](../src/NATSBridge.js)**: Handles transport selection based on payload size
- **[`smartreceive()`](../src/NATSBridge.js)**: Handles both direct and link transport
+- **[`smartsend()`](src/NATSBridge.js)**: Handles transport selection based on payload size
+- **[`smartreceive()`](src/NATSBridge.js)**: Handles both direct and link transport
+
+### Python/Micropython Module: [`src/nats_bridge.py`](../src/nats_bridge.py)
+
+The Python/Micropython implementation provides:
+
+- **`MessageEnvelope` class**: For the unified JSON envelope
+- **`MessagePayload` class**: For individual payload representation
+- **[`smartsend()`](src/nats_bridge.py)**: Handles transport selection based on payload size
+- **[`smartreceive()`](src/nats_bridge.py)**: Handles both direct and link transport

 ## Installation

@@ -117,6 +194,23 @@ Pkg.add("Dates")
 npm install nats.js apache-arrow uuid base64-url
 ```

+### Python/Micropython Dependencies
+
+1. Copy [`src/nats_bridge.py`](../src/nats_bridge.py) to your device
+2. Ensure you have the following dependencies:
+
+   **For Python (desktop):**
+   ```bash
+   pip install nats-py
+   ```
+
+   **For Micropython:**
+   - `urequests` for HTTP requests
+   - `base64` for base64 encoding (built-in)
+   - `json` for JSON handling (built-in)
+   - `socket` for networking (built-in)
+   - `uuid` for UUID generation (built-in)
+
 ## Usage Tutorial

 ### Step 1: Start NATS Server
@@ -155,35 +249,31 @@ node test/scenario3_julia_to_julia.js

 ### Scenario 0: Basic Multi-Payload Example

-#### Julia (Sender)
-```julia
-using NATSBridge
+#### Python/Micropython (Sender)
+```python
+from nats_bridge import smartsend

 # Send multiple payloads in one message (type is required per payload)
 smartsend(
    "/test",
    [("dataname1", data1, "dictionary"), ("dataname2", data2, "table")],
    nats_url="nats://localhost:4222",
-    fileserver_url="http://localhost:8080",
-    metadata=Dict("custom_key" => "custom_value")
+    fileserver_url="http://localhost:8080"
 )

 # Even single payload must be wrapped in a list with type
 smartsend("/test", [("single_data", mydata, "dictionary")])
 ```

-#### Julia (Receiver)
-```julia
-using NATSBridge
+#### Python/Micropython (Receiver)
+```python
+from nats_bridge import smartreceive

-# Receive returns a dictionary envelope with all metadata and deserialized payloads
-envelope = smartreceive(msg, "http://localhost:8080")
-# envelope["payloads"] = [(dataname1, data1, "dictionary"), (dataname2, data2, "table"), ...]
-# envelope["correlationId"], envelope["msgId"], etc.
+# Receive returns a list of (dataname, data, type) tuples
+payloads = smartreceive(msg)
+# payloads = [(dataname1, data1, "dictionary"), (dataname2, data2, "table"), ...]
 ```

-### Scenario 1: Command & Control (Small JSON)
-
 #### JavaScript (Sender)
 ```javascript
 const { smartsend } = require('./src/NATSBridge');
@@ -216,27 +306,7 @@ const configs = [
 await smartsend("control", configs);
 ```

-#### Julia (Receiver)
-```julia
-using NATS
-using JSON3
-
-# Subscribe to control subject
-subscribe(nats, "control") do msg
-    env = MessageEnvelope(String(msg.data))
-    config = JSON3.read(env.payload)
-    
-    # Execute simulation with parameters
-    step_size = config.step_size
-    iterations = config.iterations
-    
-    # Send acknowledgment
-    response = Dict("status" => "Running", "correlation_id" => env.correlation_id)
-    publish(nats, "control_response", JSON3.stringify(response))
-end
-```
-
-### JavaScript (Receiver)
+#### JavaScript (Receiver)
 ```javascript
 const { smartreceive } = require('./src/NATSBridge');

@@ -291,6 +361,26 @@ const table = envelope.payloads;

 ### Scenario 3: Live Binary Processing

+#### Python/Micropython (Sender)
+```python
+from nats_bridge import smartsend
+
+# Binary data wrapped in a list
+binary_data = [
+    ("audio_chunk", binary_buffer, "binary")
+]
+
+smartsend(
+    "binary_input",
+    binary_data,
+    nats_url="nats://localhost:4222",
+    metadata={
+        "sample_rate": 44100,
+        "channels": 1
+    }
+)
+```
+
 #### JavaScript (Sender)
 ```javascript
 const { smartsend } = require('./src/NATSBridge');
@@ -310,23 +400,23 @@ await smartsend("binary_input", binaryData, {
 });
 ```

-#### Julia (Receiver)
-```julia
-using WAV
-using DSP
+#### Python/Micropython (Receiver)
+```python
+from nats_bridge import smartreceive

 # Receive binary data
-function process_binary(data)
-    # Perform FFT or AI transcription
-    spectrum = fft(data)
+def process_binary(msg):
+    envelope = smartreceive(msg)
    
-    # Send results back (JSON + Arrow table)
-    results = Dict("transcription" => "sample text", "spectrum" => spectrum)
-    await SmartSend("binary_output", results, "json")
-end
+    # Process the binary data from envelope.payloads
+    for dataname, data, type in envelope["payloads"]:
+        if type == "binary":
+            # data is bytes
+            print(f"Received binary data: {dataname}, size: {len(data)}")
+            # Perform FFT or AI transcription here
 ```

-### JavaScript (Receiver)
+#### JavaScript (Receiver)
 ```javascript
 const { smartreceive } = require('./src/NATSBridge');

@@ -383,6 +473,64 @@ for await (const msg of consumer) {
 }
 ```

+### Scenario 4: Micropython Device Control
+
+**Focus:** Sending configuration to a Micropython device over NATS. This demonstrates the lightweight nature of the Python implementation suitable for microcontrollers.
+
+**Python/Micropython (Receiver/Device):**
+```python
+from nats_bridge import smartsend, smartreceive
+import json
+
+# Device configuration handler
+def handle_device_config(msg):
+    envelope = smartreceive(msg)
+    
+    # Process configuration from payloads
+    for dataname, data, type in envelope["payloads"]:
+        if type == "dictionary":
+            print(f"Received configuration: {data}")
+            # Apply configuration to device
+            if "wifi_ssid" in data:
+                wifi_ssid = data["wifi_ssid"]
+                wifi_password = data["wifi_password"]
+                update_wifi_config(wifi_ssid, wifi_password)
+    
+    # Send confirmation back
+    config = {
+        "status": "configured",
+        "wifi_ssid": "MyNetwork",
+        "ip": get_device_ip()
+    }
+    smartsend(
+        "device/response",
+        [("config", config, "dictionary")],
+        nats_url="nats://localhost:4222",
+        reply_to=envelope.get("replyTo")
+    )
+```
+
+**JavaScript (Sender/Controller):**
+```javascript
+const { smartsend } = require('./src/NATSBridge');
+
+// Send configuration to Micropython device
+await smartsend("device/config", [
+    {
+        dataname: "config",
+        data: {
+            wifi_ssid: "MyNetwork",
+            wifi_password: "password123",
+            update_interval: 60,
+            temperature_threshold: 30.0
+        },
+        type: "dictionary"
+    }
+]);
+```
+
+**Use Case:** A controller sends WiFi and operational configuration to a Micropython device (e.g., ESP32). The device receives the configuration, applies it, and sends back a confirmation with its current status.
+
 ### Scenario 5: Selection (Low Bandwidth)

 **Focus:** Small Arrow tables, Julia to JavaScript. The Action: Julia wants to send a small DataFrame to show on a JavaScript dashboard for the user to choose.
@@ -581,7 +729,7 @@ await smartsend("chat.room123", message);
 ### Exponential Backoff
 - Maximum retry count: 5
 - Base delay: 100ms, max delay: 5000ms
- Implemented in both Julia and JavaScript implementations
+- Implemented in all three implementations (Julia, JavaScript, Python/Micropython)

 ### Correlation ID Logging
 - Log correlation_id at every stage
@@ -590,14 +738,73 @@ await smartsend("chat.room123", message);

 ## Testing

-Run the test scripts:
+Run the test scripts for each platform:
+
+### Python/Micropython Tests

 ```bash
-# Scenario 1: Command & Control (JavaScript sender)
-node test/scenario1_command_control.js
+# Basic functionality test
+python test/test_micropython_basic.py
+```

-# Scenario 2: Large Arrow Table (JavaScript sender)
-node test/scenario2_large_table.js
+### JavaScript Tests
+
+```bash
+# Text message exchange
+node test/test_js_to_js_text_sender.js
+node test/test_js_to_js_text_receiver.js
+
+# Dictionary exchange
+node test/test_js_to_js_dict_sender.js
+node test/test_js_to_js_dict_receiver.js
+
+# File transfer (direct transport)
+node test/test_js_to_js_file_sender.js
+node test/test_js_to_js_file_receiver.js
+
+# Mixed payload types
+node test/test_js_to_js_mix_payloads_sender.js
+node test/test_js_to_js_mix_payloads_receiver.js
+
+# Table (Arrow IPC) exchange
+node test/test_js_to_js_table_sender.js
+node test/test_js_to_js_table_receiver.js
+```
+
+### Julia Tests
+
+```bash
+# Text message exchange
+julia test/test_julia_to_julia_text_sender.jl
+julia test/test_julia_to_julia_text_receiver.jl
+
+# Dictionary exchange
+julia test/test_julia_to_julia_dict_sender.jl
+julia test/test_julia_to_julia_dict_receiver.jl
+
+# File transfer
+julia test/test_julia_to_julia_file_sender.jl
+julia test/test_julia_to_julia_file_receiver.jl
+
+# Mixed payload types
+julia test/test_julia_to_julia_mix_payloads_sender.jl
+julia test/test_julia_to_julia_mix_payloads_receiver.jl
+
+# Table exchange
+julia test/test_julia_to_julia_table_sender.jl
+julia test/test_julia_to_julia_table_receiver.jl
+```
+
+### Cross-Platform Tests
+
+```bash
+# Julia ↔ JavaScript communication
+julia test/test_julia_to_julia_text_sender.jl
+node test/test_js_to_js_text_receiver.js
+
+# Python ↔ JavaScript communication
+python test/test_micropython_basic.py
+node test/test_js_to_js_text_receiver.js
 ```

 ## Troubleshooting
@@ -605,18 +812,24 @@ node test/scenario2_large_table.js
 ### Common Issues

 1. **NATS Connection Failed**
-   - Ensure NATS server is running
-   - Check NATS_URL configuration
+   - **Julia/JavaScript/Python**: Ensure NATS server is running
+   - **Python/Micropython**: Check `nats_url` parameter and network connectivity

 2. **HTTP Upload Failed**
   - Ensure file server is running
-   - Check FILESERVER_URL configuration
+   - Check `fileserver_url` configuration
   - Verify upload permissions
+   - **Micropython**: Ensure `urequests` is available and network is connected

 3. **Arrow IPC Deserialization Error**
   - Ensure data is properly serialized to Arrow format
   - Check Arrow version compatibility

+4. **Python/Micropython Specific Issues**
+   - **Import Error**: Ensure `nats_bridge.py` is in the correct path
+   - **Memory Error (Micropython)**: Reduce payload size or use link transport for large payloads
+   - **Unicode Error**: Ensure proper encoding when sending text data
+
 ## License

 MIT