Files

2026-03-09 18:16:33 +07:00

17 KiB

Raw Permalink Blame History

Cross-Platform Architecture Documentation: Bi-Directional Data Bridge

Overview

This document describes the architecture for a high-performance, bi-directional data bridge using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads. The system is implemented across three platforms with high-level API parity while maintaining idiomatic implementations for each language.

Supported Platforms:

Julia - Ground truth implementation with full feature set
JavaScript - Node.js and browser-compatible implementation
Python/MicroPython - Desktop and embedded-compatible implementation

Cross-Platform Design Principles

High-Level API Parity: All three platforms expose the same smartsend() and smartreceive() functions with identical signatures and behavior
Idiomatic Implementations: Each platform uses its native patterns (multiple dispatch in Julia, async/prototype in JS, class-based in Python)
Message Format Consistency: The msg_envelope_v1 and msg_payload_v1 JSON schemas are identical across all platforms
Handler Function Abstraction: File server operations are abstracted through handler functions for backend flexibility

High-Level API Standard (Cross-Platform)

Unified API Signature

All three platforms expose the same high-level API:

Input Format (smartsend):

[(dataname1, data1, type1), (dataname2, data2, type2), ...]

Output Format (smartreceive):

{
  "correlation_id": "...",
  "msg_id": "...",
  "timestamp": "...",
  "send_to": "...",
  "msg_purpose": "...",
  "sender_name": "...",
  "sender_id": "...",
  "receiver_name": "...",
  "receiver_id": "...",
  "reply_to": "...",
  "reply_to_msg_id": "...",
  "broker_url": "...",
  "metadata": {...},
  "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
}

Supported Payload Types

Type	Julia	JavaScript	Python/MicroPython
`text`	`String`	`string`	`str`
`dictionary`	`Dict`, `NamedTuple`	`Object`, `Array`	`dict`, `list`
`arrowtable`	`DataFrame`, `Arrow.Table`	`Array<Object>` (input) → `Buffer` (Arrow IPC)	`pandas.DataFrame`, `bytes` (Arrow IPC)
`jsontable`	`Vector{NamedTuple}`, `Vector{Dict}`	`Array<Object>`	`list[dict]`, `list`
`table`	❌	❌	`pandas.DataFrame`, `bytes` (Arrow IPC)
`image`	`Vector{UInt8}`	`Uint8Array`, `Buffer`	`bytes`, `bytearray`
`audio`	`Vector{UInt8}`	`Uint8Array`, `Buffer`	`bytes`, `bytearray`
`video`	`Vector{UInt8}`	`Uint8Array`, `Buffer`	`bytes`, `bytearray`
`binary`	`Vector{UInt8}`, `IOBuffer`	`Uint8Array`, `Buffer`	`bytes`, `bytearray`, `io.BytesIO`

Note on MicroPython: MicroPython does not support table types (arrowtable or jsontable) due to memory constraints. Use dictionary or binary instead.

Cross-Platform API Examples

Julia:

using NATSBridge

# Send
env, env_json_str = smartsend(
    "/chat",
    [("message", "Hello!", "text"), ("image", image_bytes, "image")],
    broker_url="nats://localhost:4222"
)

# Receive - returns JSON.Object{String, Any}
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff)
# env is a JSON.Object{String, Any} with "payloads" field containing Vector{Tuple{String, Any, String}}
# Access payloads: for (dataname, data, type) in env["payloads]

JavaScript:

const NATSBridge = require('natsbridge');

// Send
const [env, env_json_str] = await NATSBridge.smartsend(
    "/chat",
    [
        ["message", "Hello!", "text"],
        ["image", imageBuffer, "image"]
    ],
    { broker_url: "nats://localhost:4222" }
);

// Receive - returns Promise<object>
const env = await NATSBridge.smartreceive(msg, {
    fileserver_download_handler: fetchWithBackoff
});
// env is an object with "payloads" field containing Array of arrays
// Access payloads: for (const [dataname, data, type] of env.payloads)

Python:

from natsbridge import NATSBridge

# Send
env, env_json_str = NATSBridge.smartsend(
    "/chat",
    [("message", "Hello!", "text"), ("image", image_bytes, "image")],
    broker_url="nats://localhost:4222"
)

# Receive - returns Tuple[Dict, str]
env = NATSBridge.smartreceive(
    msg,
    fileserver_download_handler=fetch_with_backoff
)
# env is a Dict with "payloads" key containing List[Tuple[str, Any, str]]
# Access payloads: for dataname, data, type_ in env["payloads"]

MicroPython:

from natsbridge import NATSBridge

# Send (limited to direct transport due to memory constraints)
env, env_json_str = NATSBridge.smartsend(
    "/chat",
    [("message", "Hello!", "text")],
    broker_url="nats://localhost:4222"
)

Architecture Diagram (Cross-Platform)

flowchart TD
    subgraph Client
        App[Julia/JS/Python/MicroPython Application]
    end

    subgraph Server
        Julia/JS/Python/MicroPython[Julia/JS/Python/MicroPython Service]
        NATS[NATS Server]
        FileServer[HTTP File Server]
    end

    App -->|NATS| NATS
    NATS -->|NATS| Julia/JS/Python/MicroPython
    Julia/JS/Python/MicroPython -->|NATS| NATS
    Julia/JS/Python/MicroPython -->|HTTP POST| FileServer

    style App fill:#e8f5e9
    style Julia/JS/Python/MicroPython fill:#e8f5e9
    style NATS fill:#fff3e0
    style FileServer fill:#f3e5f5

System Components

1. msg_envelope_v1 - Message Envelope

JSON Schema (Identical Across All Platforms):

{
  "correlation_id": "uuid-v4-string",
  "msg_id": "uuid-v4-string",
  "timestamp": "2024-01-15T10:30:00Z",
  
  "send_to": "topic/subject",
  "msg_purpose": "ACK | NACK | updateStatus | shutdown | chat",
  "sender_name": "agent-wine-web-frontend",
  "sender_id": "uuid4",
  "receiver_name": "agent-backend",
  "receiver_id": "uuid4",
  "reply_to": "topic",
  "reply_to_msg_id": "uuid4",
  "broker_url": "nats://localhost:4222",
  
  "metadata": {
    "content_type": "application/octet-stream",
    "content_length": 123456
  },
  
  "payloads": [
    {
      "id": "uuid4",
      "dataname": "login_image",
      "payload_type": "image",
      "transport": "direct",
      "encoding": "base64",
      "size": 15433,
      "data": "base64-encoded-string",
      "metadata": {
        "checksum": "sha256_hash"
      }
    },
    {
      "id": "uuid4",
      "dataname": "large_arrow_table",
      "payload_type": "arrowtable",
      "transport": "link",
      "encoding": "arrow-ipc",
      "size": 524288,
      "data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/data.arrow",
      "metadata": {}
    }
  ]
}

2. msg_payload_v1 - Payload Structure

JSON Schema (Identical Across All Platforms):

{
  "id": "uuid4",
  "dataname": "login_image",
  "payload_type": "image | dictionary | arrowtable | jsontable | table | text | audio | video | binary",
  "transport": "direct | link",
  "encoding": "none | json | base64 | arrow-ipc",
  "size": 15433,
  "data": "base64-encoded-string | http-url | json-string",
  "metadata": {
    "checksum": "sha256_hash"
  }
}

3. Transport Strategy Decision Logic (Cross-Platform)

┌─────────────────────────────────────────────────────────────┐
│              smartsend Function (All Platforms)             │
│  Accepts: [(dataname1, data1, type1), ...]                  │
│  (Type is per payload, not standalone)                      │
└─────────────────────────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│  For each payload:                                          │
│  1. Extract type from tuple/array                           │
│  2. Serialize based on type                                 │
│  3. Check payload size                                      │
└─────────────────────────────────────────────────────────────┘
                         │
              ┌───────────┴────────────┐
              ▼                        ▼
        ┌──────────────┐         ┌──────────────┐
        │  Direct Path │         │  Link Path   │
        │  (< 1MB)     │         │  (>= 1MB)    │
        │              │         │              │
        │ • Serialize  │         │ • Serialize  │
        │   to buffer  │         │   to buffer  │
        │ • Base64/JSON│         │ • Upload to  │
        │   encode     │         │   HTTP Server│
        │ • Publish to │         │ • Publish to │
        │   NATS       │         │   NATS with  │
        │   (in msg)   │         │   URL        │
        └──────────────┘         └──────────────┘

Platform Comparison Matrix

Feature	Julia	JavaScript	Python	MicroPython
Multiple Dispatch	✅ Native	❌ (Prototypes)	❌ (Overload via `@overload`)	❌
Async/Await	❌ (Tasks)	✅ Native	✅ Native	⚠️ (uasyncio)
Type Safety	✅ Strong	⚠️ (TypeScript)	✅ (Type hints)	❌
Memory Management	✅ GC	✅ GC	✅ GC	⚠️ (Manual)
Arrow IPC	✅ Native	✅ (arrow package)	✅ (pyarrow)	❌
JSON Serialization	✅ (JSON.jl)	✅ (native)	✅ (json)	✅ (json)
arrowtable Support	✅	✅	✅	❌
jsontable Support	✅	✅	✅	❌
Direct Transport	✅	✅	✅	✅
Link Transport	✅	✅	✅	⚠️ (Limited)
Handler Functions	✅	✅	✅	✅
Cross-Platform API	✅	✅	✅	✅

Platform-Specific Architecture Patterns

Julia: Multiple Dispatch Pattern

Julia leverages multiple dispatch for type-specific implementations:

Function overloading based on argument types
Struct-based data models with explicit types
Native Arrow IPC support via Arrow.jl

JavaScript: Prototype + Async Pattern

JavaScript uses async/await for non-blocking I/O:

Class-based NATS client for connection management
Module-level utility functions for serialization
Native ArrayBuffer for binary data handling

Python: Class-Based Pattern

Python uses classes for stateful operations:

Class-based NATSBridge with type hints
Dataclasses for structured data (MsgPayloadV1, MsgEnvelopeV1)
Async/await for I/O operations

MicroPython: Synchronous Pattern

MicroPython has significant constraints:

Synchronous API (no async/await)
Memory-constrained (256KB - 1MB)
Limited payload support (no tables, max 50KB)

Cross-Platform Compatibility Notes

1. Payload Type Consistency

All platforms use the same payload type values for tabular data:

Platform	Table Types
Julia	`"arrowtable"`, `"jsontable"`
JavaScript	`"arrowtable"`, `"jsontable"`
Python	`"arrowtable"`, `"jsontable"`
MicroPython	Not supported

2. Direct Transport Encoding Field

The encoding field in direct transport payloads differs between platforms:

Platform	Encoding for Direct Transport
Julia	Preserves original type: `"base64"`, `"json"`, or `"arrow-ipc"`
JavaScript	Preserves original type: `"base64"`, `"json"`, or `"arrow-ipc"`
Python	Always `"base64"` for all direct transport payloads
MicroPython	Always `"base64"` for all direct transport payloads

Impact: The encoding field may not accurately reflect the original serialization format when using Python or MicroPython.

3. MicroPython Limitations

MicroPython has significant constraints that affect feature support:

Feature	Desktop Platforms	MicroPython
`arrowtable`	✅	❌ (not supported - memory constraints)
`jsontable`	✅	❌ (not supported - memory constraints)
`table`	✅	❌ (not supported - memory constraints)
Async/await	✅	❌ (synchronous only)
File upload/download	✅	⚠️ (placeholder implementations)
MAX_PAYLOAD_SIZE	1MB+	50KB (hard limit)
DEFAULT_SIZE_THRESHOLD	1MB	100KB

Impact: MicroPython should only be used for small payloads with direct transport. File server operations are not fully implemented.

Configuration

Environment Variables

Variable	Default	Description
`NATS_URL`	`nats://localhost:4222`	NATS server URL
`FILESERVER_URL`	`http://localhost:8080`	HTTP file server URL
`SIZE_THRESHOLD`	`1000000`	Size threshold in bytes (1MB)

MicroPython-Specific Configuration

# micropython.conf
NATS_URL = "nats://broker.local:4222"
FILESERVER_URL = "http://fileserver.local:8080"
SIZE_THRESHOLD = 100000  # Lower threshold for memory-constrained devices
MAX_PAYLOAD_SIZE = 50000  # Hard limit for MicroPython

Performance Considerations

Zero-Copy Reading

Platform	Strategy
Julia	`Arrow.read()` with memory-mapped files
JavaScript	`ArrayBuffer` with `DataView`
Python	`pyarrow` memory mapping
MicroPython	Not available (streaming only)

Exponential Backoff

All platforms implement exponential backoff for HTTP downloads:

delay = base_delay
for attempt in 1:max_retries:
    try:
        response = fetch(url)
        if success: return response
    except:
        if attempt < max_retries:
            sleep(delay)
            delay = min(delay * 2, max_delay)

Correlation ID Logging

All platforms use correlation IDs for distributed tracing:

[timestamp] [Correlation: abc123] Message published to subject

Serialization Performance Comparison

Format	Use Case	Pros	Cons
`arrowtable`	Large tabular data	Fast, zero-copy, schema-preserving	Binary format, requires Arrow library
`jsontable`	Small/medium tabular data	Human-readable, universal support	Slower, larger size, no schema
`table` (Python)	Large tabular data	Fast, zero-copy, schema-preserving	Python-specific, requires pyarrow

Summary

This cross-platform NATS bridge provides:

High-Level API Parity: Identical smartsend() and smartreceive() signatures across Julia, JavaScript, and Python/MicroPython
Idiomatic Implementations:
- Julia: Multiple dispatch and struct-based design
- JavaScript: Async/await and prototype-based utilities
- Python: Class-based design with type hints
- MicroPython: Synchronous API with memory constraints
Message Format Consistency: Identical msg_envelope_v1 and msg_payload_v1 JSON schemas
Handler Abstraction: File server operations abstracted through configurable handlers
Platform-Specific Optimizations:
- Arrow IPC (arrowtable): Efficient binary format for large tabular data
- JSON (jsontable): Universal human-readable format for smaller tables
- Python table: Unified table type for Python-specific implementations
- Streaming support in MicroPython

The Julia implementation serves as the ground truth for API design and behavior, while JavaScript and Python implementations maintain interface parity while leveraging their respective language idioms.

Datatype Summary

Datatype	Serialization	Use Case	Encoding	Supported Platforms
`text`	UTF-8 bytes	Text messages, chat content	`utf-8` → `base64`	All
`dictionary`	JSON	Structured key-value data, config	`json` → `base64`	All
`arrowtable`	Apache Arrow IPC	Large tabular data, schema-preserving	`arrow-ipc` → `base64`	Julia, JavaScript, Python
`jsontable`	JSON	Small/medium tabular data, human-readable	`json` → `base64`	Julia, JavaScript, Python
`table`	Apache Arrow IPC	Python's unified table type	`arrow-ipc` → `base64`	Python
`image`	Binary	Image files (JPEG, PNG, etc.)	`binary` → `base64`	All
`audio`	Binary	Audio files (WAV, MP3, etc.)	`binary` → `base64`	All
`video`	Binary	Video files (MP4, AVI, etc.)	`binary` → `base64`	All
`binary`	Binary	Generic binary data, files	`binary` → `base64`	All

17 KiB Raw Permalink Blame History