ton/NATSBridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity
Files
34d8e3fad82b8180ffec83e1601c8f9640ca5343
NATSBridge/docs/architecture.md

21 KiB
Raw Blame History

Architecture Documentation: NATSBridge

Version: 1.0.0
Date: 2026-03-13
Status: Active
Ground Truth: src/NATSBridge.jl
Architecture Level: C4 Container Level

Executive Summary

This document defines the blueprint for NATSBridge - the cross-platform bi-directional data bridge that enables seamless communication between Julia, JavaScript, Python, and MicroPython applications using NATS as the message bus.

This architecture document serves as the single source of truth for:

  • System Structure: How components fit together and interact
  • Scaling Considerations: How the system scales horizontally and vertically
  • Failure Modes: How the system handles failures and recovers
  • Trade-off Decisions: The rationale behind architectural decisions

Architecture Overview

C4 Context Diagram

flowchart TD
    subgraph "External Systems"
        NATS_Server[NATS Server]
        File_Server[HTTP File Server<br/>Plik/AWS S3/Custom]
    end

    subgraph "Client Applications"
        Julia_App[Julia Application]
        JS_App[JavaScript Application<br/>Node.js/Browser]
        Python_App[Python Application<br/>Desktop]
        MicroPython_App[MicroPython Device]
    end

    Julia_App -->|NATS| NATS_Server
    JS_App -->|NATS| NATS_Server
    Python_App -->|NATS| NATS_Server
    MicroPython_App -->|NATS| NATS_Server

    Julia_App -->|HTTP| File_Server
    JS_App -->|HTTP| File_Server
    Python_App -->|HTTP| File_Server
    MicroPython_App -->|HTTP| File_Server

    style NATS_Server fill:#fff3e0,stroke:#f57c00
    style File_Server fill:#f3e5f5,stroke:#9c27b4
    style Julia_App fill:#e8f5e9,stroke:#4caf50
    style JS_App fill:#e3f2fd,stroke:#2196f3
    style Python_App fill:#e3f2fd,stroke:#2196f3
    style MicroPython_App fill:#fce4ec,stroke:#e91e63

C4 Container Diagram

flowchart TD
    subgraph "Client Container"
        Julia_Module[Julia NATSBridge Module]
        JS_Module[JavaScript NATSBridge Module]
        Python_Module[Python NATSBridge Module]
        MicroPython_Module[MicroPython NATSBridge Module]
    end

    subgraph "NATS Container"
        NATS_Client[NATS Client]
        NATS_Broker[NATS Broker]
    end

    subgraph "File Server Container"
        File_Client[HTTP Client]
        File_Server[File Server]
    end

    Julia_Module --> NATS_Client
    JS_Module --> NATS_Client
    Python_Module --> NATS_Client
    MicroPython_Module --> NATS_Client

    NATS_Client --> NATS_Broker

    Julia_Module --> File_Client
    JS_Module --> File_Client
    Python_Module --> File_Client
    MicroPython_Module --> File_Client

    File_Client --> File_Server

    style Julia_Module fill:#e8f5e9,stroke:#4caf50
    style JS_Module fill:#e3f2fd,stroke:#2196f3
    style Python_Module fill:#e3f2fd,stroke:#2196f3
    style MicroPython_Module fill:#fce4ec,stroke:#e91e63
    style NATS_Broker fill:#fff3e0,stroke:#f57c00
    style File_Server fill:#f3e5f5,stroke:#9c27b4

C4 Component Diagram (Julia Implementation)

flowchart TD
    subgraph "NATSBridge Module"
        SmartSend[smartsend Function]
        SmartReceive[smartreceive Function]
        
        Serialize[_serialize_data]
        Deserialize[_deserialize_data]
        
        BuildEnvelope[build_envelope]
        BuildPayload[build_payload]
        
        PublishMessage[publish_message]
        
        FileServerUpload[fileserver_upload_handler]
        FileServerDownload[fileserver_download_handler]
    end

    subgraph "Data Models"
        Payload[MsgPayloadV1 Struct]
        Envelope[MsgEnvelopeV1 Struct]
    end

    SmartSend --> Serialize
    SmartSend --> BuildEnvelope
    SmartSend --> BuildPayload
    SmartSend --> PublishMessage
    SmartSend --> FileServerUpload

    SmartReceive --> Deserialize
    SmartReceive --> FileServerDownload

    Serialize --> Payload
    BuildEnvelope --> Envelope
    BuildPayload --> Payload

    style SmartSend fill:#d1fae5,stroke:#10b981
    style SmartReceive fill:#d1fae5,stroke:#10b981
    style PublishMessage fill:#fef3c7,stroke:#f59e0b
    style FileServerUpload fill:#fef3c7,stroke:#f59e0b
    style FileServerDownload fill:#fef3c7,stroke:#f59e0b

High-Level Architecture

System Components

Component Purpose Platform Support
smartsend Send data via NATS with automatic transport selection All
smartreceive Receive and process NATS messages All
_serialize_data Serialize data according to payload type All
_deserialize_data Deserialize bytes to native data types All
_build_envelope Build message envelope from payloads All
_build_payload Build payload object from serialized data All
publish_message Publish message to NATS subject All
fileserver_upload_handler Upload large payloads to HTTP server Desktop
fileserver_download_handler Download payloads from HTTP server Desktop

Data Flow

flowchart TD
    A[User calls smartsend subject data] --> B[Process each payload]
    B --> C{Calculate serialized size}
    C -->|Size < Threshold| D[Direct Transport]
    C -->|Size >= Threshold| E[Link Transport]
    
    D --> F[Serialize data]
    F --> G[Base64 encode]
    G --> H[Build payload object]
    
    E --> I[Serialize data]
    I --> J[Upload to file server]
    J --> K[Get download URL]
    K --> H
    
    H --> L[Build envelope]
    L --> M[Convert to JSON]
    M --> N[Publish to NATS]
    
    style A fill:#f9f9f9,stroke:#333
    style N fill:#e0e7ff,stroke:#3b82f6
    style D fill:#d1fae5,stroke:#10b981
    style E fill:#fef3c7,stroke:#f59e0b

Message Envelope Architecture

msg_envelope_v1 Structure (Julia)

struct msg_envelope_v1
  correlation_id::String       # UUID v4 for distributed tracing
  msg_id::String               # UUID v4 for this message
  timestamp::String            # ISO 8601 UTC timestamp
  
  send_to::String              # NATS subject to publish to
  msg_purpose::String          # ACK, NACK, updateStatus, shutdown, chat
  sender_name::String          # Sender application name
  sender_id::String            # UUID v4 of sender
  receiver_name::String        # Receiver application name (empty = broadcast)
  receiver_id::String          # UUID v4 of receiver (empty = broadcast)
  
  reply_to::String             # Topic for reply messages
  reply_to_msg_id::String      # Message ID being replied to
  broker_url::String           # NATS broker URL
  
  metadata::Dict{String, Any}  # Message-level metadata
  payloads::Vector{msg_payload_v1}  # List of payloads
end

msg_payload_v1 Structure (Julia)

struct msg_payload_v1
  id::String                    # UUID v4 for this payload
  dataname::String              # Name of the payload
  payload_type::String          # text, dictionary, arrowtable, etc.
  transport::String             # direct or link
  encoding::String              # none, json, base64, arrow-ipc
  size::Integer                 # Size in bytes
  data::Any                     # Base64 string or URL
  metadata::Dict{String, Any}   # Payload-level metadata
end

JSON Schema (Cross-Platform)

{
  "correlation_id": "string (UUID v4)",
  "msg_id": "string (UUID v4)",
  "timestamp": "string (ISO 8601 UTC)",
  "send_to": "string",
  "msg_purpose": "string",
  "sender_name": "string",
  "sender_id": "string (UUID v4)",
  "receiver_name": "string",
  "receiver_id": "string (UUID v4)",
  "reply_to": "string",
  "reply_to_msg_id": "string",
  "broker_url": "string",
  "metadata": "object",
  "payloads": [
    {
      "id": "string (UUID v4)",
      "dataname": "string",
      "payload_type": "string",
      "transport": "string",
      "encoding": "string",
      "size": "integer",
      "data": "string or URL",
      "metadata": "object"
    }
  ]
}

Payload Type Architecture

Supported Payload Types

Type Description Serialization Encoding Platforms
text Plain text string UTF-8 bytes Base64 All
dictionary JSON object JSON string Base64/JSON All
arrowtable Apache Arrow IPC Arrow IPC stream Base64/arrow-ipc Desktop (Julia/Python/Node.js)
jsontable JSON array of objects JSON string Base64/json All (including Browser)
image Binary image data Raw bytes Base64 All
audio Binary audio data Raw bytes Base64 All
video Binary video data Raw bytes Base64 All
binary Generic binary data Raw bytes Base64 All

Serialization Logic

flowchart TD
    A[Input data + payload_type] --> B{Payload Type}
    
    B -->|"text"| C[UTF-8 encode]
    B -->|"dictionary"| D[JSON serialize]
    B -->|"arrowtable"| E[Arrow IPC serialize]
    B -->|"jsontable"| F[JSON serialize]
    B -->|"image"| G[Raw bytes]
    B -->|"audio"| H[Raw bytes]
    B -->|"video"| I[Raw bytes]
    B -->|"binary"| J[Raw bytes]
    
    C --> K[Return bytes]
    D --> K
    E --> K
    F --> K
    G --> K
    H --> K
    I --> K
    J --> K
    
    style A fill:#f9f9f9,stroke:#333
    style K fill:#e0e7ff,stroke:#3b82f6

Transport Strategy Architecture

Size Threshold Decision Logic

Platform Size Threshold Notes
Desktop (Julia/JS/Python) 500,000 bytes (0.5MB) Default threshold
MicroPython 100,000 bytes (100KB) Lower threshold for memory constraints

Transport Selection Flow

flowchart TD
    A[smartsend called] --> B[Serialize payload]
    B --> C[Calculate size]
    C --> D{Size < Threshold?}
    
    D -->|Yes| E[Direct Transport]
    D -->|No| F[Link Transport]
    
    E --> G[Base64 encode]
    G --> H[Build payload with direct transport]
    
    F --> I[Upload to file server]
    I --> J[Get download URL]
    J --> K[Build payload with link transport]
    
    H --> L[Build envelope]
    K --> L
    
    style A fill:#f9f9f9,stroke:#333
    style L fill:#e0e7ff,stroke:#3b82f6
    style E fill:#d1fae5,stroke:#10b981
    style F fill:#fef3c7,stroke:#f59e0b

Direct Transport Protocol

When transport = "direct", the data field contains a Base64-encoded string of the serialized payload.

Encoding Rules:

  • text: UTF-8 → Base64
  • dictionary: JSON → Base64 (or direct JSON)
  • arrowtable: Arrow IPC → Base64 (or arrow-ipc)
  • jsontable: JSON → Base64 (or direct JSON)
  • image/audio/video/binary: Raw bytes → Base64

Link Transport Protocol

When transport = "link", the data field contains a URL pointing to the uploaded payload.

Upload Flow:

  1. Serialize payload according to payload_type
  2. Upload to HTTP file server (e.g., Plik)
  3. Include returned URL in data field

Download Flow:

  1. Extract URL from payload
  2. Fetch with exponential backoff (max 5 retries)
  3. Deserialize based on payload_type

Platform-Specific Architecture

Julia Architecture

Julia leverages multiple dispatch for type-specific implementations:

  • Multiple Dispatch: Function overloading based on argument types
  • Struct-based Data Models: Explicit type definitions with struct
  • Native Arrow IPC: Support via Arrow.jl
  • Async/Await: Tasks for non-blocking I/O
# Multiple dispatch for serialization
function _serialize_data(data::String, payload_type::String)
    # Text serialization
end

function _serialize_data(data::Dict, payload_type::String)
    # Dictionary serialization
end

function _serialize_data(data::DataFrame, payload_type::String)
    # Arrow table serialization
end

JavaScript Architecture

JavaScript uses async/await for non-blocking I/O:

  • Class-based NATS Client: Connection management
  • Module-level Utilities: Serialization functions
  • Native ArrayBuffer: Binary data handling
  • Fetch API: HTTP file server communication
// Class-based NATS client
class NATSClient {
    constructor(url) {
        this.url = url;
        this.connection = null;
    }
    
    async connect() {
        this.connection = await nats.connect({ servers: this.url });
    }
}

Python Architecture

Python uses classes for stateful operations:

  • Class-based NATSBridge: Encapsulated API
  • Dataclasses: Structured data (MsgPayloadV1, MsgEnvelopeV1)
  • Async/await: I/O operations
  • pyarrow: Arrow IPC support
class NATSBridge:
    DEFAULT_SIZE_THRESHOLD = 500_000
    
    def __init__(self, broker_url=None, fileserver_url=None):
        self.broker_url = broker_url or self.DEFAULT_BROKER_URL
        self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL

Browser Architecture

Browser JavaScript has specific constraints due to security and compatibility:

  • Async/await: Native async/await support
  • No Apache Arrow: Arrow IPC not available in browsers
  • JSON table only: Use "jsontable" for tabular data
  • WebSocket NATS: Uses nats.ws for browser-compatible NATS connections
  • Fetch API: HTTP file server communication via fetch

MicroPython Architecture

MicroPython has significant constraints:

  • Synchronous API: No async/await
  • Memory-constrained: 256KB - 1MB
  • Limited payload support: No tables, max 50KB
  • Simplified UUID generation: Custom implementation
# MicroPython constraints
DEFAULT_SIZE_THRESHOLD = 100_000  # 100KB
MAX_PAYLOAD_SIZE = 50_000  # 50KB hard limit

Scaling Architecture

Horizontal Scaling

Component Scaling Strategy
NATS Server Cluster deployment with multiple nodes
File Server Load balancer + multiple instances
Client Applications Deploy multiple instances behind load balancer

Vertical Scaling

Component Scaling Strategy
NATS Server Increase memory, CPU, disk I/O
File Server Increase memory, CPU, disk capacity
Client Applications Increase heap size (Python/JS)

Performance Considerations

Metric Target Notes
Message serialization overhead <50ms For 10KB payload
Message deserialization overhead <50ms For 10KB payload
NATS connection establishment <100ms Connection pool recommended
File upload latency <1s For 0.5MB file
File download latency <1s For 0.5MB file

Failure Modes and Recovery

NATS Connection Failure

Scenario: NATS server unavailable

Handler:

  • Connection auto-reconnect via TCP-level reconnection
  • Retry with exponential backoff for publish operations

Recovery:

  • NATS client automatically attempts reconnection
  • Application can check connection status before publishing

File Server Unavailable

Scenario: HTTP file server unavailable during upload/download

Handler:

  • Retry up to 5 times with exponential backoff (100ms → 5000ms)
  • Fallback to direct transport for upload (MicroPython)

Recovery:

  • Exponential backoff: delay = min(delay * 2, max_delay)
  • After max retries, throw error with correlation ID

Deserialization Error

Scenario: Payload type mismatch or corrupted data

Handler:

  • Log correlation ID and throw error
  • No retry (data corruption)

Recovery:

  • Application must validate payload_type matches data type
  • Use proper serialization before sending

Memory Overflow (MicroPython)

Scenario: Payload exceeds maximum size (50KB)

Handler:

  • Reject payloads >50KB with MemoryError
  • No retry (client-side check)

Recovery:

  • Application must split large payloads
  • Use direct transport only for small payloads

Trade-off Decisions

Decision 1: Direct vs Link Transport Threshold

Trade-off: Memory vs Network I/O

Decision: Use 0.5MB threshold for desktop, 100KB for MicroPython

Rationale:

  • Direct transport uses more memory (Base64 encoding adds ~33% overhead)
  • Link transport requires network I/O for upload/download
  • 0.5MB is reasonable for desktop memory constraints
  • 100KB is necessary for MicroPython memory constraints

Decision 2: Base64 Encoding for Direct Transport

Trade-off: Bandwidth vs Simplicity

Decision: Use Base64 encoding for all direct transport payloads

Rationale:

  • Simplifies JSON serialization (all data is string-compatible)
  • Increases payload size by ~33%, but NATS can handle this
  • Alternative would be binary payload support (more complex)

Decision 3: Multiple Platform Implementations

Trade-off: Development effort vs Cross-platform support

Decision: Maintain separate implementations for each platform

Rationale:

  • Each platform has idiomatic patterns (multiple dispatch, async/await, etc.)
  • Maintains developer productivity and code quality
  • API parity ensures cross-platform compatibility

Decision 4: Handler Function Abstraction

Trade-off: Flexibility vs Simplicity

Decision: Abstract file server operations through handler functions

Rationale:

  • Allows support for different file server implementations (Plik, AWS S3, custom)
  • Maintains simplicity for common use cases
  • Enables plug-in architecture for custom backends

Deployment Architecture

Minimum Infrastructure

Component Minimum Notes
NATS Server 1 instance Single node for development
File Server 1 instance HTTP server for large payloads
Client Memory 50MB Desktop platforms
Client Memory 256KB MicroPython devices

Environment Variables

Variable Default Description
NATS_URL nats://localhost:4222 NATS server URL
FILESERVER_URL http://localhost:8080 HTTP file server URL
SIZE_THRESHOLD 1000000 Size threshold in bytes

Container Deployment

flowchart TD
    subgraph "Docker Network"
        NATS_Container[NATS Server]
        FileServer_Container[Plik File Server]
        App_Container[Application Container]
    end
    
    App_Container -->|NATS| NATS_Container
    App_Container -->|HTTP| FileServer_Container
    
    style NATS_Container fill:#fff3e0,stroke:#f57c00
    style FileServer_Container fill:#f3e5f5,stroke:#9c27b4
    style App_Container fill:#e3f2fd,stroke:#2196f3

Security Considerations

Payload Integrity

Mechanism: SHA-256 checksum via metadata

Implementation:

  • Sender calculates checksum and stores in payload metadata
  • Receiver validates checksum on receipt

Transport Security

Mechanism: TLS support for NATS connections

Implementation:

  • Use nats:// URL for plain text
  • Use tls:// URL for TLS-encrypted connections

File Server Security

Mechanism: Authentication token for file uploads

Implementation:

  • Plik uses upload token in X-UploadToken header
  • Application can implement custom authentication

Testing Architecture

Unit Test Coverage

Test Category Coverage Files
Serialization All payload types test/test_*_sender.*
Deserialization All payload types test/test_*_receiver.*
Transport selection Direct vs link test/test_*_mix_payloads.*
File server upload Plik integration Platform-specific
File server download Exponential backoff Platform-specific

Integration Test Scenarios

Scenario Platforms Payloads Transport Expected Result
Cross-platform text Julia ↔ JS ↔ Python text direct Round-trip successful
Arrow IPC round-trip Julia ↔ JS ↔ Python arrowtable direct Arrow IPC preserved
Large file transfer All image/audio/video link File server upload/download
Multi-payload mixed All text + image + file direct/link All payloads preserved

Versioning

Architecture Versioning

Component Version Notes
Architecture 1.0.0 Initial release
Protocol v1 Message envelope protocol version

Backward Compatibility

Version Supported Platforms
v1.0.x Julia 1.7+, Node.js 16+, Python 3.8+, MicroPython 1.19+

Change Log

Date Version Changes
2026-03-13 1.0.0 Initial architecture documentation

References

This architecture document is versioned and maintained in git alongside the codebase. All implementations must adhere to this architecture.

Reference in New Issue View Git Blame Copy Permalink