# Cross-Platform Architecture Documentation: Bi-Directional Data Bridge

## Overview

This document describes the architecture for a high-performance, bi-directional data bridge using **NATS (Core & JetStream)**, implementing the Claim-Check pattern for large payloads. The system is implemented across three platforms with **high-level API parity** while maintaining **idiomatic implementations** for each language.

**Supported Platforms:**
- **Julia** - Ground truth implementation with full feature set
- **JavaScript** - Node.js and browser-compatible implementation
- **Python/MicroPython** - Desktop and embedded-compatible implementation

### Cross-Platform Design Principles

1. **High-Level API Parity**: All three platforms expose the same `smartsend()` and `smartreceive()` functions with identical signatures and behavior
2. **Idiomatic Implementations**: Each platform uses its native patterns (multiple dispatch in Julia, async/prototype in JS, class-based in Python)
3. **Message Format Consistency**: The `msg_envelope_v1` and `msg_payload_v1` JSON schemas are identical across all platforms
4. **Handler Function Abstraction**: File server operations are abstracted through handler functions for backend flexibility

---

## High-Level API Standard (Cross-Platform)

### Unified API Signature

All three platforms expose the same high-level API:

**Input Format (smartsend):**
```
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
```

**Output Format (smartreceive):**
```
{
  "correlation_id": "...",
  "msg_id": "...",
  "timestamp": "...",
  "send_to": "...",
  "msg_purpose": "...",
  "sender_name": "...",
  "sender_id": "...",
  "receiver_name": "...",
  "receiver_id": "...",
  "reply_to": "...",
  "reply_to_msg_id": "...",
  "broker_url": "...",
  "metadata": {...},
  "payloads": [(dataname1, data1, type1), (dataname2, data2, type2), ...]
}
```

### Supported Payload Types

| Type | Julia | JavaScript | Python/MicroPython |
|------|-------|------------|-------------------|
| `text` | `String` | `string` | `str` |
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` |
| `arrowtable` | `DataFrame`, `Arrow.Table` | `Array<Object>` (input) → `Buffer` (Arrow IPC) | `pandas.DataFrame`, `bytes` (Arrow IPC) |
| `jsontable` | `Vector{NamedTuple}`, `Vector{Dict}` | `Array<Object>` | `list[dict]`, `list` |
| `table` | ❌ | ❌ | `pandas.DataFrame`, `bytes` (Arrow IPC) |
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` |
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` |
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` |
| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray`, `io.BytesIO` |

**Note on MicroPython:** MicroPython does not support table types (`arrowtable` or `jsontable`) due to memory constraints. Use `dictionary` or `binary` instead.

### Cross-Platform API Examples

**Julia:**
```julia
using NATSBridge

# Send
env, env_json_str = smartsend(
    "/chat",
    [("message", "Hello!", "text"), ("image", image_bytes, "image")],
    broker_url="nats://localhost:4222"
)

# Receive - returns JSON.Object{String, Any}
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff)
# env is a JSON.Object{String, Any} with "payloads" field containing Vector{Tuple{String, Any, String}}
# Access payloads: for (dataname, data, type) in env["payloads]
```

**JavaScript:**
```javascript
const NATSBridge = require('natsbridge');

// Send
const [env, env_json_str] = await NATSBridge.smartsend(
    "/chat",
    [
        ["message", "Hello!", "text"],
        ["image", imageBuffer, "image"]
    ],
    { broker_url: "nats://localhost:4222" }
);

// Receive - returns Promise<object>
const env = await NATSBridge.smartreceive(msg, {
    fileserver_download_handler: fetchWithBackoff
});
// env is an object with "payloads" field containing Array of arrays
// Access payloads: for (const [dataname, data, type] of env.payloads)
```

**Python:**
```python
from natsbridge import NATSBridge

# Send
env, env_json_str = NATSBridge.smartsend(
    "/chat",
    [("message", "Hello!", "text"), ("image", image_bytes, "image")],
    broker_url="nats://localhost:4222"
)

# Receive - returns Tuple[Dict, str]
env = NATSBridge.smartreceive(
    msg,
    fileserver_download_handler=fetch_with_backoff
)
# env is a Dict with "payloads" key containing List[Tuple[str, Any, str]]
# Access payloads: for dataname, data, type_ in env["payloads"]
```

**MicroPython:**
```python
from natsbridge import NATSBridge

# Send (limited to direct transport due to memory constraints)
env, env_json_str = NATSBridge.smartsend(
    "/chat",
    [("message", "Hello!", "text")],
    broker_url="nats://localhost:4222"
)
```

---

## Architecture Diagram (Cross-Platform)

```mermaid
flowchart TB
    subgraph JuliaApp["Julia Application"]
        JuliaAppCode[App Code]
        JuliaBridge[NATSBridge.jl]
        JuliaNATS[<b>NATS.jl</b>]
    end

    subgraph JSApp["JavaScript Application"]
        JSAppCode[App Code]
        JSBridge[NATSBridge.js]
        JSNATS[<b>nats.js</b>]
    end

    subgraph PythonApp["Python/MicroPython Application"]
        PythonAppCode[App Code]
        PythonBridge[NATSBridge.py]
        PythonNATS[<b>nats.py</b>]
    end

    subgraph Infrastructure["Infrastructure"]
        NATS[<b>NATS Server</b><br/>Message Broker]
        FileServer[<b>HTTP File Server</b><br/>Upload/Download]
    end

    JuliaAppCode --> JuliaBridge
    JuliaBridge --> JuliaNATS
    JSAppCode --> JSBridge
    JSBridge --> JSNATS
    PythonAppCode --> PythonBridge
    PythonBridge --> PythonNATS

    JuliaNATS --> NATS
    JSNATS --> NATS
    PythonNATS --> NATS

    NATS --> JuliaNATS
    NATS --> JSNATS
    NATS --> PythonNATS

    JuliaBridge -.->|HTTP POST upload| FileServer
    JSBridge -.->|HTTP POST upload| FileServer
    PythonBridge -.->|HTTP POST upload| FileServer

    FileServer -.->|HTTP GET download| JuliaBridge
    FileServer -.->|HTTP GET download| JSBridge
    FileServer -.->|HTTP GET download| PythonBridge

    style JuliaApp fill:#c5e1a5
    style JSApp fill:#bbdefb
    style PythonApp fill:#f8bbd0
    style NATS fill:#fff3e0
    style FileServer fill:#f3e5f5
```

---

## System Components

### 1. msg_envelope_v1 - Message Envelope

**JSON Schema (Identical Across All Platforms):**
```json
{
  "correlation_id": "uuid-v4-string",
  "msg_id": "uuid-v4-string",
  "timestamp": "2024-01-15T10:30:00Z",
  
  "send_to": "topic/subject",
  "msg_purpose": "ACK | NACK | updateStatus | shutdown | chat",
  "sender_name": "agent-wine-web-frontend",
  "sender_id": "uuid4",
  "receiver_name": "agent-backend",
  "receiver_id": "uuid4",
  "reply_to": "topic",
  "reply_to_msg_id": "uuid4",
  "broker_url": "nats://localhost:4222",
  
  "metadata": {
    "content_type": "application/octet-stream",
    "content_length": 123456
  },
  
  "payloads": [
    {
      "id": "uuid4",
      "dataname": "login_image",
      "payload_type": "image",
      "transport": "direct",
      "encoding": "base64",
      "size": 15433,
      "data": "base64-encoded-string",
      "metadata": {
        "checksum": "sha256_hash"
      }
    },
    {
      "id": "uuid4",
      "dataname": "large_arrow_table",
      "payload_type": "arrowtable",
      "transport": "link",
      "encoding": "arrow-ipc",
      "size": 524288,
      "data": "http://localhost:8080/file/UPLOAD_ID/FILE_ID/data.arrow",
      "metadata": {}
    }
  ]
}
```

### 2. msg_payload_v1 - Payload Structure

**JSON Schema (Identical Across All Platforms):**
```json
{
  "id": "uuid4",
  "dataname": "login_image",
  "payload_type": "image | dictionary | arrowtable | jsontable | table | text | audio | video | binary",
  "transport": "direct | link",
  "encoding": "none | json | base64 | arrow-ipc",
  "size": 15433,
  "data": "base64-encoded-string | http-url | json-string",
  "metadata": {
    "checksum": "sha256_hash"
  }
}
```

### 3. Transport Strategy Decision Logic (Cross-Platform)

```
┌─────────────────────────────────────────────────────────────┐
│              smartsend Function (All Platforms)             │
│  Accepts: [(dataname1, data1, type1), ...]                  │
│  (Type is per payload, not standalone)                      │
└─────────────────────────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│  For each payload:                                          │
│  1. Extract type from tuple/array                           │
│  2. Serialize based on type                                 │
│  3. Check payload size                                      │
└─────────────────────────────────────────────────────────────┘
                         │
              ┌───────────┴────────────┐
              ▼                        ▼
        ┌──────────────┐         ┌──────────────┐
        │  Direct Path │         │  Link Path   │
        │  (< 1MB)     │         │  (>= 1MB)    │
        │              │         │              │
        │ • Serialize  │         │ • Serialize  │
        │   to buffer  │         │   to buffer  │
        │ • Base64/JSON│         │ • Upload to  │
        │   encode     │         │   HTTP Server│
        │ • Publish to │         │ • Publish to │
        │   NATS       │         │   NATS with  │
        │   (in msg)   │         │   URL        │
        └──────────────┘         └──────────────┘
```

---

## Platform Comparison Matrix

| Feature | Julia | JavaScript | Python | MicroPython |
|---------|-------|------------|--------|-------------|
| **Multiple Dispatch** | ✅ Native | ❌ (Prototypes) | ❌ (Overload via `@overload`) | ❌ |
| **Async/Await** | ❌ (Tasks) | ✅ Native | ✅ Native | ⚠️ (uasyncio) |
| **Type Safety** | ✅ Strong | ⚠️ (TypeScript) | ✅ (Type hints) | ❌ |
| **Memory Management** | ✅ GC | ✅ GC | ✅ GC | ⚠️ (Manual) |
| **Arrow IPC** | ✅ Native | ✅ (arrow package) | ✅ (pyarrow) | ❌ |
| **JSON Serialization** | ✅ (JSON.jl) | ✅ (native) | ✅ (json) | ✅ (json) |
| **arrowtable Support** | ✅ | ✅ | ✅ | ❌ |
| **jsontable Support** | ✅ | ✅ | ✅ | ❌ |
| **table Support** | ❌ | ❌ | ✅ | ❌ |
| **Direct Transport** | ✅ | ✅ | ✅ | ✅ |
| **Link Transport** | ✅ | ✅ | ✅ | ⚠️ (Limited) |
| **Handler Functions** | ✅ | ✅ | ✅ | ✅ |
| **Cross-Platform API** | ✅ | ✅ | ✅ | ✅ |

---

## Platform-Specific Architecture Patterns

### Julia: Multiple Dispatch Pattern

Julia leverages multiple dispatch for type-specific implementations:

- **Function overloading** based on argument types
- **Struct-based data models** with explicit types
- **Native Arrow IPC** support via Arrow.jl

### JavaScript: Prototype + Async Pattern

JavaScript uses async/await for non-blocking I/O:

- **Class-based NATS client** for connection management
- **Module-level utility functions** for serialization
- **Native ArrayBuffer** for binary data handling

### Python: Class-Based Pattern

Python uses classes for stateful operations:

- **Class-based NATSBridge** with type hints
- **Dataclasses** for structured data (MsgPayloadV1, MsgEnvelopeV1)
- **Async/await** for I/O operations

### MicroPython: Synchronous Pattern

MicroPython has significant constraints:

- **Synchronous API** (no async/await)
- **Memory-constrained** (256KB - 1MB)
- **Limited payload support** (no tables, max 50KB)

---

## Cross-Platform Compatibility Notes

### 1. Python Payload Type Naming

The Python implementation uses `"table"` as a single payload type for both Arrow and JSON table serialization, while Julia and JavaScript use separate types (`"arrowtable"` and `"jsontable"`):

| Platform | Table Types |
|----------|-------------|
| Julia | `"arrowtable"`, `"jsontable"` |
| JavaScript | `"arrowtable"`, `"jsontable"` |
| Python | `"table"` (single type) |
| MicroPython | Not supported |

**Impact:** When exchanging data between Python and Julia/JavaScript, the payload type will differ. Python code should use `"table"` while Julia/JavaScript code should use `"arrowtable"` or `"jsontable"`.

### 2. Direct Transport Encoding Field

The encoding field in direct transport payloads differs between platforms:

| Platform | Encoding for Direct Transport |
|----------|-------------------------------|
| Julia | Preserves original type: `"base64"`, `"json"`, or `"arrow-ipc"` |
| JavaScript | Preserves original type: `"base64"`, `"json"`, or `"arrow-ipc"` |
| Python | Always `"base64"` for all direct transport payloads |
| MicroPython | Always `"base64"` for all direct transport payloads |

**Impact:** The encoding field may not accurately reflect the original serialization format when using Python or MicroPython.

### 3. MicroPython Limitations

MicroPython has significant constraints that affect feature support:

| Feature | Desktop Platforms | MicroPython |
|---------|-------------------|-------------|
| `arrowtable` | ✅ | ❌ (not supported - memory constraints) |
| `jsontable` | ✅ | ❌ (not supported - memory constraints) |
| `table` | ✅ | ❌ (not supported - memory constraints) |
| Async/await | ✅ | ❌ (synchronous only) |
| File upload/download | ✅ | ⚠️ (placeholder implementations) |
| MAX_PAYLOAD_SIZE | 1MB+ | 50KB (hard limit) |
| DEFAULT_SIZE_THRESHOLD | 1MB | 100KB |

**Impact:** MicroPython should only be used for small payloads with direct transport. File server operations are not fully implemented.

---

## Configuration

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `1000000` | Size threshold in bytes (1MB) |

### MicroPython-Specific Configuration

```python
# micropython.conf
NATS_URL = "nats://broker.local:4222"
FILESERVER_URL = "http://fileserver.local:8080"
SIZE_THRESHOLD = 100000  # Lower threshold for memory-constrained devices
MAX_PAYLOAD_SIZE = 50000  # Hard limit for MicroPython
```

---

## Performance Considerations

### Zero-Copy Reading

| Platform | Strategy |
|----------|----------|
| **Julia** | `Arrow.read()` with memory-mapped files |
| **JavaScript** | `ArrayBuffer` with `DataView` |
| **Python** | `pyarrow` memory mapping |
| **MicroPython** | Not available (streaming only) |

### Exponential Backoff

All platforms implement exponential backoff for HTTP downloads:

```
delay = base_delay
for attempt in 1:max_retries:
    try:
        response = fetch(url)
        if success: return response
    except:
        if attempt < max_retries:
            sleep(delay)
            delay = min(delay * 2, max_delay)
```

### Correlation ID Logging

All platforms use correlation IDs for distributed tracing:

```
[timestamp] [Correlation: abc123] Message published to subject
```

### Serialization Performance Comparison

| Format | Use Case | Pros | Cons |
|--------|----------|------|------|
| `arrowtable` | Large tabular data | Fast, zero-copy, schema-preserving | Binary format, requires Arrow library |
| `jsontable` | Small/medium tabular data | Human-readable, universal support | Slower, larger size, no schema |
| `table` (Python) | Large tabular data | Fast, zero-copy, schema-preserving | Python-specific, requires pyarrow |

---

## Summary

This cross-platform NATS bridge provides:

1. **High-Level API Parity**: Identical `smartsend()` and `smartreceive()` signatures across Julia, JavaScript, and Python/MicroPython
2. **Idiomatic Implementations**: 
   - Julia: Multiple dispatch and struct-based design
   - JavaScript: Async/await and prototype-based utilities
   - Python: Class-based design with type hints
   - MicroPython: Synchronous API with memory constraints
3. **Message Format Consistency**: Identical `msg_envelope_v1` and `msg_payload_v1` JSON schemas
4. **Handler Abstraction**: File server operations abstracted through configurable handlers
5. **Platform-Specific Optimizations**: 
   - **Arrow IPC** (`arrowtable`): Efficient binary format for large tabular data
   - **JSON** (`jsontable`): Universal human-readable format for smaller tables
   - **Python table**: Unified table type for Python-specific implementations
   - Streaming support in MicroPython

The Julia implementation serves as the **ground truth** for API design and behavior, while JavaScript and Python implementations maintain interface parity while leveraging their respective language idioms.

### Datatype Summary

| Datatype | Serialization | Use Case | Encoding | Supported Platforms |
|----------|---------------|----------|----------|---------------------|
| `text` | UTF-8 bytes | Text messages, chat content | `utf-8` → `base64` | All |
| `dictionary` | JSON | Structured key-value data, config | `json` → `base64` | All |
| `arrowtable` | Apache Arrow IPC | Large tabular data, schema-preserving | `arrow-ipc` → `base64` | Julia, JavaScript, Python |
| `jsontable` | JSON | Small/medium tabular data, human-readable | `json` → `base64` | Julia, JavaScript, Python |
| `table` | Apache Arrow IPC | Python's unified table type | `arrow-ipc` → `base64` | Python |
| `image` | Binary | Image files (JPEG, PNG, etc.) | `binary` → `base64` | All |
| `audio` | Binary | Audio files (WAV, MP3, etc.) | `binary` → `base64` | All |
| `video` | Binary | Video files (MP4, AVI, etc.) | `binary` → `base64` | All |
| `binary` | Binary | Generic binary data, files | `binary` → `base64` | All |