# Architecture Documentation: msghandler

**Version**: 1.4.0  
**Date**: 2026-05-14  
**Status**: Active  
**Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl)  
**Architecture Level**: C4 Container Level

---

## 1. Executive Summary

This document defines the **blueprint** for msghandler - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using a message broker as the transport layer.

This architecture document serves as the single source of truth for:
- **System Structure**: How components fit together and interact
- **Scaling Considerations**: How the system scales horizontally and vertically
- **Failure Modes**: How the system handles failures and recovers
- **Trade-off Decisions**: The rationale behind architectural decisions

### 1.1 Specification Traceability

| Architecture Section | Specification Reference | UI Specification Reference | Requirement ID(s) |
|---------------------|-------------------------|---------------------------|-------------------|
| Section 2 (Context Diagram) | specification.md:2 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 3 (Container Diagram) | specification.md:2, specification.md:3, specification.md:11 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 4 (Component Diagram) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 5 (High-Level) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 6 (Message Envelope) | specification.md:2, specification.md:3, specification.md:8 | - | FR-011, FR-012, FR-013, FR-014, NFR-401, NFR-403 |
| Section 7 (Payload Type) | specification.md:3, specification.md:5, specification.md:6 | - | FR-001, FR-002, FR-003, FR-006, FR-012, NFR-101, NFR-102 |
| Section 8 (Transport Strategy) | specification.md:6, specification.md:7 | - | FR-003, FR-004, FR-005, FR-010, NFR-104, NFR-105, NFR-106 |
| Section 9 (Platform-Specific) | specification.md:13, specification.md:14 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 10 (Scaling) | specification.md:7, specification.md:13 | - | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 |
| Section 11 (Failure Modes) | specification.md:9, specification.md:11 | - | FR-008, FR-009, FR-010, FR-011, NFR-201, NFR-202, NFR-203 |
| Section 12 (Trade-offs) | specification.md:2, specification.md:3, specification.md:6, specification.md:7 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 |
| Section 13 (Deployment) | specification.md:12, specification.md:18 | - | FR-013, FR-014, NFR-201, NFR-203 |
| Section 14 (Security) | specification.md:4, specification.md:9, specification.md:12 | - | NFR-301, NFR-302, NFR-303, NFR-401, NFR-402, NFR-403, NFR-404, NFR-405 |
| Section 15 (Testing) | specification.md:17 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |

---

## 2. Architecture Overview

## Architecture Overview

### C4 Context Diagram

```mermaid
flowchart TD
   subgraph "External Systems"
        Message_Broker[Message Broker<br/>NATS/MQTT/WebSocket/Custom]
        File_Server[HTTP File Server<br/>Plik/AWS S3/Custom]
    end

    Julia_App[Julia Application]
    JS_App[JavaScript Application<br/>Node.js/Browser]
    Python_App[Python Application<br/>Desktop]
    Dart_App[Dart Application<br/>Desktop/Flutter/Web]
    Rust_App[Rust Application<br/>Server/Desktop]
    MicroPython_App[MicroPython Device]
    end

    Julia_App -->|Transport| Message_Broker
    JS_App -->|Transport| Message_Broker
    Python_App -->|Transport| Message_Broker
    Dart_App -->|Transport| Message_Broker
    Rust_App -->|Transport| Message_Broker
    MicroPython_App -->|Transport| Message_Broker

    Julia_App -->|HTTP| File_Server
    JS_App -->|HTTP| File_Server
    Python_App -->|HTTP| File_Server
    Dart_App -->|HTTP| File_Server
    Rust_App -->|HTTP| File_Server
    MicroPython_App -->|HTTP| File_Server

    style Message_Broker fill:#fff3e0,stroke:#f57c00
    style File_Server fill:#f3e5f5,stroke:#9c27b4
    style Julia_App fill:#e8f5e9,stroke:#4caf50
    style JS_App fill:#e3f2fd,stroke:#2196f3
    style Python_App fill:#e3f2fd,stroke:#2196f3
    style Dart_App fill:#fff0f6,stroke:#e91e63
    style Rust_App fill:#dea584,stroke:#e65100
    style MicroPython_App fill:#fce4ec,stroke:#e91e63
```

### C4 Container Diagram

```mermaid
flowchart TD
  subgraph "Client Container"
        Julia_Module[Julia msghandler Module]
        JS_Module[JavaScript msghandler Module]
        Python_Module[Python msghandler Module]
        Dart_Module[Dart msghandler Module]
        Rust_Module[Rust msghandler Module]
        MicroPython_Module[MicroPython msghandler Module]
    end

    Julia_Module --> Transport_Client
    JS_Module --> Transport_Client
    Python_Module --> Transport_Client
    Dart_Module --> Transport_Client
    Rust_Module --> Transport_Client
    MicroPython_Module --> Transport_Client

    Transport_Client --> Message_Broker

    Julia_Module --> File_Client
    JS_Module --> File_Client
    Python_Module --> File_Client
    Dart_Module --> File_Client
    Rust_Module --> File_Client
    MicroPython_Module --> File_Client

    File_Client --> File_Server

    style Julia_Module fill:#e8f5e9,stroke:#4caf50
    style JS_Module fill:#e3f2fd,stroke:#2196f3
    style Python_Module fill:#e3f2fd,stroke:#2196f3
    style Dart_Module fill:#fff0f6,stroke:#e91e63
    style Rust_Module fill:#dea584,stroke:#e65100
    style MicroPython_Module fill:#fce4ec,stroke:#e91e63
    style Message_Broker fill:#fff3e0,stroke:#f57c00
    style File_Server fill:#f3e5f5,stroke:#9c27b4
```

### C4 Component Diagram (Julia Implementation)

```mermaid
flowchart TD
    subgraph "msghandler Module"
        smartpack[smartpack Function]
        smartunpack[smartunpack Function]
        
        Serialize[_serialize_data]
        Deserialize[_deserialize_data]
        
        EnvelopeToJson[envelope_to_json]
        
        FileServerUpload[fileserver_upload_handler]
        FileServerDownload[fileserver_download_handler]
        
        LogTrace[log_trace]
    end

    subgraph "Data Models"
        Payload[msg_payload_v1 Struct]
        Envelope[msg_envelope_v1 Struct]
    end

    smartpack --> Serialize
    smartpack --> EnvelopeToJson
    smartpack --> FileServerUpload
    
    smartunpack --> Deserialize
    smartunpack --> FileServerDownload
    
    EnvelopeToJson --> Envelope
    Serialize --> Payload

    style smartpack fill:#d1fae5,stroke:#10b981
    style smartunpack fill:#d1fae5,stroke:#10b981
    style FileServerUpload fill:#fef3c7,stroke:#f59e0b
    style FileServerDownload fill:#fef3c7,stroke:#f59e0b
```

---

## High-Level Architecture

### System Components

| Component | Purpose | Platform Support |
|-----------|---------|------------------|
| **smartpack** | Send data with automatic transport selection, returns (envelope, json_string) for caller to publish via transport | All |
| **smartunpack** | Receive and process messages from JSON string | All |
| **_serialize_data** | Serialize data according to payload type | All |
| **_deserialize_data** | Deserialize bytes to native data types | All |
| **envelope_to_json** | Convert msg_envelope_v1 struct to JSON string | All |
| **log_trace** | Log trace messages with correlation ID | All |
| **fileserver_upload_handler** | Upload large payloads to HTTP server | Desktop (Julia/JS/Python/Dart/Rust) |
| **fileserver_download_handler** | Download payloads from HTTP server with exponential backoff | Desktop (Julia/JS/Python/Dart/Rust) |
| **plik_upload_file** | Upload a local file to Plik server from disk | Rust |

### Data Flow

```mermaid
flowchart TD
    A[User calls smartpack subject data] --> B[Process each payload]
    B --> C{Calculate serialized size}
    C -->|Size < Threshold| D[Direct Transport]
    C -->|Size >= Threshold| E[Link Transport]
    
    D --> F[Serialize data]
    F --> G[Base64 encode]
    G --> H[Build payload object]
    
    E --> I[Serialize data]
    I --> J[Upload to file server]
    J --> K[Get download URL]
    K --> H
    
    H --> L[Build envelope]
    L --> M[Convert to JSON]
    M --> N[Return envelope + JSON to caller]
    
    style A fill:#f9f9f9,stroke:#333
    style N fill:#e0e7ff,stroke:#3b82f6
    style D fill:#d1fae5,stroke:#10b981
    style E fill:#fef3c7,stroke:#f59e0b
```

---

## Message Envelope Architecture

### msg_envelope_v1 Structure (Julia)

```julia
struct msg_envelope_v1
  correlation_id::String       # UUID v4 for distributed tracing
  msg_id::String               # UUID v4 for this message
  timestamp::String            # ISO 8601 UTC timestamp
  
  send_to::String              # Topic/subject to publish to
  msg_purpose::String          # ACK, NACK, updateStatus, shutdown, chat
  sender_name::String          # Sender application name
  sender_id::String            # UUID v4 of sender
  receiver_name::String        # Receiver application name (empty = broadcast)
  receiver_id::String          # UUID v4 of receiver (empty = broadcast)
  
  reply_to::String             # Topic for reply messages
  reply_to_msg_id::String      # Message ID being replied to
  broker_url::String           # Broker URL for the transport layer
  
  metadata::Dict{String, Any}  # Message-level metadata
  payloads::Vector{msg_payload_v1}  # List of payloads
end
```

### msg_payload_v1 Structure (Julia)

```julia
struct msg_payload_v1
  id::String                    # UUID v4 for this payload
  dataname::String              # Name of the payload
  payload_type::String          # text, dictionary, arrowtable, etc.
  transport::String             # direct or link
  encoding::String              # none, json, base64, arrow-ipc
  size::Integer                 # Size in bytes
  data::Any                     # Base64 string or URL
  metadata::Dict{String, Any}   # Payload-level metadata
end
```

### JSON Schema (Cross-Platform)

```json
{
  "correlation_id": "string (UUID v4)",
  "msg_id": "string (UUID v4)",
  "timestamp": "string (ISO 8601 UTC)",
  "send_to": "string",
  "msg_purpose": "string",
  "sender_name": "string",
  "sender_id": "string (UUID v4)",
  "receiver_name": "string",
  "receiver_id": "string (UUID v4)",
  "reply_to": "string",
  "reply_to_msg_id": "string",
  "broker_url": "string",
  "metadata": "object",
  "payloads": [
    {
      "id": "string (UUID v4)",
      "dataname": "string",
      "payload_type": "string",
      "transport": "string",
      "encoding": "string",
      "size": "integer",
      "data": "string or URL",
      "metadata": "object"
    }
  ]
}
```

---

## Payload Type Architecture

### Supported Payload Types

| Type | Description | Serialization | Encoding | Platforms |
|------|-------------|---------------|----------|-----------|
| `text` | Plain text string | UTF-8 bytes | Base64 | All |
| `dictionary` | JSON object | JSON string | Base64/JSON | All |
| `arrowtable` | Apache Arrow IPC | Arrow IPC stream | Base64/arrow-ipc | Desktop (Julia/Python/Node.js/Dart/Rust) |
| `jsontable` | JSON array of objects | JSON string | Base64/json | All (including Browser/Dart Web) |
| `image` | Binary image data | Raw bytes | Base64 | All |
| `audio` | Binary audio data | Raw bytes | Base64 | All |
| `video` | Binary video data | Raw bytes | Base64 | All |
| `binary` | Generic binary data | Raw bytes | Base64 | All |

### Serialization Logic

```mermaid
flowchart TD
    A[Input data + payload_type] --> B{Payload Type}
    
    B -->|"text"| C[UTF-8 encode]
    B -->|"dictionary"| D[JSON serialize]
    B -->|"arrowtable"| E[Arrow IPC serialize]
    B -->|"jsontable"| F[JSON serialize]
    B -->|"image"| G[Raw bytes]
    B -->|"audio"| H[Raw bytes]
    B -->|"video"| I[Raw bytes]
    B -->|"binary"| J[Raw bytes]
    
    C --> K[Return bytes]
    D --> K
    E --> K
    F --> K
    G --> K
    H --> K
    I --> K
    J --> K
    
    style A fill:#f9f9f9,stroke:#333
    style K fill:#e0e7ff,stroke:#3b82f6
```

---

## Transport Strategy Architecture

### Size Threshold Decision Logic

| Platform | Size Threshold | Notes |
|----------|----------------|-------|
| Desktop (Julia/JS/Python/Dart) | 500,000 bytes (0.5MB) | Default threshold |
| Dart Desktop | 500,000 bytes (0.5MB) | Default threshold |
| Dart Flutter | 500,000 bytes (0.5MB) | Default threshold |
| Dart Web | 500,000 bytes (0.5MB) | Default threshold |
| MicroPython | 100,000 bytes (100KB) | Lower threshold for memory constraints |

### Transport Selection Flow

```mermaid
flowchart TD
    A[smartpack called] --> B[Serialize payload]
    B --> C[Calculate size]
    C --> D{Size < Threshold?}
    
    D -->|Yes| E[Direct Transport]
    D -->|No| F[Link Transport]
    
    E --> G[Base64 encode]
    G --> H[Build payload with direct transport]
    
    F --> I[Upload to file server]
    I --> J[Get download URL]
    J --> K[Build payload with link transport]
    
    H --> L[Build envelope]
    K --> L
    
    style A fill:#f9f9f9,stroke:#333
    style L fill:#e0e7ff,stroke:#3b82f6
    style E fill:#d1fae5,stroke:#10b981
    style F fill:#fef3c7,stroke:#f59e0b
```

### Direct Transport Protocol

When `transport = "direct"`, the `data` field contains a Base64-encoded string of the serialized payload.

**Encoding Rules**:
- `text`: UTF-8 → Base64
- `dictionary`: JSON → Base64 (or direct JSON)
- `arrowtable`: Arrow IPC → Base64 (or arrow-ipc)
- `jsontable`: JSON → Base64 (or direct JSON)
- `image`/`audio`/`video`/`binary`: Raw bytes → Base64

### Link Transport Protocol

When `transport = "link"`, the `data` field contains a URL pointing to the uploaded payload.

**Upload Flow**:
1. Serialize payload according to `payload_type`
2. Upload to HTTP file server (e.g., Plik)
3. Include returned URL in `data` field

**Download Flow**:
1. Extract URL from payload
2. Fetch with exponential backoff (max 5 retries)
3. Deserialize based on `payload_type`

---

## Platform-Specific Architecture

### Julia Architecture

Julia leverages multiple dispatch for type-specific implementations:

- **Multiple Dispatch**: Function overloading based on argument types
- **Struct-based Data Models**: Explicit type definitions with `struct`
- **Native Arrow IPC**: Support via `Arrow.jl`
- **Async/Await**: Tasks for non-blocking I/O

```julia
# Multiple dispatch for serialization
function _serialize_data(data::String, payload_type::String)
    # Text serialization
end

function _serialize_data(data::Dict, payload_type::String)
    # Dictionary serialization
end

function _serialize_data(data::DataFrame, payload_type::String)
    # Arrow table serialization
end
```

### JavaScript Architecture

JavaScript uses async/await for non-blocking I/O:

- **Module-level Utilities**: Serialization functions
- **Native ArrayBuffer**: Binary data handling (Browser) / Buffer (Node.js)
- **Fetch API**: HTTP file server communication

#### Node.js Implementation (msghandler_ssr.js)

- **Transport connections**: Uses broker URLs (e.g., `nats://`, `mqtt://`, `ws://`)
- **Apache Arrow IPC**: Full support via `apache-arrow`
- **Buffer for binary data**: Native Node.js Buffer handling

#### Browser Implementation (msghandler_csr.js)

- **WebSocket connections**: Uses `ws://` or `wss://` URLs (transport-agnostic)
- **No Apache Arrow**: Uses `jsontable` for tabular data only
- **Uint8Array for binary data**: Browser-compatible binary handling
- **Web Crypto API**: UUID generation via `crypto.getRandomValues()`

### Python Architecture

Python uses classes for stateful operations:

- **Class-based msghandler**: Encapsulated API
- **Dataclasses**: Structured data (MsgPayloadV1, MsgEnvelopeV1)
- **Async/await**: I/O operations
- **pyarrow**: Arrow IPC support

```python
class msghandler:
    DEFAULT_SIZE_THRESHOLD = 500_000
    
    def __init__(self, broker_url=None, fileserver_url=None):
        self.broker_url = broker_url or self.DEFAULT_BROKER_URL
        self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL
```

### Dart Architecture

Dart uses classes for stateful operations with async/await:

- **Class-based msghandler**: Encapsulated API
- **Data classes**: Structured data (MsgPayloadV1, MsgEnvelopeV1)
- **Async/await**: I/O operations
- **dart-arrow**: Arrow IPC support (Desktop/Flutter only)
- **HTTP package**: HTTP file server communication
- **Transport package**: Transport client with WebSocket support (Dart Web)

```dart
class msghandler {
  static const DEFAULT_SIZE_THRESHOLD = 500000;
  
  final String brokerUrl;
  final String fileserverUrl;
  
  msghandler({
    this.brokerUrl = DEFAULT_BROKER_URL,
    this.fileserverUrl = 'http://localhost:8080',
  });
}
```

#### Dart Desktop (Dart SDK)

- **Transport connections**: Uses broker URLs (e.g., `nats://`, `mqtt://`)
- **Apache Arrow IPC**: Full support via `dart-arrow`
- **Uint8List for binary data**: Native Dart binary handling

#### Dart Flutter (Dart SDK)

- **Transport connections**: Uses broker URLs (e.g., `nats://`, `mqtt://`)
- **Apache Arrow IPC**: Full support via `dart-arrow`
- **Uint8List for binary data**: Native Dart binary handling

#### Dart Web (Dart SDK)

- **WebSocket connections**: Uses `ws://` or `wss://` URLs (transport-agnostic)
- **No Apache Arrow**: Uses `jsontable` for tabular data only
- **Uint8List for binary data**: Browser-compatible binary handling
- **Fetch API**: HTTP file server communication via `http` package

### Browser Architecture

Browser JavaScript has specific constraints due to security and compatibility:

- **Async/await**: Native async/await support
- **No Apache Arrow**: Arrow IPC not available in browsers
- **JSON table only**: Use "jsontable" for tabular data
- **WebSocket transport**: Uses transport client for browser-compatible connections
- **Fetch API**: HTTP file server communication via fetch

### MicroPython Architecture

MicroPython has significant constraints:

- **Synchronous API**: No async/await
- **Memory-constrained**: 256KB - 1MB
- **Limited payload support**: No tables, max 50KB
- **Simplified UUID generation**: Custom implementation

```python
# MicroPython constraints
DEFAULT_SIZE_THRESHOLD = 100_000  # 100KB
MAX_PAYLOAD_SIZE = 50_000  # 50KB hard limit
```

### Rust Architecture

Rust leverages compile-time type safety and async runtimes:

- **Type-safe payloads**: Rust enum discriminates between `Text`, `Dictionary`, `ArrowTable`, `Binary`, etc.
- **serde serialization**: Automatic JSON deserialization via `#[derive(Serialize, Deserialize)]`
- **tokio runtime**: Efficient async I/O for transport connections and HTTP file server operations
- **arrow2 integration**: Native Arrow IPC deserialization without intermediate format conversion
- **reqwest**: High-performance HTTP client with built-in TLS and connection pooling
- **Zero-copy patterns**: `Vec<u8>` passed directly to avoid unnecessary memory copies
- **Result<T, E>**: Idiomatic error handling with typed error types

```rust
// Type-safe payload enum (compile-time discrimination)
#[derive(Serialize, Deserialize, Clone)]
pub enum Payload {
    Text(String),
    Dictionary(serde_json::Value),
    ArrowTable(Vec<u8>),
    JsonTable(serde_json::Value),
    Image(Vec<u8>),
    Audio(Vec<u8>),
    Video(Vec<u8>),
    Binary(Vec<u8>),
}

// Configuration via builder pattern
pub struct smartpackOptions {
    pub broker_url: String,
    pub fileserver_url: String,
    pub fileserver_upload_handler: Option<Arc<dyn FileUploadHandler>>,
    pub size_threshold: usize,
    pub correlation_id: String,
    pub msg_purpose: String,
    pub sender_name: String,
    // ... other fields
}

// Transport client with tokio integration
let conn = transport_client::connect(DEFAULT_BROKER_URL).await?;

// Subscribe and process messages
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze")?;
for msg in sub.messages() {
    let envelope = smartunpack(&String::from_utf8_lossy(&msg.payload), &Default::default()).await?;
    // Access deserialized payloads by type
    for payload in &envelope.payloads {
        match payload.payload_type.as_str() {
            "arrowtable" => { /* payload.data is base64-encoded Arrow IPC */ },
            "text" => { /* payload.data is decoded text string */ },
            "binary" | "image" | "audio" | "video" => { /* payload.data is base64-encoded binary */ },
            _ => { /* other types */ }
        }
    }
}
```

---

## Scaling Architecture

### Horizontal Scaling

| Component | Scaling Strategy |
|-----------|------------------|
| **Message Broker** | Cluster deployment with multiple nodes |
| **File Server** | Load balancer + multiple instances |
| **Client Applications** | Deploy multiple instances behind load balancer |

### Vertical Scaling

| Component | Scaling Strategy |
|-----------|------------------|
| **Message Broker** | Increase memory, CPU, disk I/O |
| **File Server** | Increase memory, CPU, disk capacity |
| **Client Applications** | Increase heap size (Python/JS) |

### Performance Considerations

| Metric | Target | Notes |
|--------|--------|-------|
| Message serialization overhead | <50ms | For 10KB payload |
| Message deserialization overhead | <50ms | For 10KB payload |
| Transport connection establishment | <100ms | Connection pool recommended |
| File upload latency | <1s | For 0.5MB file |
| File download latency | <1s | For 0.5MB file |

---

## Failure Modes and Recovery

### Transport Connection Failure

**Scenario**: Message broker unavailable

**Handler**:
- Connection auto-reconnect via transport-level reconnection
- Retry with exponential backoff for publish operations

**Recovery**:
- Transport client automatically attempts reconnection
- Application can check connection status before publishing

### File Server Unavailable

**Scenario**: HTTP file server unavailable during upload/download

**Handler**:
- Retry up to 5 times with exponential backoff (100ms → 5000ms)
- Fallback to direct transport for upload (MicroPython)

**Recovery**:
- Exponential backoff: `delay = min(delay * 2, max_delay)`
- After max retries, throw error with correlation ID

### Deserialization Error

**Scenario**: Payload type mismatch or corrupted data

**Handler**:
- Log correlation ID and throw error
- No retry (data corruption)

**Recovery**:
- Application must validate payload_type matches data type
- Use proper serialization before sending

### Memory Overflow (MicroPython)

**Scenario**: Payload exceeds maximum size (50KB)

**Handler**:
- Reject payloads >50KB with MemoryError
- No retry (client-side check)

**Recovery**:
- Application must split large payloads
- Use direct transport only for small payloads

---

## Trade-off Decisions

### Decision 1: Direct vs Link Transport Threshold

**Trade-off**: Memory vs Network I/O

**Decision**: Use 0.5MB threshold for desktop, 100KB for MicroPython

**Rationale**:
- Direct transport uses more memory (Base64 encoding adds ~33% overhead)
- Link transport requires network I/O for upload/download
- 0.5MB is reasonable for desktop memory constraints
- 100KB is necessary for MicroPython memory constraints

### Decision 2: Base64 Encoding for Direct Transport

**Trade-off**: Bandwidth vs Simplicity

**Decision**: Use Base64 encoding for all direct transport payloads

**Rationale**:
- Simplifies JSON serialization (all data is string-compatible)
- Increases payload size by ~33%, but transport can handle this
- Alternative would be binary payload support (more complex)

### Decision 3: Multiple Platform Implementations

**Trade-off**: Development effort vs Cross-platform support

**Decision**: Maintain separate implementations for each platform

**Rationale**:
- Each platform has idiomatic patterns (multiple dispatch, async/await, etc.)
- Maintains developer productivity and code quality
- API parity ensures cross-platform compatibility

### Decision 4: Handler Function Abstraction

**Trade-off**: Flexibility vs Simplicity

**Decision**: Abstract file server operations through handler functions

**Rationale**:
- Allows support for different file server implementations (Plik, AWS S3, custom)
- Maintains simplicity for common use cases
- Enables plug-in architecture for custom backends

---

## Deployment Architecture

### Minimum Infrastructure

| Component | Minimum | Notes |
|-----------|---------|-------|
| Message Broker | 1 instance | Single node for development |
| File Server | 1 instance | HTTP server for large payloads |
| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices |

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `BROKER_URL` | `ws://localhost:4222` | Message broker URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) |

### Container Deployment

```mermaid
flowchart TD
    subgraph "Docker Network"
        Broker_Container[Message Broker]
        FileServer_Container[Plik File Server]
        App_Container[Application Container]
    end
    
    App_Container -->|Transport| Broker_Container
    App_Container -->|HTTP| FileServer_Container
    
    style Broker_Container fill:#fff3e0,stroke:#f57c00
    style FileServer_Container fill:#f3e5f5,stroke:#9c27b4
    style App_Container fill:#e3f2fd,stroke:#2196f3
```

---

## Security Considerations

### Payload Integrity

**Mechanism**: SHA-256 checksum via metadata

**Implementation**:
- Sender calculates checksum and stores in payload metadata
- Receiver validates checksum on receipt

### Transport Security

**Mechanism**: TLS support for transport connections

**Implementation**:
- Use `nats://` URL for plain text
- Use `tls://` URL for TLS-encrypted connections
- Use `ws://` or `wss://` for WebSocket connections

### File Server Security

**Mechanism**: Authentication token for file uploads

**Implementation**:
- Plik uses upload token in `X-UploadToken` header
- Application can implement custom authentication

---

## Testing Architecture

### Unit Test Coverage

| Test Category | Coverage | Files |
|---------------|----------|-------|
| Serialization | All payload types | `test/test_*_sender.*` |
| Deserialization | All payload types | `test/test_*_receiver.*` |
| Transport selection | Direct vs link | `test/test_*_mix_payloads.*` |
| File server upload | Plik integration | Platform-specific |
| File server download | Exponential backoff | Platform-specific |

### Integration Test Scenarios

| Scenario | Platforms | Payloads | Transport | Expected Result |
|----------|-----------|----------|-----------|-----------------|
| Cross-platform text | Julia ↔ JS ↔ Python | text | direct | Round-trip successful |
| Arrow IPC round-trip | Julia ↔ JS ↔ Python | arrowtable | direct | Arrow IPC preserved |
| Large file transfer | All | image/audio/video | link | File server upload/download |
| Multi-payload mixed | All | text + image + file | direct/link | All payloads preserved |

---

## Versioning

### Architecture Versioning

| Component | Version | Notes |
|-----------|---------|-------|
| Architecture | 1.0.0 | Initial release |
| Protocol | v1 | Message envelope protocol version |

### Backward Compatibility

| Version | Supported Platforms |
|---------|---------------------|
| v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, Dart 2.17+, Rust 1.70+, MicroPython 1.19+ |

---

## Change Log

| Date | Version | Changes |
|------|---------|---------|
| 2026-05-15 | 1.5.0 | Made transport layer agnostic | All sections |
| - | - | Removed all NATS-specific references from architecture docs | All sections |
| - | - | Updated diagrams to use generic "Message Broker" instead of "NATS Server" | All sections |
| - | - | Updated code examples to use transport-agnostic patterns | All sections |
| - | - | Removed NATS client packages from external dependencies | All sections |
| 2026-05-14 | 1.4.0 | Updated Rust API to reflect `smartunpack` deserialization changes | All sections |
| - | - | `smartunpack` now stores deserialized data in `MsgPayloadV1.data` | specification.md:8 |
| - | - | Added `plik_upload_file` convenience function to component table | specification.md:13 |
| - | - | Fixed Rust payload access pattern (data is String, not Payload enum) | All sections |
| - | - | Fixed `smartpackOptions.fileserver_upload_handler` type to `Arc<dyn FileUploadHandler>` | specification.md:13 |
| - | - | Removed `metadata` from link transport examples (now `None`/omitted) | specification.md:3 |
| - | - | Removed duplicate footer text | All sections |
| 2026-05-13 | 1.3.0 | Added Rust support with tokio, serde, and arrow2 | All sections |
| - | - | Added Rust to C4 diagrams (context, container) | All sections |
| - | - | Added Rust platform-specific architecture section | specification.md:13 |
| - | - | Updated component table with Rust support | All sections |
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/msghandler.jl) |
| - | - | Removed publish_message component (commented out in source) |
| - | - | Removed NATSClient and NATSConnectionPool classes (not in ground truth) |
| - | - | Updated smartpack to return JSON for caller to publish via transport |
| - | - | Updated component diagram to match actual module structure |
| - | - | Updated data flow to show smartpack returns JSON for caller to publish |
| - | - | Fixed SIZE_THRESHOLD default to 500,000 bytes |
| 2026-03-15 | 1.1.0 | JavaScript connection management |
| - | - | Added NATSClient with keepAlive support |
| - | - | Added NATSConnectionPool for connection reuse |
| - | - | Added publishMessage function with closeConnection option |
| (Historical - pre-transport-agnostic refactor) | | |
| 2026-03-13 | 1.0.0 | Initial architecture documentation |

---

## 16. References

### 16.1 Documentation Artifacts

| Document | Purpose | Specification Traceability | UI Specification Traceability | Requirement ID(s) |
|----------|---------|---------------------------|------------------------------|-------------------|
| [`docs/requirements.md`](./requirements.md) | Business requirements and user stories | FR-001 through FR-014, NFR-101 through NFR-405 | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/specification.md`](./specification.md) | Technical contract for msghandler | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/ui-specification.md`](./ui-specification.md) | UI specification for client applications | - | All UI components and interactions | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/walkthrough.md`](./walkthrough.md) | End-to-end system flow | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/architecture.md`](./architecture.md) | System architecture diagrams | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/validation.md`](./validation.md) | CI/CD validation rules | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/runbook.md`](./runbook.md) | Operational runbook | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |

### 16.2 Implementation Files

| File | Platform | Features | Specification Traceability | Requirement ID(s) |
|------|----------|----------|---------------------------|-------------------|
| [`src/msghandler.jl`](../src/msghandler.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_ssr.js`](../src/msghandler_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_csr.js`](../src/msghandler_csr.js) | Browser | JSON table only | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.py`](../src/msghandler.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.dart`](../src/msghandler.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler.rs`](../src/msghandler.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe, file upload helpers | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/msghandler_mpy.py`](../src/msghandler_mpy.py) | MicroPython | Limited to direct transport | specification.md:2-19 (all sections) | FR-005, FR-006, FR-012 |

### 16.3 External Dependencies

| Platform | Package | Version | Purpose | Specification Traceability | Requirement ID(s) |
|----------|---------|---------|---------|--------------------------|-------------------|
| Julia | JSON.jl | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Julia | Arrow.jl | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Julia | HTTP.jl | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Julia | UUIDs.jl | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 |
| Node.js | node-fetch | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Browser | - | - | Transport-agnostic (caller provides) | specification.md:11 | FR-013, FR-014 |
| Python | aiohttp | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Python | pyarrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Dart | http | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Dart | uuid | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 |
| Dart | dart-arrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Rust | serde | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Rust | serde_json | Latest | JSON handling | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Rust | tokio | Latest | Async runtime | specification.md:11 | FR-013, FR-014 |
| Rust | reqwest | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Rust | uuid | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 |
| Rust | arrow2 | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| MicroPython | builtin | N/A | Limited implementation | specification.md:11 | FR-005, FR-006, FR-012 |

---

## 17. Change Log

| Date | Version | Changes | Specification Reference |
|------|---------|---------|------------------------|
| 2026-03-23 | 1.1.0 | Updated to ASG Framework architecture guidelines | specification.md:2-19 (all sections) |
| 2026-03-15 | 1.1.0 | JavaScript connection management | specification.md:2-19 (all sections) |
| 2026-03-13 | 1.0.0 | Initial architecture documentation | specification.md:2-19 (all sections) |

---

## 18. Gap-Check Validation

| Stage Transition | Gap-Check Question | Status |
|------------------|-------------------|--------|
| Requirements → Specification | Does the Specification define all edge cases and conflict scenarios from the Requirements? | ✅ Verified - All FR-XXX requirements have corresponding spec rules |
| Specification → UI Specification | Does the UI Specification expose all the data and states defined in the Specification? | ⏳ Pending - UI spec not yet created |
| UI Specification → Walkthrough | Does the Walkthrough reflect the complete flow including error states and timing? | ⏳ Pending - UI spec not yet created |
| Walkthrough → Architecture | Does the Architecture support the performance and integration requirements defined in the Walkthrough? | ✅ Verified - Architecture supports all walkthrough flows |

---

*This architecture document is versioned and maintained in git alongside the codebase. All implementations must adhere to this architecture.*