419 lines
20 KiB
Markdown
419 lines
20 KiB
Markdown
# Requirements Document: NATSBridge
|
|
|
|
**Version**: 1.0.0
|
|
**Date**: 2026-03-23
|
|
**Status**: Active
|
|
**Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl)
|
|
|
|
---
|
|
|
|
## 1. Business Context & Success Metrics
|
|
|
|
### 1.1 Business Goal
|
|
|
|
NATSBridge is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, and **MicroPython** applications using NATS as the message bus. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS.
|
|
|
|
### 1.2 User Stories (with acceptance criteria)
|
|
|
|
| Story | Priority | Acceptance Criteria |
|
|
|-------|----------|---------------------|
|
|
| **As a Julia developer**, I want to send text messages to JavaScript/Dart applications that lives on a server and also on a browser | P1 | Text messages are serialized, encoded, and received correctly across platforms |
|
|
| **As a Python developer**, I want to send tabular data to Julia/Dart applications | P1 | DataFrame exchange works with both Arrow IPC and JSON formats |
|
|
| **As a JavaScript developer**, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS |
|
|
| **As a Dart developer**, I want to send text messages to other platforms | P1 | Text messages are serialized, encoded, and received correctly across platforms |
|
|
| **As a Dart developer**, I want to send dictionary data to other platforms | P1 | JSON-serializable data is exchanged correctly |
|
|
| **As a Dart developer**, I want to send tabular data (List<Map>) to other platforms | P1 | JSON table format exchange works with Arrow IPC on desktop |
|
|
| **As a Dart developer**, I want to send large files (>0.5MB) | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS |
|
|
| **As a MicroPython developer**, I want to send sensor data with minimal memory usage | P1 | Direct transport works for payloads <100KB on memory-constrained devices |
|
|
| **As a developer**, I want to send mixed-content messages (text + image + file) | P1 | NATSBridge accepts list of (dataname, data, type) tuples and handles each payload appropriately |
|
|
| **As a developer**, I want to receive multi-payload messages | P1 | NATSBridge returns payloads as list of tuples with correct types preserved |
|
|
| **As a developer**, I want to use Plik as the file server | P2 | Plik one-shot upload mode is supported with upload ID and token handling |
|
|
| **As a developer**, I want to use custom HTTP file servers | P2 | Handler function abstraction allows plugging in AWS S3 or custom implementations |
|
|
| **As a developer**, I want automatic retry on file server download failures | P1 | Exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) |
|
|
| **As a developer**, I want message tracing across distributed systems | P1 | Correlation ID is propagated through all message processing steps |
|
|
|
|
### 1.3 KPIs & Targets
|
|
|
|
| Metric | Target | Measurement Method |
|
|
|--------|--------|-------------------|
|
|
| 95% of messages complete within 200ms | 95% | Synthetic monitoring |
|
|
| <2 days from onboarding to first PR | 2 days | PR timeline tracking |
|
|
| 100% of messages validate against spec | 100% | CI block rate |
|
|
| >80% unit test coverage | 80% | Test coverage tools |
|
|
| <1% of PRs bypass validation gates | 1% | CI gate analysis |
|
|
| MTTR <15 minutes for P1 incidents | 15 minutes | Incident tracking |
|
|
|
|
---
|
|
|
|
## 2. Technical Boundaries
|
|
|
|
### 2.1 In Scope
|
|
|
|
| Feature | Description |
|
|
|---------|-------------|
|
|
| Cross-platform interoperability | Seamless data exchange between Julia, JavaScript, Python, Dart, and MicroPython |
|
|
| Intelligent transport selection | Direct transport (<0.5MB) vs Link transport (≥0.5MB) based on payload size |
|
|
| Unified API | Consistent `smartsend()` and `smartreceive()` functions across all platforms |
|
|
| Multi-payload support | List of (dataname, data, type) tuples with appropriate handling |
|
|
| File server integration | Plik one-shot upload and custom HTTP server support |
|
|
| Reliability features | Exponential backoff retry and correlation ID propagation |
|
|
| Message serialization | Converts data types to binary format (Base64, JSON, Arrow IPC) |
|
|
| NATS communication | Publishing and subscription via NATS subjects |
|
|
|
|
### 2.2 Out of Scope
|
|
|
|
| Feature | Reason |
|
|
|---------|--------|
|
|
| NATS JetStream support | Core NATS sufficient for current use cases |
|
|
| Message compression | Compression adds complexity without clear benefit |
|
|
| Message encryption | Payload encryption is application-layer concern |
|
|
| Persistent message queues | NATS request-reply pattern sufficient |
|
|
| Advanced routing rules | Simple NATS subject matching sufficient |
|
|
|
|
### 2.3 Dependencies
|
|
|
|
| Platform | Package | Version |
|
|
|----------|---------|---------|
|
|
| Julia | NATS.jl | Latest stable |
|
|
| Julia | JSON.jl | Latest stable |
|
|
| Julia | Arrow.jl | Latest stable |
|
|
| Julia | HTTP.jl | Latest stable |
|
|
| Julia | UUIDs.jl | Latest stable |
|
|
| Node.js | nats | Latest stable |
|
|
| Node.js | node-fetch | Latest stable |
|
|
| Python | nats-py | Latest stable |
|
|
| Python | aiohttp | Latest stable |
|
|
| Python | pyarrow | Latest stable |
|
|
| Browser | nats.ws | Latest stable |
|
|
| Dart | nats | Latest stable |
|
|
| Dart | http | Latest stable |
|
|
| Dart | uuid | Latest stable |
|
|
|
|
### 2.4 Platform Compatibility
|
|
|
|
| Platform | Minimum Version | Notes |
|
|
|----------|-----------------|-------|
|
|
| Julia | 1.7+ | Arrow.jl required for arrowtable support |
|
|
| Node.js | 16+ | nats.js required, Arrow IPC supported |
|
|
| Python | 3.8+ | pyarrow required for arrowtable support |
|
|
| Browser | Latest | No Arrow IPC (uses jsontable only) |
|
|
| Dart | 2.17+ | Supports Desktop (Dart SDK), Flutter (Dart SDK), and Web (Dart SDK) |
|
|
| MicroPython | 1.19+ | Limited to direct transport |
|
|
|
|
---
|
|
|
|
## 3. Functional Requirements (FR)
|
|
|
|
| ID | Requirement | Description |
|
|
|----|-------------|-------------|
|
|
| **FR-001** | Cross-platform text messaging | System shall allow users to send text messages between Julia, JavaScript, Python, and MicroPython applications |
|
|
| **FR-002** | Cross-platform tabular data | System shall support DataFrame exchange between Julia and Python applications using Arrow IPC format |
|
|
| **FR-003** | Large file handling | System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via NATS |
|
|
| **FR-004** | Direct transport for small payloads | System shall send payloads <0.5MB directly via NATS without file server upload |
|
|
| **FR-005** | MicroPython support | System shall support payloads <100KB on MicroPython devices using direct transport |
|
|
| **FR-006** | Multi-payload messages | System shall accept and process lists of (dataname, data, type) tuples |
|
|
| **FR-007** | Payload type preservation | System shall preserve payload types when returning multi-payload messages |
|
|
| **FR-008** | Plik file server integration | System shall support Plik one-shot upload mode with upload ID and token handling |
|
|
| **FR-009** | Custom file server support | System shall provide handler function abstraction for custom HTTP file server implementations |
|
|
| **FR-010** | Exponential backoff retry | System shall implement exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) for file server download failures |
|
|
| **FR-011** | Correlation ID propagation | System shall propagate correlation IDs through all message processing steps |
|
|
| **FR-012** | Message serialization | System shall serialize data types using Base64, JSON, or Arrow IPC encoding |
|
|
| **FR-013** | NATS publishing | System shall publish messages to NATS subjects |
|
|
| **FR-014** | NATS subscription | System shall receive and process NATS messages |
|
|
|
|
---
|
|
|
|
## 4. Non-Functional Requirements (NFRs)
|
|
|
|
### 4.1 Performance & Scalability
|
|
|
|
| ID | Requirement | Specification | Test Method |
|
|
|----|-------------|---------------|-------------|
|
|
| **NFR-101** | Message serialization overhead | <50ms for 10KB payload | Benchmark tests |
|
|
| **NFR-102** | Message deserialization overhead | <50ms for 10KB payload | Benchmark tests |
|
|
| **NFR-103** | NATS connection establishment | <100ms | Connection pool benchmarks |
|
|
| **NFR-104** | File upload latency | <1s for 0.5MB file | Integration tests |
|
|
| **NFR-105** | File download latency | <1s for 0.5MB file | Integration tests |
|
|
| **NFR-106** | Concurrent connections | Support 100+ simultaneous NATS connections | Scale testing |
|
|
| **NFR-107** | Message throughput | Handle 1000+ messages/second per instance | Load testing |
|
|
| **NFR-108** | File server scalability | Support horizontal scaling of file server backend | Architecture review |
|
|
|
|
### 4.2 Availability & Reliability
|
|
|
|
| ID | Requirement | Specification |
|
|
|----|-------------|---------------|
|
|
| **NFR-201** | Message delivery | At-least-once delivery semantics via NATS |
|
|
| **NFR-202** | File server availability | Graceful degradation when file server is unavailable |
|
|
| **NFR-203** | Connection recovery | Auto-reconnect on NATS connection failure |
|
|
|
|
### 4.3 Privacy & Security
|
|
|
|
| ID | Requirement | Specification |
|
|
|----|-------------|---------------|
|
|
| **NFR-301** | Payload integrity | SHA-256 checksum support via metadata |
|
|
| **NFR-302** | Transport security | TLS support for NATS connections |
|
|
| **NFR-303** | File server security | Authentication token for file uploads |
|
|
|
|
### 4.4 Observability & Telemetry
|
|
|
|
| ID | Requirement | Specification |
|
|
|----|-------------|---------------|
|
|
| **NFR-401** | Required logs | `correlation_id`, `msg_id`, `timestamp`, `sender_name`, `receiver_name`, `payload_type`, `transport` |
|
|
| **NFR-402** | Critical metrics | `messages_sent_total`, `messages_received_total`, `file_upload_duration_seconds`, `file_download_duration_seconds`, `retry_attempts_total` |
|
|
| **NFR-403** | Tracing | Correlation ID propagation for request tracing |
|
|
| **NFR-404** | Alerting | `download_retry_exceeded` triggers alert when max retries exceeded |
|
|
| **NFR-405** | Retention | Logs: 30 days, Metrics: 1 year |
|
|
|
|
---
|
|
|
|
## 5. Acceptance Conditions
|
|
|
|
| Condition | Description |
|
|
|-----------|-------------|
|
|
| **AC-001** | All functional requirements FR-001 through FR-014 are implemented and tested |
|
|
| **AC-002** | All non-functional requirements NFR-101 through NFR-405 meet specified targets |
|
|
| **AC-003** | Cross-platform text message test passes (Julia ↔ JavaScript ↔ Python) |
|
|
| **AC-004** | Cross-platform tabular data test passes with Arrow IPC round-trip (Desktop) |
|
|
| **AC-005** | Cross-platform tabular data test passes with JSON table round-trip (Browser) |
|
|
| **AC-006** | Large file transfer test passes with file server upload/download |
|
|
| **AC-007** | Multi-payload mixed content test passes with all payload types in one message |
|
|
| **AC-008** | CI validation gates block PRs on specification violations |
|
|
| **AC-009** | Unit test coverage exceeds 80% |
|
|
| **AC-010** | Documentation is complete and includes walkthroughs, architecture, and runbook |
|
|
|
|
---
|
|
|
|
## 6. Payload Type Requirements
|
|
|
|
### 6.1 Supported Payload Types
|
|
|
|
| Type | Julia | JavaScript | Python | Dart | MicroPython | Description |
|
|
|------|-------|------------|--------|------|-------------|-------------|
|
|
| `text` | `String` | `string` | `str` | `String` | `str` | Plain text strings |
|
|
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `Map` | `dict` | JSON-serializable data |
|
|
| `arrowtable` | `DataFrame`, `Arrow.Table` | ❌ (Browser), ✅ (Node.js) | `pandas.DataFrame` | `List<Map>` (Desktop), `List<dynamic>` (Flutter) | ❌ | Tabular data (Arrow IPC) |
|
|
| `jsontable` | `Vector{NamedTuple}` | `Array<Object>` | `list[dict]` | `List<Map>` | ⚠️ | Tabular data (JSON) - **Only table type in Browser** |
|
|
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `bytearray` | Image binary data |
|
|
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `bytearray` | Audio binary data |
|
|
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `bytearray` | Video binary data |
|
|
| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `Uint8List` | `bytearray` | Generic binary data |
|
|
|
|
### 6.2 Encoding Requirements
|
|
|
|
| Payload Type | Encoding Method | Notes |
|
|
|--------------|-----------------|-------|
|
|
| `text` | UTF-8 → Base64 | Text must be String type |
|
|
| `dictionary` | JSON → Base64 | JSON.jl for Julia |
|
|
| `arrowtable` | Arrow IPC → Base64 | Requires Arrow.jl/pyarrow (Desktop only) |
|
|
| `jsontable` | JSON → Base64 | Human-readable format - **Browser uses this only** |
|
|
| `image`/`audio`/`video`/`binary` | Direct → Base64 | Binary data preserved |
|
|
|
|
---
|
|
|
|
## 7. Size Threshold Requirements
|
|
|
|
### 7.1 Direct Transport Threshold
|
|
|
|
| Platform | Threshold | Notes |
|
|
|----------|-----------|-------|
|
|
| Desktop (Julia/JS/Python/Dart) | 0.5MB | Default size threshold |
|
|
| Dart Desktop | 0.5MB | Default size threshold |
|
|
| Dart Flutter | 0.5MB | Default size threshold |
|
|
| Dart Web | 0.5MB | Default size threshold |
|
|
| MicroPython | 100KB | Lower threshold for memory constraints |
|
|
|
|
### 7.2 Maximum Payload Size
|
|
|
|
| Platform | Maximum | Notes |
|
|
|----------|---------|-------|
|
|
| Desktop | Unlimited | Limited by NATS server configuration |
|
|
| Dart Desktop | Unlimited | Limited by NATS server configuration |
|
|
| Dart Flutter | Unlimited | Limited by NATS server configuration |
|
|
| Dart Web | Unlimited | Limited by NATS server configuration |
|
|
| MicroPython | 50KB | Hard limit due to 256KB-1MB memory |
|
|
|
|
---
|
|
|
|
## 8. Message Envelope Requirements
|
|
|
|
### 8.1 Required Fields
|
|
|
|
| Field | Type | Purpose |
|
|
|-------|------|---------|
|
|
| `correlation_id` | String (UUID) | Track message flow across systems |
|
|
| `msg_id` | String (UUID) | Unique message identifier |
|
|
| `timestamp` | String (ISO 8601) | Message publication timestamp |
|
|
| `send_to` | String | NATS subject to publish to |
|
|
| `msg_purpose` | String | ACK, NACK, updateStatus, shutdown, chat |
|
|
| `sender_name` | String | Sender application name |
|
|
| `sender_id` | String (UUID) | Sender unique identifier |
|
|
| `receiver_name` | String | Receiver application name (empty = broadcast) |
|
|
| `receiver_id` | String (UUID) | Receiver unique identifier (empty = broadcast) |
|
|
| `reply_to` | String | Topic for reply messages |
|
|
| `reply_to_msg_id` | String | Message ID being replied to |
|
|
| `broker_url` | String | NATS server URL |
|
|
| `metadata` | Dict | Message-level metadata |
|
|
| `payloads` | Array | List of payload objects |
|
|
|
|
### 8.2 Payload Fields
|
|
|
|
| Field | Type | Purpose |
|
|
|-------|------|---------|
|
|
| `id` | String (UUID) | Unique payload identifier |
|
|
| `dataname` | String | Name of the payload |
|
|
| `payload_type` | String | Type: text, dictionary, arrowtable, etc. |
|
|
| `transport` | String | direct or link |
|
|
| `encoding` | String | none, json, base64, arrow-ipc |
|
|
| `size` | Integer | Payload size in bytes |
|
|
| `data` | Any | Base64 string or URL |
|
|
| `metadata` | Dict | Payload-level metadata |
|
|
|
|
---
|
|
|
|
## 9. Error Handling Requirements
|
|
|
|
### 9.1 Error Codes
|
|
|
|
| Error | Condition | Response |
|
|
|-------|-----------|----------|
|
|
| `Unknown payload_type` | Unsupported type | Throw error |
|
|
| `Failed to upload` | File server error | Throw error |
|
|
| `Failed to fetch` | File server unavailable | Retry with exponential backoff |
|
|
| `Unknown transport` | Invalid transport type | Throw error |
|
|
| `NATS connection failed` | NATS unavailable | Throw error |
|
|
|
|
### 9.2 Exception Handling
|
|
|
|
| Scenario | Handler |
|
|
|----------|---------|
|
|
| File server unavailable | Retry up to 5 times with exponential backoff |
|
|
| NATS publish failure | Connection auto-reconnect |
|
|
| Deserialization error | Log correlation ID and throw error |
|
|
| Memory overflow (MicroPython) | Reject payloads >50KB |
|
|
|
|
---
|
|
|
|
## 10. Testing Requirements
|
|
|
|
### 10.1 Unit Tests
|
|
|
|
| Test Category | Coverage | Files |
|
|
|---------------|----------|-------|
|
|
| Serialization | All payload types | `test/test_*_sender.*` |
|
|
| Deserialization | All payload types | `test/test_*_receiver.*` |
|
|
| Transport selection | Direct vs link | `test/test_*_mix_payloads.*` |
|
|
| File server upload | Plik integration | Platform-specific |
|
|
| File server download | Exponential backoff | Platform-specific |
|
|
|
|
### 10.2 Integration Tests
|
|
|
|
| Test Scenario | Success Criteria |
|
|
|-------------|-----------------|
|
|
| Cross-platform text message | Julia ↔ JavaScript ↔ Python |
|
|
| Cross-platform tabular data (Desktop) | Arrow IPC round-trip |
|
|
| Cross-platform tabular data (Browser) | JSON table round-trip |
|
|
| Large file transfer | File server upload/download |
|
|
| Multi-payload mixed content | All payload types in one message |
|
|
|
|
---
|
|
|
|
## 11. API Contract
|
|
|
|
### 11.1 smartsend Signature
|
|
|
|
```julia
|
|
function smartsend(
|
|
subject::String,
|
|
data::AbstractArray{Tuple{String, Any, String}};
|
|
broker_url::String = "nats://localhost:4222",
|
|
fileserver_url::String = "http://localhost:8080",
|
|
fileserver_upload_handler::Function = plik_oneshot_upload,
|
|
size_threshold::Int = 1_000_000,
|
|
correlation_id::String = string(uuid4()),
|
|
msg_purpose::String = "chat",
|
|
sender_name::String = "NATSBridge",
|
|
receiver_name::String = "",
|
|
receiver_id::String = "",
|
|
reply_to::String = "",
|
|
reply_to_msg_id::String = "",
|
|
is_publish::Bool = true,
|
|
NATS_connection::Union{NATS.Connection, Nothing} = nothing,
|
|
msg_id::String = string(uuid4()),
|
|
sender_id::String = string(uuid4())
|
|
)::Tuple{msg_envelope_v1, String}
|
|
```
|
|
|
|
### 11.2 smartreceive Signature
|
|
|
|
```julia
|
|
function smartreceive(
|
|
msg::NATS.Msg;
|
|
fileserver_download_handler::Function = _fetch_with_backoff,
|
|
max_retries::Int = 5,
|
|
base_delay::Int = 100,
|
|
max_delay::Int = 5000
|
|
)::JSON.Object{String, Any}
|
|
```
|
|
|
|
---
|
|
|
|
## 12. Deployment Requirements
|
|
|
|
### 12.1 Minimum Infrastructure
|
|
|
|
| Component | Minimum | Notes |
|
|
|-----------|---------|-------|
|
|
| NATS Server | 1 instance | Single node for development |
|
|
| File Server | 1 instance | HTTP server for large payloads |
|
|
| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
|
|
| Client Memory | 256KB | MicroPython devices |
|
|
|
|
### 12.2 Environment Variables
|
|
|
|
| Variable | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL |
|
|
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
|
|
| `SIZE_THRESHOLD` | `1000000` | Size threshold in bytes |
|
|
|
|
---
|
|
|
|
## 13. Versioning
|
|
|
|
### 13.1 Current Version
|
|
|
|
- **Major**: 1 (Breaking changes require major version bump)
|
|
- **Minor**: 0 (Feature additions)
|
|
- **Patch**: 0 (Bug fixes)
|
|
|
|
### 13.2 Version Compatibility
|
|
|
|
| Version | Supported Platforms |
|
|
|---------|---------------------|
|
|
| v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, Dart 2.17+, Browser (latest), MicroPython 1.19+ |
|
|
|
|
---
|
|
|
|
## 14. Change Log
|
|
|
|
| Date | Version | Changes |
|
|
|------|---------|---------|
|
|
| 2026-03-23 | 1.0.0 | Updated to ASG Framework requirements structure |
|
|
|
|
---
|
|
|
|
## 15. References
|
|
|
|
- [`src/NATSBridge.jl`](../src/NATSBridge.jl) - Ground truth implementation (Julia)
|
|
- [`src/natsbridge_ssr.js`](../src/natsbridge_ssr.js) - Server-side JavaScript implementation
|
|
- [`src/natsbridge_csr.js`](../src/natsbridge_csr.js) - Client-side JavaScript implementation
|
|
- [`src/natsbridge.py`](../src/natsbridge.py) - Python implementation
|
|
- [`src/natsbridge.dart`](../src/natsbridge.dart) - Dart implementation
|
|
- [`src/natsbridge_mpy.py`](../src/natsbridge_mpy.py) - MicroPython implementation
|
|
- [`README.md`](../README.md) - Project overview
|
|
- [`docs/specification.md`](./specification.md) - Technical specification
|
|
- [`docs/ui-specification.md`](./ui-specification.md) - UI specification
|
|
- [`docs/walkthrough.md`](./walkthrough.md) - End-to-end walkthrough
|
|
- [`docs/architecture.md`](./architecture.md) - Architecture documentation
|
|
- [`docs/validation.md`](./validation.md) - Validation and CI/CD
|
|
- [`docs/runbook.md`](./runbook.md) - Operational runbook |