From 7c83c06d6c68db69ffa39881c9e744ae428cbea7 Mon Sep 17 00:00:00 2001 From: narawat Date: Fri, 13 Mar 2026 13:35:49 +0700 Subject: [PATCH] update --- docs/requirements.md | 420 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 420 insertions(+) create mode 100644 docs/requirements.md diff --git a/docs/requirements.md b/docs/requirements.md new file mode 100644 index 0000000..02d6d64 --- /dev/null +++ b/docs/requirements.md @@ -0,0 +1,420 @@ +# Requirements Document: NATSBridge + +**Version**: 1.0.0 +**Date**: 2026-03-13 +**Status**: Active +**Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl) + +--- + +## Executive Summary + +NATSBridge is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, and **MicroPython** applications using NATS as the message bus. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>1MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS. + +--- + +## Business Goals + +### Primary Objectives + +1. **Cross-Platform Interoperability**: Enable seamless data exchange between Julia, JavaScript, Python, and MicroPython applications without platform-specific barriers. + +2. **Efficient Large Payload Handling**: Implement intelligent transport selection based on payload size: + - **Direct Transport**: Small payloads (<1MB) sent directly via NATS + - **Link Transport**: Large payloads (≥1MB) uploaded to HTTP file server, URL sent via NATS + +3. **Unified API Across Platforms**: Provide consistent `smartsend()` and `smartreceive()` functions across all supported platforms while maintaining idiomatic implementations. + +4. **Developer Productivity**: Reduce onboarding time and simplify integration through comprehensive documentation and test examples. + +### Success Metrics + +| Metric | Target | Measurement Method | +|--------|--------|-------------------| +| 95% of messages complete within 200ms | 95% | Synthetic monitoring | +| <2 days from onboarding to first PR | 2 days | PR timeline tracking | +| 100% of messages validate against spec | 100% | CI block rate | +| >80% unit test coverage | 80% | Test coverage tools | +| <1% of PRs bypass validation gates | 1% | CI gate analysis | +| MTTR <15 minutes for P1 incidents | 15 minutes | Incident tracking | + +--- + +## User Stories + +### Core Functionality + +| Story | Priority | Acceptance Criteria | +|-------|----------|---------------------| +| **As a Julia developer**, I want to send text messages to JavaScript applications | P1 | Text messages are serialized, encoded, and received correctly across platforms | +| **As a Python developer**, I want to send tabular data to Julia applications | P1 | DataFrame exchange works with both Arrow IPC and JSON formats | +| **As a JavaScript developer**, I want to send large files (>1MB) to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS | +| **As a MicroPython developer**, I want to send sensor data with minimal memory usage | P1 | Direct transport works for payloads <100KB on memory-constrained devices | + +### Multi-Payload Support + +| Story | Priority | Acceptance Criteria | +|-------|----------|---------------------| +| **As a developer**, I want to send mixed-content messages (text + image + file) | P1 | NATSBridge accepts list of (dataname, data, type) tuples and handles each payload appropriately | +| **As a developer**, I want to receive multi-payload messages | P1 | NATSBridge returns payloads as list of tuples with correct types preserved | + +### File Server Integration + +| Story | Priority | Acceptance Criteria | +|-------|----------|---------------------| +| **As a developer**, I want to use Plik as the file server | P2 | Plik one-shot upload mode is supported with upload ID and token handling | +| **As a developer**, I want to use custom HTTP file servers | P2 | Handler function abstraction allows plugging in AWS S3 or custom implementations | + +### Reliability Features + +| Story | Priority | Acceptance Criteria | +|-------|----------|---------------------| +| **As a developer**, I want automatic retry on file server download failures | P1 | Exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) | +| **As a developer**, I want message tracing across distributed systems | P1 | Correlation ID is propagated through all message processing steps | + +--- + +## Non-Functional Requirements + +### Performance Requirements + +| Requirement | Specification | Test Method | +|-------------|---------------|-------------| +| Message serialization overhead | <50ms for 10KB payload | Benchmark tests | +| Message deserialization overhead | <50ms for 10KB payload | Benchmark tests | +| NATS connection establishment | <100ms | Connection pool benchmarks | +| File upload latency | <1s for 1MB file | Integration tests | +| File download latency | <1s for 1MB file | Integration tests | + +### Scalability Requirements + +| Requirement | Specification | +|-------------|---------------| +| Concurrent connections | Support 100+ simultaneous NATS connections | +| Message throughput | Handle 1000+ messages/second per instance | +| File server scalability | Support horizontal scaling of file server backend | + +### Reliability Requirements + +| Requirement | Specification | +|-------------|---------------| +| Message delivery | At-least-once delivery semantics via NATS | +| File server availability | Graceful degradation when file server is unavailable | +| Connection recovery | Auto-reconnect on NATS connection failure | + +### Security Requirements + +| Requirement | Specification | +|-------------|---------------| +| Payload integrity | SHA-256 checksum support via metadata | +| Transport security | TLS support for NATS connections | +| File server security | Authentication token for file uploads | + +### Compatibility Requirements + +| Platform | Minimum Version | Notes | +|----------|-----------------|-------| +| Julia | 1.7+ | Arrow.jl required for arrowtable support | +| Node.js | 16+ | nats.js required | +| Python | 3.8+ | pyarrow required for arrowtable support | +| MicroPython | 1.19+ | Limited to direct transport | + +--- + +## Out of Scope + +### Phase 1 (Current Implementation) + +| Feature | Reason | +|---------|--------| +| NATS JetStream support | Core NATS sufficient for current use cases | +| Message compression | Compression adds complexity without clear benefit | +| Message encryption | Payload encryption is application-layer concern | +| Persistent message queues | NATS request-reply pattern sufficient | +| Advanced routing rules | Simple NATS subject matching sufficient | + +### Future Considerations + +| Feature | Future Phase | +|---------|--------------| +| JetStream streams and consumers | Phase 2 | +| Message TTL and dead-letter queues | Phase 3 | +| Message tracing with OpenTelemetry | Phase 3 | +| Rate limiting and quota management | Phase 4 | + +--- + +## Boundary Definitions + +### What NATSBridge Handles + +| Function | Description | +|----------|-------------| +| Message serialization | Converts data types to binary format | +| Message encoding | Base64, JSON, Arrow IPC encoding | +| Transport selection | Direct vs link based on size threshold | +| NATS publishing | Publishes messages to NATS subjects | +| NATS subscription | Receives and processes NATS messages | +| File server upload | Uploads large payloads to HTTP server | +| File server download | Downloads payloads from HTTP server with retry | +| Correlation ID generation | Creates and propagates UUIDs | +| Data deserialization | Converts binary format back to native types | + +### What NATSBridge Does NOT Handle + +| Function | Handled By | +|----------|------------| +| NATS server management | External NATS deployment | +| File server management | External HTTP server deployment | +| Application business logic | Application code using NATSBridge | +| Message encryption | Application layer | +| Message compression | Application layer | +| Authentication/Authorization | NATS server configuration | + +--- + +## Payload Type Requirements + +### Supported Payload Types + +| Type | Julia | JavaScript | Python | MicroPython | Description | +|------|-------|------------|--------|-------------|-------------| +| `text` | `String` | `string` | `str` | `str` | Plain text strings | +| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable data | +| `arrowtable` | `DataFrame`, `Arrow.Table` | `Array` | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) | +| `jsontable` | `Vector{NamedTuple}` | `Array` | `list[dict]` | ⚠️ | Tabular data (JSON) | +| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image binary data | +| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio binary data | +| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video binary data | +| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `bytearray` | Generic binary data | + +### Encoding Requirements + +| Payload Type | Encoding Method | Notes | +|--------------|-----------------|-------| +| `text` | UTF-8 → Base64 | Text must be String type | +| `dictionary` | JSON → Base64 | JSON.jl for Julia | +| `arrowtable` | Arrow IPC → Base64 | Requires Arrow.jl/pyarrow | +| `jsontable` | JSON → Base64 | Human-readable format | +| `image`/`audio`/`video`/`binary` | Direct → Base64 | Binary data preserved | + +--- + +## Size Threshold Requirements + +### Direct Transport Threshold + +| Platform | Threshold | Notes | +|----------|-----------|-------| +| Desktop (Julia/JS/Python) | 1MB (1,048,576 bytes) | Default size threshold | +| MicroPython | 100KB (102,400 bytes) | Lower threshold for memory constraints | + +### Maximum Payload Size + +| Platform | Maximum | Notes | +|----------|---------|-------| +| Desktop | Unlimited | Limited by NATS server configuration | +| MicroPython | 50KB | Hard limit due to 256KB-1MB memory | + +--- + +## Message Envelope Requirements + +### Required Fields + +| Field | Type | Purpose | +|-------|------|---------| +| `correlation_id` | String (UUID) | Track message flow across systems | +| `msg_id` | String (UUID) | Unique message identifier | +| `timestamp` | String (ISO 8601) | Message publication timestamp | +| `send_to` | String | NATS subject to publish to | +| `msg_purpose` | String | ACK, NACK, updateStatus, shutdown, chat | +| `sender_name` | String | Sender application name | +| `sender_id` | String (UUID) | Sender unique identifier | +| `receiver_name` | String | Receiver application name (empty = broadcast) | +| `receiver_id` | String (UUID) | Receiver unique identifier (empty = broadcast) | +| `reply_to` | String | Topic for reply messages | +| `reply_to_msg_id` | String | Message ID being replied to | +| `broker_url` | String | NATS server URL | +| `metadata` | Dict | Message-level metadata | +| `payloads` | Array | List of payload objects | + +### Payload Fields + +| Field | Type | Purpose | +|-------|------|---------| +| `id` | String (UUID) | Unique payload identifier | +| `dataname` | String | Name of the payload | +| `payload_type` | String | Type: text, dictionary, arrowtable, etc. | +| `transport` | String | direct or link | +| `encoding` | String | none, json, base64, arrow-ipc | +| `size` | Integer | Payload size in bytes | +| `data` | Any | Base64 string or URL | +| `metadata` | Dict | Payload-level metadata | + +--- + +## Error Handling Requirements + +### Error Codes + +| Error | Condition | Response | +|-------|-----------|----------| +| `Unknown payload_type` | Unsupported type | Throw error | +| `Failed to upload` | File server error | Throw error | +| `Failed to fetch` | File server unavailable | Retry with exponential backoff | +| `Unknown transport` | Invalid transport type | Throw error | +| `NATS connection failed` | NATS unavailable | Throw error | + +### Exception Handling + +| Scenario | Handler | +|----------|---------| +| File server unavailable | Retry up to 5 times with exponential backoff | +| NATS publish failure | Connection auto-reconnect | +| Deserialization error | Log correlation ID and throw error | +| Memory overflow (MicroPython) | Reject payloads >50KB | + +--- + +## Testing Requirements + +### Unit Tests + +| Test Category | Coverage | Files | +|---------------|----------|-------| +| Serialization | All payload types | `test/test_*_sender.*` | +| Deserialization | All payload types | `test/test_*_receiver.*` | +| Transport selection | Direct vs link | `test/test_*_mix_payloads.*` | +| File server upload | Plik integration | Platform-specific | +| File server download | Exponential backoff | Platform-specific | + +### Integration Tests + +| Test Scenario | Success Criteria | +|-------------|-----------------| +| Cross-platform text message | Julia ↔ JavaScript ↔ Python | +| Cross-platform tabular data | Arrow IPC round-trip | +| Large file transfer | File server upload/download | +| Multi-payload mixed content | All payload types in one message | + +--- + +## API Contract + +### smartsend Signature + +```julia +function smartsend( + subject::String, + data::AbstractArray{Tuple{String, Any, String}}; + broker_url::String = "nats://localhost:4222", + fileserver_url::String = "http://localhost:8080", + fileserver_upload_handler::Function = plik_oneshot_upload, + size_threshold::Int = 1_000_000, + correlation_id::String = string(uuid4()), + msg_purpose::String = "chat", + sender_name::String = "NATSBridge", + receiver_name::String = "", + receiver_id::String = "", + reply_to::String = "", + reply_to_msg_id::String = "", + is_publish::Bool = true, + NATS_connection::Union{NATS.Connection, Nothing} = nothing, + msg_id::String = string(uuid4()), + sender_id::String = string(uuid4()) +)::Tuple{msg_envelope_v1, String} +``` + +### smartreceive Signature + +```julia +function smartreceive( + msg::NATS.Msg; + fileserver_download_handler::Function = _fetch_with_backoff, + max_retries::Int = 5, + base_delay::Int = 100, + max_delay::Int = 5000 +)::JSON.Object{String, Any} +``` + +--- + +## Dependencies + +### Required Dependencies + +| Platform | Package | Version | +|----------|---------|---------| +| Julia | NATS.jl | Latest stable | +| Julia | JSON.jl | Latest stable | +| Julia | Arrow.jl | Latest stable | +| Julia | HTTP.jl | Latest stable | +| Julia | UUIDs.jl | Latest stable | +| Node.js | nats | Latest stable | +| Node.js | node-fetch | Latest stable | +| Python | nats-py | Latest stable | +| Python | aiohttp | Latest stable | +| Python | pyarrow | Latest stable | + +### Optional Dependencies + +| Platform | Package | Use Case | +|----------|---------|----------| +| Julia | DataFrames.jl | DataFrame support for arrowtable | +| Python | pandas | DataFrame support for arrowtable | + +--- + +## Deployment Requirements + +### Minimum Infrastructure + +| Component | Minimum | Notes | +|-----------|---------|-------| +| NATS Server | 1 instance | Single node for development | +| File Server | 1 instance | HTTP server for large payloads | +| Client Memory | 50MB | Desktop platforms | +| Client Memory | 256KB | MicroPython devices | + +### Environment Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `NATS_URL` | `nats://localhost:4222` | NATS server URL | +| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL | +| `SIZE_THRESHOLD` | `1000000` | Size threshold in bytes | + +--- + +## Versioning + +### Current Version + +- **Major**: 1 (Breaking changes require major version bump) +- **Minor**: 0 (Feature additions) +- **Patch**: 0 (Bug fixes) + +### Version Compatibility + +| Version | Supported Platforms | +|---------|---------------------| +| v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, MicroPython 1.19+ | + +--- + +## Change Log + +| Date | Version | Changes | +|------|---------|---------| +| 2026-03-13 | 1.0.0 | Initial requirements document | + +--- + +## References + +- [`src/NATSBridge.jl`](../src/NATSBridge.jl) - Ground truth implementation +- [`README.md`](../README.md) - Project overview +- [`docs/architecture.md`](./architecture.md) - Architecture documentation +- [`docs/implementation.md`](./implementation.md) - Implementation details +- [`docs/walkthrough.md`](./walkthrough.md) - Usage examples \ No newline at end of file