ton/NATSBridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity
Files
c5a70edd57221d9bc3fcf0aa87ac6e6540e459ea
NATSBridge/docs/requirements.md
narawat c5a70edd57 rust version implemented
2026-05-13 20:24:08 +07:00

21 KiB
Raw Blame History

Requirements Document: NATSBridge

Version: 1.2.0
Date: 2026-05-13
Status: Active
Ground Truth: src/NATSBridge.jl

1. Business Context & Success Metrics

1.1 Business Goal

NATSBridge is a cross-platform, bi-directional data bridge that enables seamless communication between Julia, JavaScript, Python, Dart, Rust, and MicroPython applications using NATS as the message bus. The system implements the Claim-Check pattern for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS.

1.2 User Stories (with acceptance criteria)

Story Priority Acceptance Criteria
As a Julia developer, I want to send text messages to JavaScript/Dart applications that lives on a server and also on a browser P1 Text messages are serialized, encoded, and received correctly across platforms
As a Python developer, I want to send tabular data to Julia/Dart applications P1 DataFrame exchange works with both Arrow IPC and JSON formats
As a JavaScript developer, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications P1 Large files are automatically uploaded to file server and URLs are sent via NATS
As a Dart developer, I want to send text messages to other platforms P1 Text messages are serialized, encoded, and received correctly across platforms
As a Dart developer, I want to send dictionary data to other platforms P1 JSON-serializable data is exchanged correctly
As a Dart developer, I want to send tabular data (List) to other platforms P1 JSON table format exchange works with Arrow IPC on desktop
As a Dart developer, I want to send large files (>0.5MB) P1 Large files are automatically uploaded to file server and URLs are sent via NATS
As a MicroPython developer, I want to send sensor data with minimal memory usage P1 Direct transport works for payloads <100KB on memory-constrained devices
As a Rust developer, I want to send and receive messages with type-safe APIs P1 Rust implementation uses serde for serialization, tokio for async, and nats-io for NATS connectivity
As a developer, I want to send mixed-content messages (text + image + file) P1 NATSBridge accepts list of (dataname, data, type) tuples and handles each payload appropriately
As a developer, I want to receive multi-payload messages P1 NATSBridge returns payloads as list of tuples with correct types preserved
As a developer, I want to use Plik as the file server P2 Plik one-shot upload mode is supported with upload ID and token handling
As a developer, I want to use custom HTTP file servers P2 Handler function abstraction allows plugging in AWS S3 or custom implementations
As a developer, I want automatic retry on file server download failures P1 Exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms)
As a developer, I want message tracing across distributed systems P1 Correlation ID is propagated through all message processing steps

1.3 KPIs & Targets

Metric Target Measurement Method
95% of messages complete within 200ms 95% Synthetic monitoring
<2 days from onboarding to first PR 2 days PR timeline tracking
100% of messages validate against spec 100% CI block rate
>80% unit test coverage 80% Test coverage tools
<1% of PRs bypass validation gates 1% CI gate analysis
MTTR <15 minutes for P1 incidents 15 minutes Incident tracking

2. Technical Boundaries

2.1 In Scope

Feature Description
Cross-platform interoperability Seamless data exchange between Julia, JavaScript, Python, Dart, Rust, and MicroPython
Intelligent transport selection Direct transport (<0.5MB) vs Link transport (≥0.5MB) based on payload size
Unified API Consistent smartsend() and smartreceive() functions across all platforms
Multi-payload support List of (dataname, data, type) tuples with appropriate handling
File server integration Plik one-shot upload and custom HTTP server support
Reliability features Exponential backoff retry and correlation ID propagation
Message serialization Converts data types to binary format (Base64, JSON, Arrow IPC)
NATS communication Publishing and subscription via NATS subjects

2.2 Out of Scope

Feature Reason
NATS JetStream support Core NATS sufficient for current use cases
Message compression Compression adds complexity without clear benefit
Message encryption Payload encryption is application-layer concern
Persistent message queues NATS request-reply pattern sufficient
Advanced routing rules Simple NATS subject matching sufficient

2.3 Dependencies

Platform Package Version
Julia NATS.jl Latest stable
Julia JSON.jl Latest stable
Julia Arrow.jl Latest stable
Julia HTTP.jl Latest stable
Julia UUIDs.jl Latest stable
Node.js nats Latest stable
Node.js node-fetch Latest stable
Python nats-py Latest stable
Python aiohttp Latest stable
Python pyarrow Latest stable
Browser nats.ws Latest stable
Dart nats Latest stable
Dart http Latest stable
Dart uuid Latest stable
Rust nats Latest stable
Rust serde Latest stable
Rust serde_json Latest stable
Rust tokio Latest stable
Rust uuid Latest stable

2.4 Platform Compatibility

Platform Minimum Version Notes
Julia 1.7+ Arrow.jl required for arrowtable support
Node.js 16+ nats.js required, Arrow IPC supported
Python 3.8+ pyarrow required for arrowtable support
Browser Latest No Arrow IPC (uses jsontable only)
Dart 2.17+ Supports Desktop (Dart SDK), Flutter (Dart SDK), and Web (Dart SDK)
Rust 1.70+ Full support with async/await, Arrow IPC on desktop
MicroPython 1.19+ Limited to direct transport

3. Functional Requirements (FR)

ID Requirement Description
FR-001 Cross-platform text messaging System shall allow users to send text messages between Julia, JavaScript, Python, and MicroPython applications
FR-002 Cross-platform tabular data System shall support DataFrame exchange between Julia and Python applications using Arrow IPC format
FR-003 Large file handling System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via NATS
FR-004 Direct transport for small payloads System shall send payloads <0.5MB directly via NATS without file server upload
FR-005 MicroPython support System shall support payloads <100KB on MicroPython devices using direct transport
FR-006 Multi-payload messages System shall accept and process lists of (dataname, data, type) tuples
FR-007 Payload type preservation System shall preserve payload types when returning multi-payload messages
FR-008 Plik file server integration System shall support Plik one-shot upload mode with upload ID and token handling
FR-009 Custom file server support System shall provide handler function abstraction for custom HTTP file server implementations
FR-010 Exponential backoff retry System shall implement exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) for file server download failures
FR-011 Correlation ID propagation System shall propagate correlation IDs through all message processing steps
FR-012 Message serialization System shall serialize data types using Base64, JSON, or Arrow IPC encoding
FR-013 NATS publishing System shall return JSON string representation for caller to publish to NATS subjects (caller is responsible for actual NATS publish)
FR-014 NATS subscription System shall receive and process NATS messages by accepting JSON string from NATS payload

4. Non-Functional Requirements (NFRs)

4.1 Performance & Scalability

ID Requirement Specification Test Method
NFR-101 Message serialization overhead <50ms for 10KB payload Benchmark tests
NFR-102 Message deserialization overhead <50ms for 10KB payload Benchmark tests
NFR-103 NATS connection establishment <100ms Connection pool benchmarks
NFR-104 File upload latency <1s for 0.5MB file Integration tests
NFR-105 File download latency <1s for 0.5MB file Integration tests
NFR-106 Concurrent connections Support 100+ simultaneous NATS connections Scale testing
NFR-107 Message throughput Handle 1000+ messages/second per instance Load testing
NFR-108 File server scalability Support horizontal scaling of file server backend Architecture review

4.2 Availability & Reliability

ID Requirement Specification
NFR-201 Message delivery At-least-once delivery semantics via NATS
NFR-202 File server availability Graceful degradation when file server is unavailable
NFR-203 Connection recovery Auto-reconnect on NATS connection failure

4.3 Privacy & Security

ID Requirement Specification
NFR-301 Payload integrity SHA-256 checksum support via metadata
NFR-302 Transport security TLS support for NATS connections
NFR-303 File server security Authentication token for file uploads

4.4 Observability & Telemetry

ID Requirement Specification
NFR-401 Required logs correlation_id, msg_id, timestamp, sender_name, receiver_name, payload_type, transport
NFR-402 Critical metrics messages_sent_total, messages_received_total, file_upload_duration_seconds, file_download_duration_seconds, retry_attempts_total
NFR-403 Tracing Correlation ID propagation for request tracing
NFR-404 Alerting download_retry_exceeded triggers alert when max retries exceeded
NFR-405 Retention Logs: 30 days, Metrics: 1 year

5. Acceptance Conditions

Condition Description
AC-001 All functional requirements FR-001 through FR-014 are implemented and tested
AC-002 All non-functional requirements NFR-101 through NFR-405 meet specified targets
AC-003 Cross-platform text message test passes (Julia ↔ JavaScript ↔ Python)
AC-004 Cross-platform tabular data test passes with Arrow IPC round-trip (Desktop)
AC-005 Cross-platform tabular data test passes with JSON table round-trip (Browser)
AC-006 Large file transfer test passes with file server upload/download
AC-007 Multi-payload mixed content test passes with all payload types in one message
AC-008 CI validation gates block PRs on specification violations
AC-009 Unit test coverage exceeds 80%
AC-010 Documentation is complete and includes walkthroughs, architecture, and runbook

6. Payload Type Requirements

6.1 Supported Payload Types

Type Julia JavaScript Python Dart MicroPython Description
text String string str String String str
dictionary Dict, NamedTuple Object, Array dict, list Map, serde_json::Value String dict
arrowtable DataFrame, Arrow.Table ❌ (Browser), ✅ (Node.js) pandas.DataFrame List<Map> (Desktop), List<dynamic> (Flutter) arrow2::Table ❌
jsontable Vector{NamedTuple} Array<Object> list[dict] Vec<Map> ⚠️ Tabular data (JSON) - Only table type in Browser
image Vector{UInt8} Uint8Array, Buffer bytes Uint8List Vec<u8> bytearray
audio Vector{UInt8} Uint8Array, Buffer bytes Uint8List Vec<u8> bytearray
video Vector{UInt8} Uint8Array, Buffer bytes Uint8List Vec<u8> bytearray
binary Vector{UInt8}, IOBuffer Uint8Array, Buffer bytes, bytearray Uint8List Vec<u8> bytearray

6.2 Encoding Requirements

Payload Type Encoding Method Notes
text UTF-8 → Base64 Text must be String type
dictionary JSON → Base64 JSON.jl for Julia
arrowtable Arrow IPC → Base64 Requires Arrow.jl/pyarrow (Desktop only)
jsontable JSON → Base64 Human-readable format - Browser uses this only
image/audio/video/binary Direct → Base64 Binary data preserved

7. Size Threshold Requirements

7.1 Direct Transport Threshold

Platform Threshold Notes
Desktop (Julia/JS/Python/Dart) 0.5MB Default size threshold
Dart Desktop 0.5MB Default size threshold
Dart Flutter 0.5MB Default size threshold
Dart Web 0.5MB Default size threshold
Rust 0.5MB Default size threshold
MicroPython 100KB Lower threshold for memory constraints

7.2 Maximum Payload Size

Platform Maximum Notes
Desktop Unlimited Limited by NATS server configuration
Dart Desktop Unlimited Limited by NATS server configuration
Dart Flutter Unlimited Limited by NATS server configuration
Dart Web Unlimited Limited by NATS server configuration
Rust Unlimited Limited by NATS server configuration
MicroPython 50KB Hard limit due to 256KB-1MB memory

8. Message Envelope Requirements

8.1 Required Fields

Field Type Purpose
correlation_id String (UUID) Track message flow across systems
msg_id String (UUID) Unique message identifier
timestamp String (ISO 8601) Message publication timestamp
send_to String NATS subject to publish to
msg_purpose String ACK, NACK, updateStatus, shutdown, chat
sender_name String Sender application name
sender_id String (UUID) Sender unique identifier
receiver_name String Receiver application name (empty = broadcast)
receiver_id String (UUID) Receiver unique identifier (empty = broadcast)
reply_to String Topic for reply messages
reply_to_msg_id String Message ID being replied to
broker_url String NATS server URL
metadata Dict Message-level metadata
payloads Array List of payload objects

8.2 Payload Fields

Field Type Purpose
id String (UUID) Unique payload identifier
dataname String Name of the payload
payload_type String Type: text, dictionary, arrowtable, etc.
transport String direct or link
encoding String none, json, base64, arrow-ipc
size Integer Payload size in bytes
data Any Base64 string or URL
metadata Dict Payload-level metadata

9. Error Handling Requirements

9.1 Error Codes

Error Condition Response
Unknown payload_type Unsupported type Throw error
Failed to upload File server error Throw error
Failed to fetch File server unavailable Retry with exponential backoff
Unknown transport Invalid transport type Throw error
NATS connection failed NATS unavailable Throw error

9.2 Exception Handling

Scenario Handler
File server unavailable Retry up to 5 times with exponential backoff
NATS publish failure Connection auto-reconnect
Deserialization error Log correlation ID and throw error
Memory overflow (MicroPython) Reject payloads >50KB

10. Testing Requirements

10.1 Unit Tests

Test Category Coverage Files
Serialization All payload types test/test_*_sender.*
Deserialization All payload types test/test_*_receiver.*
Transport selection Direct vs link test/test_*_mix_payloads.*
File server upload Plik integration Platform-specific
File server download Exponential backoff Platform-specific

10.2 Integration Tests

Test Scenario Success Criteria
Cross-platform text message Julia ↔ JavaScript ↔ Python
Cross-platform tabular data (Desktop) Arrow IPC round-trip
Cross-platform tabular data (Browser) JSON table round-trip
Large file transfer File server upload/download
Multi-payload mixed content All payload types in one message

11. API Contract

11.1 smartsend Signature

function smartsend(
    subject::String,
    data::AbstractArray{Tuple{String, T1, String}, 1};
    broker_url::String = DEFAULT_BROKER_URL,
    fileserver_url::String = DEFAULT_FILESERVER_URL,
    fileserver_upload_handler::Function = plik_oneshot_upload,
    size_threshold::Int = DEFAULT_SIZE_THRESHOLD,
    correlation_id::String = string(uuid4()),
    msg_purpose::String = "chat",
    sender_name::String = "NATSBridge",
    receiver_name::String = "",
    receiver_id::String = "",
    reply_to::String = "",
    reply_to_msg_id::String = "",
    msg_id::String = string(uuid4()),
    sender_id::String = string(uuid4())
)::Tuple{msg_envelope_v1, String} where {T1<:Any}

Note: NATS publishing is the caller's responsibility. smartsend returns (env::msg_envelope_v1, env_json_str::String).

11.2 smartreceive Signature

function smartreceive(
    msg_json_str::String;
    fileserver_download_handler::Function = _fetch_with_backoff,
    max_retries::Int = 5,
    base_delay::Int = 100,
    max_delay::Int = 5000
)::JSON.Object{String, Any}

Note: Pass String(nats_msg.payload) from NATS subscription to smartreceive.

12. Deployment Requirements

12.1 Minimum Infrastructure

Component Minimum Notes
NATS Server 1 instance Single node for development
File Server 1 instance HTTP server for large payloads
Client Memory 50MB Desktop platforms (Julia/JS/Python/Dart)
Client Memory 256KB MicroPython devices

12.2 Environment Variables

Variable Default Description
NATS_URL nats://localhost:4222 NATS server URL
FILESERVER_URL http://localhost:8080 HTTP file server URL
SIZE_THRESHOLD 500000 Size threshold in bytes (0.5MB)

13. Versioning

13.1 Current Version

  • Major: 1 (Breaking changes require major version bump)
  • Minor: 0 (Feature additions)
  • Patch: 0 (Bug fixes)

13.2 Version Compatibility

Version Supported Platforms
v1.0.x Julia 1.7+, Node.js 16+, Python 3.8+, Dart 2.17+, Rust 1.70+, Browser (latest), MicroPython 1.19+

14. Change Log

Date Version Changes
2026-05-13 1.2.0 Aligned with ground truth implementation (src/NATSBridge.jl)
- - Fixed smartsend signature: removed is_publish, NATS_connection; added sender_name
- - Fixed smartreceive signature: takes msg_json_str::String instead of msg::NATS.Msg
- - Fixed size_threshold default from 1,000,000 to 500,000
- - Updated FR-013/FR-014 to reflect caller responsibility for NATS publishing
- - Updated FR-008/FR-009 to include file path upload overload
- - Updated SIZE_THRESHOLD env var default to 500000
2026-03-23 1.0.0 Updated to ASG Framework requirements structure

15. References

Reference in New Issue View Git Blame Copy Permalink