ton/NATSBridge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity
Files
b0c5ecb9423f1b2585763176d2e19200beaef6d3
NATSBridge/docs/requirements.md
narawat 1b38b3d6f1 update requirement doc
2026-03-23 11:52:38 +07:00

18 KiB
Raw Blame History

Requirements Document: NATSBridge

Version: 1.0.0
Date: 2026-03-23
Status: Active
Ground Truth: src/NATSBridge.jl

1. Business Context & Success Metrics

1.1 Business Goal

NATSBridge is a cross-platform, bi-directional data bridge that enables seamless communication between Julia, JavaScript, Python, and MicroPython applications using NATS as the message bus. The system implements the Claim-Check pattern for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS.

1.2 User Stories (with acceptance criteria)

Story Priority Acceptance Criteria
As a Julia developer, I want to send text messages to JavaScript applications that lives on a server and also on a browser P1 Text messages are serialized, encoded, and received correctly across platforms
As a Python developer, I want to send tabular data to Julia applications P1 DataFrame exchange works with both Arrow IPC and JSON formats
As a JavaScript developer, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications P1 Large files are automatically uploaded to file server and URLs are sent via NATS
As a MicroPython developer, I want to send sensor data with minimal memory usage P1 Direct transport works for payloads <100KB on memory-constrained devices
As a developer, I want to send mixed-content messages (text + image + file) P1 NATSBridge accepts list of (dataname, data, type) tuples and handles each payload appropriately
As a developer, I want to receive multi-payload messages P1 NATSBridge returns payloads as list of tuples with correct types preserved
As a developer, I want to use Plik as the file server P2 Plik one-shot upload mode is supported with upload ID and token handling
As a developer, I want to use custom HTTP file servers P2 Handler function abstraction allows plugging in AWS S3 or custom implementations
As a developer, I want automatic retry on file server download failures P1 Exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms)
As a developer, I want message tracing across distributed systems P1 Correlation ID is propagated through all message processing steps

1.3 KPIs & Targets

Metric Target Measurement Method
95% of messages complete within 200ms 95% Synthetic monitoring
<2 days from onboarding to first PR 2 days PR timeline tracking
100% of messages validate against spec 100% CI block rate
>80% unit test coverage 80% Test coverage tools
<1% of PRs bypass validation gates 1% CI gate analysis
MTTR <15 minutes for P1 incidents 15 minutes Incident tracking

2. Technical Boundaries

2.1 In Scope

Feature Description
Cross-platform interoperability Seamless data exchange between Julia, JavaScript, Python, and MicroPython
Intelligent transport selection Direct transport (<0.5MB) vs Link transport (≥0.5MB) based on payload size
Unified API Consistent smartsend() and smartreceive() functions across all platforms
Multi-payload support List of (dataname, data, type) tuples with appropriate handling
File server integration Plik one-shot upload and custom HTTP server support
Reliability features Exponential backoff retry and correlation ID propagation
Message serialization Converts data types to binary format (Base64, JSON, Arrow IPC)
NATS communication Publishing and subscription via NATS subjects

2.2 Out of Scope

Feature Reason
NATS JetStream support Core NATS sufficient for current use cases
Message compression Compression adds complexity without clear benefit
Message encryption Payload encryption is application-layer concern
Persistent message queues NATS request-reply pattern sufficient
Advanced routing rules Simple NATS subject matching sufficient

2.3 Dependencies

Platform Package Version
Julia NATS.jl Latest stable
Julia JSON.jl Latest stable
Julia Arrow.jl Latest stable
Julia HTTP.jl Latest stable
Julia UUIDs.jl Latest stable
Node.js nats Latest stable
Node.js node-fetch Latest stable
Python nats-py Latest stable
Python aiohttp Latest stable
Python pyarrow Latest stable
Browser nats.ws Latest stable

2.4 Platform Compatibility

Platform Minimum Version Notes
Julia 1.7+ Arrow.jl required for arrowtable support
Node.js 16+ nats.js required, Arrow IPC supported
Python 3.8+ pyarrow required for arrowtable support
Browser Latest No Arrow IPC (uses jsontable only)
MicroPython 1.19+ Limited to direct transport

3. Functional Requirements (FR)

ID Requirement Description
FR-001 Cross-platform text messaging System shall allow users to send text messages between Julia, JavaScript, Python, and MicroPython applications
FR-002 Cross-platform tabular data System shall support DataFrame exchange between Julia and Python applications using Arrow IPC format
FR-003 Large file handling System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via NATS
FR-004 Direct transport for small payloads System shall send payloads <0.5MB directly via NATS without file server upload
FR-005 MicroPython support System shall support payloads <100KB on MicroPython devices using direct transport
FR-006 Multi-payload messages System shall accept and process lists of (dataname, data, type) tuples
FR-007 Payload type preservation System shall preserve payload types when returning multi-payload messages
FR-008 Plik file server integration System shall support Plik one-shot upload mode with upload ID and token handling
FR-009 Custom file server support System shall provide handler function abstraction for custom HTTP file server implementations
FR-010 Exponential backoff retry System shall implement exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) for file server download failures
FR-011 Correlation ID propagation System shall propagate correlation IDs through all message processing steps
FR-012 Message serialization System shall serialize data types using Base64, JSON, or Arrow IPC encoding
FR-013 NATS publishing System shall publish messages to NATS subjects
FR-014 NATS subscription System shall receive and process NATS messages

4. Non-Functional Requirements (NFRs)

4.1 Performance & Scalability

ID Requirement Specification Test Method
NFR-101 Message serialization overhead <50ms for 10KB payload Benchmark tests
NFR-102 Message deserialization overhead <50ms for 10KB payload Benchmark tests
NFR-103 NATS connection establishment <100ms Connection pool benchmarks
NFR-104 File upload latency <1s for 0.5MB file Integration tests
NFR-105 File download latency <1s for 0.5MB file Integration tests
NFR-106 Concurrent connections Support 100+ simultaneous NATS connections Scale testing
NFR-107 Message throughput Handle 1000+ messages/second per instance Load testing
NFR-108 File server scalability Support horizontal scaling of file server backend Architecture review

4.2 Availability & Reliability

ID Requirement Specification
NFR-201 Message delivery At-least-once delivery semantics via NATS
NFR-202 File server availability Graceful degradation when file server is unavailable
NFR-203 Connection recovery Auto-reconnect on NATS connection failure

4.3 Privacy & Security

ID Requirement Specification
NFR-301 Payload integrity SHA-256 checksum support via metadata
NFR-302 Transport security TLS support for NATS connections
NFR-303 File server security Authentication token for file uploads

4.4 Observability & Telemetry

ID Requirement Specification
NFR-401 Required logs correlation_id, msg_id, timestamp, sender_name, receiver_name, payload_type, transport
NFR-402 Critical metrics messages_sent_total, messages_received_total, file_upload_duration_seconds, file_download_duration_seconds, retry_attempts_total
NFR-403 Tracing Correlation ID propagation for request tracing
NFR-404 Alerting download_retry_exceeded triggers alert when max retries exceeded
NFR-405 Retention Logs: 30 days, Metrics: 1 year

5. Acceptance Conditions

Condition Description
AC-001 All functional requirements FR-001 through FR-014 are implemented and tested
AC-002 All non-functional requirements NFR-101 through NFR-405 meet specified targets
AC-003 Cross-platform text message test passes (Julia ↔ JavaScript ↔ Python)
AC-004 Cross-platform tabular data test passes with Arrow IPC round-trip (Desktop)
AC-005 Cross-platform tabular data test passes with JSON table round-trip (Browser)
AC-006 Large file transfer test passes with file server upload/download
AC-007 Multi-payload mixed content test passes with all payload types in one message
AC-008 CI validation gates block PRs on specification violations
AC-009 Unit test coverage exceeds 80%
AC-010 Documentation is complete and includes walkthroughs, architecture, and runbook

6. Payload Type Requirements

6.1 Supported Payload Types

Type Julia JavaScript Python MicroPython Description
text String string str str Plain text strings
dictionary Dict, NamedTuple Object, Array dict, list dict JSON-serializable data
arrowtable DataFrame, Arrow.Table ❌ (Browser), ✅ (Node.js) pandas.DataFrame ❌ Tabular data (Arrow IPC)
jsontable Vector{NamedTuple} Array<Object> list[dict] ⚠️ Tabular data (JSON) - Only table type in Browser
image Vector{UInt8} Uint8Array, Buffer bytes bytearray Image binary data
audio Vector{UInt8} Uint8Array, Buffer bytes bytearray Audio binary data
video Vector{UInt8} Uint8Array, Buffer bytes bytearray Video binary data
binary Vector{UInt8}, IOBuffer Uint8Array, Buffer bytes, bytearray bytearray Generic binary data

6.2 Encoding Requirements

Payload Type Encoding Method Notes
text UTF-8 → Base64 Text must be String type
dictionary JSON → Base64 JSON.jl for Julia
arrowtable Arrow IPC → Base64 Requires Arrow.jl/pyarrow (Desktop only)
jsontable JSON → Base64 Human-readable format - Browser uses this only
image/audio/video/binary Direct → Base64 Binary data preserved

7. Size Threshold Requirements

7.1 Direct Transport Threshold

Platform Threshold Notes
Desktop (Julia/JS/Python) 0.5MB Default size threshold
MicroPython 100KB Lower threshold for memory constraints

7.2 Maximum Payload Size

Platform Maximum Notes
Desktop Unlimited Limited by NATS server configuration
MicroPython 50KB Hard limit due to 256KB-1MB memory

8. Message Envelope Requirements

8.1 Required Fields

Field Type Purpose
correlation_id String (UUID) Track message flow across systems
msg_id String (UUID) Unique message identifier
timestamp String (ISO 8601) Message publication timestamp
send_to String NATS subject to publish to
msg_purpose String ACK, NACK, updateStatus, shutdown, chat
sender_name String Sender application name
sender_id String (UUID) Sender unique identifier
receiver_name String Receiver application name (empty = broadcast)
receiver_id String (UUID) Receiver unique identifier (empty = broadcast)
reply_to String Topic for reply messages
reply_to_msg_id String Message ID being replied to
broker_url String NATS server URL
metadata Dict Message-level metadata
payloads Array List of payload objects

8.2 Payload Fields

Field Type Purpose
id String (UUID) Unique payload identifier
dataname String Name of the payload
payload_type String Type: text, dictionary, arrowtable, etc.
transport String direct or link
encoding String none, json, base64, arrow-ipc
size Integer Payload size in bytes
data Any Base64 string or URL
metadata Dict Payload-level metadata

9. Error Handling Requirements

9.1 Error Codes

Error Condition Response
Unknown payload_type Unsupported type Throw error
Failed to upload File server error Throw error
Failed to fetch File server unavailable Retry with exponential backoff
Unknown transport Invalid transport type Throw error
NATS connection failed NATS unavailable Throw error

9.2 Exception Handling

Scenario Handler
File server unavailable Retry up to 5 times with exponential backoff
NATS publish failure Connection auto-reconnect
Deserialization error Log correlation ID and throw error
Memory overflow (MicroPython) Reject payloads >50KB

10. Testing Requirements

10.1 Unit Tests

Test Category Coverage Files
Serialization All payload types test/test_*_sender.*
Deserialization All payload types test/test_*_receiver.*
Transport selection Direct vs link test/test_*_mix_payloads.*
File server upload Plik integration Platform-specific
File server download Exponential backoff Platform-specific

10.2 Integration Tests

Test Scenario Success Criteria
Cross-platform text message Julia ↔ JavaScript ↔ Python
Cross-platform tabular data (Desktop) Arrow IPC round-trip
Cross-platform tabular data (Browser) JSON table round-trip
Large file transfer File server upload/download
Multi-payload mixed content All payload types in one message

11. API Contract

11.1 smartsend Signature

function smartsend(
    subject::String,
    data::AbstractArray{Tuple{String, Any, String}};
    broker_url::String = "nats://localhost:4222",
    fileserver_url::String = "http://localhost:8080",
    fileserver_upload_handler::Function = plik_oneshot_upload,
    size_threshold::Int = 1_000_000,
    correlation_id::String = string(uuid4()),
    msg_purpose::String = "chat",
    sender_name::String = "NATSBridge",
    receiver_name::String = "",
    receiver_id::String = "",
    reply_to::String = "",
    reply_to_msg_id::String = "",
    is_publish::Bool = true,
    NATS_connection::Union{NATS.Connection, Nothing} = nothing,
    msg_id::String = string(uuid4()),
    sender_id::String = string(uuid4())
)::Tuple{msg_envelope_v1, String}

11.2 smartreceive Signature

function smartreceive(
    msg::NATS.Msg;
    fileserver_download_handler::Function = _fetch_with_backoff,
    max_retries::Int = 5,
    base_delay::Int = 100,
    max_delay::Int = 5000
)::JSON.Object{String, Any}

12. Deployment Requirements

12.1 Minimum Infrastructure

Component Minimum Notes
NATS Server 1 instance Single node for development
File Server 1 instance HTTP server for large payloads
Client Memory 50MB Desktop platforms
Client Memory 256KB MicroPython devices

12.2 Environment Variables

Variable Default Description
NATS_URL nats://localhost:4222 NATS server URL
FILESERVER_URL http://localhost:8080 HTTP file server URL
SIZE_THRESHOLD 1000000 Size threshold in bytes

13. Versioning

13.1 Current Version

  • Major: 1 (Breaking changes require major version bump)
  • Minor: 0 (Feature additions)
  • Patch: 0 (Bug fixes)

13.2 Version Compatibility

Version Supported Platforms
v1.0.x Julia 1.7+, Node.js 16+, Python 3.8+, Browser (latest), MicroPython 1.19+

14. Change Log

Date Version Changes
2026-03-23 1.0.0 Updated to ASG Framework requirements structure

15. References

Reference in New Issue View Git Blame Copy Permalink