ton/NATSBridge

Fork 0

Files

narawat 49d7898720 remove arrow support for natsbridge_csr.js

2026-03-15 11:58:08 +07:00

16 KiB

Raw Blame History

Requirements Document: NATSBridge

Version: 1.0.0
Date: 2026-03-13
Status: Active
Ground Truth: src/NATSBridge.jl

Executive Summary

NATSBridge is a cross-platform, bi-directional data bridge that enables seamless communication between Julia, JavaScript, Python, and MicroPython applications using NATS as the message bus. The system implements the Claim-Check pattern for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS.

Business Goals

Primary Objectives

Cross-Platform Interoperability: Enable seamless data exchange between Julia, JavaScript (for both Server-Side rendering and Client-Side rendering webapp), Python, and MicroPython applications without platform-specific barriers.
Efficient Large Payload Handling: Implement intelligent transport selection based on payload size:
- Direct Transport: Small payloads (<0.5MB) sent directly via NATS
- Link Transport: Large payloads (≥0.5MB) uploaded to HTTP file server, URL sent via NATS
Unified API Across Platforms: Provide consistent smartsend() and smartreceive() functions across all supported platforms while maintaining idiomatic implementations.
Developer Productivity: Reduce onboarding time and simplify integration through comprehensive documentation and test examples.

Success Metrics

Metric	Target	Measurement Method
95% of messages complete within 200ms	95%	Synthetic monitoring
<2 days from onboarding to first PR	2 days	PR timeline tracking
100% of messages validate against spec	100%	CI block rate
>80% unit test coverage	80%	Test coverage tools
<1% of PRs bypass validation gates	1%	CI gate analysis
MTTR <15 minutes for P1 incidents	15 minutes	Incident tracking

User Stories

Core Functionality

Story	Priority	Acceptance Criteria
As a Julia developer, I want to send text messages to JavaScript applications that lives on a server and also on a browser	P1	Text messages are serialized, encoded, and received correctly across platforms
As a Python developer, I want to send tabular data to Julia applications	P1	DataFrame exchange works with both Arrow IPC and JSON formats
As a JavaScript developer, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications	P1	Large files are automatically uploaded to file server and URLs are sent via NATS
As a MicroPython developer, I want to send sensor data with minimal memory usage	P1	Direct transport works for payloads <100KB on memory-constrained devices

Multi-Payload Support

Story	Priority	Acceptance Criteria
As a developer, I want to send mixed-content messages (text + image + file)	P1	NATSBridge accepts list of (dataname, data, type) tuples and handles each payload appropriately
As a developer, I want to receive multi-payload messages	P1	NATSBridge returns payloads as list of tuples with correct types preserved

File Server Integration

Story	Priority	Acceptance Criteria
As a developer, I want to use Plik as the file server	P2	Plik one-shot upload mode is supported with upload ID and token handling
As a developer, I want to use custom HTTP file servers	P2	Handler function abstraction allows plugging in AWS S3 or custom implementations

Reliability Features

Story	Priority	Acceptance Criteria
As a developer, I want automatic retry on file server download failures	P1	Exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms)
As a developer, I want message tracing across distributed systems	P1	Correlation ID is propagated through all message processing steps

Non-Functional Requirements

Performance Requirements

Requirement	Specification	Test Method
Message serialization overhead	<50ms for 10KB payload	Benchmark tests
Message deserialization overhead	<50ms for 10KB payload	Benchmark tests
NATS connection establishment	<100ms	Connection pool benchmarks
File upload latency	<1s for 0.5MB file	Integration tests
File download latency	<1s for 0.5MB file	Integration tests

Scalability Requirements

Requirement	Specification
Concurrent connections	Support 100+ simultaneous NATS connections
Message throughput	Handle 1000+ messages/second per instance
File server scalability	Support horizontal scaling of file server backend

Reliability Requirements

Requirement	Specification
Message delivery	At-least-once delivery semantics via NATS
File server availability	Graceful degradation when file server is unavailable
Connection recovery	Auto-reconnect on NATS connection failure

Security Requirements

Requirement	Specification
Payload integrity	SHA-256 checksum support via metadata
Transport security	TLS support for NATS connections
File server security	Authentication token for file uploads

Compatibility Requirements

Platform	Minimum Version	Notes
Julia	1.7+	Arrow.jl required for arrowtable support
Node.js	16+	nats.js required, Arrow IPC supported
Python	3.8+	pyarrow required for arrowtable support
Browser	Latest	No Arrow IPC (uses jsontable only)
MicroPython	1.19+	Limited to direct transport

Out of Scope

Phase 1 (Current Implementation)

Feature	Reason
NATS JetStream support	Core NATS sufficient for current use cases
Message compression	Compression adds complexity without clear benefit
Message encryption	Payload encryption is application-layer concern
Persistent message queues	NATS request-reply pattern sufficient
Advanced routing rules	Simple NATS subject matching sufficient

Future Considerations

Feature	Future Phase
JetStream streams and consumers	Phase 2
Message TTL and dead-letter queues	Phase 3
Message tracing with OpenTelemetry	Phase 3
Rate limiting and quota management	Phase 4

Boundary Definitions

What NATSBridge Handles

Function	Description
Message serialization	Converts data types to binary format
Message encoding	Base64, JSON, Arrow IPC encoding
Transport selection	Direct vs link based on size threshold
NATS publishing	Publishes messages to NATS subjects
NATS subscription	Receives and processes NATS messages
File server upload	Uploads large payloads to HTTP server
File server download	Downloads payloads from HTTP server with retry
Correlation ID generation	Creates and propagates UUIDs
Data deserialization	Converts binary format back to native types

What NATSBridge Does NOT Handle

Function	Handled By
NATS server management	External NATS deployment
File server management	External HTTP server deployment
Application business logic	Application code using NATSBridge
Message encryption	Application layer
Message compression	Application layer
Authentication/Authorization	NATS server configuration

Payload Type Requirements

Supported Payload Types

Type	Julia	JavaScript	Python	MicroPython	Description
`text`	`String`	`string`	`str`	`str`	Plain text strings
`dictionary`	`Dict`, `NamedTuple`	`Object`, `Array`	`dict`, `list`	`dict`	JSON-serializable data
`arrowtable`	`DataFrame`, `Arrow.Table`	❌ (Browser), ✅ (Node.js)	`pandas.DataFrame`	❌	Tabular data (Arrow IPC)
`jsontable`	`Vector{NamedTuple}`	`Array<Object>`	`list[dict]`	⚠️	Tabular data (JSON) - Only table type in Browser
`image`	`Vector{UInt8}`	`Uint8Array`, `Buffer`	`bytes`	`bytearray`	Image binary data
`audio`	`Vector{UInt8}`	`Uint8Array`, `Buffer`	`bytes`	`bytearray`	Audio binary data
`video`	`Vector{UInt8}`	`Uint8Array`, `Buffer`	`bytes`	`bytearray`	Video binary data
`binary`	`Vector{UInt8}`, `IOBuffer`	`Uint8Array`, `Buffer`	`bytes`, `bytearray`	`bytearray`	Generic binary data

Encoding Requirements

Payload Type	Encoding Method	Notes
`text`	UTF-8 → Base64	Text must be String type
`dictionary`	JSON → Base64	JSON.jl for Julia
`arrowtable`	Arrow IPC → Base64	Requires Arrow.jl/pyarrow (Desktop only)
`jsontable`	JSON → Base64	Human-readable format - Browser uses this only
`image`/`audio`/`video`/`binary`	Direct → Base64	Binary data preserved

Size Threshold Requirements

Direct Transport Threshold

Platform	Threshold	Notes
Desktop (Julia/JS/Python)	0.5MB	Default size threshold
MicroPython	100KB	Lower threshold for memory constraints

Maximum Payload Size

Platform	Maximum	Notes
Desktop	Unlimited	Limited by NATS server configuration
MicroPython	50KB	Hard limit due to 256KB-1MB memory

Message Envelope Requirements

Required Fields

Field	Type	Purpose
`correlation_id`	String (UUID)	Track message flow across systems
`msg_id`	String (UUID)	Unique message identifier
`timestamp`	String (ISO 8601)	Message publication timestamp
`send_to`	String	NATS subject to publish to
`msg_purpose`	String	ACK, NACK, updateStatus, shutdown, chat
`sender_name`	String	Sender application name
`sender_id`	String (UUID)	Sender unique identifier
`receiver_name`	String	Receiver application name (empty = broadcast)
`receiver_id`	String (UUID)	Receiver unique identifier (empty = broadcast)
`reply_to`	String	Topic for reply messages
`reply_to_msg_id`	String	Message ID being replied to
`broker_url`	String	NATS server URL
`metadata`	Dict	Message-level metadata
`payloads`	Array	List of payload objects

Payload Fields

Field	Type	Purpose
`id`	String (UUID)	Unique payload identifier
`dataname`	String	Name of the payload
`payload_type`	String	Type: text, dictionary, arrowtable, etc.
`transport`	String	direct or link
`encoding`	String	none, json, base64, arrow-ipc
`size`	Integer	Payload size in bytes
`data`	Any	Base64 string or URL
`metadata`	Dict	Payload-level metadata

Error Handling Requirements

Error Codes

Error	Condition	Response
`Unknown payload_type`	Unsupported type	Throw error
`Failed to upload`	File server error	Throw error
`Failed to fetch`	File server unavailable	Retry with exponential backoff
`Unknown transport`	Invalid transport type	Throw error
`NATS connection failed`	NATS unavailable	Throw error

Exception Handling

Scenario	Handler
File server unavailable	Retry up to 5 times with exponential backoff
NATS publish failure	Connection auto-reconnect
Deserialization error	Log correlation ID and throw error
Memory overflow (MicroPython)	Reject payloads >50KB

Testing Requirements

Unit Tests

Test Category	Coverage	Files
Serialization	All payload types	`test/test__sender.`
Deserialization	All payload types	`test/test__receiver.`
Transport selection	Direct vs link	`test/test__mix_payloads.`
File server upload	Plik integration	Platform-specific
File server download	Exponential backoff	Platform-specific

Integration Tests

Test Scenario	Success Criteria
Cross-platform text message	Julia ↔ JavaScript ↔ Python
Cross-platform tabular data (Desktop)	Arrow IPC round-trip
Cross-platform tabular data (Browser)	JSON table round-trip
Large file transfer	File server upload/download
Multi-payload mixed content	All payload types in one message

API Contract

smartsend Signature

function smartsend(
    subject::String,
    data::AbstractArray{Tuple{String, Any, String}};
    broker_url::String = "nats://localhost:4222",
    fileserver_url::String = "http://localhost:8080",
    fileserver_upload_handler::Function = plik_oneshot_upload,
    size_threshold::Int = 1_000_000,
    correlation_id::String = string(uuid4()),
    msg_purpose::String = "chat",
    sender_name::String = "NATSBridge",
    receiver_name::String = "",
    receiver_id::String = "",
    reply_to::String = "",
    reply_to_msg_id::String = "",
    is_publish::Bool = true,
    NATS_connection::Union{NATS.Connection, Nothing} = nothing,
    msg_id::String = string(uuid4()),
    sender_id::String = string(uuid4())
)::Tuple{msg_envelope_v1, String}

smartreceive Signature

function smartreceive(
    msg::NATS.Msg;
    fileserver_download_handler::Function = _fetch_with_backoff,
    max_retries::Int = 5,
    base_delay::Int = 100,
    max_delay::Int = 5000
)::JSON.Object{String, Any}

Dependencies

Required Dependencies

Platform	Package	Version
Julia	NATS.jl	Latest stable
Julia	JSON.jl	Latest stable
Julia	Arrow.jl	Latest stable
Julia	HTTP.jl	Latest stable
Julia	UUIDs.jl	Latest stable
Node.js	nats	Latest stable
Node.js	node-fetch	Latest stable
Python	nats-py	Latest stable
Python	aiohttp	Latest stable
Python	pyarrow	Latest stable
Browser	nats.ws	Latest stable

Optional Dependencies

Platform	Package	Use Case
Julia	DataFrames.jl	DataFrame support for arrowtable
Python	pandas	DataFrame support for arrowtable

Deployment Requirements

Minimum Infrastructure

Component	Minimum	Notes
NATS Server	1 instance	Single node for development
File Server	1 instance	HTTP server for large payloads
Client Memory	50MB	Desktop platforms
Client Memory	256KB	MicroPython devices

Environment Variables

Variable	Default	Description
`NATS_URL`	`nats://localhost:4222`	NATS server URL
`FILESERVER_URL`	`http://localhost:8080`	HTTP file server URL
`SIZE_THRESHOLD`	`1000000`	Size threshold in bytes

Versioning

Current Version

Major: 1 (Breaking changes require major version bump)
Minor: 0 (Feature additions)
Patch: 0 (Bug fixes)

Version Compatibility

Version	Supported Platforms
v1.0.x	Julia 1.7+, Node.js 16+, Python 3.8+, Browser (latest), MicroPython 1.19+

Change Log

Date	Version	Changes
2026-03-13	1.0.0	Initial requirements document

References

src/NATSBridge.jl - Ground truth implementation
README.md - Project overview
docs/architecture.md - Architecture documentation
docs/implementation.md - Implementation details
docs/walkthrough.md - Usage examples

16 KiB Raw Blame History