msghandler/docs/requirements.md

Requirements Document: msghandler

Version: 1.3.0
Date: 2026-05-22
Status: Active
Ground Truth: src/msghandler.jl

---

1. Business Context & Success Metrics

1.1 Business Goal

msghandler is a cross-platform, bi-directional data bridge that enables seamless communication between Julia, JavaScript, Python, Dart, Rust, and MicroPython applications using a message broker as the transport layer. The system implements the Claim-Check pattern for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over the transport layer.

1.2 User Stories (with acceptance criteria)

Story	Priority	Acceptance Criteria
As a Julia developer, I want to send text messages to JavaScript/Dart applications that lives on a server and also on a browser	P1	Text messages are serialized, encoded, and received correctly across platforms
As a Python developer, I want to send tabular data to Julia/Dart applications	P1	DataFrame exchange works with both Arrow IPC and JSON formats
As a JavaScript developer, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications	P1	Large files are automatically uploaded to file server and URLs are sent via the transport layer
As a Dart developer, I want to send text messages to other platforms	P1	Text messages are serialized, encoded, and received correctly across platforms
As a Dart developer, I want to send dictionary data to other platforms	P1	JSON-serializable data is exchanged correctly
As a Dart developer, I want to send tabular data (List) to other platforms	P1	JSON table format exchange works with Arrow IPC on desktop
As a Dart developer, I want to send large files (>0.5MB)	P1	Large files are automatically uploaded to file server and URLs are sent via the transport layer
As a MicroPython developer, I want to send sensor data with minimal memory usage	P1	Direct transport works for payloads <100KB on memory-constrained devices
As a Rust developer, I want to send and receive messages with type-safe APIs	P1	Rust implementation uses serde for serialization, tokio for async, and transport-agnostic client for connectivity
As a developer, I want to send mixed-content messages (text + image + file)	P1	msghandler accepts list of (dataname, data, type) tuples and handles each payload appropriately
As a developer, I want to receive multi-payload messages	P1	msghandler returns payloads as list of tuples with correct types preserved
As a developer, I want to use Plik as the file server	P2	Plik one-shot upload mode is supported with upload ID and token handling
As a developer, I want to use custom HTTP file servers	P2	Handler function abstraction allows plugging in AWS S3 or custom implementations
As a developer, I want automatic retry on file server download failures	P1	Exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms)
As a developer, I want message tracing across distributed systems	P1	Correlation ID is propagated through all message processing steps

1.3 Success Metrics & KPIs

Functional Requirements KPIs:

• FR-001 (Cross-platform text messaging): 95% of text messages delivered correctly across all platform pairs (<200ms latency) - Measured via synthetic cross-platform tests
• FR-002 (Cross-platform tabular data): 100% Arrow IPC round-trip integrity (Desktop), 100% JSON table round-trip integrity (Browser) - Measured via data validation tests
• FR-003 (Large file handling): 99% successful file uploads to server for payloads ≥0.5MB - Measured via integration tests
• FR-004 (Direct transport for small payloads): 100% of payloads <0.5MB use direct transport - Measured via transport selection tests
• FR-005 (MicroPython support): 100% of payloads <100KB delivered on MicroPython devices - Measured via MicroPython integration tests
• FR-006 (Multi-payload messages): 100% correct parsing of multi-payload message lists - Measured via multi-payload tests
• FR-007 (Payload type preservation): 100% type integrity preserved across all platforms - Measured via type validation tests
• FR-008 (Plik file server integration): 100% successful Plik upload/token handling - Measured via Plik integration tests
• FR-009 (Custom file server support): 100% handler abstraction works with custom implementations - Measured via custom server integration tests
• FR-010 (Exponential backoff retry): 95% successful downloads within retry limit - Measured via failure injection tests
• FR-011 (Correlation ID propagation): 100% correlation IDs propagated through all steps - Measured via tracing tests
• FR-012 (Message serialization): <50ms serialization overhead for 10KB payload - Measured via benchmark tests
• FR-013 (Transport publishing): 100% JSON envelope generated correctly - Measured via serialization tests
• FR-014 (Transport subscription): 100% JSON messages processed correctly - Measured via deserialization tests

Non-Functional Requirements KPIs:

• NFR-101 (Message serialization overhead): <50ms for 10KB payload - Measured via benchmark tests
• NFR-102 (Message deserialization overhead): <50ms for 10KB payload - Measured via benchmark tests
• NFR-103 (Transport connection establishment): <100ms average - Measured via connection pool benchmarks
• NFR-104 (File upload latency): <1s for 0.5MB file - Measured via integration tests
• NFR-105 (File download latency): <1s for 0.5MB file - Measured via integration tests
• NFR-106 (Concurrent connections): 100+ simultaneous transport connections - Measured via scale testing
• NFR-107 (Message throughput): 1000+ messages/second per instance - Measured via load testing
• NFR-108 (File server scalability): Horizontal scaling verified via architecture review
• NFR-201 (Message delivery): At-least-once delivery via transport - Measured via message acknowledgment tests
• NFR-202 (File server availability): <5% failure rate when file server unavailable - Measured via failure injection tests
• NFR-203 (Connection recovery): Auto-reconnect within 30s - Measured via connection failure tests
• NFR-301 (Payload integrity): 100% SHA-256 checksum validation - Measured via integrity tests
• NFR-302 (Transport security): 100% TLS connections in production - Measured via connection audits
• NFR-303 (File server security): 100% authenticated file uploads - Measured via security tests
• NFR-401 (Required logs): 100% messages logged with required fields - Measured via log validation
• NFR-402 (Critical metrics): 100% metrics collected with 1-minute granularity - Measured via metrics pipeline tests
• NFR-403 (Tracing): 100% correlation ID propagation for tracing - Measured via tracing validation
• NFR-404 (Alerting): <5min alert latency for download_retry_exceeded - Measured via alert pipeline tests
• NFR-405 (Retention): Logs: 30 days, Metrics: 1 year - Measured via storage audits

---

2. Technical Boundaries

2.1 In Scope

Feature	Description
Cross-platform interoperability	Seamless data exchange between Julia, JavaScript, Python, Dart, Rust, and MicroPython
Intelligent transport selection	Direct transport (<0.5MB) vs Link transport (≥0.5MB) based on payload size
Unified API	Consistent smartpack() and smartunpack() functions across all platforms
Multi-payload support	List of (dataname, data, type) tuples with appropriate handling
File server integration	Plik one-shot upload and custom HTTP server support
Reliability features	Exponential backoff retry and correlation ID propagation
Message serialization	Converts data types to binary format (Base64, JSON, Arrow IPC)
Transport communication	Publishing and subscription via message broker (NATS, MQTT, WebSocket, etc.)

2.2 Out of Scope

Feature	Reason
Advanced transport features	Basic transport sufficient for current use cases
Message compression	Compression adds complexity without clear benefit
Message encryption	Payload encryption is application-layer concern
Persistent message queues	Transport request-reply pattern sufficient
Advanced routing rules	Simple topic matching sufficient

2.3 Dependencies

Platform	Package	Version
Julia	JSON.jl	Latest stable
Julia	Arrow.jl	Latest stable
Julia	HTTP.jl	Latest stable
Julia	UUIDs.jl	Latest stable
Node.js	node-fetch	Latest stable
Python	aiohttp	Latest stable
Python	pyarrow	Latest stable
Browser	-	Transport-agnostic (caller provides)
Dart	http	Latest stable
Dart	uuid	Latest stable
Rust	serde	Latest stable
Rust	serde_json	Latest stable
Rust	tokio	Latest stable
Rust	uuid	Latest stable

2.4 Platform Compatibility

Platform	Minimum Version	Notes
Julia	1.7+	Arrow.jl required for arrowtable support
Node.js	16+	Transport client required, Arrow IPC supported
Python	3.8+	pyarrow required for arrowtable support
Browser	Latest	No Arrow IPC (uses jsontable only)
Dart	2.17+	Supports Desktop (Dart SDK), Flutter (Dart SDK), and Web (Dart SDK)
Rust	1.70+	Full support with async/await, Arrow IPC on desktop
MicroPython	1.19+	Limited to direct transport

---

3. Functional Requirements (FR)

ID	Requirement	Description
FR-001	Cross-platform text messaging	System shall allow users to send text messages between Julia, JavaScript, Python, and MicroPython applications
FR-002	Cross-platform tabular data	System shall support DataFrame exchange between Julia and Python applications using Arrow IPC format
FR-003	Large file handling	System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via transport
FR-004	Direct transport for small payloads	System shall send payloads <0.5MB directly via transport without file server upload
FR-005	MicroPython support	System shall support payloads <100KB on MicroPython devices using direct transport
FR-006	Multi-payload messages	System shall accept and process lists of (dataname, data, type) tuples
FR-007	Payload type preservation	System shall preserve payload types when returning multi-payload messages
FR-008	Plik file server integration	System shall support Plik one-shot upload mode with upload ID and token handling
FR-009	Custom file server support	System shall provide handler function abstraction for custom HTTP file server implementations
FR-010	Exponential backoff retry	System shall implement exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) for file server download failures
FR-011	Correlation ID propagation	System shall propagate correlation IDs through all message processing steps
FR-012	Message serialization	System shall serialize data types using Base64, JSON, or Arrow IPC encoding
FR-013	Transport publishing	System shall return JSON string representation for caller to publish via transport layer (caller is responsible for actual transport publish)
FR-014	Transport subscription	System shall receive and process messages by accepting JSON string from transport payload

---

4. Non-Functional Requirements (NFRs)

Requirement vs KPI Clarification:

• FR and NFR is a requirement — it defines what quality or constraint the system must have (e.g., "System shall support 10K TPS", "99.9% monthly uptime", "TLS 1.3+ encryption")
• KPI is a measurement — it's the actual data collected to verify if the requirement was met (e.g., "Peak traffic was 8.5K TPS", "MTTR was 8 minutes", "100% of connections use TLS 1.3")
• Requirements tell you what to build; KPIs tell you how well you built it

4.1 Performance & Scalability

ID	Requirement	Specification	KPI	Test Method
NFR-101	Message serialization overhead	<50ms for 10KB payload	<50ms for 10KB payload	Benchmark tests
NFR-102	Message deserialization overhead	<50ms for 10KB payload	<50ms for 10KB payload	Benchmark tests
NFR-103	Transport connection establishment	<100ms	<100ms average	Connection pool benchmarks
NFR-104	File upload latency	<1s for 0.5MB file	<1s for 0.5MB file	Integration tests
NFR-105	File download latency	<1s for 0.5MB file	<1s for 0.5MB file	Integration tests
NFR-106	Concurrent connections	Support 100+ simultaneous transport connections	100+ simultaneous connections	Scale testing
NFR-107	Message throughput	Handle 1000+ messages/second per instance	1000+ messages/second	Load testing
NFR-108	File server scalability	Support horizontal scaling of file server backend	Horizontal scaling verified	Architecture review

4.2 Availability & Reliability

ID	Requirement	Specification	KPI	Test Method
NFR-201	Message delivery	At-least-once delivery semantics via transport	At-least-once delivery via transport	Message acknowledgment tests
NFR-202	File server availability	Graceful degradation when file server is unavailable	<5% failure rate when file server unavailable	Failure injection tests
NFR-203	Connection recovery	Auto-reconnect on transport connection failure	Auto-reconnect within 30s	Connection failure tests

4.3 Privacy & Security

ID	Requirement	Specification	KPI	Test Method
NFR-301	Payload integrity	SHA-256 checksum support via metadata	100% SHA-256 checksum validation	Integrity tests
NFR-302	Transport security	TLS support for transport connections	100% TLS connections in production	Connection audits
NFR-303	File server security	Authentication token for file uploads	100% authenticated file uploads	Security tests

4.4 Observability & Telemetry

ID	Requirement	Specification	KPI	Test Method
NFR-401	Required logs	correlation_id, msg_id, timestamp, sender_name, receiver_name, payload_type, transport	100% messages logged with required fields	Log validation
NFR-402	Critical metrics	messages_sent_total, messages_received_total, file_upload_duration_seconds, file_download_duration_seconds, retry_attempts_total	100% metrics collected with 1-minute granularity	Metrics pipeline tests
NFR-403	Tracing	Correlation ID propagation for request tracing	100% correlation ID propagation for tracing	Tracing validation
NFR-404	Alerting	download_retry_exceeded triggers alert when max retries exceeded	<5min alert latency for download_retry_exceeded	Alert pipeline tests
NFR-405	Retention	Logs: 30 days, Metrics: 1 year	Logs: 30 days, Metrics: 1 year	Storage audits

---

5. Acceptance Conditions

Condition	Description
AC-001	All functional requirements FR-001 through FR-014 are implemented and tested
AC-002	All non-functional requirements NFR-101 through NFR-405 meet specified KPI targets
AC-003	Cross-platform text message test passes (Julia ↔ JavaScript ↔ Python)
AC-004	Cross-platform tabular data test passes with Arrow IPC round-trip (Desktop)
AC-005	Cross-platform tabular data test passes with JSON table round-trip (Browser)
AC-006	Large file transfer test passes with file server upload/download
AC-007	Multi-payload mixed content test passes with all payload types in one message
AC-008	CI validation gates block PRs on specification violations
AC-009	Unit test coverage exceeds 80%
AC-010	Documentation is complete and includes walkthroughs, architecture, and runbook

---

6. Payload Type Requirements

6.1 Supported Payload Types

Type	Julia	JavaScript	Python	Dart	MicroPython	Description
text	String	string	str	String	String	str
dictionary	Dict, NamedTuple	Object, Array	dict, list	Map, serde_json::Value	String	dict
arrowtable	DataFrame, Arrow.Table	❌ (Browser), ✅ (Node.js)	pandas.DataFrame	List<Map> (Desktop), List<dynamic> (Flutter)	arrow2::Table	❌
jsontable	Vector{NamedTuple}	Array<Object>	list[dict]	Vec<Map>	⚠️	Tabular data (JSON) - Only table type in Browser
image	Vector{UInt8}	Uint8Array, Buffer	bytes	Uint8List	Vec<u8>	bytearray
audio	Vector{UInt8}	Uint8Array, Buffer	bytes	Uint8List	Vec<u8>	bytearray
video	Vector{UInt8}	Uint8Array, Buffer	bytes	Uint8List	Vec<u8>	bytearray
binary	Vector{UInt8}, IOBuffer	Uint8Array, Buffer	bytes, bytearray	Uint8List	Vec<u8>	bytearray

6.2 Encoding Requirements

Payload Type	Encoding Method	Notes
text	UTF-8 → Base64	Text must be String type
dictionary	JSON → Base64	JSON.jl for Julia
arrowtable	Arrow IPC → Base64	Requires Arrow.jl/pyarrow (Desktop only)
jsontable	JSON → Base64	Human-readable format - Browser uses this only
image/audio/video/binary	Direct → Base64	Binary data preserved

---

7. Size Threshold Requirements

7.1 Direct Transport Threshold

Platform	Threshold	Notes
Desktop (Julia/JS/Python/Dart)	0.5MB	Default size threshold
Dart Desktop	0.5MB	Default size threshold
Dart Flutter	0.5MB	Default size threshold
Dart Web	0.5MB	Default size threshold
Rust	0.5MB	Default size threshold
MicroPython	100KB	Lower threshold for memory constraints

7.2 Maximum Payload Size

Platform	Maximum	Notes
Desktop	Unlimited	Limited by transport server configuration
Dart Desktop	Unlimited	Limited by transport server configuration
Dart Flutter	Unlimited	Limited by transport server configuration
Dart Web	Unlimited	Limited by transport server configuration
Rust	Unlimited	Limited by transport server configuration
MicroPython	50KB	Hard limit due to 256KB-1MB memory

---

8. Message Envelope Requirements

8.1 Required Fields

Field	Type	Purpose
correlation_id	String (UUID)	Track message flow across systems
msg_id	String (UUID)	Unique message identifier
timestamp	String (ISO 8601)	Message publication timestamp
send_to	String	Topic/subject to publish to
msg_purpose	String	ACK, NACK, updateStatus, shutdown, chat
sender_name	String	Sender application name
sender_id	String (UUID)	Sender unique identifier
receiver_name	String	Receiver application name (empty = broadcast)
receiver_id	String (UUID)	Receiver unique identifier (empty = broadcast)
reply_to	String	Topic for reply messages
reply_to_msg_id	String	Message ID being replied to
broker_url	String	Broker URL for the transport layer
metadata	Dict	Message-level metadata
payloads	Array	List of payload objects

8.2 Payload Fields

Field	Type	Purpose
id	String (UUID)	Unique payload identifier
dataname	String	Name of the payload
payload_type	String	Type: text, dictionary, arrowtable, etc.
transport	String	direct or link
encoding	String	none, json, base64, arrow-ipc
size	Integer	Payload size in bytes
data	Any	Base64 string or URL
metadata	Dict	Payload-level metadata

---

9. Error Handling Requirements

9.1 Error Codes

Error	Condition	Response
Unknown payload_type	Unsupported type	Throw error
Failed to upload	File server error	Throw error
Failed to fetch	File server unavailable	Retry with exponential backoff
Unknown transport	Invalid transport type	Throw error
Transport connection failed	Transport/broker unavailable	Throw error

9.2 Exception Handling

Scenario	Handler
File server unavailable	Retry up to 5 times with exponential backoff
Transport publish failure	Handled by caller
Deserialization error	Log correlation ID and throw error
Memory overflow (MicroPython)	Reject payloads >50KB

---

10. Testing Requirements

10.1 Unit Tests

Test Category	Coverage	Files
Serialization	All payload types	test/test_*_sender.*
Deserialization	All payload types	test/test_*_receiver.*
Transport selection	Direct vs link	test/test_*_mix_payloads.*
File server upload	Plik integration	Platform-specific
File server download	Exponential backoff	Platform-specific

10.2 Integration Tests

Test Scenario	Success Criteria
Cross-platform text message	Julia ↔ JavaScript ↔ Python
Cross-platform tabular data (Desktop)	Arrow IPC round-trip
Cross-platform tabular data (Browser)	JSON table round-trip
Large file transfer	File server upload/download
Multi-payload mixed content	All payload types in one message

---

11. API Contract

11.1 smartpack Signature

function smartpack(
    subject::String,
    data::AbstractArray{Tuple{String, T1, String}, 1};
    broker_url::String = DEFAULT_BROKER_URL,
    fileserver_url::String = DEFAULT_FILESERVER_URL,
    fileserver_upload_handler::Function = plik_oneshot_upload,
    size_threshold::Int = DEFAULT_SIZE_THRESHOLD,
    correlation_id::String = string(uuid4()),
    msg_purpose::String = "chat",
    sender_name::String = "msghandler",
    receiver_name::String = "",
    receiver_id::String = "",
    reply_to::String = "",
    reply_to_msg_id::String = "",
    msg_id::String = string(uuid4()),
    sender_id::String = string(uuid4())
)::Tuple{msg_envelope_v1, String} where {T1<:Any}

Note: Publishing via the transport layer is the caller's responsibility. smartpack returns (env::msg_envelope_v1, env_json_str::String).

11.2 smartunpack Signature

function smartunpack(
    msg_json_str::String;
    fileserver_download_handler::Function = _fetch_with_backoff,
    max_retries::Int = 5,
    base_delay::Int = 100,
    max_delay::Int = 5000
)::JSON.Object{String, Any}

Note: Pass the payload string from the transport subscription to smartunpack. The input is the JSON string payload from the transport message, not the transport message object directly.

Note: Pass the payload from the transport subscription to smartunpack.

---

12. Deployment Requirements

12.1 Minimum Infrastructure

Component	Minimum	Notes
Message Broker	1 instance	Single node for development
File Server	1 instance	HTTP server for large payloads
Client Memory	50MB	Desktop platforms (Julia/JS/Python/Dart)
Client Memory	256KB	MicroPython devices

12.2 Environment Variables

Variable	Default	Description
BROKER_URL	ws://localhost:4222	Message broker URL
FILESERVER_URL	http://localhost:8080	HTTP file server URL
SIZE_THRESHOLD	500000	Size threshold in bytes (0.5MB)

---

13. Versioning

13.1 Current Version

• Major: 1 (Breaking changes require major version bump)
• Minor: 0 (Feature additions)
• Patch: 0 (Bug fixes)

13.2 Version Compatibility

Version	Supported Platforms
v1.0.x	Julia 1.7+, Node.js 16+, Python 3.8+, Dart 2.17+, Rust 1.70+, Browser (latest), MicroPython 1.19+

---

14. Change Log

Date	Version	Changes
2026-05-22	1.3.0	Updated to ASG Framework v8 pillars - added KPIs to all FR and NFR requirements
-	-	Added Success Metrics & KPIs section with measurable targets for each requirement
-	-	Added NFR vs KPI clarification section
-	-	Updated NFR tables to include KPI column and Test Method column
2026-05-15	1.3.0	Made transport layer agnostic
-	-	Removed all NATS-specific dependencies and references
-	-	Updated all NATS references to generic "transport layer"/"message broker"
-	-	Removed NATS client packages from dependencies tables
2026-05-13	1.2.0	Aligned with ground truth implementation (src/msghandler.jl)
-	-	Fixed smartpack signature: removed is_publish, NATS_connection; added sender_name
-	-	Fixed smartunpack signature: takes msg_json_str::String instead of msg::NATS.Msg
-	-	Fixed size_threshold default from 1,000,000 to 500,000
-	-	Updated FR-013/FR-014 to reflect caller responsibility for NATS publishing
-	-	Updated FR-008/FR-009 to include file path upload overload
-	-	Updated SIZE_THRESHOLD env var default to 500000
2026-03-23	1.0.0	Updated to ASG Framework requirements structure

---

15. References

• src/msghandler.jl - Ground truth implementation (Julia)
• src/msghandler_ssr.js - Server-side JavaScript implementation
• src/msghandler_csr.js - Client-side JavaScript implementation
• src/msghandler.py - Python implementation
• src/msghandler.dart - Dart implementation
• src/msghandler_mpy.py - MicroPython implementation
• src/msghandler.rs - Rust implementation
• README.md - Project overview
• docs/specification.md - Technical specification
• docs/ui-specification.md - UI specification
• docs/walkthrough.md - End-to-end walkthrough
• docs/architecture.md - Architecture documentation
• docs/validation.md - Validation and CI/CD
• docs/runbook.md - Operational runbook