14 Commits

Author SHA1 Message Date
ton
6da64092ca Merge pull request 'v0.6.0-dev-seperate_natsclient_smartsend' (#13) from v0.6.0-dev-seperate_natsclient_smartsend into v0.6.0-dev
Reviewed-on: #13
2026-05-14 06:21:29 +00:00
809aea454b update 2026-05-14 13:16:13 +07:00
c5a70edd57 rust version implemented 2026-05-13 20:24:08 +07:00
b0acee053c update docs 2026-05-13 17:35:46 +07:00
c25c6a8a43 update 2026-05-13 16:57:20 +07:00
34a6d19303 update jl docstring 2026-05-13 16:25:48 +07:00
8ada1ca49c update 2026-05-13 16:08:29 +07:00
60ae464ea2 update 2026-05-13 16:02:50 +07:00
ton
c20a266e72 Merge pull request 'adopt_ASG_doc' (#12) from adopt_ASG_doc into v0.6.0-dev
Reviewed-on: #12
2026-03-23 08:00:20 +00:00
4f141b130e update 2026-03-23 14:50:00 +07:00
fa039f2820 update architecture.md 2026-03-23 13:33:26 +07:00
b0c5ecb942 update walkthrough doc 2026-03-23 13:03:27 +07:00
ba659368a5 update spec doc 2026-03-23 12:51:55 +07:00
1b38b3d6f1 update requirement doc 2026-03-23 11:52:38 +07:00
13 changed files with 4658 additions and 979 deletions

1
.gitignore vendored
View File

@@ -1,3 +1,4 @@
node_modules/ node_modules/
package.json package.json
package-lock.json package-lock.json
target/

View File

@@ -143,12 +143,60 @@ Since I develop src folder before I adopt SDD_FRAMEWORK.md approach, can you che
# ---------------------------------------------- 100 --------------------------------------------- # # ---------------------------------------------- 100 --------------------------------------------- #
I updated src/NATSBridge.jl. Check and NATSBridge/docs folder I want to update the content of the following files according to ASG_Framework/ASG_Framework.md:
- NATSBridge/docs/requirements.md
- NATSBridge/docs/specification.md
- NATSBridge/docs/ui-specification.md (you'll need to create this one)
- NATSBridge/docs/walkthrough.md
- NATSBridge/docs/architecture.md
I'll do the other docs not listed here later myself.
now help me update the following file according to ASG_Framework/ASG_Framework.md:
- NATSBridge/docs/specification.md
<!-- ------------------------------------------- 100 ------------------------------------------- -->
Check ./docs folder. I would like to expand this package (NATSBRIDGE) to include Rust support.
Can you update the content of the following files according to /home/ton/docker-apps/sommpanion/ASG_Framework/ASG_Framework.md:
- ./docs/requirements.md
- ./docs/specification.md
- ./docs/walkthrough.md
- ./docs/architecture.md
<!-- ------------------------------------------- 100 ------------------------------------------- -->
I updated ./src/NATSBridge.jl. Use it as groundtruth. Check ./docs folder I want to update the content of the following files according to /home/ton/docker-apps/sommpanion/ASG_Framework/ASG_Framework.md:
- ./docs/requirements.md
- ./docs/specification.md
- ./docs/walkthrough.md
- ./docs/architecture.md
Check the following files:
- ./docs/requirements.md
- ./docs/specification.md
- ./docs/architecture.md
- ./docs/walkthrough.md
I would like to expand this package (NATSBRIDGE) to include Rust support.
Now help me update Rust implementation of this package at ./src/natsbridge.rs.
I want to build a client-side-rendering Dioxus-based chat webapp.
Dioxus version 0.7+ should be great.
I already populate the current folder for the project.
my server REST API endpoint is sommpanion.yiem.cc/agent-fronent/api/v1/chat but I didn't run the server yet. A message format is JSON string.
I just placed my custom package for encode and decode message at ./src/natsbridge.rs. smartsend() is for encoding and smartreceive() is for decoding.
you may also check the file /home/ton/docker-apps/sommpanion/NATSBridge/docs/walkthrough.md for more info about my package.
You can test whether Dioxus webapp can be build using this command "dx bundle --web --release --debug-symbols=false"

1915
Cargo.lock generated Normal file

File diff suppressed because it is too large Load Diff

31
Cargo.toml Normal file
View File

@@ -0,0 +1,31 @@
[package]
name = "natsbridge"
version = "1.2.0"
edition = "2021"
description = "Cross-platform bi-directional data bridge for NATS communication"
[lib]
name = "natsbridge"
path = "src/natsbridge.rs"
[dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.12", features = ["json", "stream", "multipart"] }
uuid = { version = "1", features = ["v4", "serde"] }
base64 = "0.22"
chrono = { version = "0.4", features = ["serde"] }
async-trait = "0.1"
futures = "0.3"
[dev-dependencies]
tempfile = "3"
[[example]]
name = "smartsend_example"
path = "examples/smartsend_example.rs"
[[example]]
name = "smartreceive_example"
path = "examples/smartreceive_example.rs"

View File

@@ -1,402 +0,0 @@
# SDD + GitOps Documentation Framework
This document defines the documentation framework for the NATSBridge project. It establishes a structured approach to creating, maintaining, and evolving technical documentation in alignment with GitOps principles—ensuring that documentation is versioned, auditable, and continuously validated alongside the codebase.
---
## The SDD Framework: Seven Pillars of Documentation
| Document | Purpose (Rationale) | Primary Audience | Format / Content | Example (SaaS Context) | Measurement (KPI) |
|----------|---------------------|-----------------|------------------|------------------------|-------------------|
| **Requirements** | Capture the **business intent** — why we're building this and what success looks like. Defines boundaries and user-visible outcomes. | Stakeholders, Product Owners, Lead Developers | User stories, PRDs, acceptance criteria, non-functional constraints. | "System must process tabular data from Julia to SvelteKit UI with <200ms latency for 5-member teams." | 95% of requests complete <200ms (synthetic monitoring). |
| **Specification** | The **technical contract** — precise rules for inputs, outputs, and data shape. Ensures consistency across dev and test. | Developers, QA Engineers, CI/CD pipelines | OpenAPI, Protobuf, AsyncAPI. Endpoint definitions, schemas, error codes. | `contract.yaml` defining a NATS subject that accepts Arrow streams with snake_case headers. | 100% of messages validated against spec (CI block rate). |
| **Architecture** | The **blueprint** — how components fit together, interact, and scale. Guides system structure and trade-offs. | Architects, Senior Developers, DevOps | C4 diagrams, Mermaid.js, component/network/storage models. | Diagram showing 6-node cluster routing traffic via Caddy → Node.js API → Julia pods. | 100% of major decisions logged with trade-off analysis. |
| **Walkthrough** | The **story of flow** — shows how pieces connect end-to-end and why steps are sequenced. Builds intuition for new devs. | New Developers, Team Members | TOUR.md, Loom videos, sequence diagrams. Step-by-step traces with rationale. | "UI sends JSON → Node.js wraps Claim-Check → Julia pulls Arrow data (prevents NATS overflow)." | New developers ship feature in <2 days (PR timeline). |
| **Implementation** | The **real code** — business logic, helpers, tests, configs. Where design becomes executable. | Developers, Code Reviewers | Source code, README.md, unit tests, setup scripts. | Julia function for matrix calculation + SvelteKit component rendering table. | >80% unit test coverage, <5% drift from spec. |
| **Validation** | The **enforcer** — ensures implementation matches the spec. Blocks drift and human error. | Automation servers, QA, Lead Developers | CI jobs, contract tests, linting, integration checks. | CI job rejects PR with camelCase field not allowed by YAML spec. | <1% of PRs bypass validation gates. |
| **Runbook** | The **operational manual** — how the system lives in production, scales, and recovers. Guides on-call engineers. | DevOps, SREs, On-call Developers | K8s manifests, Helm charts, Markdown guides. Deployment, scaling, backup/restore, troubleshooting. | GitOps manifest ensuring 6 Julia replicas restart if memory >80%. | MTTR <15 minutes for P1 incidents. |
---
## Detailed Document Descriptions
### 1. Requirements
**Purpose**: Capture the *business intent* — why we're building this and what success looks like. Defines boundaries and user-visible outcomes.
**Why It Matters**:
- Aligns engineering efforts with business goals
- Provides a north star for feature development
- Establishes acceptance criteria before implementation begins
- Creates a contract between product and engineering
**Content Guidelines**:
- User stories with clear acceptance criteria (As a X, I want Y so that Z)
- Product Requirements Documents (PRDs) with success metrics
- Non-functional requirements (performance, security, scalability)
- Boundary definitions (what's in scope vs. out of scope)
**Best Practices**:
- Link each requirement to a measurable KPI
- Keep requirements testable and verifiable
- Maintain backward compatibility with existing requirements
- Review and update requirements as business context changes
---
### 2. Specification
**Purpose**: The *technical contract* — precise rules for inputs, outputs, and data shape. Ensures consistency across dev and test.
**Why It Matters**:
- Prevents implementation drift between components
- Enables contract testing in CI/CD pipelines
- Provides a single source of truth for data structures
- Facilitates integration between teams
**Content Guidelines**:
- API endpoint definitions (methods, paths, parameters)
- Request/response schemas (JSON, XML, Protobuf, AsyncAPI)
- Error codes and their meanings
- Data validation rules and constraints
- Rate limiting and quota definitions
**Best Practices**:
- Use formal specification languages (OpenAPI 3.0+, AsyncAPI)
- Version specifications alongside code
- Generate client SDKs from specifications
- Block CI on specification violations
- Document edge cases and error scenarios
---
### 3. Architecture
**Purpose**: The *blueprint* — how components fit together, interact, and scale. Guides system structure and trade-offs.
**Why It Matters**:
- Provides a mental model for system design
- Guides technical decision-making and trade-off analysis
- Facilitates onboarding of new architects and senior developers
- Documents scaling and performance considerations
**Content Guidelines**:
- C4 diagrams (Context, Container, Component levels)
- Mermaid.js flowcharts for sequence diagrams
- Component interaction diagrams
- Network topology and data flow
- Storage and caching strategies
- Scaling and resilience patterns
**Best Practices**:
- Use diagrams that are easy to update (Mermaid.js over static images)
- Document trade-off decisions with Rationale Documents
- Include scaling considerations for each component
- Document failure modes and recovery strategies
- Keep architecture diagrams versioned with code
---
### 4. Walkthrough
**Purpose**: The *story of flow* — shows how pieces connect end-to-end and why steps are sequenced. Builds intuition for new devs.
**Why It Matters**:
- Reduces onboarding time for new developers
- Provides context that code comments alone cannot convey
- Explains the "why" behind architectural decisions
- Helps identify gaps in the system design
**Content Guidelines**:
- Step-by-step flow descriptions with rationale
- Sequence diagrams showing request/response patterns
- "Tour of the codebase" guides
- Video walkthroughs (Loom, internal recordings)
- Debugging and tracing examples
**Best Practices**:
- Walk through real user journeys, not just technical flows
- Include "what could go wrong" scenarios
- Link walkthroughs to relevant code locations
- Keep walkthroughs updated with architecture changes
- Make walkthroughs interactive where possible
---
### 5. Implementation
**Purpose**: The *real code* — business logic, helpers, tests, configs. Where design becomes executable.
**Why It Matters**:
- This is the actual artifact that runs in production
- Code is the ultimate source of truth (when it matches spec)
- Tests validate correctness and prevent regressions
- Configuration files define runtime behavior
**Content Guidelines**:
- Business logic implementation
- Helper functions and utilities
- Unit and integration tests
- Configuration files (YAML, JSON, environment)
- Setup and development scripts
- Code organization and module structure
**Best Practices**:
- Follow consistent code style and conventions
- Write tests before or alongside implementation (TDD/BDD)
- Document complex logic with inline comments
- Keep configuration externalized and versioned
- Use type annotations where applicable
---
### 6. Validation
**Purpose**: The *enforcer* — ensures implementation matches the spec. Blocks drift and human error.
**Why It Matters**:
- Prevents breaking changes from reaching production
- Catches specification violations early in the CI pipeline
- Maintains data integrity and API consistency
- Reduces manual QA effort through automation
**Content Guidelines**:
- CI/CD pipeline configurations
- Contract testing scripts
- Linting rules and configurations
- Integration test suites
- Schema validation jobs
- Security scanning and audit jobs
**Best Practices**:
- Fail CI on specification violations
- Run validation jobs on every commit and PR
- Use automated code review tools
- Maintain validation job health dashboard
- Document validation failure remediation steps
---
### 7. Runbook
**Purpose**: The *operational manual* — how the system lives in production, scales, and recovers. Guides on-call engineers.
**Why It Matters**:
- Reduces Mean Time To Recovery (MTTR) for incidents
- Provides step-by-step guidance for common issues
- Documents scaling and deployment procedures
- Ensures operational knowledge is not siloed
**Content Guidelines**:
- Deployment procedures (manual and automated)
- Scaling instructions (horizontal/vertical)
- Backup and restore procedures
- Troubleshooting guides for common issues
- Runbook entries for specific error codes
- Contact information and escalation paths
**Best Practices**:
- Write runbooks for every P1/P2 incident
- Include exact commands and configuration snippets
- Test runbooks periodically (chaos engineering)
- Link runbook entries to relevant documentation
- Keep runbooks updated when system changes
---
## How to Use This Approach Effectively
### 1. Start with Requirements
Before writing any code or documentation, establish clear requirements. Ask:
- What business problem are we solving?
- How will we measure success?
- What are the non-negotiable constraints?
**Action**: Create a `docs/requirements/` directory and start with `PRD.md` and `KPIs.md`.
### 2. Define the Specification First
Once requirements are stable, define the technical specification. This becomes the contract for implementation.
**Action**: Create `docs/specification/` with `contract.yaml` (or appropriate format) and `error-codes.md`.
### 3. Design the Architecture
With requirements and specification in place, design the architecture. Document trade-off decisions explicitly.
**Action**: Create `docs/architecture/` with Mermaid diagrams and `trade-offs.md`.
### 4. Create Walkthroughs Early
As soon as the architecture is defined, create walkthroughs. This helps identify gaps and provides onboarding material.
**Action**: Create `docs/walkthrough/` with `TOUR.md` and sequence diagrams.
### 5. Implement with Validation in Mind
Write implementation code that adheres to the specification. Build validation into the CI pipeline from day one.
**Action**: Ensure test files are co-located with implementation and run on every commit.
### 6. Automate Validation
Build automated validation that runs in CI/CD. This ensures spec compliance and prevents drift.
**Action**: Configure CI jobs to validate against specification and block PRs on violations.
### 7. Document Operations from Day One
Create runbook entries as soon as deployment procedures are established. Update them when incidents occur.
**Action**: Create `docs/runbook/` with entries for deployment, scaling, and common issues.
---
## GitOps Integration
This documentation framework aligns with GitOps principles:
| GitOps Principle | Documentation Alignment |
|-----------------|------------------------|
| **Versioned** | All documentation lives in git, with history and audit trail |
| ** declarative** | Specifications and architecture are declarative contracts |
| **Automated** | Validation jobs automate spec compliance checks |
| **Self-Service** | Walkthroughs and runbooks enable self-service onboarding and operations |
| **Observability** | KPIs and metrics are defined for each documentation artifact |
**Git Structure**:
```
docs/
├── requirements/ # PRDs, user stories, KPIs
├── specification/ # OpenAPI, Protobuf, AsyncAPI specs
├── architecture/ # C4 diagrams, Mermaid, trade-off docs
├── walkthrough/ # TOUR.md, sequence diagrams
├── implementation/ # Source code (in src/)
├── validation/ # CI configs, test suites
└── runbook/ # Deployment, scaling, troubleshooting
```
---
## Metrics and Continuous Improvement
Each documentation artifact has associated KPIs. Track these to ensure quality:
| Document | KPI | Target |
|----------|-----|--------|
| Requirements | Requirement coverage | 100% of features have associated requirements |
| Specification | Spec compliance rate | 100% of messages validate against spec |
| Architecture | Decision documentation | 100% of major decisions logged with trade-offs |
| Walkthrough | New dev time-to-first-PR | <2 days from onboarding to first contribution |
| Implementation | Test coverage | >80% unit test coverage |
| Validation | Bypass rate | <1% of PRs bypass validation gates |
| Runbook | MTTR | <15 minutes for P1 incidents |
**Review Cadence**:
- Weekly: Review KPI dashboards and documentation gaps
- Monthly: Update documentation based on incident learnings
- Quarterly: Full framework review and improvement
---
## Template Examples
### Requirements Template
```markdown
# PRD: Feature Name
## Business Goal
[What problem are we solving?]
## Success Metrics
- [Metric 1]: Target [value]
- [Metric 2]: Target [value]
## User Stories
- As a [role], I want [feature] so that [benefit]
- Acceptance Criteria: [details]
## Non-Functional Requirements
- Performance: [details]
- Security: [details]
- Scalability: [details]
## Out of Scope
- [What's explicitly excluded]
```
### Specification Template
```yaml
# contract.yaml
openapi: 3.0.0
info:
title: NATSBridge API
version: 1.0.0
paths:
/api/v1/endpoint:
post:
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/Request'
responses:
'200':
description: Success
content:
application/json:
schema:
$ref: '#/components/schemas/Response'
```
### Architecture Template
```mermaid
%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#3b82f6'}}}%%
flowchart TD
A[Client] --> B[Caddy]
B --> C[Node.js API]
C --> D[Julia Worker]
D --> E[NATS Cluster]
E --> F[Storage]
style A fill:#f9f9f9,stroke:#333
style E fill:#e0e7ff,stroke:#3b82f6
```
### Runbook Template
```markdown
# Runbook: Service Restart
**Severity**: P2
**Estimated Time**: 5 minutes
## Symptoms
- Service is unresponsive
- Health checks are failing
## Steps
1. SSH to the host
2. Run: `kubectl rollout restart deployment/natsbridge`
3. Monitor: `kubectl get pods -l app=natsbridge -w`
## Rollback
- Run: `kubectl rollout undo deployment/natsbridge`
## Post-Incident
- [ ] Review logs for root cause
- [ ] Update runbook if needed
```
---
## Conclusion
This SDD + GitOps Documentation Framework ensures that documentation is:
- **Structured**: Seven distinct artifacts with clear purposes
- **Automated**: Validation and CI/CD integration
- **Versioned**: All documentation in git with history
- **Measurable**: KPIs for quality and effectiveness
- **Actionable**: Practical templates and examples
Use this framework as a living document—update it as your team's needs evolve.

View File

@@ -1,16 +1,16 @@
# Architecture Documentation: NATSBridge # Architecture Documentation: NATSBridge
**Version**: 1.1.0 **Version**: 1.2.0
**Date**: 2026-03-15 **Date**: 2026-05-13
**Status**: Active **Status**: Active
**Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl) **Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl)
**Architecture Level**: C4 Container Level **Architecture Level**: C4 Container Level
--- ---
## Executive Summary ## 1. Executive Summary
This document defines the **blueprint** for NATSBridge - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, and **MicroPython** applications using NATS as the message bus. This document defines the **blueprint** for NATSBridge - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus.
This architecture document serves as the single source of truth for: This architecture document serves as the single source of truth for:
- **System Structure**: How components fit together and interact - **System Structure**: How components fit together and interact
@@ -18,8 +18,29 @@ This architecture document serves as the single source of truth for:
- **Failure Modes**: How the system handles failures and recovers - **Failure Modes**: How the system handles failures and recovers
- **Trade-off Decisions**: The rationale behind architectural decisions - **Trade-off Decisions**: The rationale behind architectural decisions
### 1.1 Specification Traceability
| Architecture Section | Specification Reference | UI Specification Reference | Requirement ID(s) |
|---------------------|-------------------------|---------------------------|-------------------|
| Section 2 (Context Diagram) | specification.md:2 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 3 (Container Diagram) | specification.md:2, specification.md:3, specification.md:11 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 4 (Component Diagram) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 5 (High-Level) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 6 (Message Envelope) | specification.md:2, specification.md:3, specification.md:8 | - | FR-011, FR-012, FR-013, FR-014, NFR-401, NFR-403 |
| Section 7 (Payload Type) | specification.md:3, specification.md:5, specification.md:6 | - | FR-001, FR-002, FR-003, FR-006, FR-012, NFR-101, NFR-102 |
| Section 8 (Transport Strategy) | specification.md:6, specification.md:7 | - | FR-003, FR-004, FR-005, FR-010, NFR-104, NFR-105, NFR-106 |
| Section 9 (Platform-Specific) | specification.md:13, specification.md:14 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
| Section 10 (Scaling) | specification.md:7, specification.md:13 | - | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 |
| Section 11 (Failure Modes) | specification.md:9, specification.md:11 | - | FR-008, FR-009, FR-010, FR-011, NFR-201, NFR-202, NFR-203 |
| Section 12 (Trade-offs) | specification.md:2, specification.md:3, specification.md:6, specification.md:7 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 |
| Section 13 (Deployment) | specification.md:12, specification.md:18 | - | FR-013, FR-014, NFR-201, NFR-203 |
| Section 14 (Security) | specification.md:4, specification.md:9, specification.md:12 | - | NFR-301, NFR-302, NFR-303, NFR-401, NFR-402, NFR-403, NFR-404, NFR-405 |
| Section 15 (Testing) | specification.md:17 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
--- ---
## 2. Architecture Overview
## Architecture Overview ## Architecture Overview
### C4 Context Diagram ### C4 Context Diagram
@@ -35,17 +56,23 @@ flowchart TD
Julia_App[Julia Application] Julia_App[Julia Application]
JS_App[JavaScript Application<br/>Node.js/Browser] JS_App[JavaScript Application<br/>Node.js/Browser]
Python_App[Python Application<br/>Desktop] Python_App[Python Application<br/>Desktop]
Dart_App[Dart Application<br/>Desktop/Flutter/Web]
Rust_App[Rust Application<br/>Server/Desktop]
MicroPython_App[MicroPython Device] MicroPython_App[MicroPython Device]
end end
Julia_App -->|NATS| NATS_Server Julia_App -->|NATS| NATS_Server
JS_App -->|NATS| NATS_Server JS_App -->|NATS| NATS_Server
Python_App -->|NATS| NATS_Server Python_App -->|NATS| NATS_Server
Dart_App -->|NATS| NATS_Server
Rust_App -->|NATS| NATS_Server
MicroPython_App -->|NATS| NATS_Server MicroPython_App -->|NATS| NATS_Server
Julia_App -->|HTTP| File_Server Julia_App -->|HTTP| File_Server
JS_App -->|HTTP| File_Server JS_App -->|HTTP| File_Server
Python_App -->|HTTP| File_Server Python_App -->|HTTP| File_Server
Dart_App -->|HTTP| File_Server
Rust_App -->|HTTP| File_Server
MicroPython_App -->|HTTP| File_Server MicroPython_App -->|HTTP| File_Server
style NATS_Server fill:#fff3e0,stroke:#f57c00 style NATS_Server fill:#fff3e0,stroke:#f57c00
@@ -53,6 +80,8 @@ flowchart TD
style Julia_App fill:#e8f5e9,stroke:#4caf50 style Julia_App fill:#e8f5e9,stroke:#4caf50
style JS_App fill:#e3f2fd,stroke:#2196f3 style JS_App fill:#e3f2fd,stroke:#2196f3
style Python_App fill:#e3f2fd,stroke:#2196f3 style Python_App fill:#e3f2fd,stroke:#2196f3
style Dart_App fill:#fff0f6,stroke:#e91e63
style Rust_App fill:#dea584,stroke:#e65100
style MicroPython_App fill:#fce4ec,stroke:#e91e63 style MicroPython_App fill:#fce4ec,stroke:#e91e63
``` ```
@@ -60,26 +89,20 @@ flowchart TD
```mermaid ```mermaid
flowchart TD flowchart TD
subgraph "Client Container" subgraph "Client Container"
Julia_Module[Julia NATSBridge Module] Julia_Module[Julia NATSBridge Module]
JS_Module[JavaScript NATSBridge Module] JS_Module[JavaScript NATSBridge Module]
Python_Module[Python NATSBridge Module] Python_Module[Python NATSBridge Module]
Dart_Module[Dart NATSBridge Module]
Rust_Module[Rust NATSBridge Module]
MicroPython_Module[MicroPython NATSBridge Module] MicroPython_Module[MicroPython NATSBridge Module]
end end
subgraph "NATS Container"
NATS_Client[NATS Client]
NATS_Broker[NATS Broker]
end
subgraph "File Server Container"
File_Client[HTTP Client]
File_Server[File Server]
end
Julia_Module --> NATS_Client Julia_Module --> NATS_Client
JS_Module --> NATS_Client JS_Module --> NATS_Client
Python_Module --> NATS_Client Python_Module --> NATS_Client
Dart_Module --> NATS_Client
Rust_Module --> NATS_Client
MicroPython_Module --> NATS_Client MicroPython_Module --> NATS_Client
NATS_Client --> NATS_Broker NATS_Client --> NATS_Broker
@@ -87,6 +110,8 @@ flowchart TD
Julia_Module --> File_Client Julia_Module --> File_Client
JS_Module --> File_Client JS_Module --> File_Client
Python_Module --> File_Client Python_Module --> File_Client
Dart_Module --> File_Client
Rust_Module --> File_Client
MicroPython_Module --> File_Client MicroPython_Module --> File_Client
File_Client --> File_Server File_Client --> File_Server
@@ -94,6 +119,8 @@ flowchart TD
style Julia_Module fill:#e8f5e9,stroke:#4caf50 style Julia_Module fill:#e8f5e9,stroke:#4caf50
style JS_Module fill:#e3f2fd,stroke:#2196f3 style JS_Module fill:#e3f2fd,stroke:#2196f3
style Python_Module fill:#e3f2fd,stroke:#2196f3 style Python_Module fill:#e3f2fd,stroke:#2196f3
style Dart_Module fill:#fff0f6,stroke:#e91e63
style Rust_Module fill:#dea584,stroke:#e65100
style MicroPython_Module fill:#fce4ec,stroke:#e91e63 style MicroPython_Module fill:#fce4ec,stroke:#e91e63
style NATS_Broker fill:#fff3e0,stroke:#f57c00 style NATS_Broker fill:#fff3e0,stroke:#f57c00
style File_Server fill:#f3e5f5,stroke:#9c27b4 style File_Server fill:#f3e5f5,stroke:#9c27b4
@@ -110,36 +137,31 @@ flowchart TD
Serialize[_serialize_data] Serialize[_serialize_data]
Deserialize[_deserialize_data] Deserialize[_deserialize_data]
BuildEnvelope[build_envelope] EnvelopeToJson[envelope_to_json]
BuildPayload[build_payload]
PublishMessage[publish_message]
FileServerUpload[fileserver_upload_handler] FileServerUpload[fileserver_upload_handler]
FileServerDownload[fileserver_download_handler] FileServerDownload[fileserver_download_handler]
LogTrace[log_trace]
end end
subgraph "Data Models" subgraph "Data Models"
Payload[MsgPayloadV1 Struct] Payload[msg_payload_v1 Struct]
Envelope[MsgEnvelopeV1 Struct] Envelope[msg_envelope_v1 Struct]
end end
SmartSend --> Serialize SmartSend --> Serialize
SmartSend --> BuildEnvelope SmartSend --> EnvelopeToJson
SmartSend --> BuildPayload
SmartSend --> PublishMessage
SmartSend --> FileServerUpload SmartSend --> FileServerUpload
SmartReceive --> Deserialize SmartReceive --> Deserialize
SmartReceive --> FileServerDownload SmartReceive --> FileServerDownload
EnvelopeToJson --> Envelope
Serialize --> Payload Serialize --> Payload
BuildEnvelope --> Envelope
BuildPayload --> Payload
style SmartSend fill:#d1fae5,stroke:#10b981 style SmartSend fill:#d1fae5,stroke:#10b981
style SmartReceive fill:#d1fae5,stroke:#10b981 style SmartReceive fill:#d1fae5,stroke:#10b981
style PublishMessage fill:#fef3c7,stroke:#f59e0b
style FileServerUpload fill:#fef3c7,stroke:#f59e0b style FileServerUpload fill:#fef3c7,stroke:#f59e0b
style FileServerDownload fill:#fef3c7,stroke:#f59e0b style FileServerDownload fill:#fef3c7,stroke:#f59e0b
``` ```
@@ -152,15 +174,14 @@ flowchart TD
| Component | Purpose | Platform Support | | Component | Purpose | Platform Support |
|-----------|---------|------------------| |-----------|---------|------------------|
| **smartsend** | Send data via NATS with automatic transport selection | All | | **smartsend** | Send data via NATS with automatic transport selection, returns (envelope, json_string) for caller to publish | All |
| **smartreceive** | Receive and process NATS messages | All | | **smartreceive** | Receive and process NATS messages from JSON string | All |
| **_serialize_data** | Serialize data according to payload type | All | | **_serialize_data** | Serialize data according to payload type | All |
| **_deserialize_data** | Deserialize bytes to native data types | All | | **_deserialize_data** | Deserialize bytes to native data types | All |
| **_build_envelope** | Build message envelope from payloads | All | | **envelope_to_json** | Convert msg_envelope_v1 struct to JSON string | All |
| **_build_payload** | Build payload object from serialized data | All | | **log_trace** | Log trace messages with correlation ID | All |
| **publish_message** | Publish message to NATS subject | All | | **fileserver_upload_handler** | Upload large payloads to HTTP server | Desktop (Julia/JS/Python/Dart/Rust) |
| **fileserver_upload_handler** | Upload large payloads to HTTP server | Desktop | | **fileserver_download_handler** | Download payloads from HTTP server with exponential backoff | Desktop (Julia/JS/Python/Dart/Rust) |
| **fileserver_download_handler** | Download payloads from HTTP server | Desktop |
### Data Flow ### Data Flow
@@ -182,7 +203,7 @@ flowchart TD
H --> L[Build envelope] H --> L[Build envelope]
L --> M[Convert to JSON] L --> M[Convert to JSON]
M --> N[Publish to NATS] M --> N[Return envelope + JSON to caller]
style A fill:#f9f9f9,stroke:#333 style A fill:#f9f9f9,stroke:#333
style N fill:#e0e7ff,stroke:#3b82f6 style N fill:#e0e7ff,stroke:#3b82f6
@@ -275,8 +296,8 @@ end
|------|-------------|---------------|----------|-----------| |------|-------------|---------------|----------|-----------|
| `text` | Plain text string | UTF-8 bytes | Base64 | All | | `text` | Plain text string | UTF-8 bytes | Base64 | All |
| `dictionary` | JSON object | JSON string | Base64/JSON | All | | `dictionary` | JSON object | JSON string | Base64/JSON | All |
| `arrowtable` | Apache Arrow IPC | Arrow IPC stream | Base64/arrow-ipc | Desktop (Julia/Python/Node.js) | | `arrowtable` | Apache Arrow IPC | Arrow IPC stream | Base64/arrow-ipc | Desktop (Julia/Python/Node.js/Dart/Rust) |
| `jsontable` | JSON array of objects | JSON string | Base64/json | All (including Browser) | | `jsontable` | JSON array of objects | JSON string | Base64/json | All (including Browser/Dart Web) |
| `image` | Binary image data | Raw bytes | Base64 | All | | `image` | Binary image data | Raw bytes | Base64 | All |
| `audio` | Binary audio data | Raw bytes | Base64 | All | | `audio` | Binary audio data | Raw bytes | Base64 | All |
| `video` | Binary video data | Raw bytes | Base64 | All | | `video` | Binary video data | Raw bytes | Base64 | All |
@@ -318,7 +339,10 @@ flowchart TD
| Platform | Size Threshold | Notes | | Platform | Size Threshold | Notes |
|----------|----------------|-------| |----------|----------------|-------|
| Desktop (Julia/JS/Python) | 500,000 bytes (0.5MB) | Default threshold | | Desktop (Julia/JS/Python/Dart) | 500,000 bytes (0.5MB) | Default threshold |
| Dart Desktop | 500,000 bytes (0.5MB) | Default threshold |
| Dart Flutter | 500,000 bytes (0.5MB) | Default threshold |
| Dart Web | 500,000 bytes (0.5MB) | Default threshold |
| MicroPython | 100,000 bytes (100KB) | Lower threshold for memory constraints | | MicroPython | 100,000 bytes (100KB) | Lower threshold for memory constraints |
### Transport Selection Flow ### Transport Selection Flow
@@ -405,11 +429,9 @@ end
JavaScript uses async/await for non-blocking I/O: JavaScript uses async/await for non-blocking I/O:
- **Class-based NATS Client**: Connection management with `keepAlive` support
- **Module-level Utilities**: Serialization functions - **Module-level Utilities**: Serialization functions
- **Native ArrayBuffer**: Binary data handling (Browser) / Buffer (Node.js) - **Native ArrayBuffer**: Binary data handling (Browser) / Buffer (Node.js)
- **Fetch API**: HTTP file server communication - **Fetch API**: HTTP file server communication
- **Connection Pooling**: `NATSConnectionPool` for high-throughput scenarios
#### Node.js Implementation (natsbridge_ssr.js) #### Node.js Implementation (natsbridge_ssr.js)
@@ -417,36 +439,6 @@ JavaScript uses async/await for non-blocking I/O:
- **Apache Arrow IPC**: Full support via `apache-arrow` - **Apache Arrow IPC**: Full support via `apache-arrow`
- **Buffer for binary data**: Native Node.js Buffer handling - **Buffer for binary data**: Native Node.js Buffer handling
```javascript
// Class-based NATS client with keepAlive support
class NATSClient {
constructor(url, keepAlive = false) {
this.url = url;
this.connection = null;
this.keepAlive = keepAlive;
}
async connect() {
if (this.connection) return this.connection;
this.connection = await nats.connect({ servers: this.url });
return this.connection;
}
}
// Connection pool for managing multiple connections
class NATSConnectionPool {
constructor(url, maxSize = 10) {
this.url = url;
this.maxSize = maxSize;
this.connections = new Map();
}
async acquire() { /* Get or create connection */ }
release(client) { /* Return to pool or close */ }
async closeAll() { /* Close all pool connections */ }
}
```
#### Browser Implementation (natsbridge_csr.js) #### Browser Implementation (natsbridge_csr.js)
- **WebSocket NATS connections**: Uses `ws://` or `wss://` URLs via `nats.ws` - **WebSocket NATS connections**: Uses `ws://` or `wss://` URLs via `nats.ws`
@@ -454,23 +446,6 @@ class NATSConnectionPool {
- **Uint8Array for binary data**: Browser-compatible binary handling - **Uint8Array for binary data**: Browser-compatible binary handling
- **Web Crypto API**: UUID generation via `crypto.getRandomValues()` - **Web Crypto API**: UUID generation via `crypto.getRandomValues()`
```javascript
// Class-based NATS client with keepAlive support
class NATSClient {
constructor(url, keepAlive = false) {
this.url = url; // ws:// or wss://
this.connection = null;
this.keepAlive = keepAlive;
}
async connect() {
if (this.connection) return this.connection;
this.connection = await nats.connect({ servers: this.url });
return this.connection;
}
}
```
### Python Architecture ### Python Architecture
Python uses classes for stateful operations: Python uses classes for stateful operations:
@@ -489,6 +464,50 @@ class NATSBridge:
self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL
``` ```
### Dart Architecture
Dart uses classes for stateful operations with async/await:
- **Class-based NATSBridge**: Encapsulated API
- **Data classes**: Structured data (MsgPayloadV1, MsgEnvelopeV1)
- **Async/await**: I/O operations
- **dart-arrow**: Arrow IPC support (Desktop/Flutter only)
- **HTTP package**: HTTP file server communication
- **nats package**: NATS client with WebSocket support (Dart Web)
```dart
class NATSBridge {
static const DEFAULT_SIZE_THRESHOLD = 500000;
final String brokerUrl;
final String fileserverUrl;
NATSBridge({
this.brokerUrl = 'nats://localhost:4222',
this.fileserverUrl = 'http://localhost:8080',
});
}
```
#### Dart Desktop (Dart SDK)
- **TCP NATS connections**: Uses `nats://` or `tls://` URLs
- **Apache Arrow IPC**: Full support via `dart-arrow`
- **Uint8List for binary data**: Native Dart binary handling
#### Dart Flutter (Dart SDK)
- **TCP NATS connections**: Uses `nats://` or `tls://` URLs
- **Apache Arrow IPC**: Full support via `dart-arrow`
- **Uint8List for binary data**: Native Dart binary handling
#### Dart Web (Dart SDK)
- **WebSocket NATS connections**: Uses `ws://` or `wss://` URLs via `nats` package
- **No Apache Arrow**: Uses `jsontable` for tabular data only
- **Uint8List for binary data**: Browser-compatible binary handling
- **Fetch API**: HTTP file server communication via `http` package
### Browser Architecture ### Browser Architecture
Browser JavaScript has specific constraints due to security and compatibility: Browser JavaScript has specific constraints due to security and compatibility:
@@ -514,6 +533,62 @@ DEFAULT_SIZE_THRESHOLD = 100_000 # 100KB
MAX_PAYLOAD_SIZE = 50_000 # 50KB hard limit MAX_PAYLOAD_SIZE = 50_000 # 50KB hard limit
``` ```
### Rust Architecture
Rust leverages compile-time type safety and async runtimes:
- **Type-safe payloads**: Rust enum discriminates between `Text`, `Dictionary`, `ArrowTable`, `Binary`, etc.
- **serde serialization**: Automatic JSON deserialization via `#[derive(Serialize, Deserialize)]`
- **tokio runtime**: Efficient async I/O for NATS connections and HTTP file server operations
- **arrow2 integration**: Native Arrow IPC deserialization without intermediate format conversion
- **reqwest**: High-performance HTTP client with built-in TLS and connection pooling
- **Zero-copy patterns**: `Vec<u8>` passed directly to avoid unnecessary memory copies
- **Result<T, E>**: Idiomatic error handling with typed error types
```rust
// Type-safe payload enum (compile-time discrimination)
#[derive(Serialize, Deserialize, Clone)]
pub enum Payload {
Text(String),
Dictionary(serde_json::Value),
ArrowTable(Vec<u8>),
JsonTable(serde_json::Value),
Image(Vec<u8>),
Audio(Vec<u8>),
Video(Vec<u8>),
Binary(Vec<u8>),
}
// Configuration via builder pattern
pub struct SmartsendOptions {
pub broker_url: String,
pub fileserver_url: String,
pub fileserver_upload_handler: Option<UploadHandler>,
pub size_threshold: usize,
pub correlation_id: String,
pub msg_purpose: String,
pub sender_name: String,
// ... other fields
}
// NATS client with tokio integration
let conn = nats::connect("nats://localhost:4222").await?;
// Subscribe and process messages
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze")?;
for msg in sub.messages() {
let envelope: MsgEnvelopeV1 = serde_json::from_slice(&msg.payload)?;
// Type-safe access to payloads
for payload in &envelope.payloads {
match &payload.data {
Payload::ArrowTable(bytes) => { /* process */ },
Payload::Text(text) => { /* process */ },
_ => {}
}
}
}
```
--- ---
## Scaling Architecture ## Scaling Architecture
@@ -655,7 +730,7 @@ MAX_PAYLOAD_SIZE = 50_000 # 50KB hard limit
|-----------|---------|-------| |-----------|---------|-------|
| NATS Server | 1 instance | Single node for development | | NATS Server | 1 instance | Single node for development |
| File Server | 1 instance | HTTP server for large payloads | | File Server | 1 instance | HTTP server for large payloads |
| Client Memory | 50MB | Desktop platforms | | Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices | | Client Memory | 256KB | MicroPython devices |
### Environment Variables ### Environment Variables
@@ -664,7 +739,7 @@ MAX_PAYLOAD_SIZE = 50_000 # 50KB hard limit
|----------|---------|-------------| |----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL | | `NATS_URL` | `nats://localhost:4222` | NATS server URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL | | `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `1000000` | Size threshold in bytes | | `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) |
### Container Deployment ### Container Deployment
@@ -750,7 +825,7 @@ flowchart TD
| Version | Supported Platforms | | Version | Supported Platforms |
|---------|---------------------| |---------|---------------------|
| v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, MicroPython 1.19+ | | v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, Dart 2.17+, Rust 1.70+, MicroPython 1.19+ |
--- ---
@@ -758,6 +833,16 @@ flowchart TD
| Date | Version | Changes | | Date | Version | Changes |
|------|---------|---------| |------|---------|---------|
| 2026-05-13 | 1.3.0 | Added Rust support with tokio, serde, and arrow2 | All sections |
| - | - | Added Rust to C4 diagrams (context, container) | All sections |
| - | - | Added Rust platform-specific architecture section | specification.md:13 |
| - | - | Updated component table with Rust support | All sections |
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/NATSBridge.jl) |
| - | - | Removed publish_message component (commented out in source) |
| - | - | Removed NATSClient and NATSConnectionPool classes (not in ground truth) |
| - | - | Updated component diagram to match actual module structure |
| - | - | Updated data flow to show smartsend returns JSON for caller to publish |
| - | - | Fixed SIZE_THRESHOLD default to 500,000 bytes |
| 2026-03-15 | 1.1.0 | JavaScript connection management | | 2026-03-15 | 1.1.0 | JavaScript connection management |
| - | - | Added NATSClient with keepAlive support | | - | - | Added NATSClient with keepAlive support |
| - | - | Added NATSConnectionPool for connection reuse | | - | - | Added NATSConnectionPool for connection reuse |
@@ -766,12 +851,85 @@ flowchart TD
--- ---
## References ## 16. References
- [`docs/requirements.md`](./requirements.md) - Business requirements and user stories ### 16.1 Documentation Artifacts
- [`docs/spec.md`](./spec.md) - Technical specification and contracts
- [`src/NATSBridge.jl`](../src/NATSBridge.jl) - Ground truth implementation | Document | Purpose | Specification Traceability | UI Specification Traceability | Requirement ID(s) |
- [`README.md`](../README.md) - Project overview |----------|---------|---------------------------|------------------------------|-------------------|
| [`docs/requirements.md`](./requirements.md) | Business requirements and user stories | FR-001 through FR-014, NFR-101 through NFR-405 | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/specification.md`](./specification.md) | Technical contract for NATSBridge | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/ui-specification.md`](./ui-specification.md) | UI specification for client applications | - | All UI components and interactions | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/walkthrough.md`](./walkthrough.md) | End-to-end system flow | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/architecture.md`](./architecture.md) | System architecture diagrams | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/validation.md`](./validation.md) | CI/CD validation rules | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/runbook.md`](./runbook.md) | Operational runbook | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 |
### 16.2 Implementation Files
| File | Platform | Features | Specification Traceability | Requirement ID(s) |
|------|----------|----------|---------------------------|-------------------|
| [`src/NATSBridge.jl`](../src/NATSBridge.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/natsbridge_ssr.js`](../src/natsbridge_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/natsbridge_csr.js`](../src/natsbridge_csr.js) | Browser | JSON table only, WebSocket NATS | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/natsbridge.py`](../src/natsbridge.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/natsbridge.dart`](../src/natsbridge.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/natsbridge.rs`](../src/natsbridge.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/natsbridge_mpy.py`](../src/natsbridge_mpy.py) | MicroPython | Limited to direct transport | specification.md:2-19 (all sections) | FR-005, FR-006, FR-012 |
### 16.3 External Dependencies
| Platform | Package | Version | Purpose | Specification Traceability | Requirement ID(s) |
|----------|---------|---------|---------|---------------------------|-------------------|
| Julia | NATS.jl | Latest | NATS client | specification.md:11 | FR-013, FR-014, NFR-201 |
| Julia | JSON.jl | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Julia | Arrow.jl | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Julia | HTTP.jl | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Julia | UUIDs.jl | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 |
| Node.js | nats | Latest | NATS client (TCP) | specification.md:11 | FR-013, FR-014 |
| Node.js | node-fetch | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Browser | nats.ws | Latest | NATS client (WebSocket) | specification.md:11 | FR-013, FR-014 |
| Browser | nats | Latest | NATS client (for bundling) | specification.md:11 | FR-013, FR-014 |
| Python | nats-py | Latest | NATS client | specification.md:11 | FR-013, FR-014 |
| Python | aiohttp | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Python | pyarrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Dart | nats | Latest | NATS client | specification.md:11 | FR-013, FR-014 |
| Dart | http | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Dart | uuid | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 |
| Dart | dart-arrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| Rust | nats | Latest | NATS client | specification.md:11 | FR-013, FR-014 |
| Rust | serde | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Rust | serde_json | Latest | JSON handling | specification.md:11 | FR-012, NFR-101, NFR-102 |
| Rust | tokio | Latest | Async runtime | specification.md:11 | FR-013, FR-014 |
| Rust | reqwest | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 |
| Rust | uuid | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 |
| Rust | arrow2 | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 |
| MicroPython | builtin | N/A | Limited implementation | specification.md:11 | FR-005, FR-006, FR-012 |
---
## 17. Change Log
| Date | Version | Changes | Specification Reference |
|------|---------|---------|------------------------|
| 2026-03-23 | 1.1.0 | Updated to ASG Framework architecture guidelines | specification.md:2-19 (all sections) |
| 2026-03-15 | 1.1.0 | JavaScript connection management | specification.md:2-19 (all sections) |
| 2026-03-13 | 1.0.0 | Initial architecture documentation | specification.md:2-19 (all sections) |
---
## 18. Gap-Check Validation
| Stage Transition | Gap-Check Question | Status |
|------------------|-------------------|--------|
| Requirements → Specification | Does the Specification define all edge cases and conflict scenarios from the Requirements? | ✅ Verified - All FR-XXX requirements have corresponding spec rules |
| Specification → UI Specification | Does the UI Specification expose all the data and states defined in the Specification? | ⏳ Pending - UI spec not yet created |
| UI Specification → Walkthrough | Does the Walkthrough reflect the complete flow including error states and timing? | ⏳ Pending - UI spec not yet created |
| Walkthrough → Architecture | Does the Architecture support the performance and integration requirements defined in the Walkthrough? | ✅ Verified - Architecture supports all walkthrough flows |
---
*This architecture document is versioned and maintained in git alongside the codebase. All implementations must adhere to this architecture.*
--- ---

View File

@@ -1,33 +1,39 @@
# Requirements Document: NATSBridge # Requirements Document: NATSBridge
**Version**: 1.0.0 **Version**: 1.2.0
**Date**: 2026-03-13 **Date**: 2026-05-13
**Status**: Active **Status**: Active
**Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl) **Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl)
--- ---
## Executive Summary ## 1. Business Context & Success Metrics
NATSBridge is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, and **MicroPython** applications using NATS as the message bus. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS. ### 1.1 Business Goal
--- NATSBridge is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS.
## Business Goals ### 1.2 User Stories (with acceptance criteria)
### Primary Objectives | Story | Priority | Acceptance Criteria |
|-------|----------|---------------------|
| **As a Julia developer**, I want to send text messages to JavaScript/Dart applications that lives on a server and also on a browser | P1 | Text messages are serialized, encoded, and received correctly across platforms |
| **As a Python developer**, I want to send tabular data to Julia/Dart applications | P1 | DataFrame exchange works with both Arrow IPC and JSON formats |
| **As a JavaScript developer**, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS |
| **As a Dart developer**, I want to send text messages to other platforms | P1 | Text messages are serialized, encoded, and received correctly across platforms |
| **As a Dart developer**, I want to send dictionary data to other platforms | P1 | JSON-serializable data is exchanged correctly |
| **As a Dart developer**, I want to send tabular data (List<Map>) to other platforms | P1 | JSON table format exchange works with Arrow IPC on desktop |
| **As a Dart developer**, I want to send large files (>0.5MB) | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS |
| **As a MicroPython developer**, I want to send sensor data with minimal memory usage | P1 | Direct transport works for payloads <100KB on memory-constrained devices |
| **As a Rust developer**, I want to send and receive messages with type-safe APIs | P1 | Rust implementation uses serde for serialization, tokio for async, and nats-io for NATS connectivity |
| **As a developer**, I want to send mixed-content messages (text + image + file) | P1 | NATSBridge accepts list of (dataname, data, type) tuples and handles each payload appropriately |
| **As a developer**, I want to receive multi-payload messages | P1 | NATSBridge returns payloads as list of tuples with correct types preserved |
| **As a developer**, I want to use Plik as the file server | P2 | Plik one-shot upload mode is supported with upload ID and token handling |
| **As a developer**, I want to use custom HTTP file servers | P2 | Handler function abstraction allows plugging in AWS S3 or custom implementations |
| **As a developer**, I want automatic retry on file server download failures | P1 | Exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) |
| **As a developer**, I want message tracing across distributed systems | P1 | Correlation ID is propagated through all message processing steps |
1. **Cross-Platform Interoperability**: Enable seamless data exchange between Julia, JavaScript (for both Server-Side rendering and Client-Side rendering webapp), Python, and MicroPython applications without platform-specific barriers. ### 1.3 KPIs & Targets
2. **Efficient Large Payload Handling**: Implement intelligent transport selection based on payload size:
- **Direct Transport**: Small payloads (<0.5MB) sent directly via NATS
- **Link Transport**: Large payloads (≥0.5MB) uploaded to HTTP file server, URL sent via NATS
3. **Unified API Across Platforms**: Provide consistent `smartsend()` and `smartreceive()` functions across all supported platforms while maintaining idiomatic implementations.
4. **Developer Productivity**: Reduce onboarding time and simplify integration through comprehensive documentation and test examples.
### Success Metrics
| Metric | Target | Measurement Method | | Metric | Target | Measurement Method |
|--------|--------|-------------------| |--------|--------|-------------------|
@@ -40,91 +46,22 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless
--- ---
## User Stories ## 2. Technical Boundaries
### Core Functionality ### 2.1 In Scope
| Story | Priority | Acceptance Criteria | | Feature | Description |
|-------|----------|---------------------| |---------|-------------|
| **As a Julia developer**, I want to send text messages to JavaScript applications that lives on a server and also on a browser | P1 | Text messages are serialized, encoded, and received correctly across platforms | | Cross-platform interoperability | Seamless data exchange between Julia, JavaScript, Python, Dart, Rust, and MicroPython |
| **As a Python developer**, I want to send tabular data to Julia applications | P1 | DataFrame exchange works with both Arrow IPC and JSON formats | | Intelligent transport selection | Direct transport (<0.5MB) vs Link transport (≥0.5MB) based on payload size |
| **As a JavaScript developer**, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS | | Unified API | Consistent `smartsend()` and `smartreceive()` functions across all platforms |
| **As a MicroPython developer**, I want to send sensor data with minimal memory usage | P1 | Direct transport works for payloads <100KB on memory-constrained devices | | Multi-payload support | List of (dataname, data, type) tuples with appropriate handling |
| File server integration | Plik one-shot upload and custom HTTP server support |
| Reliability features | Exponential backoff retry and correlation ID propagation |
| Message serialization | Converts data types to binary format (Base64, JSON, Arrow IPC) |
| NATS communication | Publishing and subscription via NATS subjects |
### Multi-Payload Support ### 2.2 Out of Scope
| Story | Priority | Acceptance Criteria |
|-------|----------|---------------------|
| **As a developer**, I want to send mixed-content messages (text + image + file) | P1 | NATSBridge accepts list of (dataname, data, type) tuples and handles each payload appropriately |
| **As a developer**, I want to receive multi-payload messages | P1 | NATSBridge returns payloads as list of tuples with correct types preserved |
### File Server Integration
| Story | Priority | Acceptance Criteria |
|-------|----------|---------------------|
| **As a developer**, I want to use Plik as the file server | P2 | Plik one-shot upload mode is supported with upload ID and token handling |
| **As a developer**, I want to use custom HTTP file servers | P2 | Handler function abstraction allows plugging in AWS S3 or custom implementations |
### Reliability Features
| Story | Priority | Acceptance Criteria |
|-------|----------|---------------------|
| **As a developer**, I want automatic retry on file server download failures | P1 | Exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) |
| **As a developer**, I want message tracing across distributed systems | P1 | Correlation ID is propagated through all message processing steps |
---
## Non-Functional Requirements
### Performance Requirements
| Requirement | Specification | Test Method |
|-------------|---------------|-------------|
| Message serialization overhead | <50ms for 10KB payload | Benchmark tests |
| Message deserialization overhead | <50ms for 10KB payload | Benchmark tests |
| NATS connection establishment | <100ms | Connection pool benchmarks |
| File upload latency | <1s for 0.5MB file | Integration tests |
| File download latency | <1s for 0.5MB file | Integration tests |
### Scalability Requirements
| Requirement | Specification |
|-------------|---------------|
| Concurrent connections | Support 100+ simultaneous NATS connections |
| Message throughput | Handle 1000+ messages/second per instance |
| File server scalability | Support horizontal scaling of file server backend |
### Reliability Requirements
| Requirement | Specification |
|-------------|---------------|
| Message delivery | At-least-once delivery semantics via NATS |
| File server availability | Graceful degradation when file server is unavailable |
| Connection recovery | Auto-reconnect on NATS connection failure |
### Security Requirements
| Requirement | Specification |
|-------------|---------------|
| Payload integrity | SHA-256 checksum support via metadata |
| Transport security | TLS support for NATS connections |
| File server security | Authentication token for file uploads |
### Compatibility Requirements
| Platform | Minimum Version | Notes |
|----------|-----------------|-------|
| Julia | 1.7+ | Arrow.jl required for arrowtable support |
| Node.js | 16+ | nats.js required, Arrow IPC supported |
| Python | 3.8+ | pyarrow required for arrowtable support |
| Browser | Latest | No Arrow IPC (uses jsontable only) |
| MicroPython | 1.19+ | Limited to direct transport |
---
## Out of Scope
### Phase 1 (Current Implementation)
| Feature | Reason | | Feature | Reason |
|---------|--------| |---------|--------|
@@ -134,62 +71,141 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless
| Persistent message queues | NATS request-reply pattern sufficient | | Persistent message queues | NATS request-reply pattern sufficient |
| Advanced routing rules | Simple NATS subject matching sufficient | | Advanced routing rules | Simple NATS subject matching sufficient |
### Future Considerations ### 2.3 Dependencies
| Feature | Future Phase | | Platform | Package | Version |
|---------|--------------| |----------|---------|---------|
| JetStream streams and consumers | Phase 2 | | Julia | NATS.jl | Latest stable |
| Message TTL and dead-letter queues | Phase 3 | | Julia | JSON.jl | Latest stable |
| Message tracing with OpenTelemetry | Phase 3 | | Julia | Arrow.jl | Latest stable |
| Rate limiting and quota management | Phase 4 | | Julia | HTTP.jl | Latest stable |
| Julia | UUIDs.jl | Latest stable |
| Node.js | nats | Latest stable |
| Node.js | node-fetch | Latest stable |
| Python | nats-py | Latest stable |
| Python | aiohttp | Latest stable |
| Python | pyarrow | Latest stable |
| Browser | nats.ws | Latest stable |
| Dart | nats | Latest stable |
| Dart | http | Latest stable |
| Dart | uuid | Latest stable |
| Rust | nats | Latest stable |
| Rust | serde | Latest stable |
| Rust | serde_json | Latest stable |
| Rust | tokio | Latest stable |
| Rust | uuid | Latest stable |
### 2.4 Platform Compatibility
| Platform | Minimum Version | Notes |
|----------|-----------------|-------|
| Julia | 1.7+ | Arrow.jl required for arrowtable support |
| Node.js | 16+ | nats.js required, Arrow IPC supported |
| Python | 3.8+ | pyarrow required for arrowtable support |
| Browser | Latest | No Arrow IPC (uses jsontable only) |
| Dart | 2.17+ | Supports Desktop (Dart SDK), Flutter (Dart SDK), and Web (Dart SDK) |
| Rust | 1.70+ | Full support with async/await, Arrow IPC on desktop |
| MicroPython | 1.19+ | Limited to direct transport |
--- ---
## Boundary Definitions ## 3. Functional Requirements (FR)
### What NATSBridge Handles | ID | Requirement | Description |
|----|-------------|-------------|
| Function | Description | | **FR-001** | Cross-platform text messaging | System shall allow users to send text messages between Julia, JavaScript, Python, and MicroPython applications |
|----------|-------------| | **FR-002** | Cross-platform tabular data | System shall support DataFrame exchange between Julia and Python applications using Arrow IPC format |
| Message serialization | Converts data types to binary format | | **FR-003** | Large file handling | System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via NATS |
| Message encoding | Base64, JSON, Arrow IPC encoding | | **FR-004** | Direct transport for small payloads | System shall send payloads <0.5MB directly via NATS without file server upload |
| Transport selection | Direct vs link based on size threshold | | **FR-005** | MicroPython support | System shall support payloads <100KB on MicroPython devices using direct transport |
| NATS publishing | Publishes messages to NATS subjects | | **FR-006** | Multi-payload messages | System shall accept and process lists of (dataname, data, type) tuples |
| NATS subscription | Receives and processes NATS messages | | **FR-007** | Payload type preservation | System shall preserve payload types when returning multi-payload messages |
| File server upload | Uploads large payloads to HTTP server | | **FR-008** | Plik file server integration | System shall support Plik one-shot upload mode with upload ID and token handling |
| File server download | Downloads payloads from HTTP server with retry | | **FR-009** | Custom file server support | System shall provide handler function abstraction for custom HTTP file server implementations |
| Correlation ID generation | Creates and propagates UUIDs | | **FR-010** | Exponential backoff retry | System shall implement exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) for file server download failures |
| Data deserialization | Converts binary format back to native types | | **FR-011** | Correlation ID propagation | System shall propagate correlation IDs through all message processing steps |
| **FR-012** | Message serialization | System shall serialize data types using Base64, JSON, or Arrow IPC encoding |
### What NATSBridge Does NOT Handle | **FR-013** | NATS publishing | System shall return JSON string representation for caller to publish to NATS subjects (caller is responsible for actual NATS publish) |
| **FR-014** | NATS subscription | System shall receive and process NATS messages by accepting JSON string from NATS payload |
| Function | Handled By |
|----------|------------|
| NATS server management | External NATS deployment |
| File server management | External HTTP server deployment |
| Application business logic | Application code using NATSBridge |
| Message encryption | Application layer |
| Message compression | Application layer |
| Authentication/Authorization | NATS server configuration |
--- ---
## Payload Type Requirements ## 4. Non-Functional Requirements (NFRs)
### Supported Payload Types ### 4.1 Performance & Scalability
| Type | Julia | JavaScript | Python | MicroPython | Description | | ID | Requirement | Specification | Test Method |
|------|-------|------------|--------|-------------|-------------| |----|-------------|---------------|-------------|
| `text` | `String` | `string` | `str` | `str` | Plain text strings | | **NFR-101** | Message serialization overhead | <50ms for 10KB payload | Benchmark tests |
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable data | | **NFR-102** | Message deserialization overhead | <50ms for 10KB payload | Benchmark tests |
| `arrowtable` | `DataFrame`, `Arrow.Table` | ❌ (Browser), ✅ (Node.js) | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) | | **NFR-103** | NATS connection establishment | <100ms | Connection pool benchmarks |
| `jsontable` | `Vector{NamedTuple}` | `Array<Object>` | `list[dict]` | ⚠️ | Tabular data (JSON) - **Only table type in Browser** | | **NFR-104** | File upload latency | <1s for 0.5MB file | Integration tests |
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image binary data | | **NFR-105** | File download latency | <1s for 0.5MB file | Integration tests |
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio binary data | | **NFR-106** | Concurrent connections | Support 100+ simultaneous NATS connections | Scale testing |
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video binary data | | **NFR-107** | Message throughput | Handle 1000+ messages/second per instance | Load testing |
| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `bytearray` | Generic binary data | | **NFR-108** | File server scalability | Support horizontal scaling of file server backend | Architecture review |
### Encoding Requirements ### 4.2 Availability & Reliability
| ID | Requirement | Specification |
|----|-------------|---------------|
| **NFR-201** | Message delivery | At-least-once delivery semantics via NATS |
| **NFR-202** | File server availability | Graceful degradation when file server is unavailable |
| **NFR-203** | Connection recovery | Auto-reconnect on NATS connection failure |
### 4.3 Privacy & Security
| ID | Requirement | Specification |
|----|-------------|---------------|
| **NFR-301** | Payload integrity | SHA-256 checksum support via metadata |
| **NFR-302** | Transport security | TLS support for NATS connections |
| **NFR-303** | File server security | Authentication token for file uploads |
### 4.4 Observability & Telemetry
| ID | Requirement | Specification |
|----|-------------|---------------|
| **NFR-401** | Required logs | `correlation_id`, `msg_id`, `timestamp`, `sender_name`, `receiver_name`, `payload_type`, `transport` |
| **NFR-402** | Critical metrics | `messages_sent_total`, `messages_received_total`, `file_upload_duration_seconds`, `file_download_duration_seconds`, `retry_attempts_total` |
| **NFR-403** | Tracing | Correlation ID propagation for request tracing |
| **NFR-404** | Alerting | `download_retry_exceeded` triggers alert when max retries exceeded |
| **NFR-405** | Retention | Logs: 30 days, Metrics: 1 year |
---
## 5. Acceptance Conditions
| Condition | Description |
|-----------|-------------|
| **AC-001** | All functional requirements FR-001 through FR-014 are implemented and tested |
| **AC-002** | All non-functional requirements NFR-101 through NFR-405 meet specified targets |
| **AC-003** | Cross-platform text message test passes (Julia ↔ JavaScript ↔ Python) |
| **AC-004** | Cross-platform tabular data test passes with Arrow IPC round-trip (Desktop) |
| **AC-005** | Cross-platform tabular data test passes with JSON table round-trip (Browser) |
| **AC-006** | Large file transfer test passes with file server upload/download |
| **AC-007** | Multi-payload mixed content test passes with all payload types in one message |
| **AC-008** | CI validation gates block PRs on specification violations |
| **AC-009** | Unit test coverage exceeds 80% |
| **AC-010** | Documentation is complete and includes walkthroughs, architecture, and runbook |
---
## 6. Payload Type Requirements
### 6.1 Supported Payload Types
| Type | Julia | JavaScript | Python | Dart | MicroPython | Description |
|------|-------|------------|--------|------|-------------|-------------|
| `text` | `String` | `string` | `str` | `String` | `String` | `str` | Plain text strings |
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `Map`, `serde_json::Value` | `String` | `dict` | JSON-serializable data |
| `arrowtable` | `DataFrame`, `Arrow.Table` | ❌ (Browser), ✅ (Node.js) | `pandas.DataFrame` | `List<Map>` (Desktop), `List<dynamic>` (Flutter) | `arrow2::Table` | ❌ | Tabular data (Arrow IPC) |
| `jsontable` | `Vector{NamedTuple}` | `Array<Object>` | `list[dict]` | `Vec<Map>` | ⚠️ | Tabular data (JSON) - **Only table type in Browser** |
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `Vec<u8>` | `bytearray` | Image binary data |
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `Vec<u8>` | `bytearray` | Audio binary data |
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `Vec<u8>` | `bytearray` | Video binary data |
| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `Uint8List` | `Vec<u8>` | `bytearray` | Generic binary data |
### 6.2 Encoding Requirements
| Payload Type | Encoding Method | Notes | | Payload Type | Encoding Method | Notes |
|--------------|-----------------|-------| |--------------|-----------------|-------|
@@ -201,27 +217,35 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless
--- ---
## Size Threshold Requirements ## 7. Size Threshold Requirements
### Direct Transport Threshold ### 7.1 Direct Transport Threshold
| Platform | Threshold | Notes | | Platform | Threshold | Notes |
|----------|-----------|-------| |----------|-----------|-------|
| Desktop (Julia/JS/Python) | 0.5MB | Default size threshold | | Desktop (Julia/JS/Python/Dart) | 0.5MB | Default size threshold |
| Dart Desktop | 0.5MB | Default size threshold |
| Dart Flutter | 0.5MB | Default size threshold |
| Dart Web | 0.5MB | Default size threshold |
| Rust | 0.5MB | Default size threshold |
| MicroPython | 100KB | Lower threshold for memory constraints | | MicroPython | 100KB | Lower threshold for memory constraints |
### Maximum Payload Size ### 7.2 Maximum Payload Size
| Platform | Maximum | Notes | | Platform | Maximum | Notes |
|----------|---------|-------| |----------|---------|-------|
| Desktop | Unlimited | Limited by NATS server configuration | | Desktop | Unlimited | Limited by NATS server configuration |
| Dart Desktop | Unlimited | Limited by NATS server configuration |
| Dart Flutter | Unlimited | Limited by NATS server configuration |
| Dart Web | Unlimited | Limited by NATS server configuration |
| Rust | Unlimited | Limited by NATS server configuration |
| MicroPython | 50KB | Hard limit due to 256KB-1MB memory | | MicroPython | 50KB | Hard limit due to 256KB-1MB memory |
--- ---
## Message Envelope Requirements ## 8. Message Envelope Requirements
### Required Fields ### 8.1 Required Fields
| Field | Type | Purpose | | Field | Type | Purpose |
|-------|------|---------| |-------|------|---------|
@@ -240,7 +264,7 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless
| `metadata` | Dict | Message-level metadata | | `metadata` | Dict | Message-level metadata |
| `payloads` | Array | List of payload objects | | `payloads` | Array | List of payload objects |
### Payload Fields ### 8.2 Payload Fields
| Field | Type | Purpose | | Field | Type | Purpose |
|-------|------|---------| |-------|------|---------|
@@ -255,9 +279,9 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless
--- ---
## Error Handling Requirements ## 9. Error Handling Requirements
### Error Codes ### 9.1 Error Codes
| Error | Condition | Response | | Error | Condition | Response |
|-------|-----------|----------| |-------|-----------|----------|
@@ -267,7 +291,7 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless
| `Unknown transport` | Invalid transport type | Throw error | | `Unknown transport` | Invalid transport type | Throw error |
| `NATS connection failed` | NATS unavailable | Throw error | | `NATS connection failed` | NATS unavailable | Throw error |
### Exception Handling ### 9.2 Exception Handling
| Scenario | Handler | | Scenario | Handler |
|----------|---------| |----------|---------|
@@ -278,9 +302,9 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless
--- ---
## Testing Requirements ## 10. Testing Requirements
### Unit Tests ### 10.1 Unit Tests
| Test Category | Coverage | Files | | Test Category | Coverage | Files |
|---------------|----------|-------| |---------------|----------|-------|
@@ -290,7 +314,7 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless
| File server upload | Plik integration | Platform-specific | | File server upload | Plik integration | Platform-specific |
| File server download | Exponential backoff | Platform-specific | | File server download | Exponential backoff | Platform-specific |
### Integration Tests ### 10.2 Integration Tests
| Test Scenario | Success Criteria | | Test Scenario | Success Criteria |
|-------------|-----------------| |-------------|-----------------|
@@ -302,18 +326,18 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless
--- ---
## API Contract ## 11. API Contract
### smartsend Signature ### 11.1 smartsend Signature
```julia ```julia
function smartsend( function smartsend(
subject::String, subject::String,
data::AbstractArray{Tuple{String, Any, String}}; data::AbstractArray{Tuple{String, T1, String}, 1};
broker_url::String = "nats://localhost:4222", broker_url::String = DEFAULT_BROKER_URL,
fileserver_url::String = "http://localhost:8080", fileserver_url::String = DEFAULT_FILESERVER_URL,
fileserver_upload_handler::Function = plik_oneshot_upload, fileserver_upload_handler::Function = plik_oneshot_upload,
size_threshold::Int = 1_000_000, size_threshold::Int = DEFAULT_SIZE_THRESHOLD,
correlation_id::String = string(uuid4()), correlation_id::String = string(uuid4()),
msg_purpose::String = "chat", msg_purpose::String = "chat",
sender_name::String = "NATSBridge", sender_name::String = "NATSBridge",
@@ -321,18 +345,18 @@ function smartsend(
receiver_id::String = "", receiver_id::String = "",
reply_to::String = "", reply_to::String = "",
reply_to_msg_id::String = "", reply_to_msg_id::String = "",
is_publish::Bool = true,
NATS_connection::Union{NATS.Connection, Nothing} = nothing,
msg_id::String = string(uuid4()), msg_id::String = string(uuid4()),
sender_id::String = string(uuid4()) sender_id::String = string(uuid4())
)::Tuple{msg_envelope_v1, String} )::Tuple{msg_envelope_v1, String} where {T1<:Any}
``` ```
### smartreceive Signature **Note**: NATS publishing is the caller's responsibility. `smartsend` returns `(env::msg_envelope_v1, env_json_str::String)`.
### 11.2 smartreceive Signature
```julia ```julia
function smartreceive( function smartreceive(
msg::NATS.Msg; msg_json_str::String;
fileserver_download_handler::Function = _fetch_with_backoff, fileserver_download_handler::Function = _fetch_with_backoff,
max_retries::Int = 5, max_retries::Int = 5,
base_delay::Int = 100, base_delay::Int = 100,
@@ -340,84 +364,75 @@ function smartreceive(
)::JSON.Object{String, Any} )::JSON.Object{String, Any}
``` ```
--- **Note**: Pass `String(nats_msg.payload)` from NATS subscription to `smartreceive`.
## Dependencies
### Required Dependencies
| Platform | Package | Version |
|----------|---------|---------|
| Julia | NATS.jl | Latest stable |
| Julia | JSON.jl | Latest stable |
| Julia | Arrow.jl | Latest stable |
| Julia | HTTP.jl | Latest stable |
| Julia | UUIDs.jl | Latest stable |
| Node.js | nats | Latest stable |
| Node.js | node-fetch | Latest stable |
| Python | nats-py | Latest stable |
| Python | aiohttp | Latest stable |
| Python | pyarrow | Latest stable |
| Browser | nats.ws | Latest stable |
### Optional Dependencies
| Platform | Package | Use Case |
|----------|---------|----------|
| Julia | DataFrames.jl | DataFrame support for arrowtable |
| Python | pandas | DataFrame support for arrowtable |
--- ---
## Deployment Requirements ## 12. Deployment Requirements
### Minimum Infrastructure ### 12.1 Minimum Infrastructure
| Component | Minimum | Notes | | Component | Minimum | Notes |
|-----------|---------|-------| |-----------|---------|-------|
| NATS Server | 1 instance | Single node for development | | NATS Server | 1 instance | Single node for development |
| File Server | 1 instance | HTTP server for large payloads | | File Server | 1 instance | HTTP server for large payloads |
| Client Memory | 50MB | Desktop platforms | | Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices | | Client Memory | 256KB | MicroPython devices |
### Environment Variables ### 12.2 Environment Variables
| Variable | Default | Description | | Variable | Default | Description |
|----------|---------|-------------| |----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL | | `NATS_URL` | `nats://localhost:4222` | NATS server URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL | | `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `1000000` | Size threshold in bytes | | `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) |
--- ---
## Versioning ## 13. Versioning
### Current Version ### 13.1 Current Version
- **Major**: 1 (Breaking changes require major version bump) - **Major**: 1 (Breaking changes require major version bump)
- **Minor**: 0 (Feature additions) - **Minor**: 0 (Feature additions)
- **Patch**: 0 (Bug fixes) - **Patch**: 0 (Bug fixes)
### Version Compatibility ### 13.2 Version Compatibility
| Version | Supported Platforms | | Version | Supported Platforms |
|---------|---------------------| |---------|---------------------|
| v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, Browser (latest), MicroPython 1.19+ | | v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, Dart 2.17+, Rust 1.70+, Browser (latest), MicroPython 1.19+ |
--- ---
## Change Log ## 14. Change Log
| Date | Version | Changes | | Date | Version | Changes |
|------|---------|---------| |------|---------|---------|
| 2026-03-13 | 1.0.0 | Initial requirements document | | 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/NATSBridge.jl) |
| - | - | Fixed smartsend signature: removed is_publish, NATS_connection; added sender_name |
| - | - | Fixed smartreceive signature: takes msg_json_str::String instead of msg::NATS.Msg |
| - | - | Fixed size_threshold default from 1,000,000 to 500,000 |
| - | - | Updated FR-013/FR-014 to reflect caller responsibility for NATS publishing |
| - | - | Updated FR-008/FR-009 to include file path upload overload |
| - | - | Updated SIZE_THRESHOLD env var default to 500000 |
| 2026-03-23 | 1.0.0 | Updated to ASG Framework requirements structure |
--- ---
## References ## 15. References
- [`src/NATSBridge.jl`](../src/NATSBridge.jl) - Ground truth implementation - [`src/NATSBridge.jl`](../src/NATSBridge.jl) - Ground truth implementation (Julia)
- [`src/natsbridge_ssr.js`](../src/natsbridge_ssr.js) - Server-side JavaScript implementation
- [`src/natsbridge_csr.js`](../src/natsbridge_csr.js) - Client-side JavaScript implementation
- [`src/natsbridge.py`](../src/natsbridge.py) - Python implementation
- [`src/natsbridge.dart`](../src/natsbridge.dart) - Dart implementation
- [`src/natsbridge_mpy.py`](../src/natsbridge_mpy.py) - MicroPython implementation
- [`src/natsbridge.rs`](../src/natsbridge.rs) - Rust implementation
- [`README.md`](../README.md) - Project overview - [`README.md`](../README.md) - Project overview
- [`docs/specification.md`](./specification.md) - Technical specification
- [`docs/ui-specification.md`](./ui-specification.md) - UI specification
- [`docs/walkthrough.md`](./walkthrough.md) - End-to-end walkthrough
- [`docs/architecture.md`](./architecture.md) - Architecture documentation - [`docs/architecture.md`](./architecture.md) - Architecture documentation
- [`docs/implementation.md`](./implementation.md) - Implementation details - [`docs/validation.md`](./validation.md) - Validation and CI/CD
- [`docs/walkthrough.md`](./walkthrough.md) - Usage examples - [`docs/runbook.md`](./runbook.md) - Operational runbook

View File

@@ -1,16 +1,16 @@
# Specification: NATSBridge # Specification: NATSBridge
**Version**: 1.1.0 **Version**: 1.2.0
**Date**: 2026-03-15 **Date**: 2026-05-13
**Status**: Active **Status**: Active
**Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl) **Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl)
**Specification Format**: JSON Schema + AsyncAPI **Specification Format**: JSON Schema + AsyncAPI
--- ---
## Executive Summary ## 1. Technical Contract Overview
This document defines the **technical contract** for NATSBridge - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, and **MicroPython** applications using NATS as the message bus. This document defines the **technical contract** for NATSBridge - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus.
This specification serves as the single source of truth for: This specification serves as the single source of truth for:
- **Inputs**: What data structures are accepted by `smartsend()` - **Inputs**: What data structures are accepted by `smartsend()`
@@ -18,8 +18,33 @@ This specification serves as the single source of truth for:
- **Data Shapes**: Exact field names, types, and constraints - **Data Shapes**: Exact field names, types, and constraints
- **Error Codes**: Standardized error responses for failure scenarios - **Error Codes**: Standardized error responses for failure scenarios
### 1.1 Requirements Traceability
| Specification Section | Requirement ID(s) | Description |
|----------------------|-------------------|-------------|
| Section 2 (Message Envelope) | FR-012, FR-013, NFR-101, NFR-102 | Message envelope structure and validation |
| Section 3 (Payload Schema) | FR-001, FR-002, FR-003, FR-004, NFR-101, NFR-102 | Payload structure and field definitions |
| Section 4 (Payload Format) | FR-006, FR-007 | Tuple format for smartsend() |
| Section 5 (Enumerations) | FR-003, FR-004, FR-006, NFR-101 | Enumerations for transport and encoding |
| Section 6 (Transport Protocols) | FR-003, FR-004, NFR-104, NFR-105 | Direct and link transport protocols |
| Section 7 (Size Thresholds) | FR-004, FR-005, NFR-104, NFR-105 | Size thresholds for transport selection |
| Section 8 (NATS Subject Convention) | FR-013, FR-014 | NATS subject naming patterns |
| Section 9 (Error Handling) | FR-010, FR-011, NFR-201, NFR-202, NFR-203 | Error codes and exception handling |
| Section 10 (Serialization Rules) | FR-001, FR-002, FR-003, FR-012, NFR-101, NFR-102 | Serialization and encoding rules |
| Section 11 (API Contract) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Function signatures for all platforms |
| Section 12 (File Server Interface) | FR-008, FR-009, FR-010 | Upload and download handler contracts |
| Section 13 (Platform-Specific Constraints) | FR-005, FR-006, NFR-106, NFR-107 | Platform-specific feature support |
| Section 14 (Implementation Files) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007 | Implementation file mapping |
| Section 15 (Message Flow) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Mermaid diagrams for send/receive flows |
| Section 16 (Validation Rules) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Envelope and payload validation rules |
| Section 17 (Test Contracts) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Unit and integration test scenarios |
| Section 18 (Dependencies) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Platform-specific dependencies |
| Section 19 (Change Log) | N/A | Version history and changes |
--- ---
## 2. Message Envelope Schema
## Specification Versioning ## Specification Versioning
| Component | Version | Notes | | Component | Version | Notes |
@@ -65,22 +90,22 @@ This specification serves as the single source of truth for:
### Field Definitions ### Field Definitions
| Field | Type | Required | Validation | Description | | Field | Type | Required | Validation | Description | Requirement ID |
|-------|------|----------|------------|-------------| |-------|------|----------|------------|-------------|----------------|
| `correlation_id` | `string` | Yes | UUID v4 format | Track message flow across distributed systems | | `correlation_id` | `string` | Yes | UUID v4 format | Track message flow across distributed systems | FR-011, NFR-401 |
| `msg_id` | `string` | Yes | UUID v4 format | Unique identifier for this specific message | | `msg_id` | `string` | Yes | UUID v4 format | Unique identifier for this specific message | FR-012, NFR-401 |
| `timestamp` | `string` | Yes | ISO 8601 UTC | Message publication timestamp (e.g., `2026-03-13T07:02:50.443Z`) | | `timestamp` | `string` | Yes | ISO 8601 UTC | Message publication timestamp | FR-012, NFR-401 |
| `send_to` | `string` | Yes | Non-empty string | NATS subject/topic to publish the message to | | `send_to` | `string` | Yes | Non-empty string | NATS subject/topic to publish the message to | FR-013 |
| `msg_purpose` | `string` | Yes | Enum | Purpose of the message (see `msg_purpose` enum) | | `msg_purpose` | `string` | Yes | Enum | Purpose of the message | FR-013 |
| `sender_name` | `string` | Yes | Non-empty string | Name of the sender application | | `sender_name` | `string` | Yes | Non-empty string | Name of the sender application | FR-013 |
| `sender_id` | `string` | Yes | UUID v4 format | Unique identifier for the sender | | `sender_id` | `string` | Yes | UUID v4 format | Unique identifier for the sender | FR-013 |
| `receiver_name` | `string` | Yes | Any string | Name of the receiver (empty = broadcast) | | `receiver_name` | `string` | Yes | Any string | Name of the receiver (empty = broadcast) | FR-013 |
| `receiver_id` | `string` | Yes | Any string | UUID of the receiver (empty = broadcast) | | `receiver_id` | `string` | Yes | Any string | UUID of the receiver (empty = broadcast) | FR-013 |
| `reply_to` | `string` | Yes | Any string | Topic where receiver should reply (empty = no reply expected) | | `reply_to` | `string` | Yes | Any string | Topic where receiver should reply | FR-013 |
| `reply_to_msg_id` | `string` | Yes | Any string | Message ID this message is replying to | | `reply_to_msg_id` | `string` | Yes | Any string | Message ID this message is replying to | FR-013 |
| `broker_url` | `string` | Yes | Valid URL | NATS broker URL | | `broker_url` | `string` | Yes | Valid URL | NATS broker URL | FR-013 |
| `metadata` | `object` | No | Any JSON object | Message-level metadata | | `metadata` | `object` | No | Any JSON object | Message-level metadata | NFR-401 |
| `payloads` | `array` | Yes | Non-empty array | List of payload objects | | `payloads` | `array` | Yes | Non-empty array | List of payload objects | FR-012, FR-013 |
--- ---
@@ -103,16 +128,16 @@ This specification serves as the single source of truth for:
### Payload Field Definitions ### Payload Field Definitions
| Field | Type | Required | Validation | Description | | Field | Type | Required | Validation | Description | Requirement ID |
|-------|------|----------|------------|-------------| |-------|------|----------|------------|-------------|----------------|
| `id` | `string` | Yes | UUID v4 format | Unique identifier for this payload | | `id` | `string` | Yes | UUID v4 format | Unique identifier for this payload | FR-012 |
| `dataname` | `string` | Yes | Non-empty string | Name of the payload (e.g., `login_image`, `user_data`) | | `dataname` | `string` | Yes | Non-empty string | Name of the payload (e.g., `login_image`, `user_data`) | FR-001, FR-002, FR-003 |
| `payload_type` | `string` | Yes | Enum | Type of payload (see `payload_type` enum) | | `payload_type` | `string` | Yes | Enum | Type of payload (see `payload_type` enum) | FR-001, FR-002, FR-003, FR-006 |
| `transport` | `string` | Yes | Enum | Transport method: `direct` or `link` | | `transport` | `string` | Yes | Enum | Transport method: `direct` or `link` | FR-003, FR-004, NFR-104, NFR-105 |
| `encoding` | `string` | Yes | Enum | Encoding method (see `encoding` enum) | | `encoding` | `string` | Yes | Enum | Encoding method (see `encoding` enum) | FR-001, FR-002, FR-003, FR-012 |
| `size` | `integer` | Yes | Positive integer | Size of the payload in bytes | | `size` | `integer` | Yes | Positive integer | Size of the payload in bytes | FR-003, FR-004, NFR-104, NFR-105 |
| `data` | `string` or `URL` | Yes | Base64 string or URL | Payload data (base64 for direct, URL for link) | | `data` | `string` or `URL` | Yes | Base64 string or URL | Payload data (base64 for direct, URL for link) | FR-003, FR-004, FR-008, FR-009, FR-010 |
| `metadata` | `object` | No | Any JSON object | Payload-level metadata | | `metadata` | `object` | No | Any JSON object | Payload-level metadata | NFR-401 |
--- ---
@@ -204,8 +229,8 @@ await smartsend("/agent/v1/process", data)
|-------|-------------|---------------------|------------------| |-------|-------------|---------------------|------------------|
| `text` | Plain text string | All | `base64` | | `text` | Plain text string | All | `base64` |
| `dictionary` | JSON object/dictionary | All | `base64`, `json` | | `dictionary` | JSON object/dictionary | All | `base64`, `json` |
| `arrowtable` | Apache Arrow IPC table | Desktop (Julia/Python/Node.js) | `base64`, `arrow-ipc` | | `arrowtable` | Apache Arrow IPC table | Desktop (Julia/Python/Node.js/Dart) | `base64`, `arrow-ipc` |
| `jsontable` | JSON array of objects | All (including Browser) | `base64`, `json` | | `jsontable` | JSON array of objects | All (including Browser/Dart Web) | `base64`, `json` |
| `image` | Binary image data | All | `base64` | | `image` | Binary image data | All | `base64` |
| `audio` | Binary audio data | All | `base64` | | `audio` | Binary audio data | All | `base64` |
| `video` | Binary video data | All | `base64` | | `video` | Binary video data | All | `base64` |
@@ -325,25 +350,25 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa
### Error Codes ### Error Codes
| Code | HTTP Status | Description | Recovery | | Code | HTTP Status | Description | Recovery | Requirement ID |
|------|-------------|-------------|----------| |------|-------------|-------------|----------|----------------|
| `INVALID_ENVELOPE` | 400 | Message envelope validation failed | Fix envelope structure | | `INVALID_ENVELOPE` | 400 | Message envelope validation failed | Fix envelope structure | FR-012, FR-013, FR-014 |
| `INVALID_PAYLOAD_TYPE` | 400 | Unsupported payload type | Use supported payload_type | | `INVALID_PAYLOAD_TYPE` | 400 | Unsupported payload type | Use supported payload_type | FR-001, FR-002, FR-003, FR-006 |
| `INVALID_TRANSPORT` | 400 | Unsupported transport type | Use `direct` or `link` | | `INVALID_TRANSPORT` | 400 | Unsupported transport type | Use `direct` or `link` | FR-003, FR-004, FR-006 |
| `UPLOAD_FAILED` | 500 | File server upload failed | Retry or use direct transport | | `UPLOAD_FAILED` | 500 | File server upload failed | Retry or use direct transport | FR-008, FR-009 |
| `DOWNLOAD_FAILED` | 503 | File server download failed | Retry with exponential backoff | | `DOWNLOAD_FAILED` | 503 | File server download failed | Retry with exponential backoff | FR-010, FR-011, NFR-201, NFR-202 |
| `NATS_CONNECTION_FAILED` | 503 | NATS connection failed | Check NATS server availability | | `NATS_CONNECTION_FAILED` | 503 | NATS connection failed | Check NATS server availability | FR-013, FR-014, NFR-201, NFR-203 |
| `DESERIALIZATION_ERROR` | 500 | Payload deserialization failed | Check payload_type matches data | | `DESERIALIZATION_ERROR` | 500 | Payload deserialization failed | Check payload_type matches data | FR-001, FR-002, FR-003, FR-012 |
| `SIZE_EXCEEDED` | 413 | Payload exceeds maximum size | Split payload or use link transport | | `SIZE_EXCEEDED` | 413 | Payload exceeds maximum size | Split payload or use link transport | FR-003, FR-004, FR-005, NFR-104, NFR-105 |
### Exception Handling ### Exception Handling
| Scenario | Handler | Retry Policy | | Scenario | Handler | Retry Policy | Requirement ID |
|----------|---------|--------------| |----------|---------|--------------|----------------|
| File server unavailable | Retry up to 5 times | Exponential backoff (100ms → 5000ms) | | File server unavailable | Retry up to 5 times | Exponential backoff (100ms → 5000ms) | FR-010, NFR-202 |
| NATS publish failure | Connection auto-reconnect | TCP-level reconnection | | NATS publish failure | Connection auto-reconnect | TCP-level reconnection | FR-013, FR-014, NFR-201, NFR-203 |
| Deserialization error | Log correlation ID and throw | No retry (data corruption) | | Deserialization error | Log correlation ID and throw | No retry (data corruption) | FR-001, FR-002, FR-003, FR-012, NFR-401 |
| Memory overflow (MicroPython) | Reject payloads >50KB | No retry (client-side check) | | Memory overflow (MicroPython) | Reject payloads >50KB | No retry (client-side check) | FR-005, NFR-106 |
--- ---
@@ -393,11 +418,11 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa
```julia ```julia
function smartsend( function smartsend(
subject::String, subject::String,
data::AbstractArray{Tuple{String, Any, String}}; data::AbstractArray{Tuple{String, T1, String}, 1};
broker_url::String = "nats://localhost:4222", broker_url::String = DEFAULT_BROKER_URL,
fileserver_url::String = "http://localhost:8080", fileserver_url::String = DEFAULT_FILESERVER_URL,
fileserver_upload_handler::Function = plik_oneshot_upload, fileserver_upload_handler::Function = plik_oneshot_upload,
size_threshold::Int = 500_000, size_threshold::Int = DEFAULT_SIZE_THRESHOLD,
correlation_id::String = string(uuid4()), correlation_id::String = string(uuid4()),
msg_purpose::String = "chat", msg_purpose::String = "chat",
sender_name::String = "NATSBridge", sender_name::String = "NATSBridge",
@@ -405,13 +430,13 @@ function smartsend(
receiver_id::String = "", receiver_id::String = "",
reply_to::String = "", reply_to::String = "",
reply_to_msg_id::String = "", reply_to_msg_id::String = "",
is_publish::Bool = true,
NATS_connection::Union{NATS.Connection, Nothing} = nothing,
msg_id::String = string(uuid4()), msg_id::String = string(uuid4()),
sender_id::String = string(uuid4()) sender_id::String = string(uuid4())
)::Tuple{msg_envelope_v1, String} )::Tuple{msg_envelope_v1, String} where {T1<:Any}
``` ```
**Note**: NATS publishing is the caller's responsibility. Returns `(env::msg_envelope_v1, env_json_str::String)`.
#### Python #### Python
```python ```python
@@ -429,13 +454,13 @@ async def smartsend(
receiver_id: str = "", receiver_id: str = "",
reply_to: str = "", reply_to: str = "",
reply_to_msg_id: str = "", reply_to_msg_id: str = "",
is_publish: bool = True,
nats_connection: Any = None,
msg_id: str = None, msg_id: str = None,
sender_id: str = None sender_id: str = None
) -> Tuple[Dict, str]: ) -> Tuple[Dict, str]:
``` ```
**Note**: NATS publishing is the caller's responsibility.
#### JavaScript (Node.js) #### JavaScript (Node.js)
```typescript ```typescript
@@ -454,14 +479,14 @@ async function smartsend(
receiver_id?: string; receiver_id?: string;
reply_to?: string; reply_to?: string;
reply_to_msg_id?: string; reply_to_msg_id?: string;
is_publish?: boolean;
nats_connection?: NATS.Connection;
msg_id?: string; msg_id?: string;
sender_id?: string; sender_id?: string;
} }
): Promise<[Object, string]>; ): Promise<[Object, string]>;
``` ```
**Note**: NATS publishing is the caller's responsibility.
#### JavaScript (Browser) #### JavaScript (Browser)
```typescript ```typescript
@@ -480,58 +505,145 @@ async function smartsend(
receiver_id?: string; receiver_id?: string;
reply_to?: string; reply_to?: string;
reply_to_msg_id?: string; reply_to_msg_id?: string;
is_publish?: boolean;
nats_connection?: NATSClient | NATS.Connection;
msg_id?: string; msg_id?: string;
sender_id?: string; sender_id?: string;
} }
): Promise<[Object, string]>; ): Promise<[Object, string]>;
// NATSClient class for connection management
class NATSClient {
constructor(url: string, keepAlive?: boolean);
connect(): Promise<NATS.Connection>;
publish(subject: string, message: string, correlationId: string): Promise<void>;
close(): Promise<void>;
getConnection(): NATS.Connection | null;
isConnected(): boolean;
}
// NATSConnectionPool for managing multiple connections
class NATSConnectionPool {
constructor(url: string, maxSize?: number);
acquire(): Promise<NATSClient>;
release(client: NATSClient): void;
closeAll(): Promise<void>;
}
// publishMessage function for manual publishing
async function publishMessage(
brokerUrlOrClient: string | NATSClient | NATS.Connection,
subject: string,
message: string,
correlationId: string,
closeConnection?: boolean
): Promise<void>;
``` ```
**Note**: NATS publishing is the caller's responsibility.
#### MicroPython #### MicroPython
```python ```python
def smartsend( def smartsend(
subject: str, subject: str,
data: List[Tuple[str, Any, str]], data: List[Tuple[str, Any, str]],
size_threshold: int = 100_000, # Lower threshold for memory constraints
**kwargs **kwargs
) -> Tuple[Dict, str]: ) -> Tuple[Dict, str]:
``` ```
**Note**: NATS publishing is the caller's responsibility.
#### Dart (Desktop/Flutter)
```dart
Future<[Map<String, dynamic>, String]> smartsend(
String subject,
List<List<dynamic>> data, {
String brokerUrl = 'nats://localhost:4222',
String fileserverUrl = 'http://localhost:8080',
Function? fileserverUploadHandler,
int sizeThreshold = 500000,
String? correlationId,
String msgPurpose = 'chat',
String senderName = 'NATSBridge',
String receiverName = '',
String receiverId = '',
String replyTo = '',
String replyToMsgId = '',
String? msgId,
String? senderId,
}) async {
// Returns [envelope, jsonString]
// NATS publishing is caller's responsibility
}
```
#### Dart Web
```dart
Future<[Map<String, dynamic>, String]> smartsend(
String subject,
List<List<dynamic>> data, {
String brokerUrl = 'nats://localhost:4222',
String fileserverUrl = 'http://localhost:8080',
Function? fileserverUploadHandler,
int sizeThreshold = 500000,
String? correlationId,
String msgPurpose = 'chat',
String senderName = 'NATSBridge',
String receiverName = '',
String receiverId = '',
String replyTo = '',
String replyToMsgId = '',
String? msgId,
String? senderId,
}) async {
// Returns [envelope, jsonString]
// NATS publishing is caller's responsibility
}
```
#### Rust
```rust
pub async fn smartsend(
subject: &str,
data: &[(String, Payload, String)],
options: &SmartsendOptions,
) -> Result<(MsgEnvelopeV1, String), NatSBridgeError>
// SmartsendOptions struct
pub struct SmartsendOptions {
pub broker_url: String,
pub fileserver_url: String,
pub fileserver_upload_handler: Option<UploadHandler>,
pub size_threshold: usize,
pub correlation_id: String,
pub msg_purpose: String,
pub sender_name: String,
pub receiver_name: String,
pub receiver_id: String,
pub reply_to: String,
pub reply_to_msg_id: String,
pub msg_id: String,
pub sender_id: String,
}
// Payload enum for type-safe data handling
#[derive(Serialize, Deserialize, Clone)]
pub enum Payload {
Text(String),
Dictionary(serde_json::Value),
ArrowTable(Vec<u8>),
JsonTable(serde_json::Value),
Image(Vec<u8>),
Audio(Vec<u8>),
Video(Vec<u8>),
Binary(Vec<u8>),
}
// MsgEnvelopeV1 struct (serde-serializable)
#[derive(Serialize, Deserialize, Clone)]
pub struct MsgEnvelopeV1 {
pub correlation_id: String,
pub msg_id: String,
pub timestamp: String,
pub send_to: String,
pub msg_purpose: String,
pub sender_name: String,
pub sender_id: String,
pub receiver_name: String,
pub receiver_id: String,
pub reply_to: String,
pub reply_to_msg_id: String,
pub broker_url: String,
pub metadata: serde_json::Value,
pub payloads: Vec<MsgPayloadV1>,
}
```
**Note**: NATS publishing is the caller's responsibility. Returns `Result<(MsgEnvelopeV1, String), NatSBridgeError>`. Uses `serde` for JSON serialization.
### `smartreceive` Function Signature ### `smartreceive` Function Signature
#### Julia #### Julia
```julia ```julia
function smartreceive( function smartreceive(
msg::NATS.Msg; msg_json_str::String; # Pass String(nats_msg.payload) from NATS subscription
fileserver_download_handler::Function = _fetch_with_backoff, fileserver_download_handler::Function = _fetch_with_backoff,
max_retries::Int = 5, max_retries::Int = 5,
base_delay::Int = 100, base_delay::Int = 100,
@@ -539,11 +651,13 @@ function smartreceive(
)::JSON.Object{String, Any} )::JSON.Object{String, Any}
``` ```
**Note**: Input is JSON string from NATS message payload, not NATS.Msg directly.
#### Python #### Python
```python ```python
async def smartreceive( async def smartreceive(
msg: Any, msg_json_str: str, # JSON string from NATS message payload
fileserver_download_handler: Callable = fetch_with_backoff, fileserver_download_handler: Callable = fetch_with_backoff,
max_retries: int = 5, max_retries: int = 5,
base_delay: int = 100, base_delay: int = 100,
@@ -551,11 +665,13 @@ async def smartreceive(
) -> Dict[str, Any]: ) -> Dict[str, Any]:
``` ```
**Note**: Input is JSON string from NATS message payload.
#### JavaScript (Node.js) #### JavaScript (Node.js)
```typescript ```typescript
async function smartreceive( async function smartreceive(
msg: Object, msg_json_str: string, // JSON string from NATS message payload
options?: { options?: {
fileserver_download_handler?: Function; fileserver_download_handler?: Function;
max_retries?: number; max_retries?: number;
@@ -569,7 +685,7 @@ async function smartreceive(
```typescript ```typescript
async function smartreceive( async function smartreceive(
msg: Object, msg_json_str: string, // JSON string from NATS message payload
options?: { options?: {
fileserver_download_handler?: Function; fileserver_download_handler?: Function;
max_retries?: number; max_retries?: number;
@@ -579,12 +695,65 @@ async function smartreceive(
): Promise<Object>; ): Promise<Object>;
``` ```
**Note**: Input is JSON string from NATS message payload.
#### MicroPython #### MicroPython
```python ```python
def smartreceive(msg: Any, **kwargs) -> Dict[str, Any]: def smartreceive(msg_json_str: str, **kwargs) -> Dict[str, Any]:
``` ```
**Note**: Input is JSON string from NATS message payload.
#### Dart (Desktop/Flutter)
```dart
Future<Map<String, dynamic>> smartreceive(
Map<String, dynamic> msg_json_str, // JSON object from NATS message payload
{
Function? fileserverDownloadHandler,
int maxRetries = 5,
int baseDelay = 100,
int maxDelay = 5000,
}) async {
// Returns envelope with processed payloads
}
```
#### Dart Web
```dart
Future<Map<String, dynamic>> smartreceive(
Map<String, dynamic> msg_json_str, // JSON object from NATS message payload
{
Function? fileserverDownloadHandler,
int maxRetries = 5,
int baseDelay = 100,
int maxDelay = 5000,
}) async {
// Returns envelope with processed payloads
}
```
#### Rust
```rust
pub async fn smartreceive(
msg_json_str: &str, // JSON string from NATS message payload
options: &SmartreceiveOptions,
) -> Result<MsgEnvelopeV1, NatSBridgeError>
// SmartreceiveOptions struct
pub struct SmartreceiveOptions {
pub fileserver_download_handler: Option<DownloadHandler>,
pub max_retries: u32,
pub base_delay: u64,
pub max_delay: u64,
}
```
**Note**: Input is JSON string from NATS message payload. Returns `Result<MsgEnvelopeV1, NatSBridgeError>`.
--- ---
## File Server Interface ## File Server Interface
@@ -598,6 +767,12 @@ function fileserver_upload_handler(
dataname::String, dataname::String,
data::Vector{UInt8} data::Vector{UInt8}
)::Dict{String, Any} )::Dict{String, Any}
# Overload: Upload file from disk
function fileserver_upload_handler(
file_server_url::String,
filepath::String
)::Dict{String, Any}
``` ```
**Return Format**: **Return Format**:
@@ -641,11 +816,11 @@ function fileserver_download_handler(
## Platform-Specific Constraints ## Platform-Specific Constraints
### Desktop (Julia/Python/Node.js) ### Desktop (Julia/Python/Node.js/Dart)
| Feature | Status | Notes | | Feature | Status | Notes |
|---------|--------|-------| |---------|--------|-------|
| Arrow IPC | ✅ Supported | Requires Arrow.jl/pyarrow | | Arrow IPC | ✅ Supported | Requires Arrow.jl/pyarrow/dart-arrow |
| JSON table | ✅ Supported | Human-readable format | | JSON table | ✅ Supported | Human-readable format |
| File server upload | ✅ Supported | HTTP/HTTPS | | File server upload | ✅ Supported | HTTP/HTTPS |
| File server download | ✅ Supported | HTTP/HTTPS | | File server download | ✅ Supported | HTTP/HTTPS |
@@ -661,6 +836,48 @@ function fileserver_download_handler(
| File server download | ✅ Supported | HTTP/HTTPS | | File server download | ✅ Supported | HTTP/HTTPS |
| Size threshold | 500KB | Configurable | | Size threshold | 500KB | Configurable |
### Dart Desktop (Dart SDK)
| Feature | Status | Notes |
|---------|--------|-------|
| Arrow IPC | ✅ Supported | Requires dart-arrow package |
| JSON table | ✅ Supported | Human-readable format |
| File server upload | ✅ Supported | HTTP/HTTPS |
| File server download | ✅ Supported | HTTP/HTTPS |
| Size threshold | 500KB | Configurable |
### Dart Flutter (Dart SDK)
| Feature | Status | Notes |
|---------|--------|-------|
| Arrow IPC | ✅ Supported | Requires dart-arrow package |
| JSON table | ✅ Supported | Human-readable format |
| File server upload | ✅ Supported | HTTP/HTTPS |
| File server download | ✅ Supported | HTTP/HTTPS |
| Size threshold | 500KB | Configurable |
### Dart Web (Dart SDK)
| Feature | Status | Notes |
|---------|--------|-------|
| Arrow IPC | ❌ Not supported | Apache Arrow not browser-compatible |
| JSON table | ✅ Supported | Only table type available in browser |
| File server upload | ✅ Supported | HTTP/HTTPS |
| File server download | ✅ Supported | HTTP/HTTPS |
| Size threshold | 500KB | Configurable |
### Rust
| Feature | Status | Notes |
|---------|--------|-------|
| Arrow IPC | ✅ Supported | Requires `arrow2` crate |
| JSON table | ✅ Supported | Uses `serde_json` |
| File server upload | ✅ Supported | HTTP/HTTPS via `reqwest` |
| File server download | ✅ Supported | HTTP/HTTPS via `reqwest` with retry |
| Size threshold | 500KB | Configurable |
| Async runtime | ✅ Supported | Uses `tokio` for async I/O |
| Type safety | ✅ Supported | Compile-time type checking via Rust enums |
### MicroPython ### MicroPython
| Feature | Status | Notes | | Feature | Status | Notes |
@@ -682,6 +899,8 @@ function fileserver_download_handler(
| [`src/natsbridge_ssr.js`](../src/natsbridge_ssr.js) | Node.js | Arrow IPC, async/await | Server-side JavaScript | | [`src/natsbridge_ssr.js`](../src/natsbridge_ssr.js) | Node.js | Arrow IPC, async/await | Server-side JavaScript |
| [`src/natsbridge_csr.js`](../src/natsbridge_csr.js) | Browser | JSON table only, WebSocket NATS | Client-side rendering | | [`src/natsbridge_csr.js`](../src/natsbridge_csr.js) | Browser | JSON table only, WebSocket NATS | Client-side rendering |
| [`src/natsbridge.py`](../src/natsbridge.py) | Python | Arrow IPC, async/await | Desktop Python | | [`src/natsbridge.py`](../src/natsbridge.py) | Python | Arrow IPC, async/await | Desktop Python |
| [`src/natsbridge.dart`](../src/natsbridge.dart) | Dart | Full feature set, Arrow IPC, async/await | Desktop/Flutter/Web |
| [`src/natsbridge.rs`](../src/natsbridge.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe | Uses tokio + serde + arrow2 |
| [`src/natsbridge_mpy.py`](../src/natsbridge_mpy.py) | MicroPython | Limited to direct transport | Memory-constrained | | [`src/natsbridge_mpy.py`](../src/natsbridge_mpy.py) | MicroPython | Limited to direct transport | Memory-constrained |
### Browser Implementation Notes ### Browser Implementation Notes
@@ -696,16 +915,16 @@ The browser implementation ([`src/natsbridge_csr.js`](../src/natsbridge_csr.js))
### Payload Type Availability by Platform ### Payload Type Availability by Platform
| Payload Type | Julia | Node.js | Browser | Python | MicroPython | | Payload Type | Julia | Node.js | Browser | Python | Dart | Rust | MicroPython |
|--------------|-------|---------|---------|--------|-------------| |--------------|-------|---------|---------|--------|------|------|-------------|
| `text` | ✅ | ✅ | ✅ | ✅ | ✅ | | `text` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `dictionary` | ✅ | ✅ | ✅ | ✅ | ✅ | | `dictionary` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `arrowtable` | ✅ | ✅ | ❌ | ✅ | ❌ | | `arrowtable` | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ |
| `jsontable` | ✅ | ✅ | ✅ | ✅ | ⚠️ | | `jsontable` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ⚠️ |
| `image` | ✅ | ✅ | ✅ | ✅ | ✅ | | `image` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `audio` | ✅ | ✅ | ✅ | ✅ | ✅ | | `audio` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `video` | ✅ | ✅ | ✅ | ✅ | ✅ | | `video` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `binary` | ✅ | ✅ | ✅ | ✅ | ✅ | | `binary` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
--- ---
@@ -759,23 +978,23 @@ flowchart TD
### Envelope Validation ### Envelope Validation
| Rule | Condition | Error Code | | Rule | Condition | Error Code | Requirement ID |
|------|-----------|------------| |------|-----------|------------|----------------|
| Required fields present | `correlation_id`, `msg_id`, `timestamp`, `send_to`, `payloads` | `INVALID_ENVELOPE` | | Required fields present | `correlation_id`, `msg_id`, `timestamp`, `send_to`, `payloads` | `INVALID_ENVELOPE` | FR-012, FR-013 |
| Valid UUID format | `correlation_id`, `msg_id`, `sender_id`, `receiver_id` | `INVALID_ENVELOPE` | | Valid UUID format | `correlation_id`, `msg_id`, `sender_id`, `receiver_id` | `INVALID_ENVELOPE` | FR-011, FR-012, NFR-401 |
| Valid timestamp format | ISO 8601 UTC | `INVALID_ENVELOPE` | | Valid timestamp format | ISO 8601 UTC | `INVALID_ENVELOPE` | FR-012, NFR-401 |
| Non-empty payloads array | `length(payloads) > 0` | `INVALID_ENVELOPE` | | Non-empty payloads array | `length(payloads) > 0` | `INVALID_ENVELOPE` | FR-012, FR-013 |
### Payload Validation ### Payload Validation
| Rule | Condition | Error Code | | Rule | Condition | Error Code | Requirement ID |
|------|-----------|------------| |------|-----------|------------|----------------|
| Valid payload_type | Must be in `payload_type` enum | `INVALID_PAYLOAD_TYPE` | | Valid payload_type | Must be in `payload_type` enum | `INVALID_PAYLOAD_TYPE` | FR-001, FR-002, FR-003, FR-006 |
| Valid transport | Must be `direct` or `link` | `INVALID_TRANSPORT` | | Valid transport | Must be `direct` or `link` | `INVALID_TRANSPORT` | FR-003, FR-004, FR-006 |
| Valid encoding | Must match payload_type and transport | `INVALID_TRANSPORT` | | Valid encoding | Must match payload_type and transport | `INVALID_TRANSPORT` | FR-001, FR-002, FR-003, FR-012 |
| Positive size | `size > 0` | `INVALID_PAYLOAD` | | Positive size | `size > 0` | `INVALID_PAYLOAD` | FR-003, FR-004, NFR-104, NFR-105 |
| Valid Base64 for direct | `data` matches Base64 pattern | `DESERIALIZATION_ERROR` | | Valid Base64 for direct | `data` matches Base64 pattern | `DESERIALIZATION_ERROR` | FR-001, FR-002, FR-003, FR-012 |
| Valid URL for link | `data` matches HTTP(S) URL pattern | `DOWNLOAD_FAILED` | | Valid URL for link | `data` matches HTTP(S) URL pattern | `DOWNLOAD_FAILED` | FR-008, FR-009, FR-010 |
--- ---
@@ -783,14 +1002,14 @@ flowchart TD
### Unit Test Validation ### Unit Test Validation
| Test | Input | Expected Output | Notes | | Test | Input | Expected Output | Notes | Requirement ID |
|------|-------|-----------------|-------| |------|-------|-----------------|-------|----------------|
| Text round-trip | `("msg", "Hello", "text")` | `("msg", "Hello", "text")` | String serialization | | Text round-trip | `("msg", "Hello", "text")` | `("msg", "Hello", "text")` | String serialization | FR-001, FR-012, NFR-101, NFR-102 |
| Dictionary round-trip | `("data", {"key": "value"}, "dictionary")` | `("data", {"key": "value"}, "dictionary")` | JSON object round-trip | | Dictionary round-trip | `("data", {"key": "value"}, "dictionary")` | `("data", {"key": "value"}, "dictionary")` | JSON object round-trip | FR-002, FR-012, NFR-101, NFR-102 |
| Arrow table round-trip | `("table", arrow_table_data, "arrowtable")` | `("table", arrow_table_data, "arrowtable")` | Arrow IPC round-trip | | Arrow table round-trip | `("table", arrow_table_data, "arrowtable")` | `("table", arrow_table_data, "arrowtable")` | Arrow IPC round-trip | FR-002, FR-012, NFR-101, NFR-102 |
| JSON table round-trip | `("table", [{"a":1},{"b":2}], "jsontable")` | `("table", [{"a":1},{"b":2}], "jsontable")` | JSON array of objects | | JSON table round-trip | `("table", [{"a":1},{"b":2}], "jsontable")` | `("table", [{"a":1},{"b":2}], "jsontable")` | JSON array of objects | FR-001, FR-002, FR-006, FR-012 |
| Mixed payloads | `[("msg", "Hello", "text"), ("imgname", bytes, "binary")]` | `[("msg", "Hello", "text"), ("imgname", bytes, "binary")]` | Multiple payload types | | Mixed payloads | `[("msg", "Hello", "text"), ("imgname", bytes, "binary")]` | `[("msg", "Hello", "text"), ("imgname", bytes, "binary")]` | Multiple payload types | FR-006, FR-007 |
| Large payload | `("data", rand(10_000_000), "arrowtable")` | `("data", URL, "arrowtable")` with link transport | File server upload | | Large payload | `("data", rand(10_000_000), "arrowtable")` | `("data", URL, "arrowtable")` with link transport | File server upload | FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105 |
**Platform-Specific Notes:** **Platform-Specific Notes:**
- **Julia**: Use `Dict`, `Vector{Dict}`, or convert `DataFrame` to dictionary for testing - **Julia**: Use `Dict`, `Vector{Dict}`, or convert `DataFrame` to dictionary for testing
@@ -800,24 +1019,24 @@ flowchart TD
### Integration Test Scenarios ### Integration Test Scenarios
| Scenario | Platforms | Payloads | Size Mix | Transport | Expected Result | | Scenario | Platforms | Payloads | Size Mix | Transport | Expected Result | Requirement ID |
|----------|-----------|----------|----------|-----------|-----------------| |----------|-----------|----------|----------|-----------|-----------------|----------------|
| Single text (small) | All | `text` | Small | direct | Round-trip successful | | Single text (small) | All | `text` | Small | direct | Round-trip successful | FR-001, FR-012, NFR-101, NFR-102 |
| Single dictionary (small) | All | `dictionary` | Small | direct | Round-trip successful | | Single dictionary (small) | All | `dictionary` | Small | direct | Round-trip successful | FR-002, FR-012, NFR-101, NFR-102 |
| Single arrow table (small) | Julia/JS/Python | `arrowtable` | Small | direct | Arrow IPC round-trip | | Single arrow table (small) | Julia/JS/Python | `arrowtable` | Small | direct | Arrow IPC round-trip | FR-002, FR-012, NFR-101, NFR-102 |
| Single JSON table (small) | All | `jsontable` | Small | direct | Dictionary array round-trip | | Single JSON table (small) | All | `jsontable` | Small | direct | Dictionary array round-trip | FR-001, FR-002, FR-006, FR-012 |
| Single image (small) | All | `image` | Small | direct | Binary round-trip | | Single image (small) | All | `image` | Small | direct | Binary round-trip | FR-001, FR-006, FR-012 |
| Single audio (small) | All | `audio` | Small | direct | Binary round-trip | | Single audio (small) | All | `audio` | Small | direct | Binary round-trip | FR-001, FR-006, FR-012 |
| Single video (small) | All | `video` | Small | direct | Binary round-trip | | Single video (small) | All | `video` | Small | direct | Binary round-trip | FR-001, FR-006, FR-012 |
| Single binary (small) | All | `binary` | Small | direct | Binary round-trip | | Single binary (small) | All | `binary` | Small | direct | Binary round-trip | FR-001, FR-006, FR-012 |
| Single text (large) | All | `text` | Large | link | File server upload/download | | Single text (large) | All | `text` | Large | link | File server upload/download | FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105 |
| Single JSON table (large) | All | `jsontable` | Large | link | File server upload/download | | Single JSON table (large) | All | `jsontable` | Large | link | File server upload/download | FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105 |
| Single image (large) | All | `image` | Large | link | File server upload/download | | Single image (large) | All | `image` | Large | link | File server upload/download | FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105 |
| **Ultimate Test** | Julia/JS/Python | `text` (small) + `dictionary` (small) + `arrowtable` (small) + `jsontable` (small) + `image` (small) + `audio` (small) + `video` (small) + `binary` (small) + `text` (large) + `dictionary` (large) + `arrowtable` (large) + `jsontable` (large) + `image` (large) | Mixed | direct/link | All payloads preserved with correct transport | | **Ultimate Test** | Julia/JS/Python | `text` (small) + `dictionary` (small) + `arrowtable` (small) + `jsontable` (small) + `image` (small) + `audio` (small) + `video` (small) + `binary` (small) + `text` (large) + `dictionary` (large) + `arrowtable` (large) + `jsontable` (large) + `image` (large) | Mixed | direct/link | All payloads preserved with correct transport | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 |
| **Ultimate Test** | MicroPython | `text` (small) + `dictionary` (small) + `text` (large) + `dictionary` (large) | Mixed | direct | Limited to text/dictionary with direct transport only | | **Ultimate Test** | MicroPython | `text` (small) + `dictionary` (small) + `text` (large) + `dictionary` (large) | Mixed | direct | Limited to text/dictionary with direct transport only | FR-005, FR-006, FR-012 |
| Cross-platform JSON table | All | `jsontable` | Small | direct | Dictionary array round-trip | | Cross-platform JSON table | All | `jsontable` | Small | direct | Dictionary array round-trip | FR-001, FR-002, FR-006, FR-012 |
| MicroPython ↔ Desktop | MicroPython ↔ Desktop | `text`/`dictionary` | Small | direct | Limited payload types | | MicroPython ↔ Desktop | MicroPython ↔ Desktop | `text`/`dictionary` | Small | direct | Limited payload types | FR-005, FR-006, FR-012 |
| Desktop ↔ Desktop (all combos) | Julia↔JS↔Python | All types | Small/Large | direct/link | Full compatibility | | Desktop ↔ Desktop (all combos) | Julia↔JS↔Python | All types | Small/Large | direct/link | Full compatibility | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 |
--- ---
@@ -839,6 +1058,17 @@ flowchart TD
| Python | nats-py | Latest | NATS client | | Python | nats-py | Latest | NATS client |
| Python | aiohttp | Latest | HTTP file server | | Python | aiohttp | Latest | HTTP file server |
| Python | pyarrow | Latest | Arrow IPC support | | Python | pyarrow | Latest | Arrow IPC support |
| Dart | nats | Latest | NATS client |
| Dart | http | Latest | HTTP file server |
| Dart | uuid | Latest | UUID generation |
| Dart | dart-arrow | Latest | Arrow IPC support (Desktop/Flutter) |
| Rust | nats | Latest | NATS client |
| Rust | serde | Latest | JSON serialization |
| Rust | serde_json | Latest | JSON handling |
| Rust | tokio | Latest | Async runtime |
| Rust | reqwest | Latest | HTTP file server |
| Rust | uuid | Latest | UUID generation |
| Rust | arrow2 | Latest | Arrow IPC support |
| MicroPython | builtin | N/A | Limited implementation | | MicroPython | builtin | N/A | Limited implementation |
### Optional Dependencies ### Optional Dependencies
@@ -871,11 +1101,74 @@ flowchart TD
## References ## References
- [`docs/requirements.md`](./requirements.md) - Business requirements and user stories ### 20.1 Documentation Artifacts
- [`docs/architecture.md`](./architecture.md) - System architecture diagrams
- [`docs/implementation.md`](./implementation.md) - Implementation details | Document | Purpose | Requirements Traceability |
- [`src/NATSBridge.jl`](../src/NATSBridge.jl) - Ground truth implementation |----------|---------|--------------------------|
- [`README.md`](../README.md) - Project overview | [`docs/requirements.md`](./requirements.md) | Business requirements and user stories | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/specification.md`](./specification.md) | Technical contract for NATSBridge | This document |
| [`docs/ui-specification.md`](./ui-specification.md) | UI specification for client applications | UI components for data entry and display |
| [`docs/walkthrough.md`](./walkthrough.md) | End-to-end system flow | Traceability from user journey to technical implementation |
| [`docs/architecture.md`](./architecture.md) | System architecture diagrams | Component interaction and data flow |
| [`docs/validation.md`](./validation.md) | CI/CD validation rules | Contract testing and spec compliance |
| [`docs/runbook.md`](./runbook.md) | Operational runbook | Deployment, scaling, and troubleshooting |
### 20.2 Implementation Files
| File | Platform | Features | Requirements Traceability |
|------|----------|----------|--------------------------|
| [`src/NATSBridge.jl`](../src/NATSBridge.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/natsbridge_ssr.js`](../src/natsbridge_ssr.js) | Node.js | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/natsbridge_csr.js`](../src/natsbridge_csr.js) | Browser | JSON table only, WebSocket NATS | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/natsbridge.py`](../src/natsbridge.py) | Python | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/natsbridge.dart`](../src/natsbridge.dart) | Dart | Full feature set, Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/natsbridge.rs`](../src/natsbridge.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`src/natsbridge_mpy.py`](../src/natsbridge_mpy.py) | MicroPython | Limited to direct transport | FR-005, FR-006, FR-012 |
### 20.3 External Dependencies
| Platform | Package | Version | Purpose | Requirements Traceability |
|----------|---------|---------|---------|--------------------------|
| Julia | NATS.jl | Latest | NATS client | FR-013, FR-014, NFR-201 |
| Julia | JSON.jl | Latest | JSON serialization | FR-012, NFR-101, NFR-102 |
| Julia | Arrow.jl | Latest | Arrow IPC support | FR-002, FR-012 |
| Julia | HTTP.jl | Latest | HTTP file server | FR-008, FR-009 |
| Julia | UUIDs.jl | Latest | UUID generation | FR-011, NFR-401 |
| Node.js | nats | Latest | NATS client (TCP) | FR-013, FR-014 |
| Node.js | node-fetch | Latest | HTTP file server | FR-008, FR-009 |
| Browser | nats.ws | Latest | NATS client (WebSocket) | FR-013, FR-014 |
| Browser | nats | Latest | NATS client (for bundling) | FR-013, FR-014 |
| Python | nats-py | Latest | NATS client | FR-013, FR-014 |
| Python | aiohttp | Latest | HTTP file server | FR-008, FR-009 |
| Python | pyarrow | Latest | Arrow IPC support | FR-002, FR-012 |
| Dart | nats | Latest | NATS client | FR-013, FR-014 |
| Dart | http | Latest | HTTP file server | FR-008, FR-009 |
| Dart | uuid | Latest | UUID generation | FR-011, NFR-401 |
| Dart | dart-arrow | Latest | Arrow IPC support | FR-002, FR-012 |
| Rust | nats | Latest | NATS client | FR-013, FR-014 |
| Rust | serde | Latest | JSON serialization | FR-012, NFR-101, NFR-102 |
| Rust | serde_json | Latest | JSON handling | FR-012, NFR-101, NFR-102 |
| Rust | tokio | Latest | Async runtime | FR-013, FR-014 |
| Rust | reqwest | Latest | HTTP file server | FR-008, FR-009 |
| Rust | uuid | Latest | UUID generation | FR-011, NFR-401 |
| Rust | arrow2 | Latest | Arrow IPC support | FR-002, FR-012 |
| MicroPython | builtin | N/A | Limited implementation | FR-005, FR-006 |
---
## 21. Change Log
| Date | Version | Changes | Requirement ID(s) |
|------|---------|---------|-------------------|
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/NATSBridge.jl) | All |
| - | - | Updated smartsend signatures: removed is_publish, nats_connection; added sender_name | FR-001 through FR-014 |
| - | - | Updated smartreceive signatures: takes msg_json_str::String instead of msg | FR-001 through FR-014 |
| - | - | Removed publishMessage function and NATSClient/NATSConnectionPool classes from browser section | FR-013, FR-014 |
| - | - | Added plik_oneshot_upload(filepath) overload to file server interface | FR-008, FR-009 |
| - | - | Fixed SIZE_THRESHOLD default to 500,000 bytes | FR-003, FR-004 |
| 2026-03-23 | 1.1.0 | Updated to ASG Framework specification guidelines | All |
| 2026-03-15 | 1.1.0 | Browser connection management | FR-001 through FR-014 |
| 2026-03-13 | 1.0.0 | Initial specification | FR-001 through FR-014, NFR-101 through NFR-405 |
--- ---

View File

@@ -1,23 +1,40 @@
# Walkthrough: NATSBridge # Walkthrough: NATSBridge
**Version**: 1.0.0 **Version**: 1.2.0
**Date**: 2026-03-13 **Date**: 2026-05-13
**Status**: Active **Status**: Active
**Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl) **Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl)
--- ---
## Executive Summary ## 1. Executive Summary
This document provides the **story of flow** for NATSBridge - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, and **MicroPython** applications using NATS as the message bus. This document provides the **end-to-end trace** for NATSBridge - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, **Rust**, and **MicroPython** applications using NATS as the message bus.
This walkthrough serves as the primary onboarding guide for new developers and explains: This walkthrough serves as the primary onboarding guide for new developers and explains:
- **User scenarios** - Real-world use cases from developer perspective - **User scenarios** - Real-world use cases from developer perspective
- **Why steps are sequenced** - The rationale behind architectural decisions - **Why steps are sequenced** - The rationale behind architectural decisions
- **What could go wrong** - Common failure scenarios and recovery strategies - **What could go wrong** - Common failure scenarios and recovery strategies
### 1.1 Specification Traceability
| Walkthrough Section | Specification Reference | Requirement ID(s) | Description |
|---------------------|-------------------------|-------------------|-------------|
| Section 2 (Big Picture) | specification.md:2, specification.md:15 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | End-to-end system flow diagrams |
| Section 3 (Chat Scenario) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | FR-001, FR-006, FR-007, FR-012, FR-013, FR-014 | Chat webapp ↔ Julia backend with mixed payloads |
| Section 4 (Large File) | specification.md:6, specification.md:7 | FR-003, FR-004, FR-008, FR-009, FR-010, NFR-104, NFR-105 | Large file transfer with link transport |
| Section 5 (Tabular Data) | specification.md:5, specification.md:10 | FR-002, FR-012, NFR-101, NFR-102 | Arrow IPC tabular data exchange |
| Section 6 (MicroPython) | specification.md:13, specification.md:17 | FR-005, FR-006, FR-012, NFR-106 | Memory-constrained device communication |
| Section 7 (Cross-Platform) | specification.md:3, specification.md:4, specification.md:5, specification.md:11 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | Multi-platform chat application |
| Section 8 (Error Handling) | specification.md:9 | FR-008, FR-009, FR-010, NFR-201, NFR-202, NFR-203 | Common error scenarios and recovery |
| Section 9 (Debugging) | specification.md:4, specification.md:11 | FR-011, NFR-401, NFR-403 | Correlation ID tracking |
| Section 10 (Performance) | specification.md:7, specification.md:13 | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 | Optimization strategies |
| Section 11 (Deployment) | specification.md:12, specification.md:18 | FR-013, FR-014, NFR-201, NFR-203 | Infrastructure requirements |
--- ---
## 2. Overview: The Big Picture
## Overview: The Big Picture ## Overview: The Big Picture
NATSBridge implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB): NATSBridge implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB):
@@ -196,23 +213,25 @@ NATSBridge builds the message envelope:
- **reply_to**: Tells backend where to send response - **reply_to**: Tells backend where to send response
- **payloads array**: Contains all data with metadata for proper handling - **payloads array**: Contains all data with metadata for proper handling
#### Step 5: Publish to NATS #### Step 5: Publish to NATS (Caller's Responsibility)
```javascript ```javascript
await NATSBridge.NATSClient.connect("ws://localhost:4222"); // NATS publishing is the caller's responsibility
await NATSBridge.NATSClient.publish("/agent/wine/api/v1/prompt", msgJson); const conn = await NATS.connect({ servers: "ws://localhost:4222" });
await conn.publish("/agent/wine/api/v1/prompt", msgJson);
``` ```
**Rationale**: **Rationale**:
- NATS provides low-latency message delivery - NATS provides low-latency message delivery
- JSON format ensures cross-platform compatibility - JSON format ensures cross-platform compatibility
- `smartsend()` returns `(env, msgJson)` - caller handles publishing
#### Step 6: Julia Backend Receives Message #### Step 6: Julia Backend Receives Message
```julia ```julia
# Julia backend # Julia backend
msg = NATS.subscription.next() # Get message from NATS nats_msg = NATS.subscription.next() # Get message from NATS
env = smartreceive(msg) env = smartreceive(String(nats_msg.payload))
# env["payloads"] is now: # env["payloads"] is now:
# [ # [
@@ -338,8 +357,8 @@ const response = await plikOneshotUpload(
```julia ```julia
# Julia backend # Julia backend
msg = NATS.subscription.next() nats_msg = NATS.subscription.next()
env = smartreceive(msg) env = smartreceive(String(nats_msg.payload))
# NATSBridge automatically: # NATSBridge automatically:
# 1. Extracts URL from payload # 1. Extracts URL from payload
@@ -411,8 +430,8 @@ arrow_bytes = buf.getvalue()
```julia ```julia
# Julia backend # Julia backend
msg = NATS.subscription.next() nats_msg = NATS.subscription.next()
env = smartreceive(msg) env = smartreceive(String(nats_msg.payload))
# env["payloads"][1] is now: # env["payloads"][1] is now:
# ("data", DataFrame with id, name, score columns, "arrowtable") # ("data", DataFrame with id, name, score columns, "arrowtable")
@@ -444,7 +463,155 @@ env, msg_json = smartsend(
--- ---
## User Scenario 4: MicroPython Device ## User Scenario 4: Rust Service with Type-Safe API
### Scenario Description
A Rust service needs to process messages from a Julia analytics pipeline and send typed results back. The Rust implementation leverages compile-time type safety via Rust enums and serde for serialization.
### Step-by-Step Flow
#### Step 1: Rust Service Receives Message
```rust
// Rust service - using tokio async runtime
use natsbridge::{smartreceive, MsgEnvelopeV1};
#[tokio::main]
async fn main() {
let conn = nats::connect("nats://localhost:4222").unwrap();
// Subscribe and receive messages
let mut sub = conn.subscribe("/agent/wine/api/v1/analyze").unwrap();
for msg in sub.messages() {
let envelope: MsgEnvelopeV1 = smartreceive(
&String::from_utf8_lossy(&msg.payload),
&Default::default(),
).await.unwrap();
// Type-safe payload access
for payload in &envelope.payloads {
match &payload.data {
Payload::ArrowTable(arrow_bytes) => {
// Process Arrow IPC data using arrow2
let table = arrow2::io::ipc::read::Reader::new(
std::io::Cursor::new(arrow_bytes.clone()),
);
println!("Received {} rows", table.len());
},
Payload::Text(text) => {
println!("Message: {}", text);
},
_ => println!("Received {} bytes of {} data",
match &payload.data {
Payload::Binary(b) => b.len(),
_ => 0,
},
payload.payload_type),
}
}
}
}
```
**Rationale**:
- **Type-safe payloads**: Rust enum discriminates between payload types at compile time
- **serde serialization**: Automatic JSON deserialization to `MsgEnvelopeV1`
- **tokio runtime**: Efficient async I/O for NATS and HTTP operations
- **arrow2 integration**: Direct Arrow IPC deserialization without intermediate format
#### Step 2: Rust Service Sends Processed Results
```rust
// Rust service sends results back with mixed payload types
use natsbridge::{smartsend, Payload, SmartsendOptions};
let results_df = /* processed Arrow table */;
let result_bytes = /* serialize to Arrow IPC */;
let (envelope, json_str) = smartsend(
"/agent/wine/api/v1/results",
&[
(
"results".to_string(),
Payload::ArrowTable(result_bytes),
"arrowtable".to_string(),
),
(
"summary".to_string(),
Payload::Text("Analysis complete: 1500 rows processed".to_string()),
"text".to_string(),
),
],
&SmartsendOptions {
broker_url: "nats://localhost:4222".to_string(),
reply_to: "/python/worker/v1/results".to_string(),
msg_purpose: "chat".to_string(),
..Default::default()
},
).await?;
// Caller publishes to NATS
conn.publish("/agent/wine/api/v1/results", &json_str)?;
```
**Rationale**:
- **Builder pattern**: `SmartsendOptions` provides clean configuration
- **Enum-based payloads**: Type safety prevents sending incorrect data types
- **Default options**: sensible defaults reduce boilerplate
- **Result<T, E>**: idiomatic Rust error handling
#### Step 3: Python/Julia Receives Rust Response
```python
# Python backend receives Rust response
env = await smartreceive(str(nats_msg.payload))
# env["payloads"][0] is now:
# ("results", arrow_table_data, "arrowtable")
# env["payloads"][1] is now:
# ("summary", "Analysis complete: 1500 rows processed", "text")
```
**Rationale**:
- **Cross-platform parity**: Rust envelope matches other platform envelopes exactly
- **Same JSON wire format**: No protocol translation needed
- **Type preservation**: Arrow IPC and text types preserved across all platforms
#### Step 4: Large File Transfer from Rust
```rust
// Rust service sends large binary file via link transport
let large_file_data: Vec<u8> = std::fs::read("/data/large_dataset.parquet")?;
let (envelope, json_str) = smartsend(
"/agent/wine/api/v1/upload",
&[
(
"dataset".to_string(),
Payload::Binary(large_file_data),
"binary".to_string(),
),
],
&SmartsendOptions {
broker_url: "nats://localhost:4222".to_string(),
fileserver_url: "http://localhost:8080".to_string(),
size_threshold: 500_000, // 0.5MB triggers link transport
..Default::default()
},
).await?;
```
**Rationale**:
- **Automatic transport selection**: Same 0.5MB threshold as other desktop platforms
- **reqwest integration**: Efficient HTTP client for file server upload/download
- **Exponential backoff**: Built-in retry with configurable parameters
- **Zero-copy where possible**: `Vec<u8>` passed directly without intermediate copies
---
## User Scenario 5: MicroPython Device
### Scenario Description ### Scenario Description
@@ -495,8 +662,8 @@ payload_b64 = base64.b64encode(json_bytes).decode('ascii')
```python ```python
# Python backend # Python backend
msg = await nats_consumer.next() nats_msg = await nats_consumer.next()
env = await smartreceive(msg) env = await smartreceive(str(nats_msg.payload))
# env["payloads"][0] is now: # env["payloads"][0] is now:
# ("data", {"temperature": 25.5, "humidity": 60.0, ...}, "dictionary") # ("data", {"temperature": 25.5, "humidity": 60.0, ...}, "dictionary")
@@ -509,7 +676,7 @@ env = await smartreceive(msg)
--- ---
## User Scenario 5: Cross-Platform Chat with Mixed Payloads ## User Scenario 6: Cross-Platform Chat with Mixed Payloads
### Scenario Description ### Scenario Description
@@ -544,8 +711,8 @@ const [env, msgJson] = await NATSBridge.smartsend(
```python ```python
# Python (Backend) # Python (Backend)
msg = await nats_consumer.next() nats_msg = await nats_consumer.next()
env = await smartreceive(msg) env = await smartreceive(str(nats_msg.payload))
# env["payloads"] is now: # env["payloads"] is now:
# [ # [
@@ -563,8 +730,8 @@ env = await smartreceive(msg)
```julia ```julia
# Julia (Backend) # Julia (Backend)
msg = NATS.subscription.next() nats_msg = NATS.subscription.next()
env = smartreceive(msg) env = smartreceive(String(nats_msg.payload))
# env["payloads"] is now: # env["payloads"] is now:
# [ # [
@@ -684,7 +851,10 @@ log_trace(correlation_id, "Published to NATS")
| Platform | Threshold | Notes | | Platform | Threshold | Notes |
|----------|-----------|-------| |----------|-----------|-------|
| Desktop (Julia/JS/Python) | 500,000 bytes (0.5MB) | Default threshold | | Desktop (Julia/JS/Python/Dart) | 500,000 bytes (0.5MB) | Default threshold |
| Dart Desktop | 500,000 bytes (0.5MB) | Default threshold |
| Dart Flutter | 500,000 bytes (0.5MB) | Default threshold |
| Dart Web | 500,000 bytes (0.5MB) | Default threshold |
| MicroPython | 100,000 bytes (100KB) | Lower threshold for memory constraints | | MicroPython | 100,000 bytes (100KB) | Lower threshold for memory constraints |
--- ---
@@ -697,7 +867,7 @@ log_trace(correlation_id, "Published to NATS")
|-----------|---------|-------| |-----------|---------|-------|
| NATS Server | 1 instance | Single node for development | | NATS Server | 1 instance | Single node for development |
| File Server | 1 instance | HTTP server for large payloads | | File Server | 1 instance | HTTP server for large payloads |
| Client Memory | 50MB | Desktop platforms | | Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
| Client Memory | 256KB | MicroPython devices | | Client Memory | 256KB | MicroPython devices |
### Environment Variables ### Environment Variables
@@ -706,7 +876,7 @@ log_trace(correlation_id, "Published to NATS")
|----------|---------|-------------| |----------|---------|-------------|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL | | `NATS_URL` | `nats://localhost:4222` | NATS server URL |
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL | | `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
| `SIZE_THRESHOLD` | `1000000` | Size threshold in bytes | | `SIZE_THRESHOLD` | `500000` | Size threshold in bytes (0.5MB) |
--- ---
@@ -718,13 +888,62 @@ log_trace(correlation_id, "Published to NATS")
--- ---
## References ## 12. References
- [`docs/requirements.md`](./requirements.md) - Business requirements and user stories ### 12.1 Documentation Artifacts
- [`docs/spec.md`](./spec.md) - Technical specification and contracts
- [`docs/architecture.md`](./architecture.md) - System architecture diagrams | Document | Purpose | Specification Traceability |
- [`src/NATSBridge.jl`](../src/NATSBridge.jl) - Ground truth implementation |----------|---------|---------------------------|
- [`README.md`](../README.md) - Project overview | [`docs/requirements.md`](./requirements.md) | Business requirements and user stories | FR-001 through FR-014, NFR-101 through NFR-405 |
| [`docs/specification.md`](./specification.md) | Technical contract for NATSBridge | specification.md:2-19 (all sections) |
| [`docs/ui-specification.md`](./ui-specification.md) | UI specification for client applications | UI components for data entry and display |
| [`docs/walkthrough.md`](./walkthrough.md) | End-to-end system flow | This document |
| [`docs/architecture.md`](./architecture.md) | System architecture diagrams | Component interaction and data flow |
| [`docs/validation.md`](./validation.md) | CI/CD validation rules | Contract testing and spec compliance |
| [`docs/runbook.md`](./runbook.md) | Operational runbook | Deployment, scaling, and troubleshooting |
### 12.2 Implementation Files
| File | Platform | Features | Specification Traceability |
|------|----------|----------|---------------------------|
| [`src/NATSBridge.jl`](../src/NATSBridge.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) |
| [`src/natsbridge_ssr.js`](../src/natsbridge_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/natsbridge_csr.js`](../src/natsbridge_csr.js) | Browser | JSON table only, WebSocket NATS | specification.md:2-19 (all sections) |
| [`src/natsbridge.py`](../src/natsbridge.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/natsbridge.dart`](../src/natsbridge.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) |
| [`src/natsbridge.rs`](../src/natsbridge.rs) | Rust | Full feature set, Arrow IPC, async/await, type-safe | specification.md:2-19 (all sections) |
| [`src/natsbridge_mpy.py`](../src/natsbridge_mpy.py) | MicroPython | Limited to direct transport | specification.md:2-19 (all sections) |
---
## 13. Change Log
| Date | Version | Changes | Specification Reference |
|------|---------|---------|------------------------|
| 2026-05-13 | 1.3.0 | Added Rust support with tokio, serde, and arrow2 | All sections |
| - | - | Added Rust user scenario (User Scenario 4) | specification.md:11 (Rust API) |
| - | - | Updated scenario numbering (MicroPython → Scenario 5, Cross-Platform → Scenario 6) | All sections |
| 2026-05-13 | 1.2.0 | Aligned with ground truth implementation (src/NATSBridge.jl) | All sections |
| - | - | Updated smartreceive calls to use String(nats_msg.payload) pattern | All sections |
| - | - | Removed NATSClient.publish() calls (caller responsible for NATS publishing) | All sections |
| - | - | Removed is_publish and nats_connection parameter references | All sections |
| 2026-03-23 | 1.0.0 | Updated to ASG Framework walkthrough guidelines | All sections |
| 2026-03-13 | 1.0.0 | Initial walkthrough documentation | specification.md:2-19 (all sections) |
---
## 14. Gap-Check Validation
| Stage Transition | Gap-Check Question | Status |
|------------------|-------------------|--------|
| Requirements → Specification | Does the Specification define all edge cases and conflict scenarios from the Requirements? | ✅ Verified - All FR-XXX requirements have corresponding spec rules |
| Specification → UI Specification | Does the UI Specification expose all the data and states defined in the Specification? | ⏳ Pending - UI spec not yet created |
| UI Specification → Walkthrough | Does the Walkthrough reflect the complete flow including error states and timing? | ⏳ Pending - UI spec not yet created |
| Walkthrough → Architecture | Does the Architecture support the performance and integration requirements defined in the Walkthrough? | ⏳ Pending - Architecture not yet created |
---
*This walkthrough document is versioned and maintained in git alongside the codebase. All implementations must adhere to this documentation.*
--- ---

View File

@@ -0,0 +1,96 @@
use natsbridge::{smartreceive, SmartreceiveOptions};
#[tokio::main]
async fn main() {
// Simulated NATS message JSON (received from NATS subscription)
let msg_json_str = r#"{
"correlation_id": "abc123-def456-ghi789",
"msg_id": "msg-uuid-001",
"timestamp": "2026-05-13T12:00:00.000Z",
"send_to": "/agent/wine/api/v1/prompt",
"msg_purpose": "chat",
"sender_name": "js-webapp",
"sender_id": "sender-uuid-001",
"receiver_name": "rust-backend",
"receiver_id": "",
"reply_to": "/agent/wine/api/v1/response",
"reply_to_msg_id": "",
"broker_url": "nats://localhost:4222",
"metadata": {},
"payloads": [
{
"id": "payload-uuid-001",
"dataname": "message",
"payload_type": "text",
"transport": "direct",
"encoding": "base64",
"size": 29,
"data": "SGVsbG8gZnJvbSBKYXZhU2NyaXB0ISE=",
"metadata": {"payload_bytes": 29}
},
{
"id": "payload-uuid-002",
"dataname": "user_data",
"payload_type": "dictionary",
"transport": "direct",
"encoding": "json",
"size": 58,
"data": "eyJ0eXBlIjoiY2hhdCIsInNlbmRlciI6InNlcnZpY2VBIiwicmVjZWl2ZXIiOiJzZXJ2aWNlQiJ9",
"metadata": {"payload_bytes": 58}
}
]
}"#;
let options = SmartreceiveOptions::default();
match smartreceive(msg_json_str, &options).await {
Ok(envelope) => {
println!("=== Envelope Received ===");
println!("Correlation ID: {}", envelope.correlation_id);
println!("Message ID: {}", envelope.msg_id);
println!("Subject: {}", envelope.send_to);
println!("Purpose: {}", envelope.msg_purpose);
println!("Sender: {}", envelope.sender_name);
println!("Receiver: {}", envelope.receiver_name);
println!("Payloads: {}", envelope.payloads.len());
println!();
for payload in &envelope.payloads {
println!("--- Payload: {} ---", payload.dataname);
println!(" Type: {}", payload.payload_type);
println!(" Transport: {}", payload.transport);
println!(" Encoding: {}", payload.encoding);
println!(" Size: {} bytes", payload.size);
// In a real scenario, you would deserialize payload.data here
// based on payload_type to get the actual data
match payload.payload_type.as_str() {
"text" => {
// For demonstration, decode the base64
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64};
if payload.transport == "direct" {
let decoded = BASE64.decode(&payload.data).unwrap();
println!(" Data: {}", String::from_utf8_lossy(&decoded));
} else {
println!(" URL: {}", payload.data);
}
}
"dictionary" => {
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64};
if payload.transport == "direct" {
let decoded = BASE64.decode(&payload.data).unwrap();
let json: serde_json::Value = serde_json::from_slice(&decoded).unwrap();
println!(" Data: {}", serde_json::to_string_pretty(&json).unwrap());
}
}
other => {
println!(" Data type: {}", other);
}
}
}
}
Err(e) => {
eprintln!("Error: {}", e);
}
}
}

View File

@@ -0,0 +1,70 @@
use natsbridge::{smartsend, Payload, SmartsendOptions};
#[tokio::main]
async fn main() {
// Create mixed payload data
let payloads = vec![
(
"message".to_string(),
Payload::Text("Hello from Rust!".to_string()),
"text".to_string(),
),
(
"user_data".to_string(),
Payload::Dictionary(serde_json::json!({
"name": "Alice",
"role": "admin",
"scores": [95, 88, 92]
})),
"dictionary".to_string(),
),
(
"avatar".to_string(),
Payload::Binary(vec![0x89, 0x50, 0x4E, 0x47]), // PNG header
"image".to_string(),
),
];
let options = SmartsendOptions {
broker_url: "nats://localhost:4222".to_string(),
fileserver_url: "http://localhost:8080".to_string(),
msg_purpose: "chat".to_string(),
sender_name: "rust-example".to_string(),
..Default::default()
};
match smartsend("/agent/wine/api/v1/prompt", &payloads, &options).await {
Ok((envelope, json_str)) => {
println!("=== Envelope Created ===");
println!("Correlation ID: {}", envelope.correlation_id);
println!("Message ID: {}", envelope.msg_id);
println!("Timestamp: {}", envelope.timestamp);
println!("Subject: {}", envelope.send_to);
println!("Purpose: {}", envelope.msg_purpose);
println!("Sender: {}", envelope.sender_name);
println!("Payloads: {}", envelope.payloads.len());
println!();
for payload in &envelope.payloads {
println!("Payload: {} (type: {}, transport: {}, encoding: {})",
payload.dataname,
payload.payload_type,
payload.transport,
payload.encoding);
println!(" Size: {} bytes", payload.size);
println!(" Data: {}", if payload.transport == "direct" {
&payload.data[..payload.data.len().min(40)]
} else {
&payload.data[..payload.data.len().min(60)]
});
}
println!();
println!("=== JSON String for NATS Publishing ===");
println!("{}", json_str);
}
Err(e) => {
eprintln!("Error: {}", e);
}
}
}

View File

@@ -43,7 +43,7 @@
module NATSBridge module NATSBridge
using NATS, JSON, Arrow, HTTP, UUIDs, Dates, Base64, PrettyPrinting, DataFrames using JSON, Arrow, HTTP, UUIDs, Dates, Base64, PrettyPrinting, DataFrames
# ---------------------------------------------- 100 --------------------------------------------- # # ---------------------------------------------- 100 --------------------------------------------- #
# Constants # Constants
@@ -340,25 +340,29 @@ end
""" smartsend - Send data either directly via NATS or via a fileserver URL, depending on payload size """ smartsend - Send data either directly via NATS or via a fileserver URL, depending on payload size
This function intelligently routes data delivery based on payload size relative to a threshold. This function intelligently routes data delivery based on payload size relative to a threshold.
If the serialized payload is smaller than `size_threshold`, it encodes the data as Base64 and publishes directly over NATS. If the serialized payload is smaller than `size_threshold`, it encodes the data as Base64 and constructs a "direct" msg_payload_v1.
Otherwise, it uploads the data to a fileserver (by default using `plik_oneshot_upload`) and publishes only the download URL over NATS. Otherwise, it uploads the data to a fileserver (by default using `plik_oneshot_upload`) and constructs a "link" msg_payload_v1 with the download URL.
The function accepts a list of (dataname, data, type) tuples as input and processes each payload individually. The function accepts a list of (dataname, data, type) tuples as input and processes each payload individually.
Each payload can have a different type, enabling mixed-content messages (e.g., chat with text, images, audio). Each payload can have a different type, enabling mixed-content messages (e.g., chat with text, images, audio).
This function creates and returns the msg_envelope_v1 and its JSON string representation only.
NATS publishing must be performed by the caller.
# Function Workflow: # Function Workflow:
1. Iterates through the list of (dataname, data, type) tuples 1. Iterates through the list of (dataname, data, type) tuples
2. For each payload: extracts the type from the tuple and serializes accordingly 2. For each payload: extracts the type from the tuple and serializes accordingly
3. Compares the serialized size against `size_threshold` 3. Compares the serialized size against `size_threshold`
4. For small payloads: encodes as Base64, constructs a "direct" msg_payload_v1 4. For small payloads: encodes as Base64, constructs a "direct" msg_payload_v1
5. For large payloads: uploads to the fileserver, constructs a "link" msg_payload_v1 with the URL 5. For large payloads: uploads to the fileserver, constructs a "link" msg_payload_v1 with the URL
6. Converts envelope to JSON string and optionally publishes to NATS 6. Constructs msg_envelope_v1 with all payloads and metadata
7. Converts envelope to JSON string and returns (NATS publishing is handled by the caller)
# Arguments: # Arguments:
- `subject::String` - NATS subject to publish the message to - `subject::String` - NATS subject to publish the message to
- `data::AbstractArray{Tuple{String, Any, String}}` - List of (dataname, data, type) tuples to send - `data::AbstractArray{Tuple{String, T1, String}, 1}` - List of (dataname, data, type) tuples to send
- `dataname::String` - Name of the payload - `dataname::String` - Name of the payload
- `data::Any` - The actual data to send - `data::T1` - The actual data to send (any type supported by `_serialize_data`)
- `payload_type::String` - Payload type: "text", "dictionary", "arrowtable", "jsontable", "image", "audio", "video", "binary" - `payload_type::String` - Payload type: "text", "dictionary", "arrowtable", "jsontable", "image", "audio", "video", "binary"
- No standalone `type` parameter - type is specified per payload - No standalone `type` parameter - type is specified per payload
@@ -367,17 +371,15 @@ Each payload can have a different type, enabling mixed-content messages (e.g., c
- `fileserver_url = DEFAULT_FILESERVER_URL` - URL of the HTTP file server for large payloads - `fileserver_url = DEFAULT_FILESERVER_URL` - URL of the HTTP file server for large payloads
- `fileserver_upload_handler::Function = plik_oneshot_upload` - Function to handle fileserver uploads (must return Dict with "status", "uploadid", "fileid", "url" keys) - `fileserver_upload_handler::Function = plik_oneshot_upload` - Function to handle fileserver uploads (must return Dict with "status", "uploadid", "fileid", "url" keys)
- `size_threshold::Int = DEFAULT_SIZE_THRESHOLD` - Threshold in bytes separating direct vs link transport - `size_threshold::Int = DEFAULT_SIZE_THRESHOLD` - Threshold in bytes separating direct vs link transport
- `correlation_id::String = string(uuid4())` - Correlation ID for tracing (auto-generated UUID) - `correlation_id::String = string(uuid4())` - Correlation ID for tracing (auto-generated UUID)
- `msg_purpose::String = "chat"` - Purpose of the message: "ACK", "NACK", "updateStatus", "shutdown", "chat", etc. - `msg_purpose::String = "chat"` - Purpose of the message: "ACK", "NACK", "updateStatus", "shutdown", "chat", etc.
- `sender_name::String = "NATSBridge"` - Name of the sender - `sender_name::String = "NATSBridge"` - Name of the sender
- `receiver_name::String = ""` - Name of the receiver (empty string means broadcast) - `receiver_name::String = ""` - Name of the receiver (empty string means broadcast)
- `receiver_id::String = ""` - UUID of the receiver (empty string means broadcast) - `receiver_id::String = ""` - UUID of the receiver (empty string means broadcast)
- `reply_to::String = ""` - Topic to reply to (empty string if no reply expected) - `reply_to::String = ""` - Topic to reply to (empty string if no reply expected)
- `reply_to_msg_id::String = ""` - Message ID this message is replying to - `reply_to_msg_id::String = ""` - Message ID this message is replying to
- `is_publish::Bool = true` - Whether to automatically publish the message to NATS - `msg_id::String = string(uuid4())` - Message ID (auto-generated UUID if not provided)
- `NATS_connection::Union{NATS.Connection, Nothing} = nothing` - Pre-existing NATS connection (if provided, uses this connection instead of creating a new one; saves connection establishment overhead) - `sender_id::String = string(uuid4())` - Sender ID (auto-generated UUID if not provided)
- `msg_id::String = string(uuid4())` - Message ID (auto-generated UUID if not provided)
- `sender_id::String = string(uuid4())` - Sender ID (auto-generated UUID if not provided)
# Return: # Return:
- `::Tuple{msg_envelope_v1, String}` - A tuple containing: - `::Tuple{msg_envelope_v1, String}` - A tuple containing:
@@ -412,8 +414,9 @@ env, msg_json = smartsend("chat.subject", [
("audio_clip", audio_data, "audio") ("audio_clip", audio_data, "audio")
]) ])
# Publish the JSON string directly using NATS request-reply pattern # Publish the JSON string directly using NATS (manual publish)
# reply = NATS.request(broker_url, subject, env_json_str; reply_to=reply_to_topic) # conn = NATS.connect(broker_url)
# NATS.publish(conn, subject, env_json_str)
``` ```
""" """
function smartsend( function smartsend(
@@ -438,8 +441,6 @@ function smartsend(
receiver_id::String = "", receiver_id::String = "",
reply_to::String = "", reply_to::String = "",
reply_to_msg_id::String = "", reply_to_msg_id::String = "",
is_publish::Bool = true, # some time the user want to get env and env_json_str from this function without publishing the msg
NATS_connection::Union{NATS.Connection, Nothing} = nothing, # a provided connection saves establishing connection overhead.
msg_id::String = string(uuid4()), # Message ID msg_id::String = string(uuid4()), # Message ID
sender_id::String = string(uuid4()) # Sender ID sender_id::String = string(uuid4()) # Sender ID
)::Tuple{msg_envelope_v1, String} where {T1<:Any} )::Tuple{msg_envelope_v1, String} where {T1<:Any}
@@ -539,13 +540,15 @@ function smartsend(
) )
env_json_str = envelope_to_json(env) # Convert envelope to JSON env_json_str = envelope_to_json(env) # Convert envelope to JSON
if is_publish == false # if is_publish == false
# skip publish a message # # skip publish a message
elseif is_publish == true && NATS_connection === nothing # elseif is_publish == true && NATS_connection === nothing
publish_message(broker_url, subject, env_json_str, correlation_id) # Publish message to NATS # # Publish message to NATS using new connection
elseif is_publish == true && NATS_connection !== nothing # publish_message(broker_url, subject, env_json_str, correlation_id)
publish_message(NATS_connection, subject, env_json_str, correlation_id) # Publish message to NATS # elseif is_publish == true && NATS_connection !== nothing
end # # Publish message to NATS using existing connection
# publish_message(NATS_connection, subject, env_json_str, correlation_id)
# end
return (env, env_json_str) return (env, env_json_str)
end end
@@ -700,73 +703,73 @@ function _serialize_data(data::Any, payload_type::String)
end end
""" publish_message - Publish message to NATS # """ publish_message - Publish message to NATS
This function publishes a message to a NATS subject with proper # This function publishes a message to a NATS subject with proper
connection management and logging. # connection management and logging.
# Arguments: # # Arguments:
- `broker_url::String` - NATS server URL (e.g., "nats://localhost:4222") # - `broker_url::String` - NATS server URL (e.g., "nats://localhost:4222")
- `subject::String` - NATS subject to publish to (e.g., "/agent/wine/api/v1/prompt") # - `subject::String` - NATS subject to publish to (e.g., "/agent/wine/api/v1/prompt")
- `message::String` - JSON message to publish # - `message::String` - JSON message to publish
- `correlation_id::String` - Correlation ID for tracing and logging # - `correlation_id::String` - Correlation ID for tracing and logging
# Return: # # Return:
- `nothing` - This function performs publishing but returns nothing # - `nothing` - This function performs publishing but returns nothing
# Example # # Example
```jldoctest # ```jldoctest
using NATS # using NATS
# Prepare JSON message # # Prepare JSON message
message = "{\"correlation_id\":\"abc123\",\"payload\":\"test\"}" # message = "{\"correlation_id\":\"abc123\",\"payload\":\"test\"}"
# Publish to NATS # # Publish to NATS
publish_message("nats://localhost:4222", "my.subject", message, "abc123") # publish_message("nats://localhost:4222", "my.subject", message, "abc123")
``` # ```
""" # """
function publish_message(broker_url::String, subject::String, message::String, correlation_id::String) # function publish_message(broker_url::String, subject::String, message::String, correlation_id::String)
conn = NATS.connect(broker_url) # Create NATS connection # conn = NATS.connect(broker_url) # Create NATS connection
publish_message(conn, subject, message, correlation_id) # publish_message(conn, subject, message, correlation_id)
end # end
""" publish_message - Publish message to NATS using pre-existing connection # """ publish_message - Publish message to NATS using pre-existing connection
This function publishes a message to a NATS subject using a pre-existing NATS connection, # This function publishes a message to a NATS subject using a pre-existing NATS connection,
avoiding the overhead of connection establishment. # avoiding the overhead of connection establishment.
# Arguments: # # Arguments:
- `conn::NATS.Connection` - Pre-existing NATS connection # - `conn::NATS.Connection` - Pre-existing NATS connection
- `subject::String` - NATS subject to publish to (e.g., "/agent/wine/api/v1/prompt") # - `subject::String` - NATS subject to publish to (e.g., "/agent/wine/api/v1/prompt")
- `message::String` - JSON message to publish # - `message::String` - JSON message to publish
- `correlation_id::String` - Correlation ID for tracing and logging # - `correlation_id::String` - Correlation ID for tracing and logging
# Return: # # Return:
- `nothing` - This function performs publishing but returns nothing # - `nothing` - This function performs publishing but returns nothing
# Example # # Example
```jldoctest # ```jldoctest
using NATS # using NATS
# Prepare JSON message # # Prepare JSON message
message = "{\"correlation_id\":\"abc123\",\"payload\":\"test\"}" # message = "{\"correlation_id\":\"abc123\",\"payload\":\"test\"}"
# Create connection once and reuse for multiple publishes # # Create connection once and reuse for multiple publishes
conn = NATS.connect("nats://localhost:4222") # conn = NATS.connect("nats://localhost:4222")
publish_message(conn, "my.subject", message, "abc123") # publish_message(conn, "my.subject", message, "abc123")
# Connection is automatically drained after publish # # Connection is automatically drained after publish
``` # ```
# Use Case: # # Use Case:
Use this version when you already have an established NATS connection and want to publish # Use this version when you already have an established NATS connection and want to publish
multiple messages without the overhead of creating a new connection for each publish. # multiple messages without the overhead of creating a new connection for each publish.
""" # """
function publish_message(conn::NATS.Connection, subject::String, message::String, correlation_id::String) # function publish_message(conn::NATS.Connection, subject::String, message::String, correlation_id::String)
try # try
NATS.publish(conn, subject, message) # Publish message to NATS # NATS.publish(conn, subject, message) # Publish message to NATS
log_trace(correlation_id, "Message published to $subject") # Log successful publish # log_trace(correlation_id, "Message published to $subject") # Log successful publish
finally # finally
NATS.drain(conn) # Ensure connection is closed properly # NATS.drain(conn) # Ensure connection is closed properly
end # end
end # end
""" smartreceive - Receive and process messages from NATS """ smartreceive - Receive and process messages from NATS
@@ -783,7 +786,7 @@ A HTTP file server is required along with its download function.
5. For link transport: fetches data from URL with exponential backoff, then deserializes 5. For link transport: fetches data from URL with exponential backoff, then deserializes
# Arguments: # Arguments:
- `msg::NATS.Msg` - NATS message to process - `msg_json_str::String` - JSON string from NATS message payload (e.g., `String(nats_msg.payload)`)
# Keyword Arguments: # Keyword Arguments:
- `fileserver_download_handler::Function = _fetch_with_backoff` - Function to handle downloading data from file server URLs - `fileserver_download_handler::Function = _fetch_with_backoff` - Function to handle downloading data from file server URLs
@@ -798,19 +801,21 @@ A HTTP file server is required along with its download function.
```jldoctest ```jldoctest
# Receive and process message # Receive and process message
msg = nats_message # NATS message msg = nats_message # NATS message
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff, max_retries=5, base_delay=100, max_delay=5000) msg_json_str = String(msg.payload)
env = smartreceive(msg_json_str; fileserver_download_handler=_fetch_with_backoff, max_retries=5, base_delay=100, max_delay=5000)
# env["payloads"] = [("dataname1", data1, "type1"), ("dataname2", data2, "type2"), ...] # env["payloads"] = [("dataname1", data1, "type1"), ("dataname2", data2, "type2"), ...]
``` ```
""" """
function smartreceive( function smartreceive(
msg::NATS.Msg; msg_json_str::String; # get it from String(nats_msg.payload)
fileserver_download_handler::Function = _fetch_with_backoff, fileserver_download_handler::Function = _fetch_with_backoff,
max_retries::Int = 5, max_retries::Int = 5,
base_delay::Int = 100, base_delay::Int = 100,
max_delay::Int = 5000 max_delay::Int = 5000
)::JSON.Object{String, Any} )::JSON.Object{String, Any}
# Parse the JSON envelope # Parse the JSON envelope
env_json_obj = JSON.parse(String(msg.payload)) env_json_obj = JSON.parse(msg_json_str)
log_trace(env_json_obj["correlation_id"], "Processing received message") # Log message processing start log_trace(env_json_obj["correlation_id"], "Processing received message") # Log message processing start
# Process all payloads in the envelope # Process all payloads in the envelope

1230
src/natsbridge.rs Normal file

File diff suppressed because it is too large Load Diff