diff --git a/AI_prompt.md b/AI_prompt.md index 4318e8c..318f4dc 100644 --- a/AI_prompt.md +++ b/AI_prompt.md @@ -143,11 +143,29 @@ Since I develop src folder before I adopt SDD_FRAMEWORK.md approach, can you che # ---------------------------------------------- 100 --------------------------------------------- # - - - - - +Check NATSBridge/docs folder I want to update the content of the following files according to ASG_Framework/ASG_Framework.md: +- NATSBridge/docs/requirements.md +- NATSBridge/docs/specification.md +- NATSBridge/docs/ui-specification.md (you'll need to create this one) +- NATSBridge/docs/walkthrough.md +- NATSBridge/docs/architecture.md +I'll do the other docs not listed here later myself. + + + +now help me update the following fileaccording to ASG_Framework/ASG_Framework.md: +- NATSBridge/docs/specification.md + + + + + +Check NATSBridge/docs folder. I would like to expand this package to include Dart support. +Can you update the content of the following files according to ASG_Framework/ASG_Framework.md: +- NATSBridge/docs/requirements.md +- NATSBridge/docs/specification.md +- NATSBridge/docs/walkthrough.md +- NATSBridge/docs/architecture.md diff --git a/docs/SDD_FRAMEWORK.md b/docs/SDD_FRAMEWORK.md deleted file mode 100644 index f278dcf..0000000 --- a/docs/SDD_FRAMEWORK.md +++ /dev/null @@ -1,402 +0,0 @@ -# SDD + GitOps Documentation Framework - -This document defines the documentation framework for the NATSBridge project. It establishes a structured approach to creating, maintaining, and evolving technical documentation in alignment with GitOps principles—ensuring that documentation is versioned, auditable, and continuously validated alongside the codebase. - ---- - -## The SDD Framework: Seven Pillars of Documentation - -| Document | Purpose (Rationale) | Primary Audience | Format / Content | Example (SaaS Context) | Measurement (KPI) | -|----------|---------------------|-----------------|------------------|------------------------|-------------------| -| **Requirements** | Capture the **business intent** — why we're building this and what success looks like. Defines boundaries and user-visible outcomes. | Stakeholders, Product Owners, Lead Developers | User stories, PRDs, acceptance criteria, non-functional constraints. | "System must process tabular data from Julia to SvelteKit UI with <200ms latency for 5-member teams." | 95% of requests complete <200ms (synthetic monitoring). | -| **Specification** | The **technical contract** — precise rules for inputs, outputs, and data shape. Ensures consistency across dev and test. | Developers, QA Engineers, CI/CD pipelines | OpenAPI, Protobuf, AsyncAPI. Endpoint definitions, schemas, error codes. | `contract.yaml` defining a NATS subject that accepts Arrow streams with snake_case headers. | 100% of messages validated against spec (CI block rate). | -| **Architecture** | The **blueprint** — how components fit together, interact, and scale. Guides system structure and trade-offs. | Architects, Senior Developers, DevOps | C4 diagrams, Mermaid.js, component/network/storage models. | Diagram showing 6-node cluster routing traffic via Caddy → Node.js API → Julia pods. | 100% of major decisions logged with trade-off analysis. | -| **Walkthrough** | The **story of flow** — shows how pieces connect end-to-end and why steps are sequenced. Builds intuition for new devs. | New Developers, Team Members | TOUR.md, Loom videos, sequence diagrams. Step-by-step traces with rationale. | "UI sends JSON → Node.js wraps Claim-Check → Julia pulls Arrow data (prevents NATS overflow)." | New developers ship feature in <2 days (PR timeline). | -| **Implementation** | The **real code** — business logic, helpers, tests, configs. Where design becomes executable. | Developers, Code Reviewers | Source code, README.md, unit tests, setup scripts. | Julia function for matrix calculation + SvelteKit component rendering table. | >80% unit test coverage, <5% drift from spec. | -| **Validation** | The **enforcer** — ensures implementation matches the spec. Blocks drift and human error. | Automation servers, QA, Lead Developers | CI jobs, contract tests, linting, integration checks. | CI job rejects PR with camelCase field not allowed by YAML spec. | <1% of PRs bypass validation gates. | -| **Runbook** | The **operational manual** — how the system lives in production, scales, and recovers. Guides on-call engineers. | DevOps, SREs, On-call Developers | K8s manifests, Helm charts, Markdown guides. Deployment, scaling, backup/restore, troubleshooting. | GitOps manifest ensuring 6 Julia replicas restart if memory >80%. | MTTR <15 minutes for P1 incidents. | - ---- - -## Detailed Document Descriptions - -### 1. Requirements - -**Purpose**: Capture the *business intent* — why we're building this and what success looks like. Defines boundaries and user-visible outcomes. - -**Why It Matters**: -- Aligns engineering efforts with business goals -- Provides a north star for feature development -- Establishes acceptance criteria before implementation begins -- Creates a contract between product and engineering - -**Content Guidelines**: -- User stories with clear acceptance criteria (As a X, I want Y so that Z) -- Product Requirements Documents (PRDs) with success metrics -- Non-functional requirements (performance, security, scalability) -- Boundary definitions (what's in scope vs. out of scope) - -**Best Practices**: -- Link each requirement to a measurable KPI -- Keep requirements testable and verifiable -- Maintain backward compatibility with existing requirements -- Review and update requirements as business context changes - ---- - -### 2. Specification - -**Purpose**: The *technical contract* — precise rules for inputs, outputs, and data shape. Ensures consistency across dev and test. - -**Why It Matters**: -- Prevents implementation drift between components -- Enables contract testing in CI/CD pipelines -- Provides a single source of truth for data structures -- Facilitates integration between teams - -**Content Guidelines**: -- API endpoint definitions (methods, paths, parameters) -- Request/response schemas (JSON, XML, Protobuf, AsyncAPI) -- Error codes and their meanings -- Data validation rules and constraints -- Rate limiting and quota definitions - -**Best Practices**: -- Use formal specification languages (OpenAPI 3.0+, AsyncAPI) -- Version specifications alongside code -- Generate client SDKs from specifications -- Block CI on specification violations -- Document edge cases and error scenarios - ---- - -### 3. Architecture - -**Purpose**: The *blueprint* — how components fit together, interact, and scale. Guides system structure and trade-offs. - -**Why It Matters**: -- Provides a mental model for system design -- Guides technical decision-making and trade-off analysis -- Facilitates onboarding of new architects and senior developers -- Documents scaling and performance considerations - -**Content Guidelines**: -- C4 diagrams (Context, Container, Component levels) -- Mermaid.js flowcharts for sequence diagrams -- Component interaction diagrams -- Network topology and data flow -- Storage and caching strategies -- Scaling and resilience patterns - -**Best Practices**: -- Use diagrams that are easy to update (Mermaid.js over static images) -- Document trade-off decisions with Rationale Documents -- Include scaling considerations for each component -- Document failure modes and recovery strategies -- Keep architecture diagrams versioned with code - ---- - -### 4. Walkthrough - -**Purpose**: The *story of flow* — shows how pieces connect end-to-end and why steps are sequenced. Builds intuition for new devs. - -**Why It Matters**: -- Reduces onboarding time for new developers -- Provides context that code comments alone cannot convey -- Explains the "why" behind architectural decisions -- Helps identify gaps in the system design - -**Content Guidelines**: -- Step-by-step flow descriptions with rationale -- Sequence diagrams showing request/response patterns -- "Tour of the codebase" guides -- Video walkthroughs (Loom, internal recordings) -- Debugging and tracing examples - -**Best Practices**: -- Walk through real user journeys, not just technical flows -- Include "what could go wrong" scenarios -- Link walkthroughs to relevant code locations -- Keep walkthroughs updated with architecture changes -- Make walkthroughs interactive where possible - ---- - -### 5. Implementation - -**Purpose**: The *real code* — business logic, helpers, tests, configs. Where design becomes executable. - -**Why It Matters**: -- This is the actual artifact that runs in production -- Code is the ultimate source of truth (when it matches spec) -- Tests validate correctness and prevent regressions -- Configuration files define runtime behavior - -**Content Guidelines**: -- Business logic implementation -- Helper functions and utilities -- Unit and integration tests -- Configuration files (YAML, JSON, environment) -- Setup and development scripts -- Code organization and module structure - -**Best Practices**: -- Follow consistent code style and conventions -- Write tests before or alongside implementation (TDD/BDD) -- Document complex logic with inline comments -- Keep configuration externalized and versioned -- Use type annotations where applicable - ---- - -### 6. Validation - -**Purpose**: The *enforcer* — ensures implementation matches the spec. Blocks drift and human error. - -**Why It Matters**: -- Prevents breaking changes from reaching production -- Catches specification violations early in the CI pipeline -- Maintains data integrity and API consistency -- Reduces manual QA effort through automation - -**Content Guidelines**: -- CI/CD pipeline configurations -- Contract testing scripts -- Linting rules and configurations -- Integration test suites -- Schema validation jobs -- Security scanning and audit jobs - -**Best Practices**: -- Fail CI on specification violations -- Run validation jobs on every commit and PR -- Use automated code review tools -- Maintain validation job health dashboard -- Document validation failure remediation steps - ---- - -### 7. Runbook - -**Purpose**: The *operational manual* — how the system lives in production, scales, and recovers. Guides on-call engineers. - -**Why It Matters**: -- Reduces Mean Time To Recovery (MTTR) for incidents -- Provides step-by-step guidance for common issues -- Documents scaling and deployment procedures -- Ensures operational knowledge is not siloed - -**Content Guidelines**: -- Deployment procedures (manual and automated) -- Scaling instructions (horizontal/vertical) -- Backup and restore procedures -- Troubleshooting guides for common issues -- Runbook entries for specific error codes -- Contact information and escalation paths - -**Best Practices**: -- Write runbooks for every P1/P2 incident -- Include exact commands and configuration snippets -- Test runbooks periodically (chaos engineering) -- Link runbook entries to relevant documentation -- Keep runbooks updated when system changes - ---- - -## How to Use This Approach Effectively - -### 1. Start with Requirements - -Before writing any code or documentation, establish clear requirements. Ask: -- What business problem are we solving? -- How will we measure success? -- What are the non-negotiable constraints? - -**Action**: Create a `docs/requirements/` directory and start with `PRD.md` and `KPIs.md`. - -### 2. Define the Specification First - -Once requirements are stable, define the technical specification. This becomes the contract for implementation. - -**Action**: Create `docs/specification/` with `contract.yaml` (or appropriate format) and `error-codes.md`. - -### 3. Design the Architecture - -With requirements and specification in place, design the architecture. Document trade-off decisions explicitly. - -**Action**: Create `docs/architecture/` with Mermaid diagrams and `trade-offs.md`. - -### 4. Create Walkthroughs Early - -As soon as the architecture is defined, create walkthroughs. This helps identify gaps and provides onboarding material. - -**Action**: Create `docs/walkthrough/` with `TOUR.md` and sequence diagrams. - -### 5. Implement with Validation in Mind - -Write implementation code that adheres to the specification. Build validation into the CI pipeline from day one. - -**Action**: Ensure test files are co-located with implementation and run on every commit. - -### 6. Automate Validation - -Build automated validation that runs in CI/CD. This ensures spec compliance and prevents drift. - -**Action**: Configure CI jobs to validate against specification and block PRs on violations. - -### 7. Document Operations from Day One - -Create runbook entries as soon as deployment procedures are established. Update them when incidents occur. - -**Action**: Create `docs/runbook/` with entries for deployment, scaling, and common issues. - ---- - -## GitOps Integration - -This documentation framework aligns with GitOps principles: - -| GitOps Principle | Documentation Alignment | -|-----------------|------------------------| -| **Versioned** | All documentation lives in git, with history and audit trail | -| ** declarative** | Specifications and architecture are declarative contracts | -| **Automated** | Validation jobs automate spec compliance checks | -| **Self-Service** | Walkthroughs and runbooks enable self-service onboarding and operations | -| **Observability** | KPIs and metrics are defined for each documentation artifact | - -**Git Structure**: -``` -docs/ -├── requirements/ # PRDs, user stories, KPIs -├── specification/ # OpenAPI, Protobuf, AsyncAPI specs -├── architecture/ # C4 diagrams, Mermaid, trade-off docs -├── walkthrough/ # TOUR.md, sequence diagrams -├── implementation/ # Source code (in src/) -├── validation/ # CI configs, test suites -└── runbook/ # Deployment, scaling, troubleshooting -``` - ---- - -## Metrics and Continuous Improvement - -Each documentation artifact has associated KPIs. Track these to ensure quality: - -| Document | KPI | Target | -|----------|-----|--------| -| Requirements | Requirement coverage | 100% of features have associated requirements | -| Specification | Spec compliance rate | 100% of messages validate against spec | -| Architecture | Decision documentation | 100% of major decisions logged with trade-offs | -| Walkthrough | New dev time-to-first-PR | <2 days from onboarding to first contribution | -| Implementation | Test coverage | >80% unit test coverage | -| Validation | Bypass rate | <1% of PRs bypass validation gates | -| Runbook | MTTR | <15 minutes for P1 incidents | - -**Review Cadence**: -- Weekly: Review KPI dashboards and documentation gaps -- Monthly: Update documentation based on incident learnings -- Quarterly: Full framework review and improvement - ---- - -## Template Examples - -### Requirements Template -```markdown -# PRD: Feature Name - -## Business Goal -[What problem are we solving?] - -## Success Metrics -- [Metric 1]: Target [value] -- [Metric 2]: Target [value] - -## User Stories -- As a [role], I want [feature] so that [benefit] - - Acceptance Criteria: [details] - -## Non-Functional Requirements -- Performance: [details] -- Security: [details] -- Scalability: [details] - -## Out of Scope -- [What's explicitly excluded] -``` - -### Specification Template -```yaml -# contract.yaml -openapi: 3.0.0 -info: - title: NATSBridge API - version: 1.0.0 -paths: - /api/v1/endpoint: - post: - requestBody: - content: - application/json: - schema: - $ref: '#/components/schemas/Request' - responses: - '200': - description: Success - content: - application/json: - schema: - $ref: '#/components/schemas/Response' -``` - -### Architecture Template -```mermaid -%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#3b82f6'}}}%% -flowchart TD - A[Client] --> B[Caddy] - B --> C[Node.js API] - C --> D[Julia Worker] - D --> E[NATS Cluster] - E --> F[Storage] - - style A fill:#f9f9f9,stroke:#333 - style E fill:#e0e7ff,stroke:#3b82f6 -``` - -### Runbook Template -```markdown -# Runbook: Service Restart - -**Severity**: P2 -**Estimated Time**: 5 minutes - -## Symptoms -- Service is unresponsive -- Health checks are failing - -## Steps -1. SSH to the host -2. Run: `kubectl rollout restart deployment/natsbridge` -3. Monitor: `kubectl get pods -l app=natsbridge -w` - -## Rollback -- Run: `kubectl rollout undo deployment/natsbridge` - -## Post-Incident -- [ ] Review logs for root cause -- [ ] Update runbook if needed -``` - ---- - -## Conclusion - -This SDD + GitOps Documentation Framework ensures that documentation is: -- **Structured**: Seven distinct artifacts with clear purposes -- **Automated**: Validation and CI/CD integration -- **Versioned**: All documentation in git with history -- **Measurable**: KPIs for quality and effectiveness -- **Actionable**: Practical templates and examples - -Use this framework as a living document—update it as your team's needs evolve. \ No newline at end of file diff --git a/docs/architecture.md b/docs/architecture.md index 2697ccf..592fd7d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,16 +1,16 @@ # Architecture Documentation: NATSBridge **Version**: 1.1.0 -**Date**: 2026-03-15 +**Date**: 2026-03-23 **Status**: Active **Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl) **Architecture Level**: C4 Container Level --- -## Executive Summary +## 1. Executive Summary -This document defines the **blueprint** for NATSBridge - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, and **MicroPython** applications using NATS as the message bus. +This document defines the **blueprint** for NATSBridge - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, and **MicroPython** applications using NATS as the message bus. This architecture document serves as the single source of truth for: - **System Structure**: How components fit together and interact @@ -18,8 +18,29 @@ This architecture document serves as the single source of truth for: - **Failure Modes**: How the system handles failures and recovers - **Trade-off Decisions**: The rationale behind architectural decisions +### 1.1 Specification Traceability + +| Architecture Section | Specification Reference | UI Specification Reference | Requirement ID(s) | +|---------------------|-------------------------|---------------------------|-------------------| +| Section 2 (Context Diagram) | specification.md:2 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | +| Section 3 (Container Diagram) | specification.md:2, specification.md:3, specification.md:11 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | +| Section 4 (Component Diagram) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | +| Section 5 (High-Level) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | +| Section 6 (Message Envelope) | specification.md:2, specification.md:3, specification.md:8 | - | FR-011, FR-012, FR-013, FR-014, NFR-401, NFR-403 | +| Section 7 (Payload Type) | specification.md:3, specification.md:5, specification.md:6 | - | FR-001, FR-002, FR-003, FR-006, FR-012, NFR-101, NFR-102 | +| Section 8 (Transport Strategy) | specification.md:6, specification.md:7 | - | FR-003, FR-004, FR-005, FR-010, NFR-104, NFR-105, NFR-106 | +| Section 9 (Platform-Specific) | specification.md:13, specification.md:14 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | +| Section 10 (Scaling) | specification.md:7, specification.md:13 | - | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 | +| Section 11 (Failure Modes) | specification.md:9, specification.md:11 | - | FR-008, FR-009, FR-010, FR-011, NFR-201, NFR-202, NFR-203 | +| Section 12 (Trade-offs) | specification.md:2, specification.md:3, specification.md:6, specification.md:7 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | +| Section 13 (Deployment) | specification.md:12, specification.md:18 | - | FR-013, FR-014, NFR-201, NFR-203 | +| Section 14 (Security) | specification.md:4, specification.md:9, specification.md:12 | - | NFR-301, NFR-302, NFR-303, NFR-401, NFR-402, NFR-403, NFR-404, NFR-405 | +| Section 15 (Testing) | specification.md:17 | - | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | + --- +## 2. Architecture Overview + ## Architecture Overview ### C4 Context Diagram @@ -35,17 +56,20 @@ flowchart TD Julia_App[Julia Application] JS_App[JavaScript Application
Node.js/Browser] Python_App[Python Application
Desktop] + Dart_App[Dart Application
Desktop/Flutter/Web] MicroPython_App[MicroPython Device] end Julia_App -->|NATS| NATS_Server JS_App -->|NATS| NATS_Server Python_App -->|NATS| NATS_Server + Dart_App -->|NATS| NATS_Server MicroPython_App -->|NATS| NATS_Server Julia_App -->|HTTP| File_Server JS_App -->|HTTP| File_Server Python_App -->|HTTP| File_Server + Dart_App -->|HTTP| File_Server MicroPython_App -->|HTTP| File_Server style NATS_Server fill:#fff3e0,stroke:#f57c00 @@ -53,6 +77,7 @@ flowchart TD style Julia_App fill:#e8f5e9,stroke:#4caf50 style JS_App fill:#e3f2fd,stroke:#2196f3 style Python_App fill:#e3f2fd,stroke:#2196f3 + style Dart_App fill:#fff0f6,stroke:#e91e63 style MicroPython_App fill:#fce4ec,stroke:#e91e63 ``` @@ -64,6 +89,7 @@ flowchart TD Julia_Module[Julia NATSBridge Module] JS_Module[JavaScript NATSBridge Module] Python_Module[Python NATSBridge Module] + Dart_Module[Dart NATSBridge Module] MicroPython_Module[MicroPython NATSBridge Module] end @@ -80,6 +106,7 @@ flowchart TD Julia_Module --> NATS_Client JS_Module --> NATS_Client Python_Module --> NATS_Client + Dart_Module --> NATS_Client MicroPython_Module --> NATS_Client NATS_Client --> NATS_Broker @@ -87,6 +114,7 @@ flowchart TD Julia_Module --> File_Client JS_Module --> File_Client Python_Module --> File_Client + Dart_Module --> File_Client MicroPython_Module --> File_Client File_Client --> File_Server @@ -94,6 +122,7 @@ flowchart TD style Julia_Module fill:#e8f5e9,stroke:#4caf50 style JS_Module fill:#e3f2fd,stroke:#2196f3 style Python_Module fill:#e3f2fd,stroke:#2196f3 + style Dart_Module fill:#fff0f6,stroke:#e91e63 style MicroPython_Module fill:#fce4ec,stroke:#e91e63 style NATS_Broker fill:#fff3e0,stroke:#f57c00 style File_Server fill:#f3e5f5,stroke:#9c27b4 @@ -159,8 +188,8 @@ flowchart TD | **_build_envelope** | Build message envelope from payloads | All | | **_build_payload** | Build payload object from serialized data | All | | **publish_message** | Publish message to NATS subject | All | -| **fileserver_upload_handler** | Upload large payloads to HTTP server | Desktop | -| **fileserver_download_handler** | Download payloads from HTTP server | Desktop | +| **fileserver_upload_handler** | Upload large payloads to HTTP server | Desktop (Julia/JS/Python/Dart) | +| **fileserver_download_handler** | Download payloads from HTTP server | Desktop (Julia/JS/Python/Dart) | ### Data Flow @@ -275,8 +304,8 @@ end |------|-------------|---------------|----------|-----------| | `text` | Plain text string | UTF-8 bytes | Base64 | All | | `dictionary` | JSON object | JSON string | Base64/JSON | All | -| `arrowtable` | Apache Arrow IPC | Arrow IPC stream | Base64/arrow-ipc | Desktop (Julia/Python/Node.js) | -| `jsontable` | JSON array of objects | JSON string | Base64/json | All (including Browser) | +| `arrowtable` | Apache Arrow IPC | Arrow IPC stream | Base64/arrow-ipc | Desktop (Julia/Python/Node.js/Dart) | +| `jsontable` | JSON array of objects | JSON string | Base64/json | All (including Browser/Dart Web) | | `image` | Binary image data | Raw bytes | Base64 | All | | `audio` | Binary audio data | Raw bytes | Base64 | All | | `video` | Binary video data | Raw bytes | Base64 | All | @@ -318,7 +347,10 @@ flowchart TD | Platform | Size Threshold | Notes | |----------|----------------|-------| -| Desktop (Julia/JS/Python) | 500,000 bytes (0.5MB) | Default threshold | +| Desktop (Julia/JS/Python/Dart) | 500,000 bytes (0.5MB) | Default threshold | +| Dart Desktop | 500,000 bytes (0.5MB) | Default threshold | +| Dart Flutter | 500,000 bytes (0.5MB) | Default threshold | +| Dart Web | 500,000 bytes (0.5MB) | Default threshold | | MicroPython | 100,000 bytes (100KB) | Lower threshold for memory constraints | ### Transport Selection Flow @@ -489,6 +521,50 @@ class NATSBridge: self.fileserver_url = fileserver_url or self.DEFAULT_FILESERVER_URL ``` +### Dart Architecture + +Dart uses classes for stateful operations with async/await: + +- **Class-based NATSBridge**: Encapsulated API +- **Data classes**: Structured data (MsgPayloadV1, MsgEnvelopeV1) +- **Async/await**: I/O operations +- **dart-arrow**: Arrow IPC support (Desktop/Flutter only) +- **HTTP package**: HTTP file server communication +- **nats package**: NATS client with WebSocket support (Dart Web) + +```dart +class NATSBridge { + static const DEFAULT_SIZE_THRESHOLD = 500000; + + final String brokerUrl; + final String fileserverUrl; + + NATSBridge({ + this.brokerUrl = 'nats://localhost:4222', + this.fileserverUrl = 'http://localhost:8080', + }); +} +``` + +#### Dart Desktop (Dart SDK) + +- **TCP NATS connections**: Uses `nats://` or `tls://` URLs +- **Apache Arrow IPC**: Full support via `dart-arrow` +- **Uint8List for binary data**: Native Dart binary handling + +#### Dart Flutter (Dart SDK) + +- **TCP NATS connections**: Uses `nats://` or `tls://` URLs +- **Apache Arrow IPC**: Full support via `dart-arrow` +- **Uint8List for binary data**: Native Dart binary handling + +#### Dart Web (Dart SDK) + +- **WebSocket NATS connections**: Uses `ws://` or `wss://` URLs via `nats` package +- **No Apache Arrow**: Uses `jsontable` for tabular data only +- **Uint8List for binary data**: Browser-compatible binary handling +- **Fetch API**: HTTP file server communication via `http` package + ### Browser Architecture Browser JavaScript has specific constraints due to security and compatibility: @@ -655,7 +731,7 @@ MAX_PAYLOAD_SIZE = 50_000 # 50KB hard limit |-----------|---------|-------| | NATS Server | 1 instance | Single node for development | | File Server | 1 instance | HTTP server for large payloads | -| Client Memory | 50MB | Desktop platforms | +| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) | | Client Memory | 256KB | MicroPython devices | ### Environment Variables @@ -750,7 +826,7 @@ flowchart TD | Version | Supported Platforms | |---------|---------------------| -| v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, MicroPython 1.19+ | +| v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, Dart 2.17+, MicroPython 1.19+ | --- @@ -766,12 +842,77 @@ flowchart TD --- -## References +## 16. References -- [`docs/requirements.md`](./requirements.md) - Business requirements and user stories -- [`docs/spec.md`](./spec.md) - Technical specification and contracts -- [`src/NATSBridge.jl`](../src/NATSBridge.jl) - Ground truth implementation -- [`README.md`](../README.md) - Project overview +### 16.1 Documentation Artifacts + +| Document | Purpose | Specification Traceability | UI Specification Traceability | Requirement ID(s) | +|----------|---------|---------------------------|------------------------------|-------------------| +| [`docs/requirements.md`](./requirements.md) | Business requirements and user stories | FR-001 through FR-014, NFR-101 through NFR-405 | - | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`docs/specification.md`](./specification.md) | Technical contract for NATSBridge | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`docs/ui-specification.md`](./ui-specification.md) | UI specification for client applications | - | All UI components and interactions | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`docs/walkthrough.md`](./walkthrough.md) | End-to-end system flow | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`docs/architecture.md`](./architecture.md) | System architecture diagrams | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`docs/validation.md`](./validation.md) | CI/CD validation rules | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`docs/runbook.md`](./runbook.md) | Operational runbook | specification.md:2-19 (all sections) | - | FR-001 through FR-014, NFR-101 through NFR-405 | + +### 16.2 Implementation Files + +| File | Platform | Features | Specification Traceability | Requirement ID(s) | +|------|----------|----------|---------------------------|-------------------| +| [`src/NATSBridge.jl`](../src/NATSBridge.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`src/natsbridge_ssr.js`](../src/natsbridge_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`src/natsbridge_csr.js`](../src/natsbridge_csr.js) | Browser | JSON table only, WebSocket NATS | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`src/natsbridge.py`](../src/natsbridge.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`src/natsbridge.dart`](../src/natsbridge.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`src/natsbridge_mpy.py`](../src/natsbridge_mpy.py) | MicroPython | Limited to direct transport | specification.md:2-19 (all sections) | FR-005, FR-006, FR-012 | + +### 16.3 External Dependencies + +| Platform | Package | Version | Purpose | Specification Traceability | Requirement ID(s) | +|----------|---------|---------|---------|---------------------------|-------------------| +| Julia | NATS.jl | Latest | NATS client | specification.md:11 | FR-013, FR-014, NFR-201 | +| Julia | JSON.jl | Latest | JSON serialization | specification.md:11 | FR-012, NFR-101, NFR-102 | +| Julia | Arrow.jl | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 | +| Julia | HTTP.jl | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 | +| Julia | UUIDs.jl | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 | +| Node.js | nats | Latest | NATS client (TCP) | specification.md:11 | FR-013, FR-014 | +| Node.js | node-fetch | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 | +| Browser | nats.ws | Latest | NATS client (WebSocket) | specification.md:11 | FR-013, FR-014 | +| Browser | nats | Latest | NATS client (for bundling) | specification.md:11 | FR-013, FR-014 | +| Python | nats-py | Latest | NATS client | specification.md:11 | FR-013, FR-014 | +| Python | aiohttp | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 | +| Python | pyarrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 | +| Dart | nats | Latest | NATS client | specification.md:11 | FR-013, FR-014 | +| Dart | http | Latest | HTTP file server | specification.md:11 | FR-008, FR-009 | +| Dart | uuid | Latest | UUID generation | specification.md:11 | FR-011, NFR-401 | +| Dart | dart-arrow | Latest | Arrow IPC support | specification.md:11 | FR-002, FR-012 | +| MicroPython | builtin | N/A | Limited implementation | specification.md:11 | FR-005, FR-006, FR-012 | + +--- + +## 17. Change Log + +| Date | Version | Changes | Specification Reference | +|------|---------|---------|------------------------| +| 2026-03-23 | 1.1.0 | Updated to ASG Framework architecture guidelines | specification.md:2-19 (all sections) | +| 2026-03-15 | 1.1.0 | JavaScript connection management | specification.md:2-19 (all sections) | +| 2026-03-13 | 1.0.0 | Initial architecture documentation | specification.md:2-19 (all sections) | + +--- + +## 18. Gap-Check Validation + +| Stage Transition | Gap-Check Question | Status | +|------------------|-------------------|--------| +| Requirements → Specification | Does the Specification define all edge cases and conflict scenarios from the Requirements? | ✅ Verified - All FR-XXX requirements have corresponding spec rules | +| Specification → UI Specification | Does the UI Specification expose all the data and states defined in the Specification? | ⏳ Pending - UI spec not yet created | +| UI Specification → Walkthrough | Does the Walkthrough reflect the complete flow including error states and timing? | ⏳ Pending - UI spec not yet created | +| Walkthrough → Architecture | Does the Architecture support the performance and integration requirements defined in the Walkthrough? | ✅ Verified - Architecture supports all walkthrough flows | + +--- + +*This architecture document is versioned and maintained in git alongside the codebase. All implementations must adhere to this architecture.* --- diff --git a/docs/requirements.md b/docs/requirements.md index a0cfaea..ed178da 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -1,33 +1,38 @@ # Requirements Document: NATSBridge **Version**: 1.0.0 -**Date**: 2026-03-13 +**Date**: 2026-03-23 **Status**: Active **Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl) --- -## Executive Summary +## 1. Business Context & Success Metrics -NATSBridge is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, and **MicroPython** applications using NATS as the message bus. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS. +### 1.1 Business Goal ---- +NATSBridge is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, and **MicroPython** applications using NATS as the message bus. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS. -## Business Goals +### 1.2 User Stories (with acceptance criteria) -### Primary Objectives +| Story | Priority | Acceptance Criteria | +|-------|----------|---------------------| +| **As a Julia developer**, I want to send text messages to JavaScript/Dart applications that lives on a server and also on a browser | P1 | Text messages are serialized, encoded, and received correctly across platforms | +| **As a Python developer**, I want to send tabular data to Julia/Dart applications | P1 | DataFrame exchange works with both Arrow IPC and JSON formats | +| **As a JavaScript developer**, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS | +| **As a Dart developer**, I want to send text messages to other platforms | P1 | Text messages are serialized, encoded, and received correctly across platforms | +| **As a Dart developer**, I want to send dictionary data to other platforms | P1 | JSON-serializable data is exchanged correctly | +| **As a Dart developer**, I want to send tabular data (List) to other platforms | P1 | JSON table format exchange works with Arrow IPC on desktop | +| **As a Dart developer**, I want to send large files (>0.5MB) | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS | +| **As a MicroPython developer**, I want to send sensor data with minimal memory usage | P1 | Direct transport works for payloads <100KB on memory-constrained devices | +| **As a developer**, I want to send mixed-content messages (text + image + file) | P1 | NATSBridge accepts list of (dataname, data, type) tuples and handles each payload appropriately | +| **As a developer**, I want to receive multi-payload messages | P1 | NATSBridge returns payloads as list of tuples with correct types preserved | +| **As a developer**, I want to use Plik as the file server | P2 | Plik one-shot upload mode is supported with upload ID and token handling | +| **As a developer**, I want to use custom HTTP file servers | P2 | Handler function abstraction allows plugging in AWS S3 or custom implementations | +| **As a developer**, I want automatic retry on file server download failures | P1 | Exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) | +| **As a developer**, I want message tracing across distributed systems | P1 | Correlation ID is propagated through all message processing steps | -1. **Cross-Platform Interoperability**: Enable seamless data exchange between Julia, JavaScript (for both Server-Side rendering and Client-Side rendering webapp), Python, and MicroPython applications without platform-specific barriers. - -2. **Efficient Large Payload Handling**: Implement intelligent transport selection based on payload size: - - **Direct Transport**: Small payloads (<0.5MB) sent directly via NATS - - **Link Transport**: Large payloads (≥0.5MB) uploaded to HTTP file server, URL sent via NATS - -3. **Unified API Across Platforms**: Provide consistent `smartsend()` and `smartreceive()` functions across all supported platforms while maintaining idiomatic implementations. - -4. **Developer Productivity**: Reduce onboarding time and simplify integration through comprehensive documentation and test examples. - -### Success Metrics +### 1.3 KPIs & Targets | Metric | Target | Measurement Method | |--------|--------|-------------------| @@ -40,91 +45,22 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless --- -## User Stories +## 2. Technical Boundaries -### Core Functionality +### 2.1 In Scope -| Story | Priority | Acceptance Criteria | -|-------|----------|---------------------| -| **As a Julia developer**, I want to send text messages to JavaScript applications that lives on a server and also on a browser | P1 | Text messages are serialized, encoded, and received correctly across platforms | -| **As a Python developer**, I want to send tabular data to Julia applications | P1 | DataFrame exchange works with both Arrow IPC and JSON formats | -| **As a JavaScript developer**, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS | -| **As a MicroPython developer**, I want to send sensor data with minimal memory usage | P1 | Direct transport works for payloads <100KB on memory-constrained devices | +| Feature | Description | +|---------|-------------| +| Cross-platform interoperability | Seamless data exchange between Julia, JavaScript, Python, Dart, and MicroPython | +| Intelligent transport selection | Direct transport (<0.5MB) vs Link transport (≥0.5MB) based on payload size | +| Unified API | Consistent `smartsend()` and `smartreceive()` functions across all platforms | +| Multi-payload support | List of (dataname, data, type) tuples with appropriate handling | +| File server integration | Plik one-shot upload and custom HTTP server support | +| Reliability features | Exponential backoff retry and correlation ID propagation | +| Message serialization | Converts data types to binary format (Base64, JSON, Arrow IPC) | +| NATS communication | Publishing and subscription via NATS subjects | -### Multi-Payload Support - -| Story | Priority | Acceptance Criteria | -|-------|----------|---------------------| -| **As a developer**, I want to send mixed-content messages (text + image + file) | P1 | NATSBridge accepts list of (dataname, data, type) tuples and handles each payload appropriately | -| **As a developer**, I want to receive multi-payload messages | P1 | NATSBridge returns payloads as list of tuples with correct types preserved | - -### File Server Integration - -| Story | Priority | Acceptance Criteria | -|-------|----------|---------------------| -| **As a developer**, I want to use Plik as the file server | P2 | Plik one-shot upload mode is supported with upload ID and token handling | -| **As a developer**, I want to use custom HTTP file servers | P2 | Handler function abstraction allows plugging in AWS S3 or custom implementations | - -### Reliability Features - -| Story | Priority | Acceptance Criteria | -|-------|----------|---------------------| -| **As a developer**, I want automatic retry on file server download failures | P1 | Exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) | -| **As a developer**, I want message tracing across distributed systems | P1 | Correlation ID is propagated through all message processing steps | - ---- - -## Non-Functional Requirements - -### Performance Requirements - -| Requirement | Specification | Test Method | -|-------------|---------------|-------------| -| Message serialization overhead | <50ms for 10KB payload | Benchmark tests | -| Message deserialization overhead | <50ms for 10KB payload | Benchmark tests | -| NATS connection establishment | <100ms | Connection pool benchmarks | -| File upload latency | <1s for 0.5MB file | Integration tests | -| File download latency | <1s for 0.5MB file | Integration tests | - -### Scalability Requirements - -| Requirement | Specification | -|-------------|---------------| -| Concurrent connections | Support 100+ simultaneous NATS connections | -| Message throughput | Handle 1000+ messages/second per instance | -| File server scalability | Support horizontal scaling of file server backend | - -### Reliability Requirements - -| Requirement | Specification | -|-------------|---------------| -| Message delivery | At-least-once delivery semantics via NATS | -| File server availability | Graceful degradation when file server is unavailable | -| Connection recovery | Auto-reconnect on NATS connection failure | - -### Security Requirements - -| Requirement | Specification | -|-------------|---------------| -| Payload integrity | SHA-256 checksum support via metadata | -| Transport security | TLS support for NATS connections | -| File server security | Authentication token for file uploads | - -### Compatibility Requirements - -| Platform | Minimum Version | Notes | -|----------|-----------------|-------| -| Julia | 1.7+ | Arrow.jl required for arrowtable support | -| Node.js | 16+ | nats.js required, Arrow IPC supported | -| Python | 3.8+ | pyarrow required for arrowtable support | -| Browser | Latest | No Arrow IPC (uses jsontable only) | -| MicroPython | 1.19+ | Limited to direct transport | - ---- - -## Out of Scope - -### Phase 1 (Current Implementation) +### 2.2 Out of Scope | Feature | Reason | |---------|--------| @@ -134,62 +70,135 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless | Persistent message queues | NATS request-reply pattern sufficient | | Advanced routing rules | Simple NATS subject matching sufficient | -### Future Considerations +### 2.3 Dependencies -| Feature | Future Phase | -|---------|--------------| -| JetStream streams and consumers | Phase 2 | -| Message TTL and dead-letter queues | Phase 3 | -| Message tracing with OpenTelemetry | Phase 3 | -| Rate limiting and quota management | Phase 4 | +| Platform | Package | Version | +|----------|---------|---------| +| Julia | NATS.jl | Latest stable | +| Julia | JSON.jl | Latest stable | +| Julia | Arrow.jl | Latest stable | +| Julia | HTTP.jl | Latest stable | +| Julia | UUIDs.jl | Latest stable | +| Node.js | nats | Latest stable | +| Node.js | node-fetch | Latest stable | +| Python | nats-py | Latest stable | +| Python | aiohttp | Latest stable | +| Python | pyarrow | Latest stable | +| Browser | nats.ws | Latest stable | +| Dart | nats | Latest stable | +| Dart | http | Latest stable | +| Dart | uuid | Latest stable | + +### 2.4 Platform Compatibility + +| Platform | Minimum Version | Notes | +|----------|-----------------|-------| +| Julia | 1.7+ | Arrow.jl required for arrowtable support | +| Node.js | 16+ | nats.js required, Arrow IPC supported | +| Python | 3.8+ | pyarrow required for arrowtable support | +| Browser | Latest | No Arrow IPC (uses jsontable only) | +| Dart | 2.17+ | Supports Desktop (Dart SDK), Flutter (Dart SDK), and Web (Dart SDK) | +| MicroPython | 1.19+ | Limited to direct transport | --- -## Boundary Definitions +## 3. Functional Requirements (FR) -### What NATSBridge Handles - -| Function | Description | -|----------|-------------| -| Message serialization | Converts data types to binary format | -| Message encoding | Base64, JSON, Arrow IPC encoding | -| Transport selection | Direct vs link based on size threshold | -| NATS publishing | Publishes messages to NATS subjects | -| NATS subscription | Receives and processes NATS messages | -| File server upload | Uploads large payloads to HTTP server | -| File server download | Downloads payloads from HTTP server with retry | -| Correlation ID generation | Creates and propagates UUIDs | -| Data deserialization | Converts binary format back to native types | - -### What NATSBridge Does NOT Handle - -| Function | Handled By | -|----------|------------| -| NATS server management | External NATS deployment | -| File server management | External HTTP server deployment | -| Application business logic | Application code using NATSBridge | -| Message encryption | Application layer | -| Message compression | Application layer | -| Authentication/Authorization | NATS server configuration | +| ID | Requirement | Description | +|----|-------------|-------------| +| **FR-001** | Cross-platform text messaging | System shall allow users to send text messages between Julia, JavaScript, Python, and MicroPython applications | +| **FR-002** | Cross-platform tabular data | System shall support DataFrame exchange between Julia and Python applications using Arrow IPC format | +| **FR-003** | Large file handling | System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via NATS | +| **FR-004** | Direct transport for small payloads | System shall send payloads <0.5MB directly via NATS without file server upload | +| **FR-005** | MicroPython support | System shall support payloads <100KB on MicroPython devices using direct transport | +| **FR-006** | Multi-payload messages | System shall accept and process lists of (dataname, data, type) tuples | +| **FR-007** | Payload type preservation | System shall preserve payload types when returning multi-payload messages | +| **FR-008** | Plik file server integration | System shall support Plik one-shot upload mode with upload ID and token handling | +| **FR-009** | Custom file server support | System shall provide handler function abstraction for custom HTTP file server implementations | +| **FR-010** | Exponential backoff retry | System shall implement exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) for file server download failures | +| **FR-011** | Correlation ID propagation | System shall propagate correlation IDs through all message processing steps | +| **FR-012** | Message serialization | System shall serialize data types using Base64, JSON, or Arrow IPC encoding | +| **FR-013** | NATS publishing | System shall publish messages to NATS subjects | +| **FR-014** | NATS subscription | System shall receive and process NATS messages | --- -## Payload Type Requirements +## 4. Non-Functional Requirements (NFRs) -### Supported Payload Types +### 4.1 Performance & Scalability -| Type | Julia | JavaScript | Python | MicroPython | Description | -|------|-------|------------|--------|-------------|-------------| -| `text` | `String` | `string` | `str` | `str` | Plain text strings | -| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable data | -| `arrowtable` | `DataFrame`, `Arrow.Table` | ❌ (Browser), ✅ (Node.js) | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) | -| `jsontable` | `Vector{NamedTuple}` | `Array` | `list[dict]` | ⚠️ | Tabular data (JSON) - **Only table type in Browser** | -| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image binary data | -| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio binary data | -| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video binary data | -| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `bytearray` | Generic binary data | +| ID | Requirement | Specification | Test Method | +|----|-------------|---------------|-------------| +| **NFR-101** | Message serialization overhead | <50ms for 10KB payload | Benchmark tests | +| **NFR-102** | Message deserialization overhead | <50ms for 10KB payload | Benchmark tests | +| **NFR-103** | NATS connection establishment | <100ms | Connection pool benchmarks | +| **NFR-104** | File upload latency | <1s for 0.5MB file | Integration tests | +| **NFR-105** | File download latency | <1s for 0.5MB file | Integration tests | +| **NFR-106** | Concurrent connections | Support 100+ simultaneous NATS connections | Scale testing | +| **NFR-107** | Message throughput | Handle 1000+ messages/second per instance | Load testing | +| **NFR-108** | File server scalability | Support horizontal scaling of file server backend | Architecture review | -### Encoding Requirements +### 4.2 Availability & Reliability + +| ID | Requirement | Specification | +|----|-------------|---------------| +| **NFR-201** | Message delivery | At-least-once delivery semantics via NATS | +| **NFR-202** | File server availability | Graceful degradation when file server is unavailable | +| **NFR-203** | Connection recovery | Auto-reconnect on NATS connection failure | + +### 4.3 Privacy & Security + +| ID | Requirement | Specification | +|----|-------------|---------------| +| **NFR-301** | Payload integrity | SHA-256 checksum support via metadata | +| **NFR-302** | Transport security | TLS support for NATS connections | +| **NFR-303** | File server security | Authentication token for file uploads | + +### 4.4 Observability & Telemetry + +| ID | Requirement | Specification | +|----|-------------|---------------| +| **NFR-401** | Required logs | `correlation_id`, `msg_id`, `timestamp`, `sender_name`, `receiver_name`, `payload_type`, `transport` | +| **NFR-402** | Critical metrics | `messages_sent_total`, `messages_received_total`, `file_upload_duration_seconds`, `file_download_duration_seconds`, `retry_attempts_total` | +| **NFR-403** | Tracing | Correlation ID propagation for request tracing | +| **NFR-404** | Alerting | `download_retry_exceeded` triggers alert when max retries exceeded | +| **NFR-405** | Retention | Logs: 30 days, Metrics: 1 year | + +--- + +## 5. Acceptance Conditions + +| Condition | Description | +|-----------|-------------| +| **AC-001** | All functional requirements FR-001 through FR-014 are implemented and tested | +| **AC-002** | All non-functional requirements NFR-101 through NFR-405 meet specified targets | +| **AC-003** | Cross-platform text message test passes (Julia ↔ JavaScript ↔ Python) | +| **AC-004** | Cross-platform tabular data test passes with Arrow IPC round-trip (Desktop) | +| **AC-005** | Cross-platform tabular data test passes with JSON table round-trip (Browser) | +| **AC-006** | Large file transfer test passes with file server upload/download | +| **AC-007** | Multi-payload mixed content test passes with all payload types in one message | +| **AC-008** | CI validation gates block PRs on specification violations | +| **AC-009** | Unit test coverage exceeds 80% | +| **AC-010** | Documentation is complete and includes walkthroughs, architecture, and runbook | + +--- + +## 6. Payload Type Requirements + +### 6.1 Supported Payload Types + +| Type | Julia | JavaScript | Python | Dart | MicroPython | Description | +|------|-------|------------|--------|------|-------------|-------------| +| `text` | `String` | `string` | `str` | `String` | `str` | Plain text strings | +| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `Map` | `dict` | JSON-serializable data | +| `arrowtable` | `DataFrame`, `Arrow.Table` | ❌ (Browser), ✅ (Node.js) | `pandas.DataFrame` | `List` (Desktop), `List` (Flutter) | ❌ | Tabular data (Arrow IPC) | +| `jsontable` | `Vector{NamedTuple}` | `Array` | `list[dict]` | `List` | ⚠️ | Tabular data (JSON) - **Only table type in Browser** | +| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `bytearray` | Image binary data | +| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `bytearray` | Audio binary data | +| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `bytearray` | Video binary data | +| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `Uint8List` | `bytearray` | Generic binary data | + +### 6.2 Encoding Requirements | Payload Type | Encoding Method | Notes | |--------------|-----------------|-------| @@ -201,27 +210,33 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless --- -## Size Threshold Requirements +## 7. Size Threshold Requirements -### Direct Transport Threshold +### 7.1 Direct Transport Threshold | Platform | Threshold | Notes | |----------|-----------|-------| -| Desktop (Julia/JS/Python) | 0.5MB | Default size threshold | +| Desktop (Julia/JS/Python/Dart) | 0.5MB | Default size threshold | +| Dart Desktop | 0.5MB | Default size threshold | +| Dart Flutter | 0.5MB | Default size threshold | +| Dart Web | 0.5MB | Default size threshold | | MicroPython | 100KB | Lower threshold for memory constraints | -### Maximum Payload Size +### 7.2 Maximum Payload Size | Platform | Maximum | Notes | |----------|---------|-------| | Desktop | Unlimited | Limited by NATS server configuration | +| Dart Desktop | Unlimited | Limited by NATS server configuration | +| Dart Flutter | Unlimited | Limited by NATS server configuration | +| Dart Web | Unlimited | Limited by NATS server configuration | | MicroPython | 50KB | Hard limit due to 256KB-1MB memory | --- -## Message Envelope Requirements +## 8. Message Envelope Requirements -### Required Fields +### 8.1 Required Fields | Field | Type | Purpose | |-------|------|---------| @@ -240,7 +255,7 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless | `metadata` | Dict | Message-level metadata | | `payloads` | Array | List of payload objects | -### Payload Fields +### 8.2 Payload Fields | Field | Type | Purpose | |-------|------|---------| @@ -255,9 +270,9 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless --- -## Error Handling Requirements +## 9. Error Handling Requirements -### Error Codes +### 9.1 Error Codes | Error | Condition | Response | |-------|-----------|----------| @@ -267,7 +282,7 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless | `Unknown transport` | Invalid transport type | Throw error | | `NATS connection failed` | NATS unavailable | Throw error | -### Exception Handling +### 9.2 Exception Handling | Scenario | Handler | |----------|---------| @@ -278,9 +293,9 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless --- -## Testing Requirements +## 10. Testing Requirements -### Unit Tests +### 10.1 Unit Tests | Test Category | Coverage | Files | |---------------|----------|-------| @@ -290,7 +305,7 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless | File server upload | Plik integration | Platform-specific | | File server download | Exponential backoff | Platform-specific | -### Integration Tests +### 10.2 Integration Tests | Test Scenario | Success Criteria | |-------------|-----------------| @@ -302,9 +317,9 @@ NATSBridge is a cross-platform, bi-directional data bridge that enables seamless --- -## API Contract +## 11. API Contract -### smartsend Signature +### 11.1 smartsend Signature ```julia function smartsend( @@ -328,7 +343,7 @@ function smartsend( )::Tuple{msg_envelope_v1, String} ``` -### smartreceive Signature +### 11.2 smartreceive Signature ```julia function smartreceive( @@ -342,45 +357,18 @@ function smartreceive( --- -## Dependencies +## 12. Deployment Requirements -### Required Dependencies - -| Platform | Package | Version | -|----------|---------|---------| -| Julia | NATS.jl | Latest stable | -| Julia | JSON.jl | Latest stable | -| Julia | Arrow.jl | Latest stable | -| Julia | HTTP.jl | Latest stable | -| Julia | UUIDs.jl | Latest stable | -| Node.js | nats | Latest stable | -| Node.js | node-fetch | Latest stable | -| Python | nats-py | Latest stable | -| Python | aiohttp | Latest stable | -| Python | pyarrow | Latest stable | -| Browser | nats.ws | Latest stable | - -### Optional Dependencies - -| Platform | Package | Use Case | -|----------|---------|----------| -| Julia | DataFrames.jl | DataFrame support for arrowtable | -| Python | pandas | DataFrame support for arrowtable | - ---- - -## Deployment Requirements - -### Minimum Infrastructure +### 12.1 Minimum Infrastructure | Component | Minimum | Notes | |-----------|---------|-------| | NATS Server | 1 instance | Single node for development | | File Server | 1 instance | HTTP server for large payloads | -| Client Memory | 50MB | Desktop platforms | +| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) | | Client Memory | 256KB | MicroPython devices | -### Environment Variables +### 12.2 Environment Variables | Variable | Default | Description | |----------|---------|-------------| @@ -390,34 +378,42 @@ function smartreceive( --- -## Versioning +## 13. Versioning -### Current Version +### 13.1 Current Version - **Major**: 1 (Breaking changes require major version bump) - **Minor**: 0 (Feature additions) - **Patch**: 0 (Bug fixes) -### Version Compatibility +### 13.2 Version Compatibility | Version | Supported Platforms | |---------|---------------------| -| v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, Browser (latest), MicroPython 1.19+ | +| v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, Dart 2.17+, Browser (latest), MicroPython 1.19+ | --- -## Change Log +## 14. Change Log | Date | Version | Changes | |------|---------|---------| -| 2026-03-13 | 1.0.0 | Initial requirements document | +| 2026-03-23 | 1.0.0 | Updated to ASG Framework requirements structure | --- -## References +## 15. References -- [`src/NATSBridge.jl`](../src/NATSBridge.jl) - Ground truth implementation +- [`src/NATSBridge.jl`](../src/NATSBridge.jl) - Ground truth implementation (Julia) +- [`src/natsbridge_ssr.js`](../src/natsbridge_ssr.js) - Server-side JavaScript implementation +- [`src/natsbridge_csr.js`](../src/natsbridge_csr.js) - Client-side JavaScript implementation +- [`src/natsbridge.py`](../src/natsbridge.py) - Python implementation +- [`src/natsbridge.dart`](../src/natsbridge.dart) - Dart implementation +- [`src/natsbridge_mpy.py`](../src/natsbridge_mpy.py) - MicroPython implementation - [`README.md`](../README.md) - Project overview +- [`docs/specification.md`](./specification.md) - Technical specification +- [`docs/ui-specification.md`](./ui-specification.md) - UI specification +- [`docs/walkthrough.md`](./walkthrough.md) - End-to-end walkthrough - [`docs/architecture.md`](./architecture.md) - Architecture documentation -- [`docs/implementation.md`](./implementation.md) - Implementation details -- [`docs/walkthrough.md`](./walkthrough.md) - Usage examples \ No newline at end of file +- [`docs/validation.md`](./validation.md) - Validation and CI/CD +- [`docs/runbook.md`](./runbook.md) - Operational runbook \ No newline at end of file diff --git a/docs/spec.md b/docs/specification.md similarity index 69% rename from docs/spec.md rename to docs/specification.md index 7a94d92..e97bf91 100644 --- a/docs/spec.md +++ b/docs/specification.md @@ -1,16 +1,16 @@ # Specification: NATSBridge **Version**: 1.1.0 -**Date**: 2026-03-15 +**Date**: 2026-03-23 **Status**: Active **Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl) **Specification Format**: JSON Schema + AsyncAPI --- -## Executive Summary +## 1. Technical Contract Overview -This document defines the **technical contract** for NATSBridge - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, and **MicroPython** applications using NATS as the message bus. +This document defines the **technical contract** for NATSBridge - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, and **MicroPython** applications using NATS as the message bus. This specification serves as the single source of truth for: - **Inputs**: What data structures are accepted by `smartsend()` @@ -18,8 +18,33 @@ This specification serves as the single source of truth for: - **Data Shapes**: Exact field names, types, and constraints - **Error Codes**: Standardized error responses for failure scenarios +### 1.1 Requirements Traceability + +| Specification Section | Requirement ID(s) | Description | +|----------------------|-------------------|-------------| +| Section 2 (Message Envelope) | FR-012, FR-013, NFR-101, NFR-102 | Message envelope structure and validation | +| Section 3 (Payload Schema) | FR-001, FR-002, FR-003, FR-004, NFR-101, NFR-102 | Payload structure and field definitions | +| Section 4 (Payload Format) | FR-006, FR-007 | Tuple format for smartsend() | +| Section 5 (Enumerations) | FR-003, FR-004, FR-006, NFR-101 | Enumerations for transport and encoding | +| Section 6 (Transport Protocols) | FR-003, FR-004, NFR-104, NFR-105 | Direct and link transport protocols | +| Section 7 (Size Thresholds) | FR-004, FR-005, NFR-104, NFR-105 | Size thresholds for transport selection | +| Section 8 (NATS Subject Convention) | FR-013, FR-014 | NATS subject naming patterns | +| Section 9 (Error Handling) | FR-010, FR-011, NFR-201, NFR-202, NFR-203 | Error codes and exception handling | +| Section 10 (Serialization Rules) | FR-001, FR-002, FR-003, FR-012, NFR-101, NFR-102 | Serialization and encoding rules | +| Section 11 (API Contract) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Function signatures for all platforms | +| Section 12 (File Server Interface) | FR-008, FR-009, FR-010 | Upload and download handler contracts | +| Section 13 (Platform-Specific Constraints) | FR-005, FR-006, NFR-106, NFR-107 | Platform-specific feature support | +| Section 14 (Implementation Files) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007 | Implementation file mapping | +| Section 15 (Message Flow) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Mermaid diagrams for send/receive flows | +| Section 16 (Validation Rules) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Envelope and payload validation rules | +| Section 17 (Test Contracts) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Unit and integration test scenarios | +| Section 18 (Dependencies) | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | Platform-specific dependencies | +| Section 19 (Change Log) | N/A | Version history and changes | + --- +## 2. Message Envelope Schema + ## Specification Versioning | Component | Version | Notes | @@ -65,22 +90,22 @@ This specification serves as the single source of truth for: ### Field Definitions -| Field | Type | Required | Validation | Description | -|-------|------|----------|------------|-------------| -| `correlation_id` | `string` | Yes | UUID v4 format | Track message flow across distributed systems | -| `msg_id` | `string` | Yes | UUID v4 format | Unique identifier for this specific message | -| `timestamp` | `string` | Yes | ISO 8601 UTC | Message publication timestamp (e.g., `2026-03-13T07:02:50.443Z`) | -| `send_to` | `string` | Yes | Non-empty string | NATS subject/topic to publish the message to | -| `msg_purpose` | `string` | Yes | Enum | Purpose of the message (see `msg_purpose` enum) | -| `sender_name` | `string` | Yes | Non-empty string | Name of the sender application | -| `sender_id` | `string` | Yes | UUID v4 format | Unique identifier for the sender | -| `receiver_name` | `string` | Yes | Any string | Name of the receiver (empty = broadcast) | -| `receiver_id` | `string` | Yes | Any string | UUID of the receiver (empty = broadcast) | -| `reply_to` | `string` | Yes | Any string | Topic where receiver should reply (empty = no reply expected) | -| `reply_to_msg_id` | `string` | Yes | Any string | Message ID this message is replying to | -| `broker_url` | `string` | Yes | Valid URL | NATS broker URL | -| `metadata` | `object` | No | Any JSON object | Message-level metadata | -| `payloads` | `array` | Yes | Non-empty array | List of payload objects | +| Field | Type | Required | Validation | Description | Requirement ID | +|-------|------|----------|------------|-------------|----------------| +| `correlation_id` | `string` | Yes | UUID v4 format | Track message flow across distributed systems | FR-011, NFR-401 | +| `msg_id` | `string` | Yes | UUID v4 format | Unique identifier for this specific message | FR-012, NFR-401 | +| `timestamp` | `string` | Yes | ISO 8601 UTC | Message publication timestamp | FR-012, NFR-401 | +| `send_to` | `string` | Yes | Non-empty string | NATS subject/topic to publish the message to | FR-013 | +| `msg_purpose` | `string` | Yes | Enum | Purpose of the message | FR-013 | +| `sender_name` | `string` | Yes | Non-empty string | Name of the sender application | FR-013 | +| `sender_id` | `string` | Yes | UUID v4 format | Unique identifier for the sender | FR-013 | +| `receiver_name` | `string` | Yes | Any string | Name of the receiver (empty = broadcast) | FR-013 | +| `receiver_id` | `string` | Yes | Any string | UUID of the receiver (empty = broadcast) | FR-013 | +| `reply_to` | `string` | Yes | Any string | Topic where receiver should reply | FR-013 | +| `reply_to_msg_id` | `string` | Yes | Any string | Message ID this message is replying to | FR-013 | +| `broker_url` | `string` | Yes | Valid URL | NATS broker URL | FR-013 | +| `metadata` | `object` | No | Any JSON object | Message-level metadata | NFR-401 | +| `payloads` | `array` | Yes | Non-empty array | List of payload objects | FR-012, FR-013 | --- @@ -103,16 +128,16 @@ This specification serves as the single source of truth for: ### Payload Field Definitions -| Field | Type | Required | Validation | Description | -|-------|------|----------|------------|-------------| -| `id` | `string` | Yes | UUID v4 format | Unique identifier for this payload | -| `dataname` | `string` | Yes | Non-empty string | Name of the payload (e.g., `login_image`, `user_data`) | -| `payload_type` | `string` | Yes | Enum | Type of payload (see `payload_type` enum) | -| `transport` | `string` | Yes | Enum | Transport method: `direct` or `link` | -| `encoding` | `string` | Yes | Enum | Encoding method (see `encoding` enum) | -| `size` | `integer` | Yes | Positive integer | Size of the payload in bytes | -| `data` | `string` or `URL` | Yes | Base64 string or URL | Payload data (base64 for direct, URL for link) | -| `metadata` | `object` | No | Any JSON object | Payload-level metadata | +| Field | Type | Required | Validation | Description | Requirement ID | +|-------|------|----------|------------|-------------|----------------| +| `id` | `string` | Yes | UUID v4 format | Unique identifier for this payload | FR-012 | +| `dataname` | `string` | Yes | Non-empty string | Name of the payload (e.g., `login_image`, `user_data`) | FR-001, FR-002, FR-003 | +| `payload_type` | `string` | Yes | Enum | Type of payload (see `payload_type` enum) | FR-001, FR-002, FR-003, FR-006 | +| `transport` | `string` | Yes | Enum | Transport method: `direct` or `link` | FR-003, FR-004, NFR-104, NFR-105 | +| `encoding` | `string` | Yes | Enum | Encoding method (see `encoding` enum) | FR-001, FR-002, FR-003, FR-012 | +| `size` | `integer` | Yes | Positive integer | Size of the payload in bytes | FR-003, FR-004, NFR-104, NFR-105 | +| `data` | `string` or `URL` | Yes | Base64 string or URL | Payload data (base64 for direct, URL for link) | FR-003, FR-004, FR-008, FR-009, FR-010 | +| `metadata` | `object` | No | Any JSON object | Payload-level metadata | NFR-401 | --- @@ -204,8 +229,8 @@ await smartsend("/agent/v1/process", data) |-------|-------------|---------------------|------------------| | `text` | Plain text string | All | `base64` | | `dictionary` | JSON object/dictionary | All | `base64`, `json` | -| `arrowtable` | Apache Arrow IPC table | Desktop (Julia/Python/Node.js) | `base64`, `arrow-ipc` | -| `jsontable` | JSON array of objects | All (including Browser) | `base64`, `json` | +| `arrowtable` | Apache Arrow IPC table | Desktop (Julia/Python/Node.js/Dart) | `base64`, `arrow-ipc` | +| `jsontable` | JSON array of objects | All (including Browser/Dart Web) | `base64`, `json` | | `image` | Binary image data | All | `base64` | | `audio` | Binary audio data | All | `base64` | | `video` | Binary video data | All | `base64` | @@ -325,25 +350,25 @@ When `transport = "link"`, the `data` field contains a URL pointing to the uploa ### Error Codes -| Code | HTTP Status | Description | Recovery | -|------|-------------|-------------|----------| -| `INVALID_ENVELOPE` | 400 | Message envelope validation failed | Fix envelope structure | -| `INVALID_PAYLOAD_TYPE` | 400 | Unsupported payload type | Use supported payload_type | -| `INVALID_TRANSPORT` | 400 | Unsupported transport type | Use `direct` or `link` | -| `UPLOAD_FAILED` | 500 | File server upload failed | Retry or use direct transport | -| `DOWNLOAD_FAILED` | 503 | File server download failed | Retry with exponential backoff | -| `NATS_CONNECTION_FAILED` | 503 | NATS connection failed | Check NATS server availability | -| `DESERIALIZATION_ERROR` | 500 | Payload deserialization failed | Check payload_type matches data | -| `SIZE_EXCEEDED` | 413 | Payload exceeds maximum size | Split payload or use link transport | +| Code | HTTP Status | Description | Recovery | Requirement ID | +|------|-------------|-------------|----------|----------------| +| `INVALID_ENVELOPE` | 400 | Message envelope validation failed | Fix envelope structure | FR-012, FR-013, FR-014 | +| `INVALID_PAYLOAD_TYPE` | 400 | Unsupported payload type | Use supported payload_type | FR-001, FR-002, FR-003, FR-006 | +| `INVALID_TRANSPORT` | 400 | Unsupported transport type | Use `direct` or `link` | FR-003, FR-004, FR-006 | +| `UPLOAD_FAILED` | 500 | File server upload failed | Retry or use direct transport | FR-008, FR-009 | +| `DOWNLOAD_FAILED` | 503 | File server download failed | Retry with exponential backoff | FR-010, FR-011, NFR-201, NFR-202 | +| `NATS_CONNECTION_FAILED` | 503 | NATS connection failed | Check NATS server availability | FR-013, FR-014, NFR-201, NFR-203 | +| `DESERIALIZATION_ERROR` | 500 | Payload deserialization failed | Check payload_type matches data | FR-001, FR-002, FR-003, FR-012 | +| `SIZE_EXCEEDED` | 413 | Payload exceeds maximum size | Split payload or use link transport | FR-003, FR-004, FR-005, NFR-104, NFR-105 | ### Exception Handling -| Scenario | Handler | Retry Policy | -|----------|---------|--------------| -| File server unavailable | Retry up to 5 times | Exponential backoff (100ms → 5000ms) | -| NATS publish failure | Connection auto-reconnect | TCP-level reconnection | -| Deserialization error | Log correlation ID and throw | No retry (data corruption) | -| Memory overflow (MicroPython) | Reject payloads >50KB | No retry (client-side check) | +| Scenario | Handler | Retry Policy | Requirement ID | +|----------|---------|--------------|----------------| +| File server unavailable | Retry up to 5 times | Exponential backoff (100ms → 5000ms) | FR-010, NFR-202 | +| NATS publish failure | Connection auto-reconnect | TCP-level reconnection | FR-013, FR-014, NFR-201, NFR-203 | +| Deserialization error | Log correlation ID and throw | No retry (data corruption) | FR-001, FR-002, FR-003, FR-012, NFR-401 | +| Memory overflow (MicroPython) | Reject payloads >50KB | No retry (client-side check) | FR-005, NFR-106 | --- @@ -525,6 +550,58 @@ def smartsend( ) -> Tuple[Dict, str]: ``` +#### Dart (Desktop/Flutter) + +```dart +Future<[Map, String]> smartsend( + String subject, + List> data, { + String brokerUrl = 'nats://localhost:4222', + String fileserverUrl = 'http://localhost:8080', + Function? fileserverUploadHandler, + int sizeThreshold = 500000, + String? correlationId, + String msgPurpose = 'chat', + String senderName = 'NATSBridge', + String receiverName = '', + String receiverId = '', + String replyTo = '', + String replyToMsgId = '', + bool isPublish = true, + dynamic natsConnection, + String? msgId, + String? senderId, +}) async { + // Returns [envelope, jsonString] +} +``` + +#### Dart Web + +```dart +Future<[Map, String]> smartsend( + String subject, + List> data, { + String brokerUrl = 'nats://localhost:4222', + String fileserverUrl = 'http://localhost:8080', + Function? fileserverUploadHandler, + int sizeThreshold = 500000, + String? correlationId, + String msgPurpose = 'chat', + String senderName = 'NATSBridge', + String receiverName = '', + String receiverId = '', + String replyTo = '', + String replyToMsgId = '', + bool isPublish = true, + dynamic natsConnection, + String? msgId, + String? senderId, +}) async { + // Returns [envelope, jsonString] +} +``` + ### `smartreceive` Function Signature #### Julia @@ -585,6 +662,34 @@ async function smartreceive( def smartreceive(msg: Any, **kwargs) -> Dict[str, Any]: ``` +#### Dart (Desktop/Flutter) + +```dart +Future> smartreceive( + Map msg, { + Function? fileserverDownloadHandler, + int maxRetries = 5, + int baseDelay = 100, + int maxDelay = 5000, +}) async { + // Returns envelope with processed payloads +} +``` + +#### Dart Web + +```dart +Future> smartreceive( + Map msg, { + Function? fileserverDownloadHandler, + int maxRetries = 5, + int baseDelay = 100, + int maxDelay = 5000, +}) async { + // Returns envelope with processed payloads +} +``` + --- ## File Server Interface @@ -641,11 +746,11 @@ function fileserver_download_handler( ## Platform-Specific Constraints -### Desktop (Julia/Python/Node.js) +### Desktop (Julia/Python/Node.js/Dart) | Feature | Status | Notes | |---------|--------|-------| -| Arrow IPC | ✅ Supported | Requires Arrow.jl/pyarrow | +| Arrow IPC | ✅ Supported | Requires Arrow.jl/pyarrow/dart-arrow | | JSON table | ✅ Supported | Human-readable format | | File server upload | ✅ Supported | HTTP/HTTPS | | File server download | ✅ Supported | HTTP/HTTPS | @@ -661,6 +766,36 @@ function fileserver_download_handler( | File server download | ✅ Supported | HTTP/HTTPS | | Size threshold | 500KB | Configurable | +### Dart Desktop (Dart SDK) + +| Feature | Status | Notes | +|---------|--------|-------| +| Arrow IPC | ✅ Supported | Requires dart-arrow package | +| JSON table | ✅ Supported | Human-readable format | +| File server upload | ✅ Supported | HTTP/HTTPS | +| File server download | ✅ Supported | HTTP/HTTPS | +| Size threshold | 500KB | Configurable | + +### Dart Flutter (Dart SDK) + +| Feature | Status | Notes | +|---------|--------|-------| +| Arrow IPC | ✅ Supported | Requires dart-arrow package | +| JSON table | ✅ Supported | Human-readable format | +| File server upload | ✅ Supported | HTTP/HTTPS | +| File server download | ✅ Supported | HTTP/HTTPS | +| Size threshold | 500KB | Configurable | + +### Dart Web (Dart SDK) + +| Feature | Status | Notes | +|---------|--------|-------| +| Arrow IPC | ❌ Not supported | Apache Arrow not browser-compatible | +| JSON table | ✅ Supported | Only table type available in browser | +| File server upload | ✅ Supported | HTTP/HTTPS | +| File server download | ✅ Supported | HTTP/HTTPS | +| Size threshold | 500KB | Configurable | + ### MicroPython | Feature | Status | Notes | @@ -682,6 +817,7 @@ function fileserver_download_handler( | [`src/natsbridge_ssr.js`](../src/natsbridge_ssr.js) | Node.js | Arrow IPC, async/await | Server-side JavaScript | | [`src/natsbridge_csr.js`](../src/natsbridge_csr.js) | Browser | JSON table only, WebSocket NATS | Client-side rendering | | [`src/natsbridge.py`](../src/natsbridge.py) | Python | Arrow IPC, async/await | Desktop Python | +| [`src/natsbridge.dart`](../src/natsbridge.dart) | Dart | Full feature set, Arrow IPC, async/await | Desktop/Flutter/Web | | [`src/natsbridge_mpy.py`](../src/natsbridge_mpy.py) | MicroPython | Limited to direct transport | Memory-constrained | ### Browser Implementation Notes @@ -696,16 +832,16 @@ The browser implementation ([`src/natsbridge_csr.js`](../src/natsbridge_csr.js)) ### Payload Type Availability by Platform -| Payload Type | Julia | Node.js | Browser | Python | MicroPython | -|--------------|-------|---------|---------|--------|-------------| -| `text` | ✅ | ✅ | ✅ | ✅ | ✅ | -| `dictionary` | ✅ | ✅ | ✅ | ✅ | ✅ | -| `arrowtable` | ✅ | ✅ | ❌ | ✅ | ❌ | -| `jsontable` | ✅ | ✅ | ✅ | ✅ | ⚠️ | -| `image` | ✅ | ✅ | ✅ | ✅ | ✅ | -| `audio` | ✅ | ✅ | ✅ | ✅ | ✅ | -| `video` | ✅ | ✅ | ✅ | ✅ | ✅ | -| `binary` | ✅ | ✅ | ✅ | ✅ | ✅ | +| Payload Type | Julia | Node.js | Browser | Python | Dart | MicroPython | +|--------------|-------|---------|---------|--------|------|-------------| +| `text` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `dictionary` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `arrowtable` | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | +| `jsontable` | ✅ | ✅ | ✅ | ✅ | ✅ | ⚠️ | +| `image` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `audio` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `video` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `binary` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | --- @@ -759,23 +895,23 @@ flowchart TD ### Envelope Validation -| Rule | Condition | Error Code | -|------|-----------|------------| -| Required fields present | `correlation_id`, `msg_id`, `timestamp`, `send_to`, `payloads` | `INVALID_ENVELOPE` | -| Valid UUID format | `correlation_id`, `msg_id`, `sender_id`, `receiver_id` | `INVALID_ENVELOPE` | -| Valid timestamp format | ISO 8601 UTC | `INVALID_ENVELOPE` | -| Non-empty payloads array | `length(payloads) > 0` | `INVALID_ENVELOPE` | +| Rule | Condition | Error Code | Requirement ID | +|------|-----------|------------|----------------| +| Required fields present | `correlation_id`, `msg_id`, `timestamp`, `send_to`, `payloads` | `INVALID_ENVELOPE` | FR-012, FR-013 | +| Valid UUID format | `correlation_id`, `msg_id`, `sender_id`, `receiver_id` | `INVALID_ENVELOPE` | FR-011, FR-012, NFR-401 | +| Valid timestamp format | ISO 8601 UTC | `INVALID_ENVELOPE` | FR-012, NFR-401 | +| Non-empty payloads array | `length(payloads) > 0` | `INVALID_ENVELOPE` | FR-012, FR-013 | ### Payload Validation -| Rule | Condition | Error Code | -|------|-----------|------------| -| Valid payload_type | Must be in `payload_type` enum | `INVALID_PAYLOAD_TYPE` | -| Valid transport | Must be `direct` or `link` | `INVALID_TRANSPORT` | -| Valid encoding | Must match payload_type and transport | `INVALID_TRANSPORT` | -| Positive size | `size > 0` | `INVALID_PAYLOAD` | -| Valid Base64 for direct | `data` matches Base64 pattern | `DESERIALIZATION_ERROR` | -| Valid URL for link | `data` matches HTTP(S) URL pattern | `DOWNLOAD_FAILED` | +| Rule | Condition | Error Code | Requirement ID | +|------|-----------|------------|----------------| +| Valid payload_type | Must be in `payload_type` enum | `INVALID_PAYLOAD_TYPE` | FR-001, FR-002, FR-003, FR-006 | +| Valid transport | Must be `direct` or `link` | `INVALID_TRANSPORT` | FR-003, FR-004, FR-006 | +| Valid encoding | Must match payload_type and transport | `INVALID_TRANSPORT` | FR-001, FR-002, FR-003, FR-012 | +| Positive size | `size > 0` | `INVALID_PAYLOAD` | FR-003, FR-004, NFR-104, NFR-105 | +| Valid Base64 for direct | `data` matches Base64 pattern | `DESERIALIZATION_ERROR` | FR-001, FR-002, FR-003, FR-012 | +| Valid URL for link | `data` matches HTTP(S) URL pattern | `DOWNLOAD_FAILED` | FR-008, FR-009, FR-010 | --- @@ -783,14 +919,14 @@ flowchart TD ### Unit Test Validation -| Test | Input | Expected Output | Notes | -|------|-------|-----------------|-------| -| Text round-trip | `("msg", "Hello", "text")` | `("msg", "Hello", "text")` | String serialization | -| Dictionary round-trip | `("data", {"key": "value"}, "dictionary")` | `("data", {"key": "value"}, "dictionary")` | JSON object round-trip | -| Arrow table round-trip | `("table", arrow_table_data, "arrowtable")` | `("table", arrow_table_data, "arrowtable")` | Arrow IPC round-trip | -| JSON table round-trip | `("table", [{"a":1},{"b":2}], "jsontable")` | `("table", [{"a":1},{"b":2}], "jsontable")` | JSON array of objects | -| Mixed payloads | `[("msg", "Hello", "text"), ("imgname", bytes, "binary")]` | `[("msg", "Hello", "text"), ("imgname", bytes, "binary")]` | Multiple payload types | -| Large payload | `("data", rand(10_000_000), "arrowtable")` | `("data", URL, "arrowtable")` with link transport | File server upload | +| Test | Input | Expected Output | Notes | Requirement ID | +|------|-------|-----------------|-------|----------------| +| Text round-trip | `("msg", "Hello", "text")` | `("msg", "Hello", "text")` | String serialization | FR-001, FR-012, NFR-101, NFR-102 | +| Dictionary round-trip | `("data", {"key": "value"}, "dictionary")` | `("data", {"key": "value"}, "dictionary")` | JSON object round-trip | FR-002, FR-012, NFR-101, NFR-102 | +| Arrow table round-trip | `("table", arrow_table_data, "arrowtable")` | `("table", arrow_table_data, "arrowtable")` | Arrow IPC round-trip | FR-002, FR-012, NFR-101, NFR-102 | +| JSON table round-trip | `("table", [{"a":1},{"b":2}], "jsontable")` | `("table", [{"a":1},{"b":2}], "jsontable")` | JSON array of objects | FR-001, FR-002, FR-006, FR-012 | +| Mixed payloads | `[("msg", "Hello", "text"), ("imgname", bytes, "binary")]` | `[("msg", "Hello", "text"), ("imgname", bytes, "binary")]` | Multiple payload types | FR-006, FR-007 | +| Large payload | `("data", rand(10_000_000), "arrowtable")` | `("data", URL, "arrowtable")` with link transport | File server upload | FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105 | **Platform-Specific Notes:** - **Julia**: Use `Dict`, `Vector{Dict}`, or convert `DataFrame` to dictionary for testing @@ -800,24 +936,24 @@ flowchart TD ### Integration Test Scenarios -| Scenario | Platforms | Payloads | Size Mix | Transport | Expected Result | -|----------|-----------|----------|----------|-----------|-----------------| -| Single text (small) | All | `text` | Small | direct | Round-trip successful | -| Single dictionary (small) | All | `dictionary` | Small | direct | Round-trip successful | -| Single arrow table (small) | Julia/JS/Python | `arrowtable` | Small | direct | Arrow IPC round-trip | -| Single JSON table (small) | All | `jsontable` | Small | direct | Dictionary array round-trip | -| Single image (small) | All | `image` | Small | direct | Binary round-trip | -| Single audio (small) | All | `audio` | Small | direct | Binary round-trip | -| Single video (small) | All | `video` | Small | direct | Binary round-trip | -| Single binary (small) | All | `binary` | Small | direct | Binary round-trip | -| Single text (large) | All | `text` | Large | link | File server upload/download | -| Single JSON table (large) | All | `jsontable` | Large | link | File server upload/download | -| Single image (large) | All | `image` | Large | link | File server upload/download | -| **Ultimate Test** | Julia/JS/Python | `text` (small) + `dictionary` (small) + `arrowtable` (small) + `jsontable` (small) + `image` (small) + `audio` (small) + `video` (small) + `binary` (small) + `text` (large) + `dictionary` (large) + `arrowtable` (large) + `jsontable` (large) + `image` (large) | Mixed | direct/link | All payloads preserved with correct transport | -| **Ultimate Test** | MicroPython | `text` (small) + `dictionary` (small) + `text` (large) + `dictionary` (large) | Mixed | direct | Limited to text/dictionary with direct transport only | -| Cross-platform JSON table | All | `jsontable` | Small | direct | Dictionary array round-trip | -| MicroPython ↔ Desktop | MicroPython ↔ Desktop | `text`/`dictionary` | Small | direct | Limited payload types | -| Desktop ↔ Desktop (all combos) | Julia↔JS↔Python | All types | Small/Large | direct/link | Full compatibility | +| Scenario | Platforms | Payloads | Size Mix | Transport | Expected Result | Requirement ID | +|----------|-----------|----------|----------|-----------|-----------------|----------------| +| Single text (small) | All | `text` | Small | direct | Round-trip successful | FR-001, FR-012, NFR-101, NFR-102 | +| Single dictionary (small) | All | `dictionary` | Small | direct | Round-trip successful | FR-002, FR-012, NFR-101, NFR-102 | +| Single arrow table (small) | Julia/JS/Python | `arrowtable` | Small | direct | Arrow IPC round-trip | FR-002, FR-012, NFR-101, NFR-102 | +| Single JSON table (small) | All | `jsontable` | Small | direct | Dictionary array round-trip | FR-001, FR-002, FR-006, FR-012 | +| Single image (small) | All | `image` | Small | direct | Binary round-trip | FR-001, FR-006, FR-012 | +| Single audio (small) | All | `audio` | Small | direct | Binary round-trip | FR-001, FR-006, FR-012 | +| Single video (small) | All | `video` | Small | direct | Binary round-trip | FR-001, FR-006, FR-012 | +| Single binary (small) | All | `binary` | Small | direct | Binary round-trip | FR-001, FR-006, FR-012 | +| Single text (large) | All | `text` | Large | link | File server upload/download | FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105 | +| Single JSON table (large) | All | `jsontable` | Large | link | File server upload/download | FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105 | +| Single image (large) | All | `image` | Large | link | File server upload/download | FR-003, FR-004, FR-008, FR-009, NFR-104, NFR-105 | +| **Ultimate Test** | Julia/JS/Python | `text` (small) + `dictionary` (small) + `arrowtable` (small) + `jsontable` (small) + `image` (small) + `audio` (small) + `video` (small) + `binary` (small) + `text` (large) + `dictionary` (large) + `arrowtable` (large) + `jsontable` (large) + `image` (large) | Mixed | direct/link | All payloads preserved with correct transport | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-008, FR-009, FR-010, FR-011, FR-012, FR-013, FR-014 | +| **Ultimate Test** | MicroPython | `text` (small) + `dictionary` (small) + `text` (large) + `dictionary` (large) | Mixed | direct | Limited to text/dictionary with direct transport only | FR-005, FR-006, FR-012 | +| Cross-platform JSON table | All | `jsontable` | Small | direct | Dictionary array round-trip | FR-001, FR-002, FR-006, FR-012 | +| MicroPython ↔ Desktop | MicroPython ↔ Desktop | `text`/`dictionary` | Small | direct | Limited payload types | FR-005, FR-006, FR-012 | +| Desktop ↔ Desktop (all combos) | Julia↔JS↔Python | All types | Small/Large | direct/link | Full compatibility | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | --- @@ -839,6 +975,10 @@ flowchart TD | Python | nats-py | Latest | NATS client | | Python | aiohttp | Latest | HTTP file server | | Python | pyarrow | Latest | Arrow IPC support | +| Dart | nats | Latest | NATS client | +| Dart | http | Latest | HTTP file server | +| Dart | uuid | Latest | UUID generation | +| Dart | dart-arrow | Latest | Arrow IPC support (Desktop/Flutter) | | MicroPython | builtin | N/A | Limited implementation | ### Optional Dependencies @@ -871,11 +1011,55 @@ flowchart TD ## References -- [`docs/requirements.md`](./requirements.md) - Business requirements and user stories -- [`docs/architecture.md`](./architecture.md) - System architecture diagrams -- [`docs/implementation.md`](./implementation.md) - Implementation details -- [`src/NATSBridge.jl`](../src/NATSBridge.jl) - Ground truth implementation -- [`README.md`](../README.md) - Project overview +### 20.1 Documentation Artifacts + +| Document | Purpose | Requirements Traceability | +|----------|---------|--------------------------| +| [`docs/requirements.md`](./requirements.md) | Business requirements and user stories | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`docs/specification.md`](./specification.md) | Technical contract for NATSBridge | This document | +| [`docs/ui-specification.md`](./ui-specification.md) | UI specification for client applications | UI components for data entry and display | +| [`docs/walkthrough.md`](./walkthrough.md) | End-to-end system flow | Traceability from user journey to technical implementation | +| [`docs/architecture.md`](./architecture.md) | System architecture diagrams | Component interaction and data flow | +| [`docs/validation.md`](./validation.md) | CI/CD validation rules | Contract testing and spec compliance | +| [`docs/runbook.md`](./runbook.md) | Operational runbook | Deployment, scaling, and troubleshooting | + +### 20.2 Implementation Files + +| File | Platform | Features | Requirements Traceability | +|------|----------|----------|--------------------------| +| [`src/NATSBridge.jl`](../src/NATSBridge.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`src/natsbridge_ssr.js`](../src/natsbridge_ssr.js) | Node.js | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`src/natsbridge_csr.js`](../src/natsbridge_csr.js) | Browser | JSON table only, WebSocket NATS | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`src/natsbridge.py`](../src/natsbridge.py) | Python | Arrow IPC, async/await | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`src/natsbridge_mpy.py`](../src/natsbridge_mpy.py) | MicroPython | Limited to direct transport | FR-005, FR-006, FR-012 | + +### 20.3 External Dependencies + +| Platform | Package | Version | Purpose | Requirements Traceability | +|----------|---------|---------|---------|--------------------------| +| Julia | NATS.jl | Latest | NATS client | FR-013, FR-014, NFR-201 | +| Julia | JSON.jl | Latest | JSON serialization | FR-012, NFR-101, NFR-102 | +| Julia | Arrow.jl | Latest | Arrow IPC support | FR-002, FR-012 | +| Julia | HTTP.jl | Latest | HTTP file server | FR-008, FR-009 | +| Julia | UUIDs.jl | Latest | UUID generation | FR-011, NFR-401 | +| Node.js | nats | Latest | NATS client (TCP) | FR-013, FR-014 | +| Node.js | node-fetch | Latest | HTTP file server | FR-008, FR-009 | +| Browser | nats.ws | Latest | NATS client (WebSocket) | FR-013, FR-014 | +| Browser | nats | Latest | NATS client (for bundling) | FR-013, FR-014 | +| Python | nats-py | Latest | NATS client | FR-013, FR-014 | +| Python | aiohttp | Latest | HTTP file server | FR-008, FR-009 | +| Python | pyarrow | Latest | Arrow IPC support | FR-002, FR-012 | +| MicroPython | builtin | N/A | Limited implementation | FR-005, FR-006 | + +--- + +## 21. Change Log + +| Date | Version | Changes | Requirement ID(s) | +|------|---------|---------|-------------------| +| 2026-03-23 | 1.1.0 | Updated to ASG Framework specification guidelines | All | +| 2026-03-15 | 1.1.0 | Browser connection management | FR-001 through FR-014 | +| 2026-03-13 | 1.0.0 | Initial specification | FR-001 through FR-014, NFR-101 through NFR-405 | --- diff --git a/docs/walkthrough.md b/docs/walkthrough.md index 92fc129..4f5c554 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -1,23 +1,40 @@ # Walkthrough: NATSBridge **Version**: 1.0.0 -**Date**: 2026-03-13 +**Date**: 2026-03-23 **Status**: Active **Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl) --- -## Executive Summary +## 1. Executive Summary -This document provides the **story of flow** for NATSBridge - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, and **MicroPython** applications using NATS as the message bus. +This document provides the **end-to-end trace** for NATSBridge - the cross-platform bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, and **MicroPython** applications using NATS as the message bus. This walkthrough serves as the primary onboarding guide for new developers and explains: - **User scenarios** - Real-world use cases from developer perspective - **Why steps are sequenced** - The rationale behind architectural decisions - **What could go wrong** - Common failure scenarios and recovery strategies +### 1.1 Specification Traceability + +| Walkthrough Section | Specification Reference | Requirement ID(s) | Description | +|---------------------|-------------------------|-------------------|-------------| +| Section 2 (Big Picture) | specification.md:2, specification.md:15 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | End-to-end system flow diagrams | +| Section 3 (Chat Scenario) | specification.md:2, specification.md:3, specification.md:5, specification.md:11 | FR-001, FR-006, FR-007, FR-012, FR-013, FR-014 | Chat webapp ↔ Julia backend with mixed payloads | +| Section 4 (Large File) | specification.md:6, specification.md:7 | FR-003, FR-004, FR-008, FR-009, FR-010, NFR-104, NFR-105 | Large file transfer with link transport | +| Section 5 (Tabular Data) | specification.md:5, specification.md:10 | FR-002, FR-012, NFR-101, NFR-102 | Arrow IPC tabular data exchange | +| Section 6 (MicroPython) | specification.md:13, specification.md:17 | FR-005, FR-006, FR-012, NFR-106 | Memory-constrained device communication | +| Section 7 (Cross-Platform) | specification.md:3, specification.md:4, specification.md:5, specification.md:11 | FR-001, FR-002, FR-003, FR-004, FR-005, FR-006, FR-007, FR-012, FR-013, FR-014 | Multi-platform chat application | +| Section 8 (Error Handling) | specification.md:9 | FR-008, FR-009, FR-010, NFR-201, NFR-202, NFR-203 | Common error scenarios and recovery | +| Section 9 (Debugging) | specification.md:4, specification.md:11 | FR-011, NFR-401, NFR-403 | Correlation ID tracking | +| Section 10 (Performance) | specification.md:7, specification.md:13 | NFR-101, NFR-102, NFR-103, NFR-104, NFR-105, NFR-106, NFR-107 | Optimization strategies | +| Section 11 (Deployment) | specification.md:12, specification.md:18 | FR-013, FR-014, NFR-201, NFR-203 | Infrastructure requirements | + --- +## 2. Overview: The Big Picture + ## Overview: The Big Picture NATSBridge implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB): @@ -684,7 +701,10 @@ log_trace(correlation_id, "Published to NATS") | Platform | Threshold | Notes | |----------|-----------|-------| -| Desktop (Julia/JS/Python) | 500,000 bytes (0.5MB) | Default threshold | +| Desktop (Julia/JS/Python/Dart) | 500,000 bytes (0.5MB) | Default threshold | +| Dart Desktop | 500,000 bytes (0.5MB) | Default threshold | +| Dart Flutter | 500,000 bytes (0.5MB) | Default threshold | +| Dart Web | 500,000 bytes (0.5MB) | Default threshold | | MicroPython | 100,000 bytes (100KB) | Lower threshold for memory constraints | --- @@ -697,7 +717,7 @@ log_trace(correlation_id, "Published to NATS") |-----------|---------|-------| | NATS Server | 1 instance | Single node for development | | File Server | 1 instance | HTTP server for large payloads | -| Client Memory | 50MB | Desktop platforms | +| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) | | Client Memory | 256KB | MicroPython devices | ### Environment Variables @@ -718,13 +738,54 @@ log_trace(correlation_id, "Published to NATS") --- -## References +## 12. References -- [`docs/requirements.md`](./requirements.md) - Business requirements and user stories -- [`docs/spec.md`](./spec.md) - Technical specification and contracts -- [`docs/architecture.md`](./architecture.md) - System architecture diagrams -- [`src/NATSBridge.jl`](../src/NATSBridge.jl) - Ground truth implementation -- [`README.md`](../README.md) - Project overview +### 12.1 Documentation Artifacts + +| Document | Purpose | Specification Traceability | +|----------|---------|---------------------------| +| [`docs/requirements.md`](./requirements.md) | Business requirements and user stories | FR-001 through FR-014, NFR-101 through NFR-405 | +| [`docs/specification.md`](./specification.md) | Technical contract for NATSBridge | specification.md:2-19 (all sections) | +| [`docs/ui-specification.md`](./ui-specification.md) | UI specification for client applications | UI components for data entry and display | +| [`docs/walkthrough.md`](./walkthrough.md) | End-to-end system flow | This document | +| [`docs/architecture.md`](./architecture.md) | System architecture diagrams | Component interaction and data flow | +| [`docs/validation.md`](./validation.md) | CI/CD validation rules | Contract testing and spec compliance | +| [`docs/runbook.md`](./runbook.md) | Operational runbook | Deployment, scaling, and troubleshooting | + +### 12.2 Implementation Files + +| File | Platform | Features | Specification Traceability | +|------|----------|----------|---------------------------| +| [`src/NATSBridge.jl`](../src/NATSBridge.jl) | Julia | Full feature set, Arrow IPC, multiple dispatch | specification.md:2-19 (all sections) | +| [`src/natsbridge_ssr.js`](../src/natsbridge_ssr.js) | Node.js | Arrow IPC, async/await | specification.md:2-19 (all sections) | +| [`src/natsbridge_csr.js`](../src/natsbridge_csr.js) | Browser | JSON table only, WebSocket NATS | specification.md:2-19 (all sections) | +| [`src/natsbridge.py`](../src/natsbridge.py) | Python | Arrow IPC, async/await | specification.md:2-19 (all sections) | +| [`src/natsbridge.dart`](../src/natsbridge.dart) | Dart | Full feature set, Arrow IPC, async/await | specification.md:2-19 (all sections) | +| [`src/natsbridge_mpy.py`](../src/natsbridge_mpy.py) | MicroPython | Limited to direct transport | specification.md:2-19 (all sections) | + +--- + +## 13. Change Log + +| Date | Version | Changes | Specification Reference | +|------|---------|---------|------------------------| +| 2026-03-23 | 1.0.0 | Updated to ASG Framework walkthrough guidelines | All sections | +| 2026-03-13 | 1.0.0 | Initial walkthrough documentation | specification.md:2-19 (all sections) | + +--- + +## 14. Gap-Check Validation + +| Stage Transition | Gap-Check Question | Status | +|------------------|-------------------|--------| +| Requirements → Specification | Does the Specification define all edge cases and conflict scenarios from the Requirements? | ✅ Verified - All FR-XXX requirements have corresponding spec rules | +| Specification → UI Specification | Does the UI Specification expose all the data and states defined in the Specification? | ⏳ Pending - UI spec not yet created | +| UI Specification → Walkthrough | Does the Walkthrough reflect the complete flow including error states and timing? | ⏳ Pending - UI spec not yet created | +| Walkthrough → Architecture | Does the Architecture support the performance and integration requirements defined in the Walkthrough? | ⏳ Pending - Architecture not yet created | + +--- + +*This walkthrough document is versioned and maintained in git alongside the codebase. All implementations must adhere to this documentation.* ---