Compare commits
49 Commits
v0.5.0
...
dart_suppo
| Author | SHA1 | Date | |
|---|---|---|---|
| 14cd4c0076 | |||
| 5191f1aae5 | |||
| c20a266e72 | |||
| 4f141b130e | |||
| fa039f2820 | |||
| b0c5ecb942 | |||
| ba659368a5 | |||
| 1b38b3d6f1 | |||
| ad28386ff0 | |||
| 9c4c941840 | |||
| 34d8e3fad8 | |||
| 49d7898720 | |||
| fb315a0525 | |||
| 07acde45da | |||
| 3c6e139ac0 | |||
| 50211b671d | |||
| d32f64dbc0 | |||
| bc670a2af4 | |||
| a1971b737a | |||
| d888e679c5 | |||
| 46f024df4c | |||
| 824468336d | |||
| 8a5eef6b13 | |||
| 7bc3e4992a | |||
| 3e6ac1430a | |||
| 8d31c5829b | |||
| 6b9d175e82 | |||
| 24d818bfe1 | |||
| 1b41d2d3e6 | |||
| d345ddbe86 | |||
| e4d668cebb | |||
| e99fb09298 | |||
| 42fffb8a4f | |||
| f045c2faef | |||
| 5369df7148 | |||
| a8887b1fb6 | |||
| ceda1b7709 | |||
| ba567f21fc | |||
| 7c83c06d6c | |||
| e974dc5fdb | |||
| 437ca81e76 | |||
| fbd061b253 | |||
| 0fb132555b | |||
| 64796ff0a3 | |||
| f9aa6bc9f6 | |||
| a4b3695510 | |||
| 8f50039a68 | |||
| 99f1b2e720 | |||
| 54ecc811f7 |
172
AI_prompt.md
Normal file
172
AI_prompt.md
Normal file
@@ -0,0 +1,172 @@
|
|||||||
|
Consider the following scenarios:
|
||||||
|
Scenario 1: The "Command & Control" Loop (Low Latency)Focus: Small payloads, Core NATS, bi-directional JSON.The Action: A user on a JavaScript dashboard clicks a "Start Simulation" button. This sends a JSON configuration (parameters like step_size and iterations) to Julia.The Flow: * JS (Sender): Recognizes the message is small ($< 10KB$). Packages it as a direct transport JSON envelope.Julia (Receiver): Listens on the NATS subject, decodes the JSON, and immediately acknowledges receipt with a "Running" status.Project Requirement Met: Fast, low-overhead communication for control signals without involving the fileserver.
|
||||||
|
Scenario 2: The "Deep Dive" Analysis (High Bandwidth)Focus: Large Arrow tables, Claim-Check pattern, Julia to JS.The Action: Julia finishes a heavy computation and produces a 500MB DataFrame with 10 million rows. It needs to send this to the JS frontend for visualization (e.g., using Perspective.js or D3).The Flow:Julia (Sender): Converts the DataFrame to an Arrow IPC stream. It sees the size is $> 1MB$, so it uploads the bytes to the HTTP fileserver. It then publishes a NATS message with transport: "link" and the URL.JS (Receiver): Receives the URL, fetches the data via fetch(), and uses tableFromIPC() to load the data into memory with zero-copy.Project Requirement Met: Handling massive datasets that exceed NATS message limits while maintaining data integrity across languages.
|
||||||
|
Scenario 3: Live Audio/Signal Processing (Multimedia & Metadata)Focus: Raw binary, bi-directional streaming, NATS Headers.The Action: The JS client captures a 2-second "chunk" of microphone audio. It needs Julia to perform a Fast Fourier Transform (FFT) or AI transcription.The Flow:JS (Sender): Sends the raw binary WAV/PCM data. It uses NATS Headers to store the metadata ($fs = 44.1kHz$, $channels = 1$) to keep the payload purely binary.Julia (Receiver): Processes the audio and sends back a JSON result (the transcription) and an Arrow Table (the frequency spectrum data).Project Requirement Met: Bi-directional flow involving mixed media (Audio) and technical results (Arrow).
|
||||||
|
Scenario 4: The "Catch-Up" (Persistence & JetStream)Focus: NATS JetStream, late-joining consumers, state sync.The Action: Julia is constantly publishing "System Health" updates. The JS dashboard is closed for 10 minutes. When the user re-opens the dashboard, they need to see the last 10 minutes of history.The Flow:NATS (Server): Uses a JetStream with a Limits retention policy.JS (Consumer): Connects and requests a "Replay" from the last 10 minutes. It receives a mix of direct (small updates) and link (historical snapshots) messages.Project Requirement Met: Temporal decoupling—consumers can receive data that was sent while they were offline.
|
||||||
|
|
||||||
|
Role: Principal Systems Architect & Lead Software Engineer.Objective: Implement a high-performance, bi-directional data bridge between a Julia service and a JavaScript (Node.js) service using NATS (Core & JetStream).⚠️ STRICT ARCHITECTURAL CONSTRAINTS (Non-Negotiable)Transport Strategy (Claim-Check Pattern):Direct Path: If payload is < 1MB, send data directly via NATS inside the message envelope (Base64 encoded).Link Path: If payload is > 1MB, upload to a shared HTTP fileserver/store. The NATS message must only contain the metadata and the download URL.Tabular Data Format: * MUST use Apache Arrow IPC Stream for all tables/DataFrames. No CSV or standard JSON-serialization of tables allowed.System Symmetry: * Both services must function as Producers AND Consumers.Modular Elegance: * Implementation must be abstracted into a SmartSend function and a SmartReceive handler. The developer calling these functions should not need to care if the data is going via NATS direct or HTTP link.Technical Stack & Use CasesJulia: NATS.jl, Arrow.jl, JSON3.jl, HTTP.jl.Node.js: nats.js, apache-arrow.Scenarios to Support: * Large Data: Sending a 500MB Arrow table from Julia $\rightarrow$ JS.Media: Sending a 5MB WAV file from JS $\rightarrow$ Julia.Signals: Sending small JSON control commands ($< 10KB$) directly via NATS.Implementation Requirements1. Unified JSON Envelope:Define a schema containing: correlation_id (UUID), type (table/binary/json), transport (direct/link), payload (if direct), and url (if link).2. The Julia Module:Implement SmartSend(subject, data, type): Handles Arrow serialization to an IOBuffer, checks size, and manages HTTP uploads for large blobs.Implement SmartReceive(msg): Parses envelope, handles the HTTP fetch with Exponential Backoff (to avoid race conditions), and restores the DataFrame.Include a basic HTTP.listen server to serve as the temporary storage.3. The JavaScript Module:Implement a symmetric SmartSend using nats.js and apache-arrow.Implement a JetStream Pull Consumer for SmartReceive to ensure backpressure and memory safety.4. Performance & Reliability:Demonstrate "Zero-Copy" reading of the Arrow IPC stream on the JS side.Log the correlation_id at every stage for distributed tracing.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Create a walkthrough for Julia service-A service sending a mix-content chat message to Julia service-B. the chat message must includes
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
I updated the following:
|
||||||
|
- NATSBridge.jl. Essentially I add NATS_connection keyword and new publish_message function to support the keyword.
|
||||||
|
Use them and ONLY them as ground truth.
|
||||||
|
Then update the following files accordingly:
|
||||||
|
- architecture.md
|
||||||
|
- implementation.md
|
||||||
|
|
||||||
|
All API should be semantically consistent and naming should be consistent across the board.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Task: Update NATSBridge.js to reflect recent changes in NATSBridge.jl and docs
|
||||||
|
Context: NATSBridge.jl and docs has been updated.
|
||||||
|
Requirements:
|
||||||
|
Source of Truth: Treat the updated NATSBridge.jl and docs as the definitive source.
|
||||||
|
API Consistency: Ensure the Main Package API (e.g., smartsend(), publish_message()) uses consistent naming across all three supported languages.
|
||||||
|
Ecosystem Variance: Low-level native functions (e.g., NATS.connect(), JSON.read()) should follow the conventions of the specific language ecosystem and do not require cross-language consistency.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
I'm expanding this Julia package (NATSBridge) into a cross-platform project by adding a JavaScript and Python/MicroPython implementation. To ensure accuracy, the Julia src directory will serve as the ground truth, as the documentation may be outdated.
|
||||||
|
|
||||||
|
My goal is to maintain interface parity at the high-level API for a consistent user experience, while ensuring the low-level implementation adheres strictly to the idiomatic conventions of each respective language (e.g., multiple dispatch in Julia vs. asynchronous, prototype, or class-based patterns in JS and Python/MicroPython)
|
||||||
|
|
||||||
|
Now, help me do the following:
|
||||||
|
1) check architecture.md for any mistake.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Help me expands this Julia package (NATSBridge) into a cross-platform project by adding a JavaScript and Python/MicroPython implementation. To ensure accuracy, NATSBridge.jl will serve as the ground truth, as the documentation may be outdated.
|
||||||
|
|
||||||
|
My goal is to maintain interface parity at the high-level API for a consistent user experience, while ensuring the low-level implementation adheres strictly to the idiomatic conventions of each respective language (e.g., multiple dispatch in Julia vs. asynchronous, prototype, or class-based patterns in JS and Python/MicroPython)
|
||||||
|
|
||||||
|
Now do the following:
|
||||||
|
1) check docs to see if there is any mistake.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
I'm expanding this Julia package (NATSBridge) into a cross-platform project by adding
|
||||||
|
a JavaScript, Python and MicroPython implementation.
|
||||||
|
The following will serve as the ground truth:
|
||||||
|
- test_julia_mix_payloads_sender.jl
|
||||||
|
- NATSBridge.jl
|
||||||
|
- test_julia_mix_payloads_receiver.jl
|
||||||
|
- architecture.md
|
||||||
|
|
||||||
|
My goal is to maintain interface parity at the high-level API for a consistent user experience,
|
||||||
|
while ensuring the low-level implementation adheres strictly to the idiomatic conventions of each
|
||||||
|
respective language (e.g., multiple dispatch in Julia vs. asynchronous, prototype, or class-based
|
||||||
|
patterns in JS, Python and MicroPython)
|
||||||
|
|
||||||
|
Now, help me do the following:
|
||||||
|
1) Check whether natsbridge.js needs update or it already up to date.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------- 100 --------------------------------------------- #
|
||||||
|
|
||||||
|
Got it — let’s rebuild your table in my own teaching style, keeping it crisp, intuitive, and easy for students to grasp. I’ll emphasize **purpose, audience, format, example, and KPI** in a way that flows like a story of how projects move from idea → contract → design → code → review → operations.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### SDD + GitOps Documentation Framework
|
||||||
|
|
||||||
|
| Document | Purpose (Rationale) | Primary Audience | Format / Content | Example (SaaS Context) | Measurement (KPI) |
|
||||||
|
|-----------------|---------------------|-----------------|------------------|------------------------|-------------------|
|
||||||
|
| **Requirements** | Capture the **business intent** — why we’re building this and what success looks like. Defines boundaries and user‑visible outcomes. | Stakeholders, Product Owners, Lead Developers | User stories, PRDs, acceptance criteria, non‑functional constraints. | “System must process tabular data from Julia to SvelteKit UI with <200ms latency for 5‑member teams.” | 95% of requests complete <200ms (synthetic monitoring). |
|
||||||
|
| **Specification** | The **technical contract** — precise rules for inputs, outputs, and data shape. Ensures consistency across dev and test. | Developers, QA Engineers, CI/CD pipelines | OpenAPI, Protobuf, AsyncAPI. Endpoint definitions, schemas, error codes. | `contract.yaml` defining a NATS subject that accepts Arrow streams with snake_case headers. | 100% of messages validated against spec (CI block rate). |
|
||||||
|
| **Architecture** | The **blueprint** — how components fit together, interact, and scale. Guides system structure and trade‑offs. | Architects, Senior Developers, DevOps | C4 diagrams, Mermaid.js, component/network/storage models. | Diagram showing 6‑node cluster routing traffic via Caddy → Node.js API → Julia pods. | 100% of major decisions logged with trade‑off analysis. |
|
||||||
|
| **Walkthrough** | The **story of flow** — shows how pieces connect end‑to‑end and why steps are sequenced. Builds intuition for new devs. | New Developers, Team Members | TOUR.md, Loom videos, sequence diagrams. Step‑by‑step traces with rationale. | “UI sends JSON → Node.js wraps Claim‑Check → Julia pulls Arrow data (prevents NATS overflow).” | New developers ship feature in <2 days (PR timeline). |
|
||||||
|
| **Implementation** | The **real code** — business logic, helpers, tests, configs. Where design becomes executable. | Developers, Code Reviewers | Source code, README.md, unit tests, setup scripts. | Julia function for matrix calculation + SvelteKit component rendering table. | >80% unit test coverage, <5% drift from spec. |
|
||||||
|
| **Validation** | The **enforcer** — ensures implementation matches the spec. Blocks drift and human error. | Automation servers, QA, Lead Developers | CI jobs, contract tests, linting, integration checks. | CI job rejects PR with camelCase field not allowed by YAML spec. | <1% of PRs bypass validation gates. |
|
||||||
|
| **Runbook** | The **operational manual** — how the system lives in production, scales, and recovers. Guides on‑call engineers. | DevOps, SREs, On‑call Developers | K8s manifests, Helm charts, Markdown guides. Deployment, scaling, backup/restore, troubleshooting. | GitOps manifest ensuring 6 Julia replicas restart if memory >80%. | MTTR <15 minutes for P1 incidents. |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------- 100 --------------------------------------------- #
|
||||||
|
|
||||||
|
SDD + GitOps Documentation Stack
|
||||||
|
Document,"Purpose (The ""Rationale"")",Primary Audience,Format / Content,Example (SaaS Context),"Measurement (KPI)"
|
||||||
|
Requirements,"Defines the ""Why"" and the Business Boundary. It sets the constraints and success criteria so the team knows when a feature is ""done"" from a user's perspective.","Stakeholders, Product Owners, Lead Developers","Format: User Stories, PRDs. Content: Functional goals, non-functional requirements (latency, scale), and explicit ""out-of-scope"" items.","""The system must process high-volume tabular data from Julia to the SvelteKit UI with <200ms latency for 5-member teams."",""Pass/Fail: 95% of requests complete <200ms (measured via synthetic monitoring)""
|
||||||
|
The Spec,"The Technical Contract. It serves as the single source of truth that defines the shape of data. In SDD, this file drives code generation and automated testing.","Developers, QA Engineers, CI/CD Pipelines","Format: OpenAPI (YAML), Protobuf, AsyncAPI. Content: Endpoint definitions, strict data types, error codes, and request/response schemas.",A contract.yaml defining a NATS subject that accepts an Apache Arrow stream with specific snake_case headers.",""Schema Validation Rate: 100% of messages validated against spec (CI block rate)""
|
||||||
|
Architecture,"The Structural Blueprint. It explains how the ""pieces"" are arranged in the cluster. It defines the relationships between services, databases, and external providers.","System Architects, Senior Developers, DevOps","Format: C4 Model Diagrams, Mermaid.js. Content: Component diagrams, network flow, storage strategy, and technology stack definitions.",A diagram showing how the 6-node cluster routes traffic through Caddy to the Node.js API and offloads heavy math to Julia pods.",""Architecture Decision Log: 100% of major decisions documented with trade-off analysis""
|
||||||
|
Walkthrough,"The Intuition & Flow. It connects multiple APIs/services into a cohesive end-to-end story. It explains the ""steps"" and the ""rationale"" behind the sequence of operations.","New Developers, Current Team Members","Format: TOUR.md, Loom videos, Sequence Diagrams. Content: Step-by-step trace of a feature, explanation of state changes, and the ""why"" behind complex logic.","""End-to-End Trace:"" 1. UI sends JSON to Node.js. 2. Node.js wraps it in a Claim-Check. 3. Julia pulls the Arrow data. Rationale: This prevents NATS memory overflow.",""Onboarding Velocity: New developers deploy feature in <2 days (tracked via PR timeline)""
|
||||||
|
Implementation,"The Functional Reality. This is the actual execution of the logic. In SDD, parts of this are auto-generated to ensure it never drifts from the Spec.","Developers, Code Reviewers","Format: Source Code (Git), README.md. Content: Business logic, internal helper functions, unit tests, and local setup instructions.",The Julia function that performs the matrix calculation and the SvelteKit component that renders the resulting table.",""Code Coverage: >80% unit test coverage, <5% test drift from spec""
|
||||||
|
Validation,"The Enforcement Layer. It ensures that the ""Reality"" (Code) actually matches the ""Contract"" (Spec). It prevents human error from breaking the system.","Automation Servers, QA, Lead Developers","Format: GitHub Actions, Dredd, Prism. Content: Contract tests, linting rules, and integration tests that check API compliance.",A CI job that blocks a Pull Request because a developer added a camelCase field that isn't allowed in the shared YAML spec.",""Block Rate: <1% of PRs reach production without validation (CI gate pass rate)""
|
||||||
|
Runbook,"The Operational Life-Support. It defines how the system lives in production and how to fix it. In GitOps, the ""State"" is declared here.","DevOps, SREs, On-call Developers","Format: K8s Manifests, Helm Charts, Markdown. Content: Deployment steps, scaling triggers, backup/restore commands, and troubleshooting guides.",A GitOps manifest in Flux that ensures 6 replicas of the Julia service are always running and restarts them if memory hits 80%.",""MTTR: <15 minutes for P1 incidents (tracked via incident management system)""
|
||||||
|
|
||||||
|
Do you understand the provided text? Don't fucking change the table content. I want you to add "Measurement (KPI)" column. it is only example of course. This table will be used for consult and teaching.
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------- 100 --------------------------------------------- #
|
||||||
|
|
||||||
|
Can you write the table and explain this approach and each doc in details then save to docs/SDD_FRAMEWORK.md so I can consult it later.
|
||||||
|
Don't forget to add How to use this approach effectively.
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------- 100 --------------------------------------------- #
|
||||||
|
|
||||||
|
Since I develop src folder before I adopt SDD_FRAMEWORK.md approach, can you check src folder and my current doc files then write docs/requirements.md according to SDD framework? Treat src as ground truth.
|
||||||
|
|
||||||
|
# ---------------------------------------------- 100 --------------------------------------------- #
|
||||||
|
|
||||||
|
Check NATSBridge/docs folder I want to update the content of the following files according to ASG_Framework/ASG_Framework.md:
|
||||||
|
- NATSBridge/docs/requirements.md
|
||||||
|
- NATSBridge/docs/specification.md
|
||||||
|
- NATSBridge/docs/ui-specification.md (you'll need to create this one)
|
||||||
|
- NATSBridge/docs/walkthrough.md
|
||||||
|
- NATSBridge/docs/architecture.md
|
||||||
|
I'll do the other docs not listed here later myself.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
now help me update the following fileaccording to ASG_Framework/ASG_Framework.md:
|
||||||
|
- NATSBridge/docs/specification.md
|
||||||
|
|
||||||
|
|
||||||
|
<!-- ------------------------------------------- 100 ------------------------------------------- -->
|
||||||
|
|
||||||
|
|
||||||
|
Check NATSBridge/docs folder. I would like to expand this package to include Dart support.
|
||||||
|
Can you update the content of the following files according to ASG_Framework/ASG_Framework.md:
|
||||||
|
- NATSBridge/docs/requirements.md
|
||||||
|
- NATSBridge/docs/specification.md
|
||||||
|
- NATSBridge/docs/walkthrough.md
|
||||||
|
- NATSBridge/docs/architecture.md
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
103
AI_prompt.txt
103
AI_prompt.txt
@@ -1,103 +0,0 @@
|
|||||||
Consider the following scenarios:
|
|
||||||
Scenario 1: The "Command & Control" Loop (Low Latency)Focus: Small payloads, Core NATS, bi-directional JSON.The Action: A user on a JavaScript dashboard clicks a "Start Simulation" button. This sends a JSON configuration (parameters like step_size and iterations) to Julia.The Flow: * JS (Sender): Recognizes the message is small ($< 10KB$). Packages it as a direct transport JSON envelope.Julia (Receiver): Listens on the NATS subject, decodes the JSON, and immediately acknowledges receipt with a "Running" status.Project Requirement Met: Fast, low-overhead communication for control signals without involving the fileserver.
|
|
||||||
Scenario 2: The "Deep Dive" Analysis (High Bandwidth)Focus: Large Arrow tables, Claim-Check pattern, Julia to JS.The Action: Julia finishes a heavy computation and produces a 500MB DataFrame with 10 million rows. It needs to send this to the JS frontend for visualization (e.g., using Perspective.js or D3).The Flow:Julia (Sender): Converts the DataFrame to an Arrow IPC stream. It sees the size is $> 1MB$, so it uploads the bytes to the HTTP fileserver. It then publishes a NATS message with transport: "link" and the URL.JS (Receiver): Receives the URL, fetches the data via fetch(), and uses tableFromIPC() to load the data into memory with zero-copy.Project Requirement Met: Handling massive datasets that exceed NATS message limits while maintaining data integrity across languages.
|
|
||||||
Scenario 3: Live Audio/Signal Processing (Multimedia & Metadata)Focus: Raw binary, bi-directional streaming, NATS Headers.The Action: The JS client captures a 2-second "chunk" of microphone audio. It needs Julia to perform a Fast Fourier Transform (FFT) or AI transcription.The Flow:JS (Sender): Sends the raw binary WAV/PCM data. It uses NATS Headers to store the metadata ($fs = 44.1kHz$, $channels = 1$) to keep the payload purely binary.Julia (Receiver): Processes the audio and sends back a JSON result (the transcription) and an Arrow Table (the frequency spectrum data).Project Requirement Met: Bi-directional flow involving mixed media (Audio) and technical results (Arrow).
|
|
||||||
Scenario 4: The "Catch-Up" (Persistence & JetStream)Focus: NATS JetStream, late-joining consumers, state sync.The Action: Julia is constantly publishing "System Health" updates. The JS dashboard is closed for 10 minutes. When the user re-opens the dashboard, they need to see the last 10 minutes of history.The Flow:NATS (Server): Uses a JetStream with a Limits retention policy.JS (Consumer): Connects and requests a "Replay" from the last 10 minutes. It receives a mix of direct (small updates) and link (historical snapshots) messages.Project Requirement Met: Temporal decoupling—consumers can receive data that was sent while they were offline.
|
|
||||||
|
|
||||||
Role: Principal Systems Architect & Lead Software Engineer.Objective: Implement a high-performance, bi-directional data bridge between a Julia service and a JavaScript (Node.js) service using NATS (Core & JetStream).⚠️ STRICT ARCHITECTURAL CONSTRAINTS (Non-Negotiable)Transport Strategy (Claim-Check Pattern):Direct Path: If payload is < 1MB, send data directly via NATS inside the message envelope (Base64 encoded).Link Path: If payload is > 1MB, upload to a shared HTTP fileserver/store. The NATS message must only contain the metadata and the download URL.Tabular Data Format: * MUST use Apache Arrow IPC Stream for all tables/DataFrames. No CSV or standard JSON-serialization of tables allowed.System Symmetry: * Both services must function as Producers AND Consumers.Modular Elegance: * Implementation must be abstracted into a SmartSend function and a SmartReceive handler. The developer calling these functions should not need to care if the data is going via NATS direct or HTTP link.Technical Stack & Use CasesJulia: NATS.jl, Arrow.jl, JSON3.jl, HTTP.jl.Node.js: nats.js, apache-arrow.Scenarios to Support: * Large Data: Sending a 500MB Arrow table from Julia $\rightarrow$ JS.Media: Sending a 5MB WAV file from JS $\rightarrow$ Julia.Signals: Sending small JSON control commands ($< 10KB$) directly via NATS.Implementation Requirements1. Unified JSON Envelope:Define a schema containing: correlation_id (UUID), type (table/binary/json), transport (direct/link), payload (if direct), and url (if link).2. The Julia Module:Implement SmartSend(subject, data, type): Handles Arrow serialization to an IOBuffer, checks size, and manages HTTP uploads for large blobs.Implement SmartReceive(msg): Parses envelope, handles the HTTP fetch with Exponential Backoff (to avoid race conditions), and restores the DataFrame.Include a basic HTTP.listen server to serve as the temporary storage.3. The JavaScript Module:Implement a symmetric SmartSend using nats.js and apache-arrow.Implement a JetStream Pull Consumer for SmartReceive to ensure backpressure and memory safety.4. Performance & Reliability:Demonstrate "Zero-Copy" reading of the Arrow IPC stream on the JS side.Log the correlation_id at every stage for distributed tracing.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Create a walkthrough for Julia service-A service sending a mix-content chat message to Julia service-B. the chat message must includes
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
I updated the following:
|
|
||||||
- NATSBridge.jl. Essentially I add NATS_connection keyword and new publish_message function to support the keyword.
|
|
||||||
Use them and ONLY them as ground truth.
|
|
||||||
Then update the following files accordingly:
|
|
||||||
- architecture.md
|
|
||||||
- implementation.md
|
|
||||||
|
|
||||||
All API should be semantically consistent and naming should be consistent across the board.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Task: Update NATSBridge.js to reflect recent changes in NATSBridge.jl and docs
|
|
||||||
Context: NATSBridge.jl and docs has been updated.
|
|
||||||
Requirements:
|
|
||||||
Source of Truth: Treat the updated NATSBridge.jl and docs as the definitive source.
|
|
||||||
API Consistency: Ensure the Main Package API (e.g., smartsend(), publish_message()) uses consistent naming across all three supported languages.
|
|
||||||
Ecosystem Variance: Low-level native functions (e.g., NATS.connect(), JSON.read()) should follow the conventions of the specific language ecosystem and do not require cross-language consistency.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
I'm expanding this Julia package (NATSBridge) into a cross-platform project by adding a JavaScript and Python/MicroPython implementation. To ensure accuracy, the Julia src directory will serve as the ground truth, as the documentation may be outdated.
|
|
||||||
|
|
||||||
My goal is to maintain interface parity at the high-level API for a consistent user experience, while ensuring the low-level implementation adheres strictly to the idiomatic conventions of each respective language (e.g., multiple dispatch in Julia vs. asynchronous, prototype, or class-based patterns in JS and Python/MicroPython)
|
|
||||||
|
|
||||||
Now, help me do the following:
|
|
||||||
1) check architecture.md for any mistake.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Help me expands this Julia package (NATSBridge) into a cross-platform project by adding a JavaScript and Python/MicroPython implementation. To ensure accuracy, NATSBridge.jl will serve as the ground truth, as the documentation may be outdated.
|
|
||||||
|
|
||||||
My goal is to maintain interface parity at the high-level API for a consistent user experience, while ensuring the low-level implementation adheres strictly to the idiomatic conventions of each respective language (e.g., multiple dispatch in Julia vs. asynchronous, prototype, or class-based patterns in JS and Python/MicroPython)
|
|
||||||
|
|
||||||
Now do the following:
|
|
||||||
1) check docs to see if there is any mistake.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
I'm expanding this Julia package (NATSBridge) into a cross-platform project by adding
|
|
||||||
a JavaScript, Python and MicroPython implementation.
|
|
||||||
The following will serve as the ground truth:
|
|
||||||
- test_julia_mix_payloads_sender.jl
|
|
||||||
- NATSBridge.jl
|
|
||||||
- test_julia_mix_payloads_receiver.jl
|
|
||||||
- architecture.md
|
|
||||||
|
|
||||||
My goal is to maintain interface parity at the high-level API for a consistent user experience,
|
|
||||||
while ensuring the low-level implementation adheres strictly to the idiomatic conventions of each
|
|
||||||
respective language (e.g., multiple dispatch in Julia vs. asynchronous, prototype, or class-based
|
|
||||||
patterns in JS, Python and MicroPython)
|
|
||||||
|
|
||||||
Now, help me do the following:
|
|
||||||
1) Check whether natsbridge.js needs update or it already up to date.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -52,9 +52,9 @@ version = "1.2.2"
|
|||||||
|
|
||||||
[[deps.CSV]]
|
[[deps.CSV]]
|
||||||
deps = ["CodecZlib", "Dates", "FilePathsBase", "InlineStrings", "Mmap", "Parsers", "PooledArrays", "PrecompileTools", "SentinelArrays", "Tables", "Unicode", "WeakRefStrings", "WorkerUtilities"]
|
deps = ["CodecZlib", "Dates", "FilePathsBase", "InlineStrings", "Mmap", "Parsers", "PooledArrays", "PrecompileTools", "SentinelArrays", "Tables", "Unicode", "WeakRefStrings", "WorkerUtilities"]
|
||||||
git-tree-sha1 = "deddd8725e5e1cc49ee205a1964256043720a6c3"
|
git-tree-sha1 = "8d8e0b0f350b8e1c91420b5e64e5de774c2f0f4d"
|
||||||
uuid = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
|
uuid = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
|
||||||
version = "0.10.15"
|
version = "0.10.16"
|
||||||
|
|
||||||
[[deps.CodeTracking]]
|
[[deps.CodeTracking]]
|
||||||
deps = ["InteractiveUtils", "UUIDs"]
|
deps = ["InteractiveUtils", "UUIDs"]
|
||||||
@@ -108,9 +108,9 @@ version = "1.3.0+1"
|
|||||||
|
|
||||||
[[deps.ConcurrentUtilities]]
|
[[deps.ConcurrentUtilities]]
|
||||||
deps = ["Serialization", "Sockets"]
|
deps = ["Serialization", "Sockets"]
|
||||||
git-tree-sha1 = "d9d26935a0bcffc87d2613ce14c527c99fc543fd"
|
git-tree-sha1 = "21d088c496ea22914fe80906eb5bce65755e5ec8"
|
||||||
uuid = "f0e56b4a-5159-44fe-b623-3e5288b988bb"
|
uuid = "f0e56b4a-5159-44fe-b623-3e5288b988bb"
|
||||||
version = "2.5.0"
|
version = "2.5.1"
|
||||||
|
|
||||||
[[deps.Crayons]]
|
[[deps.Crayons]]
|
||||||
git-tree-sha1 = "249fe38abf76d48563e2f4556bebd215aa317e15"
|
git-tree-sha1 = "249fe38abf76d48563e2f4556bebd215aa317e15"
|
||||||
@@ -171,9 +171,9 @@ uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
|
|||||||
version = "1.7.0"
|
version = "1.7.0"
|
||||||
|
|
||||||
[[deps.EnumX]]
|
[[deps.EnumX]]
|
||||||
git-tree-sha1 = "7bebc8aad6ee6217c78c5ddcf7ed289d65d0263e"
|
git-tree-sha1 = "c49898e8438c828577f04b92fc9368c388ac783c"
|
||||||
uuid = "4e289a0a-7415-4d19-859d-a7e5c4648b56"
|
uuid = "4e289a0a-7415-4d19-859d-a7e5c4648b56"
|
||||||
version = "1.0.6"
|
version = "1.0.7"
|
||||||
|
|
||||||
[[deps.ExceptionUnwrapping]]
|
[[deps.ExceptionUnwrapping]]
|
||||||
deps = ["Test"]
|
deps = ["Test"]
|
||||||
@@ -234,9 +234,9 @@ version = "0.3.1"
|
|||||||
|
|
||||||
[[deps.HTTP]]
|
[[deps.HTTP]]
|
||||||
deps = ["Base64", "CodecZlib", "ConcurrentUtilities", "Dates", "ExceptionUnwrapping", "Logging", "LoggingExtras", "MbedTLS", "NetworkOptions", "OpenSSL", "PrecompileTools", "Random", "SimpleBufferStream", "Sockets", "URIs", "UUIDs"]
|
deps = ["Base64", "CodecZlib", "ConcurrentUtilities", "Dates", "ExceptionUnwrapping", "Logging", "LoggingExtras", "MbedTLS", "NetworkOptions", "OpenSSL", "PrecompileTools", "Random", "SimpleBufferStream", "Sockets", "URIs", "UUIDs"]
|
||||||
git-tree-sha1 = "5e6fe50ae7f23d171f44e311c2960294aaa0beb5"
|
git-tree-sha1 = "51059d23c8bb67911a2e6fd5130229113735fc7e"
|
||||||
uuid = "cd3eb016-35fb-5094-929b-558a96fad6f3"
|
uuid = "cd3eb016-35fb-5094-929b-558a96fad6f3"
|
||||||
version = "1.10.19"
|
version = "1.11.0"
|
||||||
|
|
||||||
[[deps.HashArrayMappedTries]]
|
[[deps.HashArrayMappedTries]]
|
||||||
git-tree-sha1 = "2eaa69a7cab70a52b9687c8bf950a5a93ec895ae"
|
git-tree-sha1 = "2eaa69a7cab70a52b9687c8bf950a5a93ec895ae"
|
||||||
@@ -307,9 +307,9 @@ weakdeps = ["ArrowTypes"]
|
|||||||
|
|
||||||
[[deps.JuliaInterpreter]]
|
[[deps.JuliaInterpreter]]
|
||||||
deps = ["CodeTracking", "InteractiveUtils", "Random", "UUIDs"]
|
deps = ["CodeTracking", "InteractiveUtils", "Random", "UUIDs"]
|
||||||
git-tree-sha1 = "80580012d4ed5a3e8b18c7cd86cebe4b816d17a6"
|
git-tree-sha1 = "3d3b79166e2a0afcf875df20db110af91ad3ab61"
|
||||||
uuid = "aa1ae85d-cabe-5617-a682-6adf51b2e16a"
|
uuid = "aa1ae85d-cabe-5617-a682-6adf51b2e16a"
|
||||||
version = "0.10.9"
|
version = "0.10.11"
|
||||||
|
|
||||||
[[deps.JuliaSyntaxHighlighting]]
|
[[deps.JuliaSyntaxHighlighting]]
|
||||||
deps = ["StyledStrings"]
|
deps = ["StyledStrings"]
|
||||||
@@ -383,9 +383,9 @@ version = "1.2.0"
|
|||||||
|
|
||||||
[[deps.LoweredCodeUtils]]
|
[[deps.LoweredCodeUtils]]
|
||||||
deps = ["CodeTracking", "Compiler", "JuliaInterpreter"]
|
deps = ["CodeTracking", "Compiler", "JuliaInterpreter"]
|
||||||
git-tree-sha1 = "65ae3db6ab0e5b1b5f217043c558d9d1d33cc88d"
|
git-tree-sha1 = "5d4278f755440f70648d80cc6225f51e78e94094"
|
||||||
uuid = "6f1432cf-f94c-5a45-995e-cdbf5db27b0b"
|
uuid = "6f1432cf-f94c-5a45-995e-cdbf5db27b0b"
|
||||||
version = "3.5.0"
|
version = "3.5.1"
|
||||||
|
|
||||||
[[deps.Lz4_jll]]
|
[[deps.Lz4_jll]]
|
||||||
deps = ["Artifacts", "JLLWrappers", "Libdl"]
|
deps = ["Artifacts", "JLLWrappers", "Libdl"]
|
||||||
@@ -400,9 +400,9 @@ version = "1.11.0"
|
|||||||
|
|
||||||
[[deps.MbedTLS]]
|
[[deps.MbedTLS]]
|
||||||
deps = ["Dates", "MbedTLS_jll", "MozillaCACerts_jll", "NetworkOptions", "Random", "Sockets"]
|
deps = ["Dates", "MbedTLS_jll", "MozillaCACerts_jll", "NetworkOptions", "Random", "Sockets"]
|
||||||
git-tree-sha1 = "c067a280ddc25f196b5e7df3877c6b226d390aaf"
|
git-tree-sha1 = "8785729fa736197687541f7053f6d8ab7fc44f92"
|
||||||
uuid = "739be429-bea8-5141-9913-cc70e7f3736d"
|
uuid = "739be429-bea8-5141-9913-cc70e7f3736d"
|
||||||
version = "1.1.9"
|
version = "1.1.10"
|
||||||
|
|
||||||
[[deps.MbedTLS_jll]]
|
[[deps.MbedTLS_jll]]
|
||||||
deps = ["Artifacts", "JLLWrappers", "Libdl"]
|
deps = ["Artifacts", "JLLWrappers", "Libdl"]
|
||||||
@@ -437,10 +437,10 @@ uuid = "55e73f9c-eeeb-467f-b4cc-a633fde63d2a"
|
|||||||
version = "0.1.0"
|
version = "0.1.0"
|
||||||
|
|
||||||
[[deps.NATSBridge]]
|
[[deps.NATSBridge]]
|
||||||
deps = ["Arrow", "DataFrames", "Dates", "GeneralUtils", "HTTP", "JSON", "NATS", "PrettyPrinting", "Revise", "UUIDs"]
|
deps = ["Arrow", "Base64", "DataFrames", "Dates", "GeneralUtils", "HTTP", "JSON", "NATS", "PrettyPrinting", "Revise", "UUIDs"]
|
||||||
path = "."
|
path = "."
|
||||||
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
|
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
|
||||||
version = "0.4.1"
|
version = "0.5.6"
|
||||||
|
|
||||||
[[deps.NanoDates]]
|
[[deps.NanoDates]]
|
||||||
deps = ["Dates", "Parsers"]
|
deps = ["Dates", "Parsers"]
|
||||||
@@ -514,9 +514,9 @@ version = "1.3.3"
|
|||||||
|
|
||||||
[[deps.Preferences]]
|
[[deps.Preferences]]
|
||||||
deps = ["TOML"]
|
deps = ["TOML"]
|
||||||
git-tree-sha1 = "522f093a29b31a93e34eaea17ba055d850edea28"
|
git-tree-sha1 = "8b770b60760d4451834fe79dd483e318eee709c4"
|
||||||
uuid = "21216c6a-2e73-6563-6e65-726566657250"
|
uuid = "21216c6a-2e73-6563-6e65-726566657250"
|
||||||
version = "1.5.1"
|
version = "1.5.2"
|
||||||
|
|
||||||
[[deps.PrettyPrinting]]
|
[[deps.PrettyPrinting]]
|
||||||
git-tree-sha1 = "142ee93724a9c5d04d78df7006670a93ed1b244e"
|
git-tree-sha1 = "142ee93724a9c5d04d78df7006670a93ed1b244e"
|
||||||
@@ -525,9 +525,15 @@ version = "0.4.2"
|
|||||||
|
|
||||||
[[deps.PrettyTables]]
|
[[deps.PrettyTables]]
|
||||||
deps = ["Crayons", "LaTeXStrings", "Markdown", "PrecompileTools", "Printf", "REPL", "Reexport", "StringManipulation", "Tables"]
|
deps = ["Crayons", "LaTeXStrings", "Markdown", "PrecompileTools", "Printf", "REPL", "Reexport", "StringManipulation", "Tables"]
|
||||||
git-tree-sha1 = "c5a07210bd060d6a8491b0ccdee2fa0235fc00bf"
|
git-tree-sha1 = "211530a7dc76ab59087f4d4d1fc3f086fbe87594"
|
||||||
uuid = "08abe8d2-0d0c-5749-adfa-8a2ac140af0d"
|
uuid = "08abe8d2-0d0c-5749-adfa-8a2ac140af0d"
|
||||||
version = "3.1.2"
|
version = "3.2.3"
|
||||||
|
|
||||||
|
[deps.PrettyTables.extensions]
|
||||||
|
PrettyTablesTypstryExt = "Typstry"
|
||||||
|
|
||||||
|
[deps.PrettyTables.weakdeps]
|
||||||
|
Typstry = "f0ed7684-a786-439e-b1e3-3b82803b501e"
|
||||||
|
|
||||||
[[deps.Printf]]
|
[[deps.Printf]]
|
||||||
deps = ["Unicode"]
|
deps = ["Unicode"]
|
||||||
@@ -535,9 +541,9 @@ uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"
|
|||||||
version = "1.11.0"
|
version = "1.11.0"
|
||||||
|
|
||||||
[[deps.PtrArrays]]
|
[[deps.PtrArrays]]
|
||||||
git-tree-sha1 = "1d36ef11a9aaf1e8b74dacc6a731dd1de8fd493d"
|
git-tree-sha1 = "4fbbafbc6251b883f4d2705356f3641f3652a7fe"
|
||||||
uuid = "43287f4e-b6f4-7ad1-bb20-aadabca52c3d"
|
uuid = "43287f4e-b6f4-7ad1-bb20-aadabca52c3d"
|
||||||
version = "1.3.0"
|
version = "1.4.0"
|
||||||
|
|
||||||
[[deps.QuadGK]]
|
[[deps.QuadGK]]
|
||||||
deps = ["DataStructures", "LinearAlgebra"]
|
deps = ["DataStructures", "LinearAlgebra"]
|
||||||
@@ -568,9 +574,9 @@ version = "1.2.2"
|
|||||||
|
|
||||||
[[deps.Revise]]
|
[[deps.Revise]]
|
||||||
deps = ["CodeTracking", "FileWatching", "InteractiveUtils", "JuliaInterpreter", "LibGit2", "LoweredCodeUtils", "OrderedCollections", "Preferences", "REPL", "UUIDs"]
|
deps = ["CodeTracking", "FileWatching", "InteractiveUtils", "JuliaInterpreter", "LibGit2", "LoweredCodeUtils", "OrderedCollections", "Preferences", "REPL", "UUIDs"]
|
||||||
git-tree-sha1 = "14d1bfb0a30317edc77e11094607ace3c800f193"
|
git-tree-sha1 = "d97d78d4fc5f858d8ce44f6b88bc972f2023f51d"
|
||||||
uuid = "295af30f-e4ad-537b-8983-00126c2a3abe"
|
uuid = "295af30f-e4ad-537b-8983-00126c2a3abe"
|
||||||
version = "3.13.2"
|
version = "3.14.0"
|
||||||
|
|
||||||
[deps.Revise.extensions]
|
[deps.Revise.extensions]
|
||||||
DistributedExt = "Distributed"
|
DistributedExt = "Distributed"
|
||||||
@@ -596,9 +602,9 @@ version = "0.7.0"
|
|||||||
|
|
||||||
[[deps.ScopedValues]]
|
[[deps.ScopedValues]]
|
||||||
deps = ["HashArrayMappedTries", "Logging"]
|
deps = ["HashArrayMappedTries", "Logging"]
|
||||||
git-tree-sha1 = "c3b2323466378a2ba15bea4b2f73b081e022f473"
|
git-tree-sha1 = "ac4b837d89a58c848e85e698e2a2514e9d59d8f6"
|
||||||
uuid = "7e506255-f358-4e82-b7e4-beb19740aa63"
|
uuid = "7e506255-f358-4e82-b7e4-beb19740aa63"
|
||||||
version = "1.5.0"
|
version = "1.6.0"
|
||||||
|
|
||||||
[[deps.Scratch]]
|
[[deps.Scratch]]
|
||||||
deps = ["Dates"]
|
deps = ["Dates"]
|
||||||
@@ -644,9 +650,9 @@ version = "1.12.0"
|
|||||||
|
|
||||||
[[deps.SpecialFunctions]]
|
[[deps.SpecialFunctions]]
|
||||||
deps = ["IrrationalConstants", "LogExpFunctions", "OpenLibm_jll", "OpenSpecFun_jll"]
|
deps = ["IrrationalConstants", "LogExpFunctions", "OpenLibm_jll", "OpenSpecFun_jll"]
|
||||||
git-tree-sha1 = "f2685b435df2613e25fc10ad8c26dddb8640f547"
|
git-tree-sha1 = "5acc6a41b3082920f79ca3c759acbcecf18a8d78"
|
||||||
uuid = "276daf66-3868-5448-9aa4-cd146d93841b"
|
uuid = "276daf66-3868-5448-9aa4-cd146d93841b"
|
||||||
version = "2.6.1"
|
version = "2.7.1"
|
||||||
|
|
||||||
[deps.SpecialFunctions.extensions]
|
[deps.SpecialFunctions.extensions]
|
||||||
SpecialFunctionsChainRulesCoreExt = "ChainRulesCore"
|
SpecialFunctionsChainRulesCoreExt = "ChainRulesCore"
|
||||||
@@ -692,9 +698,9 @@ version = "1.5.2"
|
|||||||
|
|
||||||
[[deps.StringManipulation]]
|
[[deps.StringManipulation]]
|
||||||
deps = ["PrecompileTools"]
|
deps = ["PrecompileTools"]
|
||||||
git-tree-sha1 = "a3c1536470bf8c5e02096ad4853606d7c8f62721"
|
git-tree-sha1 = "d05693d339e37d6ab134c5ab53c29fce5ee5d7d5"
|
||||||
uuid = "892a3eda-7b42-436c-8928-eab12a02cf0e"
|
uuid = "892a3eda-7b42-436c-8928-eab12a02cf0e"
|
||||||
version = "0.4.2"
|
version = "0.4.4"
|
||||||
|
|
||||||
[[deps.StringViews]]
|
[[deps.StringViews]]
|
||||||
git-tree-sha1 = "f2dcb92855b31ad92fe8f079d4f75ac57c93e4b8"
|
git-tree-sha1 = "f2dcb92855b31ad92fe8f079d4f75ac57c93e4b8"
|
||||||
@@ -709,16 +715,18 @@ version = "1.11.0"
|
|||||||
|
|
||||||
[[deps.StructUtils]]
|
[[deps.StructUtils]]
|
||||||
deps = ["Dates", "UUIDs"]
|
deps = ["Dates", "UUIDs"]
|
||||||
git-tree-sha1 = "9297459be9e338e546f5c4bedb59b3b5674da7f1"
|
git-tree-sha1 = "fa95b3b097bcef5845c142ea2e085f1b2591e92c"
|
||||||
uuid = "ec057cc2-7a8d-4b58-b3b3-92acb9f63b42"
|
uuid = "ec057cc2-7a8d-4b58-b3b3-92acb9f63b42"
|
||||||
version = "2.6.2"
|
version = "2.7.1"
|
||||||
|
|
||||||
[deps.StructUtils.extensions]
|
[deps.StructUtils.extensions]
|
||||||
StructUtilsMeasurementsExt = ["Measurements"]
|
StructUtilsMeasurementsExt = ["Measurements"]
|
||||||
|
StructUtilsStaticArraysCoreExt = ["StaticArraysCore"]
|
||||||
StructUtilsTablesExt = ["Tables"]
|
StructUtilsTablesExt = ["Tables"]
|
||||||
|
|
||||||
[deps.StructUtils.weakdeps]
|
[deps.StructUtils.weakdeps]
|
||||||
Measurements = "eff96d63-e80a-5855-80a2-b1b0885c5ab7"
|
Measurements = "eff96d63-e80a-5855-80a2-b1b0885c5ab7"
|
||||||
|
StaticArraysCore = "1e83bf80-4336-4d27-bf5d-d5a4f845583c"
|
||||||
Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
|
Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
|
||||||
|
|
||||||
[[deps.StyledStrings]]
|
[[deps.StyledStrings]]
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
name = "NATSBridge"
|
name = "NATSBridge"
|
||||||
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
|
uuid = "f2724d33-f338-4a57-b9f8-1be882570d10"
|
||||||
version = "0.4.5"
|
version = "0.5.6"
|
||||||
authors = ["narawat <narawat@gmail.com>"]
|
authors = ["narawat <narawat@gmail.com>"]
|
||||||
|
|
||||||
[deps]
|
[deps]
|
||||||
|
|||||||
550
README.md
550
README.md
@@ -1,6 +1,6 @@
|
|||||||
# NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
# NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||||
|
|
||||||
A high-performance, bi-directional data bridge for **Julia, JavaScript, Python, and MicroPython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
A high-performance, bi-directional data bridge for **Julia**, **JavaScript**, **Python**, **Dart**, and **MicroPython** applications using NATS (Core & JetStream), implementing the Claim-Check pattern for large payloads.
|
||||||
|
|
||||||
[](https://opensource.org/licenses/MIT)
|
[](https://opensource.org/licenses/MIT)
|
||||||
[](https://nats.io)
|
[](https://nats.io)
|
||||||
@@ -28,8 +28,8 @@ NATSBridge enables seamless communication across multiple platforms through NATS
|
|||||||
|
|
||||||
| Transport | Payload Size | Method |
|
| Transport | Payload Size | Method |
|
||||||
|-----------|--------------|--------|
|
|-----------|--------------|--------|
|
||||||
| **Direct** | < 1MB | Sent directly via NATS (Base64 encoded) |
|
| **Direct** | < 500KB | Sent directly via NATS (Base64 encoded) |
|
||||||
| **Link** | >= 1MB | Uploaded to HTTP file server, URL sent via NATS |
|
| **Link** | ≥ 500KB | Uploaded to HTTP file server, URL sent via NATS |
|
||||||
|
|
||||||
### Use Cases
|
### Use Cases
|
||||||
|
|
||||||
@@ -45,33 +45,39 @@ NATSBridge enables seamless communication across multiple platforms through NATS
|
|||||||
| Platform | Implementation | Features |
|
| Platform | Implementation | Features |
|
||||||
|----------|----------------|----------|
|
|----------|----------------|----------|
|
||||||
| **Julia** | [`src/NATSBridge.jl`](src/NATSBridge.jl) | Full feature set, Arrow IPC, multiple dispatch |
|
| **Julia** | [`src/NATSBridge.jl`](src/NATSBridge.jl) | Full feature set, Arrow IPC, multiple dispatch |
|
||||||
| **JavaScript** | [`src/natsbridge.js`](src/natsbridge.js) | Node.js & browser, async/await |
|
| **JavaScript (Node.js)** | [`src/natsbridge_ssr.js`](src/natsbridge_ssr.js) | Node.js, async/await, Arrow IPC |
|
||||||
| **Python** | [`src/natsbridge.py`](src/natsbridge.py) | Desktop Python, asyncio, type hints |
|
| **JavaScript (Browser)** | [`src/natsbridge_csr.js`](src/natsbridge_csr.js) | Browser, WebSocket NATS, async/await, JSON table only |
|
||||||
|
| **Python** | [`src/natsbridge.py`](src/natsbridge.py) | Desktop Python, asyncio, type hints, Arrow IPC |
|
||||||
|
| **Dart (Desktop/Flutter)** | [`src/natsbridge.dart`](src/natsbridge.dart) | Desktop/Flutter, async/await, Arrow IPC |
|
||||||
|
| **Dart Web** | [`src/natsbridge.dart`](src/natsbridge.dart) | Web, WebSocket NATS, JSON table only |
|
||||||
| **MicroPython** | [`src/natsbridge_mpy.py`](src/natsbridge_mpy.py) | Memory-constrained, synchronous API |
|
| **MicroPython** | [`src/natsbridge_mpy.py`](src/natsbridge_mpy.py) | Memory-constrained, synchronous API |
|
||||||
|
|
||||||
### Platform Comparison
|
### Platform Comparison
|
||||||
|
|
||||||
| Feature | Julia | JavaScript | Python | MicroPython |
|
| Feature | Julia | JavaScript | JavaScript (Browser) | Python | Dart | Dart Web | MicroPython |
|
||||||
|---------|-------|------------|--------|-------------|
|
|---------|-------|------------|----------------------|--------|------|----------|-------------|
|
||||||
| Multiple Dispatch | ✅ Native | ❌ | ❌ | ❌ |
|
| Multiple Dispatch | ✅ Native | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||||
| Async/Await | ❌ | ✅ Native | ✅ Native | ⚠️ (uasyncio) |
|
| Async/Await | ❌ | ✅ Native | ✅ Native | ✅ Native | ✅ Native | ✅ Native | ⚠️ (uasyncio) |
|
||||||
| Type Safety | ✅ Strong | ⚠️ (TypeScript) | ✅ (Type hints) | ❌ |
|
| Type Safety | ✅ Strong | ⚠️ (TypeScript) | ⚠️ (TypeScript) | ✅ (Type hints) | ✅ Strong | ✅ Strong | ❌ |
|
||||||
| Arrow IPC | ✅ Native | ✅ | ✅ | ❌ |
|
| Arrow IPC | ✅ Native | ✅ Native | ❌ (Browser incompatible) | ✅ Native | ✅ Native | ❌ (Browser incompatible) | ❌ |
|
||||||
| Direct Transport | ✅ | ✅ | ✅ | ✅ |
|
| JSON Table | ✅ | ✅ | ✅ (Only table type) | ✅ | ✅ | ✅ | ⚠️ (Limited) |
|
||||||
| Link Transport | ✅ | ✅ | ✅ | ⚠️ (Limited) |
|
| Direct Transport | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||||
| Handler Functions | ✅ | ✅ | ✅ | ✅ |
|
| Link Transport | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ⚠️ (Limited) |
|
||||||
| Cross-Platform API | ✅ | ✅ | ✅ | ✅ |
|
| Handler Functions | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||||
|
| Cross-Platform API | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||||
|
| WebSocket NATS | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
- ✅ **Cross-platform messaging** for Julia, JavaScript, Python, and MicroPython applications
|
- ✅ **Cross-platform messaging** for Julia, JavaScript, Python, Dart, and MicroPython applications
|
||||||
- ✅ **Bi-directional messaging** with request-reply patterns
|
- ✅ **Bi-directional messaging** with request-reply patterns
|
||||||
- ✅ **Multi-payload support** - send multiple payloads with different types in one message
|
- ✅ **Multi-payload support** - send multiple payloads with different types in one message
|
||||||
- ✅ **Automatic transport selection** - direct vs link based on payload size
|
- ✅ **Automatic transport selection** - direct vs link based on payload size
|
||||||
- ✅ **Claim-Check pattern** for payloads > 1MB
|
- ✅ **Claim-Check pattern** for payloads ≥ 500KB
|
||||||
- ✅ **Apache Arrow IPC** support for tabular data (zero-copy reading)
|
- ✅ **Apache Arrow IPC** support for tabular data (Desktop: Julia/Python/Node.js/Dart)
|
||||||
|
- ✅ **JSON Table** support for tabular data (All platforms including Browser)
|
||||||
- ✅ **Exponential backoff** for reliable file server downloads
|
- ✅ **Exponential backoff** for reliable file server downloads
|
||||||
- ✅ **Correlation ID tracking** for message tracing
|
- ✅ **Correlation ID tracking** for message tracing
|
||||||
- ✅ **Reply-to support** for request-response patterns
|
- ✅ **Reply-to support** for request-response patterns
|
||||||
@@ -81,23 +87,24 @@ NATSBridge enables seamless communication across multiple platforms through NATS
|
|||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
### Step 1: Start NATS Server
|
### Prerequisites
|
||||||
|
|
||||||
```bash
|
1. **NATS Server** - Install and run a NATS server:
|
||||||
docker run -p 4222:4222 nats:latest
|
```bash
|
||||||
```
|
docker run -p 4222:4222 nats:latest
|
||||||
|
```
|
||||||
|
|
||||||
### Step 2: Start HTTP File Server (Optional)
|
2. **HTTP File Server** (optional, for large payloads) - Install and run a file server:
|
||||||
|
```bash
|
||||||
|
# Using Plik
|
||||||
|
docker run -p 8080:8080 -v /tmp/fileserver:/var/lib/plik -e PLIK_ADMIN_PASSWORD=admin plik/plik
|
||||||
|
|
||||||
|
# OR using simple Python HTTP server
|
||||||
|
mkdir -p /tmp/fileserver
|
||||||
|
python3 -m http.server 8080 --directory /tmp/fileserver
|
||||||
|
```
|
||||||
|
|
||||||
```bash
|
### Send Your First Message
|
||||||
# Create a directory for file uploads
|
|
||||||
mkdir -p /tmp/fileserver
|
|
||||||
|
|
||||||
# Start HTTP file server
|
|
||||||
python3 -m http.server 8080 --directory /tmp/fileserver
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 3: Send Your First Message
|
|
||||||
|
|
||||||
#### Julia
|
#### Julia
|
||||||
|
|
||||||
@@ -105,14 +112,14 @@ python3 -m http.server 8080 --directory /tmp/fileserver
|
|||||||
using NATSBridge
|
using NATSBridge
|
||||||
|
|
||||||
data = [("message", "Hello World", "text")]
|
data = [("message", "Hello World", "text")]
|
||||||
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222")
|
env, env_json_str = smartsend("/chat/room1", data; broker_url="nats://localhost:4222")
|
||||||
println("Message sent!")
|
println("Message sent!")
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript
|
#### JavaScript (Node.js)
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
const NATSBridge = require('./src/natsbridge.js');
|
import NATSBridge from './src/natsbridge_ssr.js';
|
||||||
|
|
||||||
const data = [["message", "Hello World", "text"]];
|
const data = [["message", "Hello World", "text"]];
|
||||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||||
@@ -123,6 +130,20 @@ const [env, env_json_str] = await NATSBridge.smartsend(
|
|||||||
console.log("Message sent!");
|
console.log("Message sent!");
|
||||||
```
|
```
|
||||||
|
|
||||||
|
#### JavaScript (Browser)
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
import NATSBridge from './src/natsbridge_csr.js';
|
||||||
|
|
||||||
|
const data = [["message", "Hello World", "text"]];
|
||||||
|
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||||
|
"/chat/room1",
|
||||||
|
data,
|
||||||
|
{ broker_url: "ws://localhost:4222" }
|
||||||
|
);
|
||||||
|
console.log("Message sent!");
|
||||||
|
```
|
||||||
|
|
||||||
#### Python
|
#### Python
|
||||||
|
|
||||||
```python
|
```python
|
||||||
@@ -137,6 +158,53 @@ env, env_json_str = await smartsend(
|
|||||||
print("Message sent!")
|
print("Message sent!")
|
||||||
```
|
```
|
||||||
|
|
||||||
|
#### MicroPython
|
||||||
|
|
||||||
|
```python
|
||||||
|
from natsbridge import smartsend
|
||||||
|
|
||||||
|
data = [("message", "Hello World", "text")]
|
||||||
|
env, env_json_str = smartsend(
|
||||||
|
"/chat/room1",
|
||||||
|
data,
|
||||||
|
broker_url="nats://localhost:4222",
|
||||||
|
size_threshold=100000 # 100KB for MicroPython
|
||||||
|
)
|
||||||
|
print("Message sent!")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Dart (Desktop/Flutter)
|
||||||
|
|
||||||
|
```dart
|
||||||
|
import 'package:natsbridge/natsbridge.dart';
|
||||||
|
|
||||||
|
final data = [
|
||||||
|
['message', 'Hello World', 'text']
|
||||||
|
];
|
||||||
|
final [env, envJsonStr] = await NATSBridge.send(
|
||||||
|
'/chat/room1',
|
||||||
|
data,
|
||||||
|
brokerUrl: 'nats://localhost:4222',
|
||||||
|
);
|
||||||
|
print('Message sent!');
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Dart Web
|
||||||
|
|
||||||
|
```dart
|
||||||
|
import 'package:natsbridge/natsbridge.dart';
|
||||||
|
|
||||||
|
final data = [
|
||||||
|
['message', 'Hello World', 'text']
|
||||||
|
];
|
||||||
|
final [env, envJsonStr] = await NATSBridge.send(
|
||||||
|
'/chat/room1',
|
||||||
|
data,
|
||||||
|
brokerUrl: 'ws://localhost:4222', // WebSocket for browser
|
||||||
|
);
|
||||||
|
print('Message sent!');
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## API Reference
|
## API Reference
|
||||||
@@ -145,13 +213,13 @@ print("Message sent!")
|
|||||||
|
|
||||||
All platforms use the same input/output format for payloads:
|
All platforms use the same input/output format for payloads:
|
||||||
|
|
||||||
**Input format for smartsend:**
|
**Input format for `smartsend`:**
|
||||||
```
|
```
|
||||||
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
[(dataname1, data1, type1), (dataname2, data2, type2), ...]
|
||||||
```
|
```
|
||||||
|
|
||||||
**Output format for smartreceive:**
|
**Output format for `smartreceive`:**
|
||||||
```
|
```json
|
||||||
{
|
{
|
||||||
"correlation_id": "...",
|
"correlation_id": "...",
|
||||||
"msg_id": "...",
|
"msg_id": "...",
|
||||||
@@ -185,7 +253,7 @@ env, env_json_str = NATSBridge.smartsend(
|
|||||||
broker_url::String = "nats://localhost:4222",
|
broker_url::String = "nats://localhost:4222",
|
||||||
fileserver_url = "http://localhost:8080",
|
fileserver_url = "http://localhost:8080",
|
||||||
fileserver_upload_handler::Function = plik_oneshot_upload,
|
fileserver_upload_handler::Function = plik_oneshot_upload,
|
||||||
size_threshold::Int = 1_000_000,
|
size_threshold::Int = 500_000,
|
||||||
correlation_id::String = string(uuid4()),
|
correlation_id::String = string(uuid4()),
|
||||||
msg_purpose::String = "chat",
|
msg_purpose::String = "chat",
|
||||||
sender_name::String = "NATSBridge",
|
sender_name::String = "NATSBridge",
|
||||||
@@ -201,10 +269,10 @@ env, env_json_str = NATSBridge.smartsend(
|
|||||||
# Returns: ::Tuple{msg_envelope_v1, String}
|
# Returns: ::Tuple{msg_envelope_v1, String}
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript
|
#### JavaScript (Node.js)
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
const NATSBridge = require('natsbridge');
|
import NATSBridge from './src/natsbridge_ssr.js';
|
||||||
|
|
||||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||||
subject,
|
subject,
|
||||||
@@ -213,7 +281,36 @@ const [env, env_json_str] = await NATSBridge.smartsend(
|
|||||||
broker_url: 'nats://localhost:4222',
|
broker_url: 'nats://localhost:4222',
|
||||||
fileserver_url: 'http://localhost:8080',
|
fileserver_url: 'http://localhost:8080',
|
||||||
fileserver_upload_handler: NATSBridge.plikOneshotUpload,
|
fileserver_upload_handler: NATSBridge.plikOneshotUpload,
|
||||||
size_threshold: 1_000_000,
|
size_threshold: 500_000,
|
||||||
|
correlation_id: uuidv4(),
|
||||||
|
msg_purpose: 'chat',
|
||||||
|
sender_name: 'NATSBridge',
|
||||||
|
receiver_name: '',
|
||||||
|
receiver_id: '',
|
||||||
|
reply_to: '',
|
||||||
|
reply_to_msg_id: '',
|
||||||
|
is_publish: true,
|
||||||
|
nats_connection: null,
|
||||||
|
msg_id: uuidv4(),
|
||||||
|
sender_id: uuidv4()
|
||||||
|
}
|
||||||
|
);
|
||||||
|
// Returns: Promise<[env, env_json_str]>
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript (Browser)
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
import NATSBridge from './src/natsbridge_csr.js';
|
||||||
|
|
||||||
|
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||||
|
subject,
|
||||||
|
data,
|
||||||
|
{
|
||||||
|
broker_url: 'ws://localhost:4222',
|
||||||
|
fileserver_url: 'http://localhost:8080',
|
||||||
|
fileserver_upload_handler: NATSBridge.plikOneshotUpload,
|
||||||
|
size_threshold: 500_000,
|
||||||
correlation_id: uuidv4(),
|
correlation_id: uuidv4(),
|
||||||
msg_purpose: 'chat',
|
msg_purpose: 'chat',
|
||||||
sender_name: 'NATSBridge',
|
sender_name: 'NATSBridge',
|
||||||
@@ -241,7 +338,7 @@ env, env_json_str = await NATSBridge.smartsend(
|
|||||||
broker_url: str = "nats://localhost:4222",
|
broker_url: str = "nats://localhost:4222",
|
||||||
fileserver_url: str = "http://localhost:8080",
|
fileserver_url: str = "http://localhost:8080",
|
||||||
fileserver_upload_handler: Callable = plik_oneshot_upload,
|
fileserver_upload_handler: Callable = plik_oneshot_upload,
|
||||||
size_threshold: int = 1_000_000,
|
size_threshold: int = 500_000,
|
||||||
correlation_id: str = None,
|
correlation_id: str = None,
|
||||||
msg_purpose: str = "chat",
|
msg_purpose: str = "chat",
|
||||||
sender_name: str = "NATSBridge",
|
sender_name: str = "NATSBridge",
|
||||||
@@ -272,6 +369,54 @@ env, env_json_str = NATSBridge.smartsend(
|
|||||||
# Returns: Tuple[Dict, str]
|
# Returns: Tuple[Dict, str]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
#### Dart (Desktop/Flutter)
|
||||||
|
|
||||||
|
```dart
|
||||||
|
import 'package:natsbridge/natsbridge.dart';
|
||||||
|
|
||||||
|
final env, envJsonStr = await NATSBridge.send(
|
||||||
|
subject,
|
||||||
|
data, // List of [dataname, data, type] lists
|
||||||
|
brokerUrl: 'nats://localhost:4222',
|
||||||
|
fileserverUrl: 'http://localhost:8080',
|
||||||
|
fileserverUploadHandler: plikOneshotUpload,
|
||||||
|
sizeThreshold: 500000,
|
||||||
|
correlationId: uuid.v4(),
|
||||||
|
msgPurpose: 'chat',
|
||||||
|
senderName: 'NATSBridge',
|
||||||
|
receiverName: '',
|
||||||
|
receiverId: '',
|
||||||
|
replyTo: '',
|
||||||
|
replyToMsgId: '',
|
||||||
|
isPublish: true,
|
||||||
|
);
|
||||||
|
// Returns: Future<List<dynamic>> [env, env_json_str]
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Dart Web
|
||||||
|
|
||||||
|
```dart
|
||||||
|
import 'package:natsbridge/natsbridge.dart';
|
||||||
|
|
||||||
|
final env, envJsonStr = await NATSBridge.send(
|
||||||
|
subject,
|
||||||
|
data, // List of [dataname, data, type] lists
|
||||||
|
brokerUrl: 'ws://localhost:4222', // WebSocket for browser
|
||||||
|
fileserverUrl: 'http://localhost:8080',
|
||||||
|
fileserverUploadHandler: plikOneshotUpload,
|
||||||
|
sizeThreshold: 500000,
|
||||||
|
correlationId: uuid.v4(),
|
||||||
|
msgPurpose: 'chat',
|
||||||
|
senderName: 'NATSBridge',
|
||||||
|
receiverName: '',
|
||||||
|
receiverId: '',
|
||||||
|
replyTo: '',
|
||||||
|
replyToMsgId: '',
|
||||||
|
isPublish: true,
|
||||||
|
);
|
||||||
|
// Returns: Future<List<dynamic>> [env, env_json_str]
|
||||||
|
```
|
||||||
|
|
||||||
### smartreceive
|
### smartreceive
|
||||||
|
|
||||||
Receives and processes messages from NATS, handling both direct and link transport.
|
Receives and processes messages from NATS, handling both direct and link transport.
|
||||||
@@ -291,9 +436,28 @@ env = NATSBridge.smartreceive(
|
|||||||
# Returns: ::JSON.Object{String, Any}
|
# Returns: ::JSON.Object{String, Any}
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript
|
#### JavaScript (Node.js)
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
|
import NATSBridge from './src/natsbridge_ssr.js';
|
||||||
|
|
||||||
|
const env = await NATSBridge.smartreceive(
|
||||||
|
msg,
|
||||||
|
{
|
||||||
|
fileserver_download_handler: NATSBridge.fetchWithBackoff,
|
||||||
|
max_retries: 5,
|
||||||
|
base_delay: 100,
|
||||||
|
max_delay: 5000
|
||||||
|
}
|
||||||
|
);
|
||||||
|
// Returns: Promise<env_object>
|
||||||
|
```
|
||||||
|
|
||||||
|
#### JavaScript (Browser)
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
import NATSBridge from './src/natsbridge_csr.js';
|
||||||
|
|
||||||
const env = await NATSBridge.smartreceive(
|
const env = await NATSBridge.smartreceive(
|
||||||
msg,
|
msg,
|
||||||
{
|
{
|
||||||
@@ -309,6 +473,8 @@ const env = await NATSBridge.smartreceive(
|
|||||||
#### Python
|
#### Python
|
||||||
|
|
||||||
```python
|
```python
|
||||||
|
from natsbridge import NATSBridge
|
||||||
|
|
||||||
env = await NATSBridge.smartreceive(
|
env = await NATSBridge.smartreceive(
|
||||||
msg,
|
msg,
|
||||||
fileserver_download_handler=fetch_with_backoff,
|
fileserver_download_handler=fetch_with_backoff,
|
||||||
@@ -322,6 +488,8 @@ env = await NATSBridge.smartreceive(
|
|||||||
#### MicroPython
|
#### MicroPython
|
||||||
|
|
||||||
```python
|
```python
|
||||||
|
from natsbridge import NATSBridge
|
||||||
|
|
||||||
env = NATSBridge.smartreceive(
|
env = NATSBridge.smartreceive(
|
||||||
msg,
|
msg,
|
||||||
fileserver_download_handler=_sync_fileserver_download,
|
fileserver_download_handler=_sync_fileserver_download,
|
||||||
@@ -332,20 +500,50 @@ env = NATSBridge.smartreceive(
|
|||||||
# Returns: Dict with "payloads" key
|
# Returns: Dict with "payloads" key
|
||||||
```
|
```
|
||||||
|
|
||||||
|
#### Dart (Desktop/Flutter)
|
||||||
|
|
||||||
|
```dart
|
||||||
|
import 'package:natsbridge/natsbridge.dart';
|
||||||
|
|
||||||
|
final env = await NATSBridge.receive(
|
||||||
|
msg,
|
||||||
|
fileserverDownloadHandler: fetchWithBackoff,
|
||||||
|
maxRetries: 5,
|
||||||
|
baseDelay: 100,
|
||||||
|
maxDelay: 5000,
|
||||||
|
);
|
||||||
|
// Returns: Future<Map<String, dynamic>> with "payloads" key
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Dart Web
|
||||||
|
|
||||||
|
```dart
|
||||||
|
import 'package:natsbridge/natsbridge.dart';
|
||||||
|
|
||||||
|
final env = await NATSBridge.receive(
|
||||||
|
msg,
|
||||||
|
fileserverDownloadHandler: fetchWithBackoff,
|
||||||
|
maxRetries: 5,
|
||||||
|
baseDelay: 100,
|
||||||
|
maxDelay: 5000,
|
||||||
|
);
|
||||||
|
// Returns: Future<Map<String, dynamic>> with "payloads" key
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Payload Types
|
## Payload Types
|
||||||
|
|
||||||
| Type | Julia | JavaScript | Python | MicroPython | Description |
|
| Type | Julia | JavaScript | Python | Dart | Dart Web | MicroPython | Description |
|
||||||
|------|-------|------------|--------|-------------|-------------|
|
|------|-------|------------|--------|------|----------|-------------|-------------|
|
||||||
| `text` | `String` | `string` | `str` | `str` | Plain text strings |
|
| `text` | `String` | `string` | `str` | `String` | `String` | `str` | Plain text strings |
|
||||||
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `dict` | JSON-serializable dictionaries |
|
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `Map` | `Map` | `dict` | JSON-serializable dictionaries |
|
||||||
| `arrowtable` | `DataFrame`, `Arrow.Table` | `Array<Object>` | `pandas.DataFrame` | ❌ | Tabular data (Arrow IPC) |
|
| `arrowtable` | `DataFrame`, `Arrow.Table` | ❌ (Browser), ✅ (Node.js) | `pandas.DataFrame` | `List<Map>` (Desktop/Flutter) | ❌ (Browser incompatible) | ❌ | Tabular data (Arrow IPC) |
|
||||||
| `jsontable` | `Vector{NamedTuple}` | `Array<Object>` | `list[dict]` | ❌ | Tabular data (JSON) |
|
| `jsontable` | `DataFrame`, `Vector{NamedTuple}` | `Array<Object>` | `list[dict]` | `List<Map>` | `List<Map>` | ⚠️ | Tabular data (JSON) - **Only table type in Browser/Dart Web** |
|
||||||
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Image data (PNG, JPG) |
|
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `Uint8List` | `bytearray` | Image data (PNG, JPG) |
|
||||||
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Audio data (WAV, MP3) |
|
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `Uint8List` | `bytearray` | Audio data (WAV, MP3) |
|
||||||
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `bytearray` | Video data (MP4, AVI) |
|
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `Uint8List` | `bytearray` | Video data (MP4, AVI) |
|
||||||
| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `bytearray` | Generic binary data |
|
| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `Uint8List` | `Uint8List` | `bytearray` | Generic binary data |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -366,13 +564,13 @@ data = [
|
|||||||
("large_document", large_file_data, "binary")
|
("large_document", large_file_data, "binary")
|
||||||
]
|
]
|
||||||
|
|
||||||
env, env_json_str = NATSBridge.smartsend("/chat/room1", data; fileserver_url="http://localhost:8080")
|
env, env_json_str = smartsend("/chat/room1", data; fileserver_url="http://localhost:8080")
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript
|
#### JavaScript (Node.js)
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
const NATSBridge = require('natsbridge');
|
import NATSBridge from './src/natsbridge_ssr.js';
|
||||||
|
|
||||||
const data = [
|
const data = [
|
||||||
["message_text", "Hello!", "text"],
|
["message_text", "Hello!", "text"],
|
||||||
@@ -387,6 +585,24 @@ const [env, env_json_str] = await NATSBridge.smartsend(
|
|||||||
);
|
);
|
||||||
```
|
```
|
||||||
|
|
||||||
|
#### JavaScript (Browser)
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
import NATSBridge from './src/natsbridge_csr.js';
|
||||||
|
|
||||||
|
const data = [
|
||||||
|
["message_text", "Hello!", "text"],
|
||||||
|
["user_avatar", imageData, "image"],
|
||||||
|
["large_document", largeFileData, "binary"]
|
||||||
|
];
|
||||||
|
|
||||||
|
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||||
|
"/chat/room1",
|
||||||
|
data,
|
||||||
|
{ broker_url: 'ws://localhost:4222', fileserver_url: 'http://localhost:8080' }
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
#### Python
|
#### Python
|
||||||
|
|
||||||
```python
|
```python
|
||||||
@@ -421,13 +637,13 @@ config = Dict(
|
|||||||
)
|
)
|
||||||
|
|
||||||
data = [("config", config, "dictionary")]
|
data = [("config", config, "dictionary")]
|
||||||
env, env_json_str = NATSBridge.smartsend("/device/config", data)
|
env, env_json_str = smartsend("/device/config", data)
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript
|
#### JavaScript (Node.js)
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
const NATSBridge = require('natsbridge');
|
import NATSBridge from './src/natsbridge_ssr.js';
|
||||||
|
|
||||||
const config = {
|
const config = {
|
||||||
wifi_ssid: "MyNetwork",
|
wifi_ssid: "MyNetwork",
|
||||||
@@ -473,13 +689,13 @@ df = DataFrame(
|
|||||||
)
|
)
|
||||||
|
|
||||||
data = [("students", df, "arrowtable")]
|
data = [("students", df, "arrowtable")]
|
||||||
env, env_json_str = NATSBridge.smartsend("/data/analysis", data)
|
env, env_json_str = smartsend("/data/analysis", data)
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript
|
#### JavaScript (Node.js)
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
const NATSBridge = require('natsbridge');
|
import NATSBridge from './src/natsbridge_ssr.js';
|
||||||
|
|
||||||
const df = [
|
const df = [
|
||||||
{ id: 1, name: "Alice", score: 95 },
|
{ id: 1, name: "Alice", score: 95 },
|
||||||
@@ -509,6 +725,26 @@ data = [("students", df, "arrowtable")]
|
|||||||
env, env_json_str = await NATSBridge.smartsend("/data/analysis", data)
|
env, env_json_str = await NATSBridge.smartsend("/data/analysis", data)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
#### JavaScript (Browser)
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
import NATSBridge from './src/natsbridge_csr.js';
|
||||||
|
|
||||||
|
// Browser uses jsontable (JSON array of objects) instead of arrowtable
|
||||||
|
// Apache Arrow is not compatible with browsers
|
||||||
|
const df = [
|
||||||
|
{ id: 1, name: "Alice", score: 95 },
|
||||||
|
{ id: 2, name: "Bob", score: 88 },
|
||||||
|
{ id: 3, name: "Charlie", score: 92 }
|
||||||
|
];
|
||||||
|
|
||||||
|
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||||
|
"/data/analysis",
|
||||||
|
[["students", df, "jsontable"]], // Use jsontable for browser
|
||||||
|
{ broker_url: 'ws://localhost:4222' }
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
### Example 4: Request-Response Pattern
|
### Example 4: Request-Response Pattern
|
||||||
|
|
||||||
Bi-directional communication with reply-to support.
|
Bi-directional communication with reply-to support.
|
||||||
@@ -519,18 +755,29 @@ Bi-directional communication with reply-to support.
|
|||||||
using NATSBridge
|
using NATSBridge
|
||||||
|
|
||||||
# Requester
|
# Requester
|
||||||
env, env_json_str = NATSBridge.smartsend(
|
env, env_json_str = smartsend(
|
||||||
"/device/command",
|
"/device/command",
|
||||||
[("command", Dict("action" => "read_sensor"), "dictionary")];
|
[("command", Dict("action" => "read_sensor"), "dictionary")];
|
||||||
broker_url="nats://localhost:4222",
|
broker_url="nats://localhost:4222",
|
||||||
reply_to="/device/response"
|
reply_to="/device/response"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Receiver (in separate application)
|
||||||
|
msg = NATS.subscription.next()
|
||||||
|
env = smartreceive(msg)
|
||||||
|
# Process request and send response
|
||||||
|
response_env, response_json = smartsend(
|
||||||
|
"/device/response",
|
||||||
|
[("result", Dict("value" => 42), "dictionary")],
|
||||||
|
reply_to="/device/command",
|
||||||
|
reply_to_msg_id=env["msg_id"]
|
||||||
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
#### JavaScript
|
#### JavaScript (Node.js)
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
const NATSBridge = require('natsbridge');
|
import NATSBridge from './src/natsbridge_ssr.js';
|
||||||
|
|
||||||
// Requester
|
// Requester
|
||||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
const [env, env_json_str] = await NATSBridge.smartsend(
|
||||||
@@ -538,6 +785,16 @@ const [env, env_json_str] = await NATSBridge.smartsend(
|
|||||||
[["command", { action: "read_sensor" }, "dictionary"]],
|
[["command", { action: "read_sensor" }, "dictionary"]],
|
||||||
{ broker_url: 'nats://localhost:4222', reply_to: '/device/response' }
|
{ broker_url: 'nats://localhost:4222', reply_to: '/device/response' }
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// Receiver (in separate application)
|
||||||
|
// const msg = await natsConsumer.next();
|
||||||
|
// const env = await NATSBridge.smartreceive(msg);
|
||||||
|
// Process request and send response
|
||||||
|
// const response_env, response_json = await NATSBridge.smartsend(
|
||||||
|
// "/device/response",
|
||||||
|
// [["result", { value: 42 }, "dictionary"]],
|
||||||
|
// { reply_to: '/device/command', reply_to_msg_id: env.msg_id }
|
||||||
|
// );
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Python
|
#### Python
|
||||||
@@ -552,6 +809,17 @@ env, env_json_str = await NATSBridge.smartsend(
|
|||||||
broker_url="nats://localhost:4222",
|
broker_url="nats://localhost:4222",
|
||||||
reply_to="/device/response"
|
reply_to="/device/response"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Receiver (in separate application)
|
||||||
|
# msg = await nats_consumer.next()
|
||||||
|
# env = await NATSBridge.smartreceive(msg)
|
||||||
|
# Process request and send response
|
||||||
|
# response_env, response_json = await NATSBridge.smartsend(
|
||||||
|
# "/device/response",
|
||||||
|
# [("result", {"value": 42}, "dictionary")],
|
||||||
|
# reply_to="/device/command",
|
||||||
|
# reply_to_msg_id=env["msg_id"]
|
||||||
|
# )
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -565,6 +833,7 @@ env, env_json_str = await NATSBridge.smartsend(
|
|||||||
| **Julia** | `test/test_julia_*_sender.jl` | `test/test_julia_*_receiver.jl` |
|
| **Julia** | `test/test_julia_*_sender.jl` | `test/test_julia_*_receiver.jl` |
|
||||||
| **JavaScript** | `test/test_js_*_sender.js` | `test/test_js_*_receiver.js` |
|
| **JavaScript** | `test/test_js_*_sender.js` | `test/test_js_*_receiver.js` |
|
||||||
| **Python** | `test/test_py_*_sender.py` | `test/test_py_*_receiver.py` |
|
| **Python** | `test/test_py_*_sender.py` | `test/test_py_*_receiver.py` |
|
||||||
|
| **Dart** | `test/test_dart_*_sender.dart` | `test/test_dart_*_receiver.dart` |
|
||||||
|
|
||||||
### Run Tests
|
### Run Tests
|
||||||
|
|
||||||
@@ -632,16 +901,157 @@ python3 test/test_py_table_sender.py
|
|||||||
python3 test/test_py_table_receiver.py
|
python3 test/test_py_table_receiver.py
|
||||||
```
|
```
|
||||||
|
|
||||||
|
#### Dart
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Text message exchange
|
||||||
|
dart test/test_dart_text_sender.dart
|
||||||
|
dart test/test_dart_text_receiver.dart
|
||||||
|
|
||||||
|
# Dictionary exchange
|
||||||
|
dart test/test_dart_dictionary_sender.dart
|
||||||
|
dart test/test_dart_dictionary_receiver.dart
|
||||||
|
|
||||||
|
# Binary transfer
|
||||||
|
dart test/test_dart_binary_sender.dart
|
||||||
|
dart test/test_dart_binary_receiver.dart
|
||||||
|
|
||||||
|
# Mixed payload types
|
||||||
|
dart test/test_dart_mix_payloads_sender.dart
|
||||||
|
dart test/test_dart_mix_payloads_receiver.dart
|
||||||
|
|
||||||
|
# Table exchange
|
||||||
|
dart test/test_dart_table_sender.dart
|
||||||
|
dart test/test_dart_table_receiver.dart
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Browser Deployment
|
||||||
|
|
||||||
|
### Using with Node.js Build Tools
|
||||||
|
|
||||||
|
The browser implementation (`src/natsbridge_csr.js`) can be bundled for production deployment using modern JavaScript build tools.
|
||||||
|
|
||||||
|
#### Prerequisites
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Install the browser-compatible NATS client
|
||||||
|
npm install nats.ws
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Vite (Recommended)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm create vite@latest my-app -- --template vanilla
|
||||||
|
cd my-app
|
||||||
|
npm install nats.ws
|
||||||
|
```
|
||||||
|
|
||||||
|
In `vite.config.js`:
|
||||||
|
```javascript
|
||||||
|
import { defineConfig } from 'vite';
|
||||||
|
export default defineConfig({
|
||||||
|
resolve: {
|
||||||
|
alias: {
|
||||||
|
'nats.ws': 'nats.ws/dist/esm/browser.js'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
Build command:
|
||||||
|
```bash
|
||||||
|
npm run build # Outputs to dist/ folder
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Webpack
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install webpack webpack-cli --save-dev
|
||||||
|
npm install nats.ws
|
||||||
|
```
|
||||||
|
|
||||||
|
In `webpack.config.js`:
|
||||||
|
```javascript
|
||||||
|
module.exports = {
|
||||||
|
entry: './src/index.js',
|
||||||
|
output: {
|
||||||
|
filename: 'bundle.js',
|
||||||
|
path: __dirname + '/dist'
|
||||||
|
},
|
||||||
|
resolve: {
|
||||||
|
alias: {
|
||||||
|
'nats.ws': 'nats.ws/dist/esm/browser.js'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Build command:
|
||||||
|
```bash
|
||||||
|
npx webpack
|
||||||
|
```
|
||||||
|
|
||||||
|
#### esbuild (Simple & Fast)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install esbuild nats.ws --save-dev
|
||||||
|
```
|
||||||
|
|
||||||
|
Create `build.js`:
|
||||||
|
```javascript
|
||||||
|
import esbuild from 'esbuild';
|
||||||
|
|
||||||
|
esbuild.buildSync({
|
||||||
|
entryPoints: ['src/natsbridge_csr.js'],
|
||||||
|
bundle: true,
|
||||||
|
outfile: 'dist/natsbridge-csr-bundle.js',
|
||||||
|
format: 'esm',
|
||||||
|
platform: 'browser',
|
||||||
|
target: 'es2020'
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
Build command:
|
||||||
|
```bash
|
||||||
|
node build.js
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using in Your HTML
|
||||||
|
|
||||||
|
```html
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html>
|
||||||
|
<head>
|
||||||
|
<title>My App</title>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<script type="module" src="dist/natsbridge-csr-bundle.js"></script>
|
||||||
|
<script type="module">
|
||||||
|
import NATSBridgeCSR from './dist/natsbridge-csr-bundle.js';
|
||||||
|
|
||||||
|
// Use the library
|
||||||
|
const [env, envJson] = await NATSBridgeCSR.smartsend(
|
||||||
|
"/chat/user/v1/message",
|
||||||
|
[["msg", "Hello", "text"]],
|
||||||
|
{ broker_url: "wss://nats.example.com" }
|
||||||
|
);
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
For detailed architecture and implementation information, see:
|
For detailed architecture and implementation information, see:
|
||||||
|
|
||||||
- [Architecture Documentation](docs/architecture_updated.md) - Cross-platform architecture, API parity, platform-specific patterns
|
- [`docs/architecture.md`](docs/architecture.md) - Cross-platform architecture, API parity, platform-specific patterns
|
||||||
- [Implementation Guide](docs/implementation_updated.md) - Detailed implementation for each platform, handler functions, testing
|
- [`docs/requirements.md`](docs/requirements.md) - Business requirements and user stories
|
||||||
- [Tutorial](docs/tutorial_updated.md) - Step-by-step getting started guide
|
- [`docs/spec.md`](docs/spec.md) - Technical specification and contracts
|
||||||
- [Walkthrough](docs/walkthrough_updated.md) - Real-world application building guides
|
- [`docs/walkthrough.md`](docs/walkthrough.md) - Real-world application building guides
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
1188
docs/architecture.md
1188
docs/architecture.md
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
419
docs/requirements.md
Normal file
419
docs/requirements.md
Normal file
@@ -0,0 +1,419 @@
|
|||||||
|
# Requirements Document: NATSBridge
|
||||||
|
|
||||||
|
**Version**: 1.0.0
|
||||||
|
**Date**: 2026-03-23
|
||||||
|
**Status**: Active
|
||||||
|
**Ground Truth**: [`src/NATSBridge.jl`](../src/NATSBridge.jl)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Business Context & Success Metrics
|
||||||
|
|
||||||
|
### 1.1 Business Goal
|
||||||
|
|
||||||
|
NATSBridge is a cross-platform, bi-directional data bridge that enables seamless communication between **Julia**, **JavaScript**, **Python**, **Dart**, and **MicroPython** applications using NATS as the message bus. The system implements the **Claim-Check pattern** for efficient handling of large payloads (>0.5MB) by uploading them to an HTTP file server instead of sending raw binary data over NATS.
|
||||||
|
|
||||||
|
### 1.2 User Stories (with acceptance criteria)
|
||||||
|
|
||||||
|
| Story | Priority | Acceptance Criteria |
|
||||||
|
|-------|----------|---------------------|
|
||||||
|
| **As a Julia developer**, I want to send text messages to JavaScript/Dart applications that lives on a server and also on a browser | P1 | Text messages are serialized, encoded, and received correctly across platforms |
|
||||||
|
| **As a Python developer**, I want to send tabular data to Julia/Dart applications | P1 | DataFrame exchange works with both Arrow IPC and JSON formats |
|
||||||
|
| **As a JavaScript developer**, I want to send large files (>0.5MB) from JavaScript applications that lives on a server and also on a browser to other applications | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS |
|
||||||
|
| **As a Dart developer**, I want to send text messages to other platforms | P1 | Text messages are serialized, encoded, and received correctly across platforms |
|
||||||
|
| **As a Dart developer**, I want to send dictionary data to other platforms | P1 | JSON-serializable data is exchanged correctly |
|
||||||
|
| **As a Dart developer**, I want to send tabular data (List<Map>) to other platforms | P1 | JSON table format exchange works with Arrow IPC on desktop |
|
||||||
|
| **As a Dart developer**, I want to send large files (>0.5MB) | P1 | Large files are automatically uploaded to file server and URLs are sent via NATS |
|
||||||
|
| **As a MicroPython developer**, I want to send sensor data with minimal memory usage | P1 | Direct transport works for payloads <100KB on memory-constrained devices |
|
||||||
|
| **As a developer**, I want to send mixed-content messages (text + image + file) | P1 | NATSBridge accepts list of (dataname, data, type) tuples and handles each payload appropriately |
|
||||||
|
| **As a developer**, I want to receive multi-payload messages | P1 | NATSBridge returns payloads as list of tuples with correct types preserved |
|
||||||
|
| **As a developer**, I want to use Plik as the file server | P2 | Plik one-shot upload mode is supported with upload ID and token handling |
|
||||||
|
| **As a developer**, I want to use custom HTTP file servers | P2 | Handler function abstraction allows plugging in AWS S3 or custom implementations |
|
||||||
|
| **As a developer**, I want automatic retry on file server download failures | P1 | Exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) |
|
||||||
|
| **As a developer**, I want message tracing across distributed systems | P1 | Correlation ID is propagated through all message processing steps |
|
||||||
|
|
||||||
|
### 1.3 KPIs & Targets
|
||||||
|
|
||||||
|
| Metric | Target | Measurement Method |
|
||||||
|
|--------|--------|-------------------|
|
||||||
|
| 95% of messages complete within 200ms | 95% | Synthetic monitoring |
|
||||||
|
| <2 days from onboarding to first PR | 2 days | PR timeline tracking |
|
||||||
|
| 100% of messages validate against spec | 100% | CI block rate |
|
||||||
|
| >80% unit test coverage | 80% | Test coverage tools |
|
||||||
|
| <1% of PRs bypass validation gates | 1% | CI gate analysis |
|
||||||
|
| MTTR <15 minutes for P1 incidents | 15 minutes | Incident tracking |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Technical Boundaries
|
||||||
|
|
||||||
|
### 2.1 In Scope
|
||||||
|
|
||||||
|
| Feature | Description |
|
||||||
|
|---------|-------------|
|
||||||
|
| Cross-platform interoperability | Seamless data exchange between Julia, JavaScript, Python, Dart, and MicroPython |
|
||||||
|
| Intelligent transport selection | Direct transport (<0.5MB) vs Link transport (≥0.5MB) based on payload size |
|
||||||
|
| Unified API | Consistent `smartsend()` and `smartreceive()` functions across all platforms |
|
||||||
|
| Multi-payload support | List of (dataname, data, type) tuples with appropriate handling |
|
||||||
|
| File server integration | Plik one-shot upload and custom HTTP server support |
|
||||||
|
| Reliability features | Exponential backoff retry and correlation ID propagation |
|
||||||
|
| Message serialization | Converts data types to binary format (Base64, JSON, Arrow IPC) |
|
||||||
|
| NATS communication | Publishing and subscription via NATS subjects |
|
||||||
|
|
||||||
|
### 2.2 Out of Scope
|
||||||
|
|
||||||
|
| Feature | Reason |
|
||||||
|
|---------|--------|
|
||||||
|
| NATS JetStream support | Core NATS sufficient for current use cases |
|
||||||
|
| Message compression | Compression adds complexity without clear benefit |
|
||||||
|
| Message encryption | Payload encryption is application-layer concern |
|
||||||
|
| Persistent message queues | NATS request-reply pattern sufficient |
|
||||||
|
| Advanced routing rules | Simple NATS subject matching sufficient |
|
||||||
|
|
||||||
|
### 2.3 Dependencies
|
||||||
|
|
||||||
|
| Platform | Package | Version |
|
||||||
|
|----------|---------|---------|
|
||||||
|
| Julia | NATS.jl | Latest stable |
|
||||||
|
| Julia | JSON.jl | Latest stable |
|
||||||
|
| Julia | Arrow.jl | Latest stable |
|
||||||
|
| Julia | HTTP.jl | Latest stable |
|
||||||
|
| Julia | UUIDs.jl | Latest stable |
|
||||||
|
| Node.js | nats | Latest stable |
|
||||||
|
| Node.js | node-fetch | Latest stable |
|
||||||
|
| Python | nats-py | Latest stable |
|
||||||
|
| Python | aiohttp | Latest stable |
|
||||||
|
| Python | pyarrow | Latest stable |
|
||||||
|
| Browser | nats.ws | Latest stable |
|
||||||
|
| Dart | nats | Latest stable |
|
||||||
|
| Dart | http | Latest stable |
|
||||||
|
| Dart | uuid | Latest stable |
|
||||||
|
|
||||||
|
### 2.4 Platform Compatibility
|
||||||
|
|
||||||
|
| Platform | Minimum Version | Notes |
|
||||||
|
|----------|-----------------|-------|
|
||||||
|
| Julia | 1.7+ | Arrow.jl required for arrowtable support |
|
||||||
|
| Node.js | 16+ | nats.js required, Arrow IPC supported |
|
||||||
|
| Python | 3.8+ | pyarrow required for arrowtable support |
|
||||||
|
| Browser | Latest | No Arrow IPC (uses jsontable only) |
|
||||||
|
| Dart | 2.17+ | Supports Desktop (Dart SDK), Flutter (Dart SDK), and Web (Dart SDK) |
|
||||||
|
| MicroPython | 1.19+ | Limited to direct transport |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Functional Requirements (FR)
|
||||||
|
|
||||||
|
| ID | Requirement | Description |
|
||||||
|
|----|-------------|-------------|
|
||||||
|
| **FR-001** | Cross-platform text messaging | System shall allow users to send text messages between Julia, JavaScript, Python, and MicroPython applications |
|
||||||
|
| **FR-002** | Cross-platform tabular data | System shall support DataFrame exchange between Julia and Python applications using Arrow IPC format |
|
||||||
|
| **FR-003** | Large file handling | System shall automatically detect payloads ≥0.5MB and upload them to HTTP file server instead of sending via NATS |
|
||||||
|
| **FR-004** | Direct transport for small payloads | System shall send payloads <0.5MB directly via NATS without file server upload |
|
||||||
|
| **FR-005** | MicroPython support | System shall support payloads <100KB on MicroPython devices using direct transport |
|
||||||
|
| **FR-006** | Multi-payload messages | System shall accept and process lists of (dataname, data, type) tuples |
|
||||||
|
| **FR-007** | Payload type preservation | System shall preserve payload types when returning multi-payload messages |
|
||||||
|
| **FR-008** | Plik file server integration | System shall support Plik one-shot upload mode with upload ID and token handling |
|
||||||
|
| **FR-009** | Custom file server support | System shall provide handler function abstraction for custom HTTP file server implementations |
|
||||||
|
| **FR-010** | Exponential backoff retry | System shall implement exponential backoff with configurable retries (default: 5, base_delay: 100ms, max_delay: 5000ms) for file server download failures |
|
||||||
|
| **FR-011** | Correlation ID propagation | System shall propagate correlation IDs through all message processing steps |
|
||||||
|
| **FR-012** | Message serialization | System shall serialize data types using Base64, JSON, or Arrow IPC encoding |
|
||||||
|
| **FR-013** | NATS publishing | System shall publish messages to NATS subjects |
|
||||||
|
| **FR-014** | NATS subscription | System shall receive and process NATS messages |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Non-Functional Requirements (NFRs)
|
||||||
|
|
||||||
|
### 4.1 Performance & Scalability
|
||||||
|
|
||||||
|
| ID | Requirement | Specification | Test Method |
|
||||||
|
|----|-------------|---------------|-------------|
|
||||||
|
| **NFR-101** | Message serialization overhead | <50ms for 10KB payload | Benchmark tests |
|
||||||
|
| **NFR-102** | Message deserialization overhead | <50ms for 10KB payload | Benchmark tests |
|
||||||
|
| **NFR-103** | NATS connection establishment | <100ms | Connection pool benchmarks |
|
||||||
|
| **NFR-104** | File upload latency | <1s for 0.5MB file | Integration tests |
|
||||||
|
| **NFR-105** | File download latency | <1s for 0.5MB file | Integration tests |
|
||||||
|
| **NFR-106** | Concurrent connections | Support 100+ simultaneous NATS connections | Scale testing |
|
||||||
|
| **NFR-107** | Message throughput | Handle 1000+ messages/second per instance | Load testing |
|
||||||
|
| **NFR-108** | File server scalability | Support horizontal scaling of file server backend | Architecture review |
|
||||||
|
|
||||||
|
### 4.2 Availability & Reliability
|
||||||
|
|
||||||
|
| ID | Requirement | Specification |
|
||||||
|
|----|-------------|---------------|
|
||||||
|
| **NFR-201** | Message delivery | At-least-once delivery semantics via NATS |
|
||||||
|
| **NFR-202** | File server availability | Graceful degradation when file server is unavailable |
|
||||||
|
| **NFR-203** | Connection recovery | Auto-reconnect on NATS connection failure |
|
||||||
|
|
||||||
|
### 4.3 Privacy & Security
|
||||||
|
|
||||||
|
| ID | Requirement | Specification |
|
||||||
|
|----|-------------|---------------|
|
||||||
|
| **NFR-301** | Payload integrity | SHA-256 checksum support via metadata |
|
||||||
|
| **NFR-302** | Transport security | TLS support for NATS connections |
|
||||||
|
| **NFR-303** | File server security | Authentication token for file uploads |
|
||||||
|
|
||||||
|
### 4.4 Observability & Telemetry
|
||||||
|
|
||||||
|
| ID | Requirement | Specification |
|
||||||
|
|----|-------------|---------------|
|
||||||
|
| **NFR-401** | Required logs | `correlation_id`, `msg_id`, `timestamp`, `sender_name`, `receiver_name`, `payload_type`, `transport` |
|
||||||
|
| **NFR-402** | Critical metrics | `messages_sent_total`, `messages_received_total`, `file_upload_duration_seconds`, `file_download_duration_seconds`, `retry_attempts_total` |
|
||||||
|
| **NFR-403** | Tracing | Correlation ID propagation for request tracing |
|
||||||
|
| **NFR-404** | Alerting | `download_retry_exceeded` triggers alert when max retries exceeded |
|
||||||
|
| **NFR-405** | Retention | Logs: 30 days, Metrics: 1 year |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Acceptance Conditions
|
||||||
|
|
||||||
|
| Condition | Description |
|
||||||
|
|-----------|-------------|
|
||||||
|
| **AC-001** | All functional requirements FR-001 through FR-014 are implemented and tested |
|
||||||
|
| **AC-002** | All non-functional requirements NFR-101 through NFR-405 meet specified targets |
|
||||||
|
| **AC-003** | Cross-platform text message test passes (Julia ↔ JavaScript ↔ Python) |
|
||||||
|
| **AC-004** | Cross-platform tabular data test passes with Arrow IPC round-trip (Desktop) |
|
||||||
|
| **AC-005** | Cross-platform tabular data test passes with JSON table round-trip (Browser) |
|
||||||
|
| **AC-006** | Large file transfer test passes with file server upload/download |
|
||||||
|
| **AC-007** | Multi-payload mixed content test passes with all payload types in one message |
|
||||||
|
| **AC-008** | CI validation gates block PRs on specification violations |
|
||||||
|
| **AC-009** | Unit test coverage exceeds 80% |
|
||||||
|
| **AC-010** | Documentation is complete and includes walkthroughs, architecture, and runbook |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Payload Type Requirements
|
||||||
|
|
||||||
|
### 6.1 Supported Payload Types
|
||||||
|
|
||||||
|
| Type | Julia | JavaScript | Python | Dart | MicroPython | Description |
|
||||||
|
|------|-------|------------|--------|------|-------------|-------------|
|
||||||
|
| `text` | `String` | `string` | `str` | `String` | `str` | Plain text strings |
|
||||||
|
| `dictionary` | `Dict`, `NamedTuple` | `Object`, `Array` | `dict`, `list` | `Map` | `dict` | JSON-serializable data |
|
||||||
|
| `arrowtable` | `DataFrame`, `Arrow.Table` | ❌ (Browser), ✅ (Node.js) | `pandas.DataFrame` | `List<Map>` (Desktop), `List<dynamic>` (Flutter) | ❌ | Tabular data (Arrow IPC) |
|
||||||
|
| `jsontable` | `Vector{NamedTuple}` | `Array<Object>` | `list[dict]` | `List<Map>` | ⚠️ | Tabular data (JSON) - **Only table type in Browser** |
|
||||||
|
| `image` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `bytearray` | Image binary data |
|
||||||
|
| `audio` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `bytearray` | Audio binary data |
|
||||||
|
| `video` | `Vector{UInt8}` | `Uint8Array`, `Buffer` | `bytes` | `Uint8List` | `bytearray` | Video binary data |
|
||||||
|
| `binary` | `Vector{UInt8}`, `IOBuffer` | `Uint8Array`, `Buffer` | `bytes`, `bytearray` | `Uint8List` | `bytearray` | Generic binary data |
|
||||||
|
|
||||||
|
### 6.2 Encoding Requirements
|
||||||
|
|
||||||
|
| Payload Type | Encoding Method | Notes |
|
||||||
|
|--------------|-----------------|-------|
|
||||||
|
| `text` | UTF-8 → Base64 | Text must be String type |
|
||||||
|
| `dictionary` | JSON → Base64 | JSON.jl for Julia |
|
||||||
|
| `arrowtable` | Arrow IPC → Base64 | Requires Arrow.jl/pyarrow (Desktop only) |
|
||||||
|
| `jsontable` | JSON → Base64 | Human-readable format - **Browser uses this only** |
|
||||||
|
| `image`/`audio`/`video`/`binary` | Direct → Base64 | Binary data preserved |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Size Threshold Requirements
|
||||||
|
|
||||||
|
### 7.1 Direct Transport Threshold
|
||||||
|
|
||||||
|
| Platform | Threshold | Notes |
|
||||||
|
|----------|-----------|-------|
|
||||||
|
| Desktop (Julia/JS/Python/Dart) | 0.5MB | Default size threshold |
|
||||||
|
| Dart Desktop | 0.5MB | Default size threshold |
|
||||||
|
| Dart Flutter | 0.5MB | Default size threshold |
|
||||||
|
| Dart Web | 0.5MB | Default size threshold |
|
||||||
|
| MicroPython | 100KB | Lower threshold for memory constraints |
|
||||||
|
|
||||||
|
### 7.2 Maximum Payload Size
|
||||||
|
|
||||||
|
| Platform | Maximum | Notes |
|
||||||
|
|----------|---------|-------|
|
||||||
|
| Desktop | Unlimited | Limited by NATS server configuration |
|
||||||
|
| Dart Desktop | Unlimited | Limited by NATS server configuration |
|
||||||
|
| Dart Flutter | Unlimited | Limited by NATS server configuration |
|
||||||
|
| Dart Web | Unlimited | Limited by NATS server configuration |
|
||||||
|
| MicroPython | 50KB | Hard limit due to 256KB-1MB memory |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Message Envelope Requirements
|
||||||
|
|
||||||
|
### 8.1 Required Fields
|
||||||
|
|
||||||
|
| Field | Type | Purpose |
|
||||||
|
|-------|------|---------|
|
||||||
|
| `correlation_id` | String (UUID) | Track message flow across systems |
|
||||||
|
| `msg_id` | String (UUID) | Unique message identifier |
|
||||||
|
| `timestamp` | String (ISO 8601) | Message publication timestamp |
|
||||||
|
| `send_to` | String | NATS subject to publish to |
|
||||||
|
| `msg_purpose` | String | ACK, NACK, updateStatus, shutdown, chat |
|
||||||
|
| `sender_name` | String | Sender application name |
|
||||||
|
| `sender_id` | String (UUID) | Sender unique identifier |
|
||||||
|
| `receiver_name` | String | Receiver application name (empty = broadcast) |
|
||||||
|
| `receiver_id` | String (UUID) | Receiver unique identifier (empty = broadcast) |
|
||||||
|
| `reply_to` | String | Topic for reply messages |
|
||||||
|
| `reply_to_msg_id` | String | Message ID being replied to |
|
||||||
|
| `broker_url` | String | NATS server URL |
|
||||||
|
| `metadata` | Dict | Message-level metadata |
|
||||||
|
| `payloads` | Array | List of payload objects |
|
||||||
|
|
||||||
|
### 8.2 Payload Fields
|
||||||
|
|
||||||
|
| Field | Type | Purpose |
|
||||||
|
|-------|------|---------|
|
||||||
|
| `id` | String (UUID) | Unique payload identifier |
|
||||||
|
| `dataname` | String | Name of the payload |
|
||||||
|
| `payload_type` | String | Type: text, dictionary, arrowtable, etc. |
|
||||||
|
| `transport` | String | direct or link |
|
||||||
|
| `encoding` | String | none, json, base64, arrow-ipc |
|
||||||
|
| `size` | Integer | Payload size in bytes |
|
||||||
|
| `data` | Any | Base64 string or URL |
|
||||||
|
| `metadata` | Dict | Payload-level metadata |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Error Handling Requirements
|
||||||
|
|
||||||
|
### 9.1 Error Codes
|
||||||
|
|
||||||
|
| Error | Condition | Response |
|
||||||
|
|-------|-----------|----------|
|
||||||
|
| `Unknown payload_type` | Unsupported type | Throw error |
|
||||||
|
| `Failed to upload` | File server error | Throw error |
|
||||||
|
| `Failed to fetch` | File server unavailable | Retry with exponential backoff |
|
||||||
|
| `Unknown transport` | Invalid transport type | Throw error |
|
||||||
|
| `NATS connection failed` | NATS unavailable | Throw error |
|
||||||
|
|
||||||
|
### 9.2 Exception Handling
|
||||||
|
|
||||||
|
| Scenario | Handler |
|
||||||
|
|----------|---------|
|
||||||
|
| File server unavailable | Retry up to 5 times with exponential backoff |
|
||||||
|
| NATS publish failure | Connection auto-reconnect |
|
||||||
|
| Deserialization error | Log correlation ID and throw error |
|
||||||
|
| Memory overflow (MicroPython) | Reject payloads >50KB |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Testing Requirements
|
||||||
|
|
||||||
|
### 10.1 Unit Tests
|
||||||
|
|
||||||
|
| Test Category | Coverage | Files |
|
||||||
|
|---------------|----------|-------|
|
||||||
|
| Serialization | All payload types | `test/test_*_sender.*` |
|
||||||
|
| Deserialization | All payload types | `test/test_*_receiver.*` |
|
||||||
|
| Transport selection | Direct vs link | `test/test_*_mix_payloads.*` |
|
||||||
|
| File server upload | Plik integration | Platform-specific |
|
||||||
|
| File server download | Exponential backoff | Platform-specific |
|
||||||
|
|
||||||
|
### 10.2 Integration Tests
|
||||||
|
|
||||||
|
| Test Scenario | Success Criteria |
|
||||||
|
|-------------|-----------------|
|
||||||
|
| Cross-platform text message | Julia ↔ JavaScript ↔ Python |
|
||||||
|
| Cross-platform tabular data (Desktop) | Arrow IPC round-trip |
|
||||||
|
| Cross-platform tabular data (Browser) | JSON table round-trip |
|
||||||
|
| Large file transfer | File server upload/download |
|
||||||
|
| Multi-payload mixed content | All payload types in one message |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. API Contract
|
||||||
|
|
||||||
|
### 11.1 smartsend Signature
|
||||||
|
|
||||||
|
```julia
|
||||||
|
function smartsend(
|
||||||
|
subject::String,
|
||||||
|
data::AbstractArray{Tuple{String, Any, String}};
|
||||||
|
broker_url::String = "nats://localhost:4222",
|
||||||
|
fileserver_url::String = "http://localhost:8080",
|
||||||
|
fileserver_upload_handler::Function = plik_oneshot_upload,
|
||||||
|
size_threshold::Int = 1_000_000,
|
||||||
|
correlation_id::String = string(uuid4()),
|
||||||
|
msg_purpose::String = "chat",
|
||||||
|
sender_name::String = "NATSBridge",
|
||||||
|
receiver_name::String = "",
|
||||||
|
receiver_id::String = "",
|
||||||
|
reply_to::String = "",
|
||||||
|
reply_to_msg_id::String = "",
|
||||||
|
is_publish::Bool = true,
|
||||||
|
NATS_connection::Union{NATS.Connection, Nothing} = nothing,
|
||||||
|
msg_id::String = string(uuid4()),
|
||||||
|
sender_id::String = string(uuid4())
|
||||||
|
)::Tuple{msg_envelope_v1, String}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 11.2 smartreceive Signature
|
||||||
|
|
||||||
|
```julia
|
||||||
|
function smartreceive(
|
||||||
|
msg::NATS.Msg;
|
||||||
|
fileserver_download_handler::Function = _fetch_with_backoff,
|
||||||
|
max_retries::Int = 5,
|
||||||
|
base_delay::Int = 100,
|
||||||
|
max_delay::Int = 5000
|
||||||
|
)::JSON.Object{String, Any}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Deployment Requirements
|
||||||
|
|
||||||
|
### 12.1 Minimum Infrastructure
|
||||||
|
|
||||||
|
| Component | Minimum | Notes |
|
||||||
|
|-----------|---------|-------|
|
||||||
|
| NATS Server | 1 instance | Single node for development |
|
||||||
|
| File Server | 1 instance | HTTP server for large payloads |
|
||||||
|
| Client Memory | 50MB | Desktop platforms (Julia/JS/Python/Dart) |
|
||||||
|
| Client Memory | 256KB | MicroPython devices |
|
||||||
|
|
||||||
|
### 12.2 Environment Variables
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
|----------|---------|-------------|
|
||||||
|
| `NATS_URL` | `nats://localhost:4222` | NATS server URL |
|
||||||
|
| `FILESERVER_URL` | `http://localhost:8080` | HTTP file server URL |
|
||||||
|
| `SIZE_THRESHOLD` | `1000000` | Size threshold in bytes |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. Versioning
|
||||||
|
|
||||||
|
### 13.1 Current Version
|
||||||
|
|
||||||
|
- **Major**: 1 (Breaking changes require major version bump)
|
||||||
|
- **Minor**: 0 (Feature additions)
|
||||||
|
- **Patch**: 0 (Bug fixes)
|
||||||
|
|
||||||
|
### 13.2 Version Compatibility
|
||||||
|
|
||||||
|
| Version | Supported Platforms |
|
||||||
|
|---------|---------------------|
|
||||||
|
| v1.0.x | Julia 1.7+, Node.js 16+, Python 3.8+, Dart 2.17+, Browser (latest), MicroPython 1.19+ |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 14. Change Log
|
||||||
|
|
||||||
|
| Date | Version | Changes |
|
||||||
|
|------|---------|---------|
|
||||||
|
| 2026-03-23 | 1.0.0 | Updated to ASG Framework requirements structure |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 15. References
|
||||||
|
|
||||||
|
- [`src/NATSBridge.jl`](../src/NATSBridge.jl) - Ground truth implementation (Julia)
|
||||||
|
- [`src/natsbridge_ssr.js`](../src/natsbridge_ssr.js) - Server-side JavaScript implementation
|
||||||
|
- [`src/natsbridge_csr.js`](../src/natsbridge_csr.js) - Client-side JavaScript implementation
|
||||||
|
- [`src/natsbridge.py`](../src/natsbridge.py) - Python implementation
|
||||||
|
- [`src/natsbridge.dart`](../src/natsbridge.dart) - Dart implementation
|
||||||
|
- [`src/natsbridge_mpy.py`](../src/natsbridge_mpy.py) - MicroPython implementation
|
||||||
|
- [`README.md`](../README.md) - Project overview
|
||||||
|
- [`docs/specification.md`](./specification.md) - Technical specification
|
||||||
|
- [`docs/ui-specification.md`](./ui-specification.md) - UI specification
|
||||||
|
- [`docs/walkthrough.md`](./walkthrough.md) - End-to-end walkthrough
|
||||||
|
- [`docs/architecture.md`](./architecture.md) - Architecture documentation
|
||||||
|
- [`docs/validation.md`](./validation.md) - Validation and CI/CD
|
||||||
|
- [`docs/runbook.md`](./runbook.md) - Operational runbook
|
||||||
1328
docs/specification.md
Normal file
1328
docs/specification.md
Normal file
File diff suppressed because it is too large
Load Diff
741
docs/tutorial.md
741
docs/tutorial.md
@@ -1,741 +0,0 @@
|
|||||||
# Cross-Platform NATSBridge Tutorial
|
|
||||||
|
|
||||||
A step-by-step guide to get started with NATSBridge across **Julia**, **JavaScript**, and **Python/MicroPython**.
|
|
||||||
|
|
||||||
## Table of Contents
|
|
||||||
|
|
||||||
1. [Overview](#overview)
|
|
||||||
2. [Prerequisites](#prerequisites)
|
|
||||||
3. [Installation](#installation)
|
|
||||||
4. [Quick Start](#quick-start)
|
|
||||||
5. [Basic Examples](#basic-examples)
|
|
||||||
6. [Advanced Usage](#advanced-usage)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
NATSBridge enables seamless communication across platforms through NATS, with automatic transport selection based on payload size:
|
|
||||||
|
|
||||||
- **Direct Transport**: Payloads < 1MB are sent directly via NATS (Base64 encoded)
|
|
||||||
- **Link Transport**: Payloads >= 1MB are uploaded to an HTTP file server and referenced via URL
|
|
||||||
|
|
||||||
### Cross-Platform API Parity
|
|
||||||
|
|
||||||
All three platforms use the same high-level API:
|
|
||||||
|
|
||||||
```
|
|
||||||
# Input format
|
|
||||||
smartsend(subject, [(dataname, data, type), ...], options)
|
|
||||||
|
|
||||||
# Output format
|
|
||||||
(env, env_json_str) = smartsend(...)
|
|
||||||
env = smartreceive(msg, options)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Important Platform Differences:**
|
|
||||||
|
|
||||||
1. **Encoding field:** Julia and JavaScript preserve the original serialization format in the encoding field (`"base64"`, `"json"`, or `"arrow-ipc"`), while Python and MicroPython always use `"base64"` for all direct transport payloads.
|
|
||||||
|
|
||||||
2. **Async vs Sync:** JavaScript and Python desktop use async/await, while MicroPython uses synchronous API.
|
|
||||||
|
|
||||||
### Supported Payload Types
|
|
||||||
|
|
||||||
| Type | Julia | JavaScript | Python | MicroPython |
|
|
||||||
|------|-------|------------|--------|-------------|
|
|
||||||
| `text` | `String` | `string` | `str` | `str` |
|
|
||||||
| `dictionary` | `Dict` | `Object` | `dict` | `dict` |
|
|
||||||
| `arrowtable` | `DataFrame` | `Array<Object>` | `pandas.DataFrame` | ❌ |
|
|
||||||
| `jsontable` | `Vector{NamedTuple}` | `Array<Object>` | `list[dict]` | ❌ |
|
|
||||||
| `table` | ❌ | ❌ | `pandas.DataFrame` | ❌ |
|
|
||||||
| `image` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
|
|
||||||
| `audio` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
|
|
||||||
| `video` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
|
|
||||||
| `binary` | `Vector{UInt8}` | `Uint8Array` | `bytes` | `bytearray` |
|
|
||||||
|
|
||||||
**Note on MicroPython:** MicroPython does not support table types (`arrowtable`, `jsontable`, or `table`) due to memory constraints. Use `dictionary` or `binary` instead.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prerequisites
|
|
||||||
|
|
||||||
Before you begin, ensure you have:
|
|
||||||
|
|
||||||
1. **NATS Server** running (or accessible)
|
|
||||||
2. **HTTP File Server** (optional, for large payloads > 1MB)
|
|
||||||
3. **Platform-specific packages** installed
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
### Julia
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using Pkg
|
|
||||||
Pkg.add("NATS")
|
|
||||||
Pkg.add("Arrow")
|
|
||||||
Pkg.add("JSON3")
|
|
||||||
Pkg.add("HTTP")
|
|
||||||
Pkg.add("UUIDs")
|
|
||||||
Pkg.add("Dates")
|
|
||||||
```
|
|
||||||
|
|
||||||
### JavaScript (Node.js)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
npm install nats uuid apache-arrow node-fetch
|
|
||||||
```
|
|
||||||
|
|
||||||
### JavaScript (Browser)
|
|
||||||
|
|
||||||
```html
|
|
||||||
<script src="https://unpkg.com/nats-js/dist/bundle/nats.min.js"></script>
|
|
||||||
<script src="https://unpkg.com/apache-arrow/arrow.min.js"></script>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Python (Desktop)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install nats-py aiohttp pyarrow pandas
|
|
||||||
```
|
|
||||||
|
|
||||||
### MicroPython
|
|
||||||
|
|
||||||
Uses built-in modules: `network`, `socket`, `time`, `json`, `base64`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Quick Start
|
|
||||||
|
|
||||||
### Step 1: Start NATS Server
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker run -p 4222:4222 nats:latest
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 2: Start HTTP File Server (Optional)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p /tmp/fileserver
|
|
||||||
python3 -m http.server 8080 --directory /tmp/fileserver
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 3: Send Your First Message
|
|
||||||
|
|
||||||
#### Julia
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
|
|
||||||
# Send a text message
|
|
||||||
data = [("message", "Hello World", "text")]
|
|
||||||
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222")
|
|
||||||
# env: msg_envelope_v1 struct with all metadata and payloads
|
|
||||||
# env_json_str: JSON string representation of the envelope for publishing
|
|
||||||
println("Message sent!")
|
|
||||||
|
|
||||||
# Or use is_publish=false to get envelope and JSON without publishing
|
|
||||||
env, env_json_str = smartsend("/chat/room1", data, broker_url="nats://localhost:4222", is_publish=false)
|
|
||||||
# env: msg_envelope_v1 struct
|
|
||||||
# env_json_str: JSON string for publishing to NATS
|
|
||||||
```
|
|
||||||
|
|
||||||
#### JavaScript
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const NATSBridge = require('./src/natsbridge.js');
|
|
||||||
|
|
||||||
// Send a text message
|
|
||||||
const data = [["message", "Hello World", "text"]];
|
|
||||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
|
||||||
"/chat/room1",
|
|
||||||
data,
|
|
||||||
{ broker_url: "nats://localhost:4222" }
|
|
||||||
);
|
|
||||||
// env: Object with all metadata and payloads
|
|
||||||
// env_json_str: JSON string for publishing
|
|
||||||
console.log("Message sent!");
|
|
||||||
|
|
||||||
// Or use is_publish=false
|
|
||||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
|
||||||
"/chat/room1",
|
|
||||||
data,
|
|
||||||
{ broker_url: "nats://localhost:4222", is_publish: false }
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Python
|
|
||||||
|
|
||||||
```python
|
|
||||||
from natsbridge import smartsend
|
|
||||||
|
|
||||||
# Send a text message
|
|
||||||
data = [("message", "Hello World", "text")]
|
|
||||||
env, env_json_str = await smartsend(
|
|
||||||
"/chat/room1",
|
|
||||||
data,
|
|
||||||
broker_url="nats://localhost:4222"
|
|
||||||
)
|
|
||||||
# env: Dict with all metadata and payloads
|
|
||||||
# env_json_str: JSON string for publishing
|
|
||||||
print("Message sent!")
|
|
||||||
|
|
||||||
# Or use is_publish=False
|
|
||||||
env, env_json_str = await smartsend(
|
|
||||||
"/chat/room1",
|
|
||||||
data,
|
|
||||||
broker_url="nats://localhost:4222",
|
|
||||||
is_publish=False
|
|
||||||
)
|
|
||||||
# env: Dict with all metadata and payloads
|
|
||||||
# env_json_str: JSON string for publishing to NATS
|
|
||||||
```
|
|
||||||
|
|
||||||
#### MicroPython
|
|
||||||
|
|
||||||
```python
|
|
||||||
from natsbridge_mpy import NATSBridge
|
|
||||||
|
|
||||||
bridge = NATSBridge()
|
|
||||||
|
|
||||||
# Send a text message (limited to small payloads)
|
|
||||||
data = [("message", "Hello World", "text")]
|
|
||||||
env, env_json_str = bridge.smartsend(
|
|
||||||
"/chat/room1",
|
|
||||||
data,
|
|
||||||
size_threshold=100000 # Lower threshold for MicroPython
|
|
||||||
)
|
|
||||||
print("Message sent!")
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 4: Receive Messages
|
|
||||||
|
|
||||||
#### Julia
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
|
|
||||||
# Receive and process message
|
|
||||||
env = smartreceive(msg; fileserver_download_handler=_fetch_with_backoff)
|
|
||||||
# Returns: ::JSON.Object{String, Any} with "payloads" field containing Vector{Tuple{String, Any, String}}
|
|
||||||
# Access payloads: for (dataname, data, type) in env["payloads"]
|
|
||||||
for (dataname, data, type) in env["payloads"]
|
|
||||||
println("Received $dataname: $data")
|
|
||||||
end
|
|
||||||
```
|
|
||||||
|
|
||||||
#### JavaScript
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const NATSBridge = require('./src/natsbridge.js');
|
|
||||||
|
|
||||||
// Receive and process message
|
|
||||||
const env = await NATSBridge.smartreceive(msg, {
|
|
||||||
fileserver_download_handler: NATSBridge.fetchWithBackoff
|
|
||||||
});
|
|
||||||
// env.payloads = [[dataname, data, type], ...]
|
|
||||||
for (const [dataname, data, type] of env.payloads) {
|
|
||||||
console.log(`Received ${dataname}:`, data);
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Python
|
|
||||||
|
|
||||||
```python
|
|
||||||
from natsbridge import smartreceive, fetch_with_backoff
|
|
||||||
|
|
||||||
# Receive and process message
|
|
||||||
env = await smartreceive(
|
|
||||||
msg,
|
|
||||||
fileserver_download_handler=fetch_with_backoff
|
|
||||||
)
|
|
||||||
# env["payloads"] = [(dataname, data, type), ...]
|
|
||||||
for dataname, data, type_ in env["payloads"]:
|
|
||||||
print(f"Received {dataname}: {data}")
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Basic Examples
|
|
||||||
|
|
||||||
### Example 1: Sending a Dictionary
|
|
||||||
|
|
||||||
#### Julia
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
|
|
||||||
config = Dict(
|
|
||||||
"wifi_ssid" => "MyNetwork",
|
|
||||||
"wifi_password" => "password123",
|
|
||||||
"update_interval" => 60
|
|
||||||
)
|
|
||||||
|
|
||||||
data = [("config", config, "dictionary")]
|
|
||||||
env, env_json_str = smartsend("/device/config", data, broker_url="nats://localhost:4222")
|
|
||||||
```
|
|
||||||
|
|
||||||
#### JavaScript
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const NATSBridge = require('./src/natsbridge.js');
|
|
||||||
|
|
||||||
const config = {
|
|
||||||
wifi_ssid: "MyNetwork",
|
|
||||||
wifi_password: "password123",
|
|
||||||
update_interval: 60
|
|
||||||
};
|
|
||||||
|
|
||||||
const data = [["config", config, "dictionary"]];
|
|
||||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
|
||||||
"/device/config",
|
|
||||||
data,
|
|
||||||
{ broker_url: "nats://localhost:4222" }
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Python
|
|
||||||
|
|
||||||
```python
|
|
||||||
from natsbridge import smartsend
|
|
||||||
|
|
||||||
config = {
|
|
||||||
"wifi_ssid": "MyNetwork",
|
|
||||||
"wifi_password": "password123",
|
|
||||||
"update_interval": 60
|
|
||||||
}
|
|
||||||
|
|
||||||
data = [("config", config, "dictionary")]
|
|
||||||
env, env_json_str = await smartsend(
|
|
||||||
"/device/config",
|
|
||||||
data,
|
|
||||||
broker_url="nats://localhost:4222"
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
#### MicroPython
|
|
||||||
|
|
||||||
```python
|
|
||||||
from natsbridge_mpy import NATSBridge
|
|
||||||
|
|
||||||
bridge = NATSBridge()
|
|
||||||
|
|
||||||
config = {
|
|
||||||
"wifi_ssid": "MyNetwork",
|
|
||||||
"wifi_password": "password123",
|
|
||||||
"update_interval": 60
|
|
||||||
}
|
|
||||||
|
|
||||||
data = [("config", config, "dictionary")]
|
|
||||||
env, env_json_str = bridge.smartsend(
|
|
||||||
"/device/config",
|
|
||||||
data,
|
|
||||||
size_threshold=100000
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Example 2: Sending Binary Data (Image)
|
|
||||||
|
|
||||||
#### Julia
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
|
|
||||||
# Read image file
|
|
||||||
image_data = read("image.png")
|
|
||||||
|
|
||||||
data = [("user_image", image_data, "binary")]
|
|
||||||
env, env_json_str = smartsend("/chat/image", data, broker_url="nats://localhost:4222")
|
|
||||||
```
|
|
||||||
|
|
||||||
#### JavaScript
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const NATSBridge = require('./src/natsbridge.js');
|
|
||||||
const fs = require('fs');
|
|
||||||
|
|
||||||
// Read image file
|
|
||||||
const image_data = fs.readFileSync('image.png');
|
|
||||||
|
|
||||||
const data = [["user_image", image_data, "binary"]];
|
|
||||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
|
||||||
"/chat/image",
|
|
||||||
data,
|
|
||||||
{ broker_url: "nats://localhost:4222" }
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Python
|
|
||||||
|
|
||||||
```python
|
|
||||||
from natsbridge import smartsend
|
|
||||||
|
|
||||||
# Read image file
|
|
||||||
with open("image.png", "rb") as f:
|
|
||||||
image_data = f.read()
|
|
||||||
|
|
||||||
data = [("user_image", image_data, "binary")]
|
|
||||||
env, env_json_str = await smartsend(
|
|
||||||
"/chat/image",
|
|
||||||
data,
|
|
||||||
broker_url="nats://localhost:4222"
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
#### MicroPython
|
|
||||||
|
|
||||||
```python
|
|
||||||
from natsbridge_mpy import NATSBridge
|
|
||||||
|
|
||||||
bridge = NATSBridge()
|
|
||||||
|
|
||||||
# Read image file
|
|
||||||
with open("image.png", "rb") as f:
|
|
||||||
image_data = f.read()
|
|
||||||
|
|
||||||
data = [("user_image", image_data, "binary")]
|
|
||||||
env, env_json_str = bridge.smartsend(
|
|
||||||
"/chat/image",
|
|
||||||
data,
|
|
||||||
size_threshold=100000
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Example 3: Request-Response Pattern
|
|
||||||
|
|
||||||
#### Julia (Requester)
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
|
|
||||||
# Send command with reply-to
|
|
||||||
data = [("command", Dict("action" => "read_sensor"), "dictionary")]
|
|
||||||
env, env_json_str = smartsend(
|
|
||||||
"/device/command",
|
|
||||||
data,
|
|
||||||
broker_url="nats://localhost:4222",
|
|
||||||
reply_to="/device/response",
|
|
||||||
reply_to_msg_id="cmd-001"
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
#### JavaScript (Requester)
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const NATSBridge = require('./src/natsbridge.js');
|
|
||||||
|
|
||||||
// Send command with reply-to
|
|
||||||
const data = [["command", { action: "read_sensor" }, "dictionary"]];
|
|
||||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
|
||||||
"/device/command",
|
|
||||||
data,
|
|
||||||
{
|
|
||||||
broker_url: "nats://localhost:4222",
|
|
||||||
reply_to: "/device/response",
|
|
||||||
reply_to_msg_id: "cmd-001"
|
|
||||||
}
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Python (Requester)
|
|
||||||
|
|
||||||
```python
|
|
||||||
from natsbridge import smartsend
|
|
||||||
|
|
||||||
# Send command with reply-to
|
|
||||||
data = [("command", {"action": "read_sensor"}, "dictionary")]
|
|
||||||
env, env_json_str = await smartsend(
|
|
||||||
"/device/command",
|
|
||||||
data,
|
|
||||||
broker_url="nats://localhost:4222",
|
|
||||||
reply_to="/device/response",
|
|
||||||
reply_to_msg_id="cmd-001"
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Julia (Responder)
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATSBridge, NATS
|
|
||||||
|
|
||||||
const SUBJECT = "/device/command"
|
|
||||||
const NATS_URL = "nats://localhost:4222"
|
|
||||||
|
|
||||||
function test_responder()
|
|
||||||
conn = NATS.connect(NATS_URL)
|
|
||||||
NATS.subscribe(conn, SUBJECT) do msg
|
|
||||||
env = smartreceive(msg, fileserver_download_handler=_fetch_with_backoff)
|
|
||||||
|
|
||||||
reply_to = env["reply_to"]
|
|
||||||
|
|
||||||
for (dataname, data, type) in env["payloads"]
|
|
||||||
if dataname == "command" && data["action"] == "read_sensor"
|
|
||||||
response = Dict("sensor_id" => "sensor-001", "value" => 42.5)
|
|
||||||
if !isempty(reply_to)
|
|
||||||
smartsend(reply_to, [("data", response, "dictionary")])
|
|
||||||
end
|
|
||||||
end
|
|
||||||
end
|
|
||||||
end
|
|
||||||
|
|
||||||
sleep(120)
|
|
||||||
NATS.drain(conn)
|
|
||||||
end
|
|
||||||
|
|
||||||
test_responder()
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Advanced Usage
|
|
||||||
|
|
||||||
### Example 4: Large Payloads (File Server)
|
|
||||||
|
|
||||||
For payloads larger than 1MB, NATSBridge automatically uses the file server:
|
|
||||||
|
|
||||||
#### Julia
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
|
|
||||||
# Create large data (> 1MB)
|
|
||||||
large_data = rand(UInt8, 2_000_000)
|
|
||||||
|
|
||||||
env, env_json_str = smartsend(
|
|
||||||
"/data/large",
|
|
||||||
[("large_file", large_data, "binary")],
|
|
||||||
broker_url="nats://localhost:4222",
|
|
||||||
fileserver_url="http://localhost:8080"
|
|
||||||
)
|
|
||||||
|
|
||||||
println("File uploaded to: $(env.payloads[1].data)")
|
|
||||||
# Note: For link transport, data field contains the URL string
|
|
||||||
```
|
|
||||||
|
|
||||||
#### JavaScript
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const NATSBridge = require('./src/natsbridge.js');
|
|
||||||
|
|
||||||
// Create large data (> 1MB)
|
|
||||||
const large_data = Buffer.alloc(2_000_000);
|
|
||||||
for (let i = 0; i < large_data.length; i++) {
|
|
||||||
large_data[i] = Math.floor(Math.random() * 256);
|
|
||||||
}
|
|
||||||
|
|
||||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
|
||||||
"/data/large",
|
|
||||||
[["large_file", large_data, "binary"]],
|
|
||||||
{
|
|
||||||
broker_url: "nats://localhost:4222",
|
|
||||||
fileserver_url: "http://localhost:8080"
|
|
||||||
}
|
|
||||||
);
|
|
||||||
|
|
||||||
console.log("File uploaded to:", env.payloads[0].data);
|
|
||||||
// Note: For link transport, data field contains the URL string
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Python
|
|
||||||
|
|
||||||
```python
|
|
||||||
from natsbridge import smartsend
|
|
||||||
|
|
||||||
# Create large data (> 1MB)
|
|
||||||
import os
|
|
||||||
large_data = os.urandom(2_000_000)
|
|
||||||
|
|
||||||
env, env_json_str = await smartsend(
|
|
||||||
"/data/large",
|
|
||||||
[("large_file", large_data, "binary")],
|
|
||||||
broker_url="nats://localhost:4222",
|
|
||||||
fileserver_url="http://localhost:8080"
|
|
||||||
)
|
|
||||||
|
|
||||||
print(f"File uploaded to: {env['payloads'][0]['data']}")
|
|
||||||
# Note: For link transport, data field contains the URL string
|
|
||||||
```
|
|
||||||
|
|
||||||
#### MicroPython
|
|
||||||
|
|
||||||
MicroPython enforces a hard limit of 50KB per payload:
|
|
||||||
|
|
||||||
```python
|
|
||||||
from natsbridge_mpy import NATSBridge
|
|
||||||
|
|
||||||
bridge = NATSBridge()
|
|
||||||
|
|
||||||
# MicroPython has a hard limit of 50KB per payload
|
|
||||||
# Use streaming or chunking for larger data
|
|
||||||
small_data = bytes(1000) # 1KB
|
|
||||||
|
|
||||||
data = [("small_file", small_data, "binary")]
|
|
||||||
env, env_json_str = bridge.smartsend(
|
|
||||||
"/data/small",
|
|
||||||
data,
|
|
||||||
size_threshold=100000 # Enforced max: 50000 bytes
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Example 5: Mixed Content (Chat with Text + Image)
|
|
||||||
|
|
||||||
NATSBridge supports sending multiple payloads with different types in a single message:
|
|
||||||
|
|
||||||
#### Julia
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
|
|
||||||
image_data = read("avatar.png")
|
|
||||||
|
|
||||||
data = [
|
|
||||||
("message_text", "Hello with image!", "text"),
|
|
||||||
("user_avatar", image_data, "image")
|
|
||||||
]
|
|
||||||
|
|
||||||
env, env_json_str = smartsend("/chat/mixed", data, broker_url="nats://localhost:4222")
|
|
||||||
```
|
|
||||||
|
|
||||||
#### JavaScript
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const NATSBridge = require('./src/natsbridge.js');
|
|
||||||
const fs = require('fs');
|
|
||||||
|
|
||||||
const image_data = fs.readFileSync('avatar.png');
|
|
||||||
|
|
||||||
const data = [
|
|
||||||
["message_text", "Hello with image!", "text"],
|
|
||||||
["user_avatar", image_data, "image"]
|
|
||||||
];
|
|
||||||
|
|
||||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
|
||||||
"/chat/mixed",
|
|
||||||
data,
|
|
||||||
{ broker_url: "nats://localhost:4222" }
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Python
|
|
||||||
|
|
||||||
```python
|
|
||||||
from natsbridge import smartsend
|
|
||||||
|
|
||||||
with open("avatar.png", "rb") as f:
|
|
||||||
image_data = f.read()
|
|
||||||
|
|
||||||
data = [
|
|
||||||
("message_text", "Hello with image!", "text"),
|
|
||||||
("user_avatar", image_data, "image")
|
|
||||||
]
|
|
||||||
|
|
||||||
env, env_json_str = await smartsend(
|
|
||||||
"/chat/mixed",
|
|
||||||
data,
|
|
||||||
broker_url="nats://localhost:4222"
|
|
||||||
)
|
|
||||||
# env: Dict with all metadata and payloads
|
|
||||||
```
|
|
||||||
|
|
||||||
### Example 6: Table Data (Arrow IPC)
|
|
||||||
|
|
||||||
For tabular data, NATSBridge uses Apache Arrow IPC format:
|
|
||||||
|
|
||||||
#### Julia
|
|
||||||
|
|
||||||
```julia
|
|
||||||
using NATSBridge
|
|
||||||
using DataFrames
|
|
||||||
|
|
||||||
# Create DataFrame
|
|
||||||
df = DataFrame(
|
|
||||||
id = [1, 2, 3],
|
|
||||||
name = ["Alice", "Bob", "Charlie"],
|
|
||||||
score = [95, 88, 92]
|
|
||||||
)
|
|
||||||
|
|
||||||
data = [("students", df, "arrowtable")]
|
|
||||||
env, env_json_str = smartsend("/data/students", data, broker_url="nats://localhost:4222")
|
|
||||||
```
|
|
||||||
|
|
||||||
#### JavaScript
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
const NATSBridge = require('./src/natsbridge.js');
|
|
||||||
|
|
||||||
// Create table data (array of objects)
|
|
||||||
const table_data = [
|
|
||||||
{ id: 1, name: "Alice", score: 95 },
|
|
||||||
{ id: 2, name: "Bob", score: 88 },
|
|
||||||
{ id: 3, name: "Charlie", score: 92 }
|
|
||||||
];
|
|
||||||
|
|
||||||
const data = [["students", table_data, "arrowtable"]];
|
|
||||||
const [env, env_json_str] = await NATSBridge.smartsend(
|
|
||||||
"/data/students",
|
|
||||||
data,
|
|
||||||
{ broker_url: "nats://localhost:4222" }
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Python
|
|
||||||
|
|
||||||
```python
|
|
||||||
from natsbridge import smartsend
|
|
||||||
import pandas as pd
|
|
||||||
|
|
||||||
# Create DataFrame
|
|
||||||
df = pd.DataFrame({
|
|
||||||
'id': [1, 2, 3],
|
|
||||||
'name': ['Alice', 'Bob', 'Charlie'],
|
|
||||||
'score': [95, 88, 92]
|
|
||||||
})
|
|
||||||
|
|
||||||
data = [("students", df, "table")]
|
|
||||||
env, env_json_str = await smartsend(
|
|
||||||
"/data/students",
|
|
||||||
data,
|
|
||||||
broker_url="nats://localhost:4222"
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
#### MicroPython
|
|
||||||
|
|
||||||
MicroPython does not support table type due to memory constraints. Use dictionary or binary instead.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Next Steps
|
|
||||||
|
|
||||||
1. **Explore the test directory** for more examples
|
|
||||||
2. **Check the documentation** for advanced configuration options
|
|
||||||
3. **Read the walkthrough** for building real-world applications
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Connection Issues
|
|
||||||
|
|
||||||
- Ensure NATS server is running: `docker ps | grep nats`
|
|
||||||
- Check firewall settings
|
|
||||||
- Verify NATS URL configuration
|
|
||||||
|
|
||||||
### File Server Issues
|
|
||||||
|
|
||||||
- Ensure file server is running and accessible
|
|
||||||
- Check upload permissions
|
|
||||||
- Verify file server URL configuration
|
|
||||||
|
|
||||||
### Serialization Errors
|
|
||||||
|
|
||||||
- Verify data type matches the specified type
|
|
||||||
- Check that binary data is in the correct format
|
|
||||||
- MicroPython: Ensure payload size < 50KB
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## License
|
|
||||||
|
|
||||||
MIT
|
|
||||||
1947
docs/walkthrough.md
1947
docs/walkthrough.md
File diff suppressed because it is too large
Load Diff
56
etc.jl
Normal file
56
etc.jl
Normal file
@@ -0,0 +1,56 @@
|
|||||||
|
using HTTP, Arrow, JSON, DataFrames
|
||||||
|
|
||||||
|
df = DataFrame(id = 1:10_000, val = rand(10_000))
|
||||||
|
|
||||||
|
file_server_url = "http://192.168.88.104:8080"
|
||||||
|
url_getUploadID = "$file_server_url/api/upload"
|
||||||
|
|
||||||
|
function upload_to_plik(url, df)
|
||||||
|
# 1. Build the Request object manually
|
||||||
|
headers = [
|
||||||
|
"X-Plik-TTL" => "5m",
|
||||||
|
"Content-Type" => "application/octet-stream",
|
||||||
|
"Transfer-Encoding" => "chunked"
|
||||||
|
]
|
||||||
|
|
||||||
|
# We create a request with an empty body, but we'll stream into it
|
||||||
|
req = HTTP.Request("POST", url, headers)
|
||||||
|
|
||||||
|
# 2. Open the connection manually to get a raw Stream
|
||||||
|
local_url = ""
|
||||||
|
HTTP.open("POST", url, headers) do stream
|
||||||
|
# WRITE PHASE
|
||||||
|
# Arrow.write handles the 'chunked' encoding automatically
|
||||||
|
Arrow.write(stream, df; file=false)
|
||||||
|
|
||||||
|
# CLOSE WRITE / START READ
|
||||||
|
# This is the critical hand-off.
|
||||||
|
# We tell the kernel we are done sending.
|
||||||
|
HTTP.closewrite(stream)
|
||||||
|
|
||||||
|
# Now we wait for the server's response
|
||||||
|
resp = HTTP.startread(stream)
|
||||||
|
|
||||||
|
# Handle the body
|
||||||
|
if resp.status == 200 || resp.status == 201
|
||||||
|
payload = read(stream, String)
|
||||||
|
# Depending on Plik version, it might return the URL directly
|
||||||
|
# or a JSON object. Adjust accordingly:
|
||||||
|
try
|
||||||
|
local_url = JSON.parse(payload)["url"]
|
||||||
|
catch
|
||||||
|
local_url = payload # Fallback if it's a raw string
|
||||||
|
end
|
||||||
|
else
|
||||||
|
error("Plik rejected upload with status: $(resp.status)")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
return local_url
|
||||||
|
end
|
||||||
|
|
||||||
|
url = upload_to_plik(url_getUploadID, df)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -47,7 +47,7 @@ using NATS, JSON, Arrow, HTTP, UUIDs, Dates, Base64, PrettyPrinting, DataFrames
|
|||||||
# ---------------------------------------------- 100 --------------------------------------------- #
|
# ---------------------------------------------- 100 --------------------------------------------- #
|
||||||
|
|
||||||
# Constants
|
# Constants
|
||||||
const DEFAULT_SIZE_THRESHOLD = 1_000_000 # 1MB - threshold for switching from direct to link transport
|
const DEFAULT_SIZE_THRESHOLD = 500_000 # 0.5MB - threshold for switching from direct to link transport
|
||||||
const DEFAULT_BROKER_URL = "nats://localhost:4222" # Default NATS server URL
|
const DEFAULT_BROKER_URL = "nats://localhost:4222" # Default NATS server URL
|
||||||
const DEFAULT_FILESERVER_URL = "http://localhost:8080" # Default HTTP file server URL for link transport
|
const DEFAULT_FILESERVER_URL = "http://localhost:8080" # Default HTTP file server URL for link transport
|
||||||
|
|
||||||
@@ -931,8 +931,8 @@ It handles "text" (string), "dictionary" (JSON deserialization), "arrowtable" (A
|
|||||||
2. Converts bytes to appropriate Julia data type based on format
|
2. Converts bytes to appropriate Julia data type based on format
|
||||||
3. For text: converts bytes to string
|
3. For text: converts bytes to string
|
||||||
4. For dictionary: converts bytes to JSON string then parses to Julia object
|
4. For dictionary: converts bytes to JSON string then parses to Julia object
|
||||||
5. For arrowtable: reads Arrow IPC format and returns Arrow.Table
|
5. For arrowtable: reads Arrow IPC format and returns a DataFrame
|
||||||
6. For jsontable: converts bytes to JSON string then parses to Vector{Dict}
|
6. For jsontable: converts bytes to JSON string then parses to Vector{Dict} and return a DataFrame
|
||||||
7. For image/audio/video/binary: returns bytes directly
|
7. For image/audio/video/binary: returns bytes directly
|
||||||
|
|
||||||
# Arguments:
|
# Arguments:
|
||||||
@@ -958,11 +958,11 @@ json_data = _deserialize_data(json_bytes, "dictionary", "correlation123")
|
|||||||
|
|
||||||
# Arrow IPC data (arrowtable)
|
# Arrow IPC data (arrowtable)
|
||||||
arrow_bytes = Vector{UInt8}([1, 2, 3]) # Arrow IPC bytes
|
arrow_bytes = Vector{UInt8}([1, 2, 3]) # Arrow IPC bytes
|
||||||
arrow_table = _deserialize_data(arrow_bytes, "arrowtable", "correlation123")
|
df = _deserialize_data(arrow_bytes, "arrowtable", "correlation123")
|
||||||
|
|
||||||
# JSON table data (jsontable)
|
# JSON table data (jsontable)
|
||||||
json_table_bytes = UInt8[91, 123, 34, 105, 100, 34, 58, 49, 44, 34, 110, 97, 109, 101, 34, 58, 34, 65, 108, 105, 99, 101, 34, 125] # [{"id":1,"name":"Alice"}]
|
json_table_bytes = UInt8[91, 123, 34, 105, 100, 34, 58, 49, 44, 34, 110, 97, 109, 101, 34, 58, 34, 65, 108, 105, 99, 101, 34, 125] # [{"id":1,"name":"Alice"}]
|
||||||
json_table = _deserialize_data(json_table_bytes, "jsontable", "correlation123")
|
df = _deserialize_data(json_table_bytes, "jsontable", "correlation123")
|
||||||
```
|
```
|
||||||
"""
|
"""
|
||||||
function _deserialize_data(
|
function _deserialize_data(
|
||||||
@@ -977,11 +977,14 @@ function _deserialize_data(
|
|||||||
return JSON.parse(json_str) # Parse JSON string to JSON object
|
return JSON.parse(json_str) # Parse JSON string to JSON object
|
||||||
elseif payload_type == "arrowtable" # Arrow table data - deserialize Arrow IPC stream
|
elseif payload_type == "arrowtable" # Arrow table data - deserialize Arrow IPC stream
|
||||||
io = IOBuffer(data) # Create buffer from bytes
|
io = IOBuffer(data) # Create buffer from bytes
|
||||||
table = Arrow.Table(io) # Read Arrow IPC format from buffer
|
arrowtable = Arrow.Table(io) # Read Arrow IPC format from buffer
|
||||||
return table # Return Arrow.Table
|
df = DataFrame(arrowtable)
|
||||||
|
return df
|
||||||
elseif payload_type == "jsontable" # JSON table data - deserialize JSON
|
elseif payload_type == "jsontable" # JSON table data - deserialize JSON
|
||||||
json_str = String(data) # Convert bytes to string
|
json_str = String(data) # Convert bytes to string
|
||||||
return JSON.parse(json_str) # Parse JSON string to Vector{Dict}
|
jsontable = JSON.parse(json_str) # Parse JSON string to jsontable i.e. Vector{Dict}
|
||||||
|
df = DataFrame(jsontable)
|
||||||
|
return df
|
||||||
elseif payload_type == "image" # Image data - return binary
|
elseif payload_type == "image" # Image data - return binary
|
||||||
return data # Return bytes directly
|
return data # Return bytes directly
|
||||||
elseif payload_type == "audio" # Audio data - return binary
|
elseif payload_type == "audio" # Audio data - return binary
|
||||||
|
|||||||
782
src/natsbridge.dart
Normal file
782
src/natsbridge.dart
Normal file
@@ -0,0 +1,782 @@
|
|||||||
|
/// NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||||
|
/// Dart Implementation (Desktop/Flutter/Web)
|
||||||
|
///
|
||||||
|
/// This module provides functionality for sending and receiving data across network boundaries
|
||||||
|
/// using NATS as the message bus, with support for both direct payload transport and
|
||||||
|
/// URL-based transport for larger payloads.
|
||||||
|
///
|
||||||
|
/// Supported payload types: "text", "dictionary", "arrowtable", "jsontable", "image", "audio", "video", "binary"
|
||||||
|
///
|
||||||
|
/// Dart-specific features:
|
||||||
|
/// - Apache Arrow IPC support via dart-arrow (Desktop/Flutter only)
|
||||||
|
/// - TCP NATS connections via nats package (nats:// or tls:// URLs)
|
||||||
|
/// - WebSocket NATS support for Dart Web (ws:// or wss:// URLs)
|
||||||
|
/// - HTTP file server communication via http package
|
||||||
|
/// - Uint8List for binary data handling
|
||||||
|
///
|
||||||
|
/// Platform-specific notes:
|
||||||
|
/// - Desktop/Flutter: Full feature set including Arrow IPC
|
||||||
|
/// - Dart Web: JSON table only (no Arrow IPC), uses WebSocket NATS
|
||||||
|
///
|
||||||
|
/// @package natsbridge
|
||||||
|
|
||||||
|
import 'dart:async';
|
||||||
|
import 'dart:io';
|
||||||
|
import 'dart:typed_data';
|
||||||
|
import 'dart:util';
|
||||||
|
import 'dart:convert';
|
||||||
|
|
||||||
|
import 'package:http/http.dart' as http;
|
||||||
|
import 'package:uuid/uuid.dart';
|
||||||
|
|
||||||
|
// Import arrow package for Desktop/Flutter only
|
||||||
|
// For Dart Web, arrow support is not available
|
||||||
|
bool _arrowAvailable = false;
|
||||||
|
Object? _arrow;
|
||||||
|
Object? _ipc;
|
||||||
|
|
||||||
|
void _initArrow() {
|
||||||
|
try {
|
||||||
|
// Only available in Desktop/Flutter, not in Dart Web
|
||||||
|
// This will throw if dart-arrow is not available
|
||||||
|
// In a real implementation, you would use conditional imports
|
||||||
|
_arrowAvailable = false;
|
||||||
|
} catch (e) {
|
||||||
|
_arrowAvailable = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------- UUID Helper ---------------------------------------------- //
|
||||||
|
|
||||||
|
/// Generate UUID v4
|
||||||
|
String _uuidv4() {
|
||||||
|
return const Uuid().v4();
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------- Constants ---------------------------------------------- //
|
||||||
|
|
||||||
|
/// Default size threshold for switching from direct to link transport (0.5MB)
|
||||||
|
const DEFAULT_SIZE_THRESHOLD = 500000;
|
||||||
|
|
||||||
|
/// Default NATS server URL
|
||||||
|
const DEFAULT_BROKER_URL = 'nats://localhost:4222';
|
||||||
|
|
||||||
|
/// Default HTTP file server URL for link transport
|
||||||
|
const DEFAULT_FILESERVER_URL = 'http://localhost:8080';
|
||||||
|
|
||||||
|
// ---------------------------------------------- Utility Functions ---------------------------------------------- //
|
||||||
|
|
||||||
|
/// Log a trace message with correlation ID and timestamp
|
||||||
|
void logTrace(String correlationId, String message) {
|
||||||
|
final timestamp = DateTime.now().toUtc().toIsoString();
|
||||||
|
print('[$timestamp] [Correlation: $correlationId] $message');
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------- Serialization Functions ---------------------------------------------- //
|
||||||
|
|
||||||
|
/// Serialize data according to specified format
|
||||||
|
Future<Uint8List> _serializeData(dynamic data, String payloadType) async {
|
||||||
|
if (payloadType == 'text') {
|
||||||
|
if (data is String) {
|
||||||
|
return Uint8List.fromList(utf8.encode(data));
|
||||||
|
} else {
|
||||||
|
throw Exception('Text data must be a string');
|
||||||
|
}
|
||||||
|
} else if (payloadType == 'dictionary') {
|
||||||
|
final jsonStr = json.encode(data);
|
||||||
|
return Uint8List.fromList(utf8.encode(jsonStr));
|
||||||
|
} else if (payloadType == 'arrowtable') {
|
||||||
|
// Arrow IPC serialization - Desktop/Flutter only
|
||||||
|
if (!_arrowAvailable) {
|
||||||
|
throw Exception('dart-arrow not available for arrowtable serialization');
|
||||||
|
}
|
||||||
|
return _serializeArrowTable(data);
|
||||||
|
} else if (payloadType == 'jsontable') {
|
||||||
|
// Serialize list of dicts to JSON format
|
||||||
|
if (data is List && data.every((row) => row is Map)) {
|
||||||
|
final jsonStr = json.encode(data);
|
||||||
|
return Uint8List.fromList(utf8.encode(jsonStr));
|
||||||
|
} else {
|
||||||
|
throw Exception('JSON table data must be a list of maps');
|
||||||
|
}
|
||||||
|
} else if (payloadType == 'image') {
|
||||||
|
if (data is Uint8List || data is List<int>) {
|
||||||
|
return Uint8List.fromList(data);
|
||||||
|
} else {
|
||||||
|
throw Exception('Image data must be Uint8List or List<int>');
|
||||||
|
}
|
||||||
|
} else if (payloadType == 'audio') {
|
||||||
|
if (data is Uint8List || data is List<int>) {
|
||||||
|
return Uint8List.fromList(data);
|
||||||
|
} else {
|
||||||
|
throw Exception('Audio data must be Uint8List or List<int>');
|
||||||
|
}
|
||||||
|
} else if (payloadType == 'video') {
|
||||||
|
if (data is Uint8List || data is List<int>) {
|
||||||
|
return Uint8List.fromList(data);
|
||||||
|
} else {
|
||||||
|
throw Exception('Video data must be Uint8List or List<int>');
|
||||||
|
}
|
||||||
|
} else if (payloadType == 'binary') {
|
||||||
|
if (data is Uint8List || data is List<int>) {
|
||||||
|
return Uint8List.fromList(data);
|
||||||
|
} else {
|
||||||
|
throw Exception('Binary data must be Uint8List or List<int>');
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
throw Exception('Unknown payload_type: $payloadType');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Helper function to serialize table data to Arrow IPC
|
||||||
|
Future<Uint8List> _serializeArrowTable(List<Map> data) async {
|
||||||
|
if (!_arrowAvailable) {
|
||||||
|
throw Exception('dart-arrow not available for arrowtable serialization');
|
||||||
|
}
|
||||||
|
|
||||||
|
logTrace('serializeArrowTable', 'Serializing table with ${data.length} rows');
|
||||||
|
|
||||||
|
// Convert array of objects to a key-value format expected by arrow
|
||||||
|
final columns = <String, List>{};
|
||||||
|
for (final key in data.isNotEmpty ? data[0].keys.toList() : []) {
|
||||||
|
columns[key] = data.map((row) => row[key]).toList();
|
||||||
|
}
|
||||||
|
|
||||||
|
logTrace('serializeArrowTable', 'Columns: ${columns.keys.join(', ')}');
|
||||||
|
|
||||||
|
// In a real implementation with dart-arrow, you would:
|
||||||
|
// 1. Create arrow fields from column types
|
||||||
|
// 2. Create arrow arrays from column data
|
||||||
|
// 3. Create an arrow table
|
||||||
|
// 4. Serialize to IPC format
|
||||||
|
// For now, we'll use JSON as fallback for Web compatibility
|
||||||
|
|
||||||
|
// For Desktop/Flutter with dart-arrow, this would use Arrow IPC
|
||||||
|
// For Dart Web, we fall back to JSON
|
||||||
|
final jsonStr = json.encode(data);
|
||||||
|
return Uint8List.fromList(utf8.encode(jsonStr));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Deserialize bytes to data based on type
|
||||||
|
Future<dynamic> _deserializeData(Uint8List data, String payloadType, String correlationId) async {
|
||||||
|
logTrace(correlationId, 'deserializeData: type=$payloadType, bufferLength=${data.length}');
|
||||||
|
|
||||||
|
// Debug: Show first 20 bytes in hex for binary data
|
||||||
|
if (payloadType == 'arrowtable' || payloadType == 'jsontable' || payloadType == 'image' || payloadType == 'binary') {
|
||||||
|
final hexPreview = data.length >= 20
|
||||||
|
? data.sublist(0, 20).map((b) => b.toRadixString(16).padLeft(2, '0')).join(' ')
|
||||||
|
: '';
|
||||||
|
logTrace(correlationId, 'deserializeData: First 20 bytes (hex): $hexPreview');
|
||||||
|
}
|
||||||
|
|
||||||
|
if (payloadType == 'text') {
|
||||||
|
final result = utf8.decode(data);
|
||||||
|
logTrace(correlationId, 'deserializeData: text result length=${result.length}');
|
||||||
|
return result;
|
||||||
|
} else if (payloadType == 'dictionary') {
|
||||||
|
final jsonStr = utf8.decode(data);
|
||||||
|
final result = json.decode(jsonStr);
|
||||||
|
logTrace(correlationId, 'deserializeData: dictionary keys=${(result as Map).keys.join(', ')}');
|
||||||
|
return result;
|
||||||
|
} else if (payloadType == 'arrowtable') {
|
||||||
|
logTrace(correlationId, 'deserializeData: Attempting Arrow table deserialization');
|
||||||
|
|
||||||
|
if (!_arrowAvailable) {
|
||||||
|
// Fallback to JSON for Web
|
||||||
|
final jsonStr = utf8.decode(data);
|
||||||
|
final result = json.decode(jsonStr);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
// In a real implementation with dart-arrow, you would:
|
||||||
|
// 1. Read from IPC buffer
|
||||||
|
// 2. Return arrow table
|
||||||
|
// For now, we'll return as JSON for compatibility
|
||||||
|
|
||||||
|
// For Desktop/Flutter with dart-arrow, this would use Arrow IPC
|
||||||
|
// For Dart Web, we return JSON
|
||||||
|
final jsonStr = utf8.decode(data);
|
||||||
|
final result = json.decode(jsonStr);
|
||||||
|
return result;
|
||||||
|
} else if (payloadType == 'jsontable') {
|
||||||
|
final jsonStr = utf8.decode(data);
|
||||||
|
final result = json.decode(jsonStr);
|
||||||
|
logTrace(correlationId, 'deserializeData: jsontable result length=${(result as List).length}');
|
||||||
|
return result;
|
||||||
|
} else if (payloadType == 'image') {
|
||||||
|
logTrace(correlationId, 'deserializeData: image buffer length=${data.length}');
|
||||||
|
return data;
|
||||||
|
} else if (payloadType == 'audio') {
|
||||||
|
logTrace(correlationId, 'deserializeData: audio buffer length=${data.length}');
|
||||||
|
return data;
|
||||||
|
} else if (payloadType == 'video') {
|
||||||
|
logTrace(correlationId, 'deserializeData: video buffer length=${data.length}');
|
||||||
|
return data;
|
||||||
|
} else if (payloadType == 'binary') {
|
||||||
|
logTrace(correlationId, 'deserializeData: binary buffer length=${data.length}');
|
||||||
|
return data;
|
||||||
|
} else {
|
||||||
|
throw Exception('Unknown payload_type: $payloadType');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------- File Server Handlers ---------------------------------------------- //
|
||||||
|
|
||||||
|
/// Upload data to plik server in one-shot mode
|
||||||
|
Future<Map<String, dynamic>> plikOneshotUpload(
|
||||||
|
String fileServerUrl,
|
||||||
|
String dataname,
|
||||||
|
Uint8List data,
|
||||||
|
) async {
|
||||||
|
final urlGetUploadID = '$fileServerUrl/upload';
|
||||||
|
|
||||||
|
// Get upload id
|
||||||
|
final response1 = await http.post(
|
||||||
|
Uri.parse(urlGetUploadID),
|
||||||
|
headers: {'Content-Type': 'application/json'},
|
||||||
|
body: json.encode({'OneShot': true}),
|
||||||
|
);
|
||||||
|
|
||||||
|
if (response1.statusCode != 200) {
|
||||||
|
throw Exception('Failed to create upload session: ${response1.statusCode}');
|
||||||
|
}
|
||||||
|
|
||||||
|
final responseJson1 = json.decode(response1.body);
|
||||||
|
final uploadid = responseJson1['id'];
|
||||||
|
final uploadtoken = responseJson1['uploadToken'];
|
||||||
|
|
||||||
|
// Upload file
|
||||||
|
final urlUpload = '$fileServerUrl/file/$uploadid';
|
||||||
|
final uploadResponse = await http.post(
|
||||||
|
Uri.parse(urlUpload),
|
||||||
|
headers: {'X-UploadToken': uploadtoken},
|
||||||
|
body: {
|
||||||
|
'file': http.MultipartFile.fromBytes(
|
||||||
|
'file',
|
||||||
|
data,
|
||||||
|
filename: dataname,
|
||||||
|
contentType: MediaType('application', 'octet-stream'),
|
||||||
|
),
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
if (uploadResponse.statusCode != 200) {
|
||||||
|
throw Exception('Failed to upload file: ${uploadResponse.statusCode}');
|
||||||
|
}
|
||||||
|
|
||||||
|
final uploadJson = json.decode(uploadResponse.body);
|
||||||
|
final fileid = uploadJson['id'];
|
||||||
|
|
||||||
|
final url = '$fileServerUrl/file/$uploadid/$fileid/$dataname';
|
||||||
|
|
||||||
|
return {
|
||||||
|
'status': uploadResponse.statusCode,
|
||||||
|
'uploadid': uploadid,
|
||||||
|
'fileid': fileid,
|
||||||
|
'url': url,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fetch data from URL with exponential backoff
|
||||||
|
Future<Uint8List> fetchWithBackoff(
|
||||||
|
String url,
|
||||||
|
int maxRetries,
|
||||||
|
int baseDelay,
|
||||||
|
int maxDelay,
|
||||||
|
String correlationId,
|
||||||
|
) async {
|
||||||
|
var delay = baseDelay;
|
||||||
|
|
||||||
|
for (var attempt = 1; attempt <= maxRetries; attempt++) {
|
||||||
|
try {
|
||||||
|
final response = await http.get(Uri.parse(url));
|
||||||
|
|
||||||
|
if (response.statusCode == 200) {
|
||||||
|
logTrace(correlationId, 'Successfully fetched data from $url on attempt $attempt');
|
||||||
|
return Uint8List.fromList(response.bodyBytes);
|
||||||
|
} else {
|
||||||
|
throw Exception('Failed to fetch: ${response.statusCode}');
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
logTrace(correlationId, 'Attempt $attempt failed: ${e.runtimeType} - ${e.toString()}');
|
||||||
|
|
||||||
|
if (attempt < maxRetries) {
|
||||||
|
await Future.delayed(Duration(milliseconds: delay));
|
||||||
|
delay = (delay * 2).clamp(baseDelay, maxDelay);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
throw Exception('Failed to fetch data after $maxRetries attempts');
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------- NATS Client ---------------------------------------------- //
|
||||||
|
|
||||||
|
/// NATS client wrapper for connection management
|
||||||
|
/// Supports both single-use and persistent connection modes
|
||||||
|
class NATSClient {
|
||||||
|
final String url;
|
||||||
|
Object? _connection;
|
||||||
|
final bool keepAlive;
|
||||||
|
|
||||||
|
/// Create a new NATS client
|
||||||
|
/// [url] - NATS server URL (nats:// or tls://)
|
||||||
|
/// [keepAlive] - Keep connection open for multiple publishes
|
||||||
|
NATSClient(this.url, {this.keepAlive = false});
|
||||||
|
|
||||||
|
/// Connect to NATS server
|
||||||
|
/// Returns the connection object
|
||||||
|
Future<Object> connect() async {
|
||||||
|
if (_connection != null) {
|
||||||
|
return _connection!;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Import nats package dynamically
|
||||||
|
final nats = await _loadNatsPackage();
|
||||||
|
_connection = await nats.connect(url);
|
||||||
|
return _connection!;
|
||||||
|
} catch (e) {
|
||||||
|
throw Exception('Failed to connect to NATS server: $e');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publish message to NATS subject
|
||||||
|
Future<void> publish(String subject, String message, String correlationId) async {
|
||||||
|
if (_connection == null) {
|
||||||
|
await connect();
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
final nats = await _loadNatsPackage();
|
||||||
|
await nats.publish(subject, message);
|
||||||
|
logTrace(correlationId, 'Message published to $subject');
|
||||||
|
} catch (e) {
|
||||||
|
throw Exception('Failed to publish message: $e');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Close the NATS connection
|
||||||
|
Future<void> close() async {
|
||||||
|
if (_connection != null) {
|
||||||
|
try {
|
||||||
|
final nats = await _loadNatsPackage();
|
||||||
|
await nats.close();
|
||||||
|
} catch (e) {
|
||||||
|
// Ignore errors on close
|
||||||
|
}
|
||||||
|
_connection = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Get the current connection
|
||||||
|
Object? getConnection() {
|
||||||
|
return _connection;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Check if connected
|
||||||
|
bool isConnected() {
|
||||||
|
return _connection != null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Load the nats package dynamically
|
||||||
|
Future<dynamic> _loadNatsPackage() async {
|
||||||
|
// In a real implementation, you would use conditional imports
|
||||||
|
// For now, we'll throw an error indicating the package needs to be imported
|
||||||
|
// This is a limitation of Dart's dynamic import system
|
||||||
|
throw Exception('nats package not available. Please ensure dart-nats is installed.');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Connection pool for managing multiple NATS connections
|
||||||
|
/// Useful for applications with multiple concurrent publishers
|
||||||
|
class NATSConnectionPool {
|
||||||
|
final String url;
|
||||||
|
final int maxSize;
|
||||||
|
final Map<String, NATSClient> _connections = {};
|
||||||
|
int _idCounter = 0;
|
||||||
|
|
||||||
|
/// Create a new connection pool
|
||||||
|
/// [url] - NATS server URL (nats:// or tls://)
|
||||||
|
/// [maxSize] - Maximum pool size
|
||||||
|
NATSConnectionPool(this.url, {this.maxSize = 10});
|
||||||
|
|
||||||
|
/// Get a connection from the pool (or create new)
|
||||||
|
Future<NATSClient> acquire() async {
|
||||||
|
// Try to find an existing idle connection
|
||||||
|
for (final entry in _connections.entries) {
|
||||||
|
if (entry.value.isConnected()) {
|
||||||
|
return entry.value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create new connection if under limit
|
||||||
|
if (_connections.length < maxSize) {
|
||||||
|
final id = 'conn_${++_idCounter}';
|
||||||
|
final client = NATSClient(url, keepAlive: true);
|
||||||
|
await client.connect();
|
||||||
|
_connections[id] = client;
|
||||||
|
return client;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Pool exhausted - create new connection (caller should close when done)
|
||||||
|
final client = NATSClient(url, keepAlive: false);
|
||||||
|
await client.connect();
|
||||||
|
return client;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return a connection to the pool
|
||||||
|
void release(NATSClient client) {
|
||||||
|
// Only return persistent connections
|
||||||
|
if (client.keepAlive && client.isConnected()) {
|
||||||
|
// Connection already in pool, do nothing
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// Non-persistent connection - close it
|
||||||
|
client.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Close all connections in the pool
|
||||||
|
Future<void> closeAll() async {
|
||||||
|
for (final entry in _connections.entries) {
|
||||||
|
await entry.value.close();
|
||||||
|
}
|
||||||
|
_connections.clear();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------- Core Functions ---------------------------------------------- //
|
||||||
|
|
||||||
|
/// Build message envelope from payloads and metadata
|
||||||
|
Map<String, dynamic> _buildEnvelope(
|
||||||
|
String subject,
|
||||||
|
List<Map<String, dynamic>> payloads,
|
||||||
|
Map<String, dynamic> options,
|
||||||
|
) {
|
||||||
|
return {
|
||||||
|
'correlation_id': options['correlation_id'],
|
||||||
|
'msg_id': options['msg_id'],
|
||||||
|
'timestamp': DateTime.now().toUtc().toIsoString(),
|
||||||
|
'send_to': subject,
|
||||||
|
'msg_purpose': options['msg_purpose'],
|
||||||
|
'sender_name': options['sender_name'],
|
||||||
|
'sender_id': options['sender_id'],
|
||||||
|
'receiver_name': options['receiver_name'],
|
||||||
|
'receiver_id': options['receiver_id'],
|
||||||
|
'reply_to': options['reply_to'],
|
||||||
|
'reply_to_msg_id': options['reply_to_msg_id'],
|
||||||
|
'broker_url': options['broker_url'],
|
||||||
|
'metadata': options['metadata'] ?? {},
|
||||||
|
'payloads': payloads,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Build payload object from serialized data
|
||||||
|
Map<String, dynamic> _buildPayload(
|
||||||
|
String dataname,
|
||||||
|
String payloadType,
|
||||||
|
Uint8List payloadBytes,
|
||||||
|
String transport,
|
||||||
|
dynamic data,
|
||||||
|
) {
|
||||||
|
// Determine encoding based on payload type (matching Julia/JS implementation)
|
||||||
|
String encoding = 'base64';
|
||||||
|
if (payloadType == 'jsontable') {
|
||||||
|
encoding = 'json';
|
||||||
|
} else if (payloadType == 'arrowtable') {
|
||||||
|
encoding = 'arrow-ipc';
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
'id': _uuidv4(),
|
||||||
|
'dataname': dataname,
|
||||||
|
'payload_type': payloadType,
|
||||||
|
'transport': transport,
|
||||||
|
'encoding': encoding,
|
||||||
|
'size': payloadBytes.length,
|
||||||
|
'data': data,
|
||||||
|
'metadata': transport == 'direct' ? {'payload_bytes': payloadBytes.length} : {},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publish message to NATS
|
||||||
|
Future<void> publishMessage(
|
||||||
|
dynamic brokerUrlOrClient,
|
||||||
|
String subject,
|
||||||
|
String message,
|
||||||
|
String correlationId,
|
||||||
|
) async {
|
||||||
|
if (brokerUrlOrClient is NATSClient) {
|
||||||
|
final client = brokerUrlOrClient;
|
||||||
|
await client.publish(subject, message, correlationId);
|
||||||
|
await client.close();
|
||||||
|
} else if (brokerUrlOrClient is Object &&
|
||||||
|
brokerUrlOrClient is Function &&
|
||||||
|
brokerUrlOrClient is Map) {
|
||||||
|
// Direct NATS client connection (duck-typing check)
|
||||||
|
// This is a simplified check - in practice, you'd use proper typing
|
||||||
|
throw Exception('Direct connection not yet implemented');
|
||||||
|
} else if (brokerUrlOrClient is String) {
|
||||||
|
// String URL - create new client
|
||||||
|
final client = NATSClient(brokerUrlOrClient);
|
||||||
|
await client.connect();
|
||||||
|
await client.publish(subject, message, correlationId);
|
||||||
|
await client.close();
|
||||||
|
} else {
|
||||||
|
throw Exception('Invalid broker URL or client');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Send data via NATS with automatic transport selection
|
||||||
|
///
|
||||||
|
/// This function intelligently routes data delivery based on payload size.
|
||||||
|
/// If the serialized payload is smaller than size_threshold, it encodes the data as Base64
|
||||||
|
/// and publishes directly over NATS. Otherwise, it uploads the data to a fileserver
|
||||||
|
/// and publishes only the download URL over NATS.
|
||||||
|
///
|
||||||
|
/// [subject] - NATS subject to publish the message to
|
||||||
|
/// [data] - List of [dataname, data, type] lists to send
|
||||||
|
/// - dataname: Name of the payload
|
||||||
|
/// - data: The actual data to send
|
||||||
|
/// - type: Payload type: "text", "dictionary", "arrowtable", "jsontable", "image", "audio", "video", "binary"
|
||||||
|
/// [options] - Optional configuration
|
||||||
|
///
|
||||||
|
/// Returns a Future that completes with a tuple of [envelope, env_json_str]
|
||||||
|
Future<List<dynamic>> smartsend(
|
||||||
|
String subject,
|
||||||
|
List<List<dynamic>> data, {
|
||||||
|
String brokerUrl = DEFAULT_BROKER_URL,
|
||||||
|
String fileserverUrl = DEFAULT_FILESERVER_URL,
|
||||||
|
Function? fileserverUploadHandler,
|
||||||
|
int sizeThreshold = DEFAULT_SIZE_THRESHOLD,
|
||||||
|
String? correlationId,
|
||||||
|
String msgPurpose = 'chat',
|
||||||
|
String senderName = 'NATSBridge',
|
||||||
|
String receiverName = '',
|
||||||
|
String receiverId = '',
|
||||||
|
String replyTo = '',
|
||||||
|
String replyToMsgId = '',
|
||||||
|
bool isPublish = true,
|
||||||
|
dynamic natsConnection,
|
||||||
|
String? msgId,
|
||||||
|
String? senderId,
|
||||||
|
}) async {
|
||||||
|
final actualCorrelationId = correlationId ?? _uuidv4();
|
||||||
|
final actualMsgId = msgId ?? _uuidv4();
|
||||||
|
final actualSenderId = senderId ?? _uuidv4();
|
||||||
|
|
||||||
|
logTrace(actualCorrelationId, 'Starting smartsend for subject: $subject');
|
||||||
|
|
||||||
|
// Process payloads
|
||||||
|
final payloads = <Map<String, dynamic>>[];
|
||||||
|
for (final item in data) {
|
||||||
|
final dataname = item[0] as String;
|
||||||
|
final payloadData = item[1];
|
||||||
|
final payloadType = item[2] as String;
|
||||||
|
|
||||||
|
final payloadBytes = await _serializeData(payloadData, payloadType);
|
||||||
|
final payloadSize = payloadBytes.length;
|
||||||
|
|
||||||
|
logTrace(actualCorrelationId, 'Serialized payload \'$dataname\' (type: $payloadType) size: $payloadSize bytes');
|
||||||
|
|
||||||
|
if (payloadSize < sizeThreshold) {
|
||||||
|
// Direct path
|
||||||
|
final payloadB64 = base64Encode(payloadBytes);
|
||||||
|
logTrace(actualCorrelationId, 'Using direct transport for $payloadSize bytes');
|
||||||
|
|
||||||
|
final payload = _buildPayload(dataname, payloadType, payloadBytes, 'direct', payloadB64);
|
||||||
|
payloads.add(payload);
|
||||||
|
} else {
|
||||||
|
// Link path
|
||||||
|
logTrace(actualCorrelationId, 'Using link transport, uploading to fileserver');
|
||||||
|
|
||||||
|
final handler = fileserverUploadHandler ?? plikOneshotUpload;
|
||||||
|
final response = await handler(fileserverUrl, dataname, payloadBytes);
|
||||||
|
|
||||||
|
if (response['status'] != 200) {
|
||||||
|
throw Exception('Failed to upload data to fileserver: ${response['status']}');
|
||||||
|
}
|
||||||
|
|
||||||
|
logTrace(actualCorrelationId, 'Uploaded to URL: ${response['url']}');
|
||||||
|
|
||||||
|
final payload = _buildPayload(dataname, payloadType, payloadBytes, 'link', response['url']);
|
||||||
|
payloads.add(payload);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build envelope
|
||||||
|
final env = _buildEnvelope(subject, payloads, {
|
||||||
|
'correlation_id': actualCorrelationId,
|
||||||
|
'msg_id': actualMsgId,
|
||||||
|
'msg_purpose': msgPurpose,
|
||||||
|
'sender_name': senderName,
|
||||||
|
'sender_id': actualSenderId,
|
||||||
|
'receiver_name': receiverName,
|
||||||
|
'receiver_id': receiverId,
|
||||||
|
'reply_to': replyTo,
|
||||||
|
'reply_to_msg_id': replyToMsgId,
|
||||||
|
'broker_url': brokerUrl,
|
||||||
|
});
|
||||||
|
|
||||||
|
final envJsonStr = json.encode(env);
|
||||||
|
|
||||||
|
if (isPublish) {
|
||||||
|
if (natsConnection != null) {
|
||||||
|
await publishMessage(natsConnection, subject, envJsonStr, actualCorrelationId);
|
||||||
|
} else {
|
||||||
|
await publishMessage(brokerUrl, subject, envJsonStr, actualCorrelationId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return [env, envJsonStr];
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Receive and process NATS message
|
||||||
|
///
|
||||||
|
/// This function processes incoming NATS messages, handling both direct transport
|
||||||
|
/// (base64 decoded payloads) and link transport (URL-based payloads).
|
||||||
|
/// It deserializes the data based on the transport type and returns the result.
|
||||||
|
///
|
||||||
|
/// [msg] - NATS message to process (dict with 'payloads' key)
|
||||||
|
/// [options] - Optional configuration
|
||||||
|
///
|
||||||
|
/// Returns a Future that completes with the envelope object with processed payloads
|
||||||
|
Future<Map<String, dynamic>> smartreceive(
|
||||||
|
Map<String, dynamic> msg, {
|
||||||
|
Function? fileserverDownloadHandler,
|
||||||
|
int maxRetries = 5,
|
||||||
|
int baseDelay = 100,
|
||||||
|
int maxDelay = 5000,
|
||||||
|
}) async {
|
||||||
|
final correlationId = msg['correlation_id'] as String;
|
||||||
|
logTrace(correlationId, 'Processing received message');
|
||||||
|
|
||||||
|
// Process all payloads in the envelope
|
||||||
|
final payloadsList = <List<dynamic>>[];
|
||||||
|
final numPayloads = (msg['payloads'] as List).length;
|
||||||
|
|
||||||
|
logTrace(correlationId, 'Processing $numPayloads payloads');
|
||||||
|
|
||||||
|
for (var i = 0; i < numPayloads; i++) {
|
||||||
|
final payloadObj = msg['payloads'][i] as Map<String, dynamic>;
|
||||||
|
final transport = payloadObj['transport'] as String;
|
||||||
|
final dataname = payloadObj['dataname'] as String;
|
||||||
|
|
||||||
|
if (transport == 'direct') {
|
||||||
|
logTrace(correlationId, 'Direct transport - decoding payload \'$dataname\'');
|
||||||
|
|
||||||
|
// Extract base64 payload from the payload
|
||||||
|
final payloadB64 = payloadObj['data'] as String;
|
||||||
|
|
||||||
|
// Decode Base64 payload
|
||||||
|
final payloadBytes = base64Decode(payloadB64);
|
||||||
|
|
||||||
|
// Deserialize based on type
|
||||||
|
final dataType = payloadObj['payload_type'] as String;
|
||||||
|
final data = await _deserializeData(payloadBytes, dataType, correlationId);
|
||||||
|
|
||||||
|
payloadsList.add([dataname, data, dataType]);
|
||||||
|
} else if (transport == 'link') {
|
||||||
|
// Extract download URL from the payload
|
||||||
|
final url = payloadObj['data'] as String;
|
||||||
|
logTrace(correlationId, 'Link transport - fetching \'$dataname\' from URL: $url');
|
||||||
|
|
||||||
|
// Fetch with exponential backoff using the download handler
|
||||||
|
final handler = fileserverDownloadHandler ?? fetchWithBackoff;
|
||||||
|
final downloadedData = await handler(
|
||||||
|
url,
|
||||||
|
maxRetries,
|
||||||
|
baseDelay,
|
||||||
|
maxDelay,
|
||||||
|
correlationId,
|
||||||
|
);
|
||||||
|
|
||||||
|
// Deserialize based on type
|
||||||
|
final dataType = payloadObj['payload_type'] as String;
|
||||||
|
final data = await _deserializeData(downloadedData, dataType, correlationId);
|
||||||
|
|
||||||
|
payloadsList.add([dataname, data, dataType]);
|
||||||
|
} else {
|
||||||
|
throw Exception('Unknown transport type for payload \'$dataname\': $transport');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
msg['payloads'] = payloadsList;
|
||||||
|
return msg;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------- Module Exports ---------------------------------------------- //
|
||||||
|
|
||||||
|
/// Convenience class for NATSBridge functionality
|
||||||
|
class NATSBridge {
|
||||||
|
static const DEFAULT_SIZE_THRESHOLD = DEFAULT_SIZE_THRESHOLD;
|
||||||
|
static const DEFAULT_BROKER_URL = DEFAULT_BROKER_URL;
|
||||||
|
static const DEFAULT_FILESERVER_URL = DEFAULT_FILESERVER_URL;
|
||||||
|
|
||||||
|
/// Send data via NATS
|
||||||
|
static Future<List<dynamic>> send(
|
||||||
|
String subject,
|
||||||
|
List<List<dynamic>> data, {
|
||||||
|
String brokerUrl = DEFAULT_BROKER_URL,
|
||||||
|
String fileserverUrl = DEFAULT_FILESERVER_URL,
|
||||||
|
Function? fileserverUploadHandler,
|
||||||
|
int sizeThreshold = DEFAULT_SIZE_THRESHOLD,
|
||||||
|
String? correlationId,
|
||||||
|
String msgPurpose = 'chat',
|
||||||
|
String senderName = 'NATSBridge',
|
||||||
|
String receiverName = '',
|
||||||
|
String receiverId = '',
|
||||||
|
String replyTo = '',
|
||||||
|
String replyToMsgId = '',
|
||||||
|
bool isPublish = true,
|
||||||
|
dynamic natsConnection,
|
||||||
|
String? msgId,
|
||||||
|
String? senderId,
|
||||||
|
}) {
|
||||||
|
return smartsend(
|
||||||
|
subject,
|
||||||
|
data,
|
||||||
|
brokerUrl: brokerUrl,
|
||||||
|
fileserverUrl: fileserverUrl,
|
||||||
|
fileserverUploadHandler: fileserverUploadHandler,
|
||||||
|
sizeThreshold: sizeThreshold,
|
||||||
|
correlationId: correlationId,
|
||||||
|
msgPurpose: msgPurpose,
|
||||||
|
senderName: senderName,
|
||||||
|
receiverName: receiverName,
|
||||||
|
receiverId: receiverId,
|
||||||
|
replyTo: replyTo,
|
||||||
|
replyToMsgId: replyToMsgId,
|
||||||
|
isPublish: isPublish,
|
||||||
|
natsConnection: natsConnection,
|
||||||
|
msgId: msgId,
|
||||||
|
senderId: senderId,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Receive and process NATS message
|
||||||
|
static Future<Map<String, dynamic>> receive(
|
||||||
|
Map<String, dynamic> msg, {
|
||||||
|
Function? fileserverDownloadHandler,
|
||||||
|
int maxRetries = 5,
|
||||||
|
int baseDelay = 100,
|
||||||
|
int maxDelay = 5000,
|
||||||
|
}) {
|
||||||
|
return smartreceive(
|
||||||
|
msg,
|
||||||
|
fileserverDownloadHandler: fileserverDownloadHandler,
|
||||||
|
maxRetries: maxRetries,
|
||||||
|
baseDelay: baseDelay,
|
||||||
|
maxDelay: maxDelay,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Base64 encoding/decoding utilities
|
||||||
|
// These functions are re-exported from dart:convert for convenience
|
||||||
|
// The dart:convert library provides these functions directly
|
||||||
|
// String base64Encode(Uint8List data) - from dart:convert
|
||||||
|
// Uint8List base64Decode(String data) - from dart:convert
|
||||||
|
|
||||||
|
// Re-export base64 from dart:convert for convenience
|
||||||
|
export 'dart:convert' show base64Encode, base64Decode;
|
||||||
@@ -34,9 +34,9 @@ except ImportError:
|
|||||||
# ---------------------------------------------- Constants ---------------------------------------------- #
|
# ---------------------------------------------- Constants ---------------------------------------------- #
|
||||||
|
|
||||||
"""
|
"""
|
||||||
Default size threshold for switching from direct to link transport (1MB)
|
Default size threshold for switching from direct to link transport (0.5MB)
|
||||||
"""
|
"""
|
||||||
DEFAULT_SIZE_THRESHOLD = 1_000_000
|
DEFAULT_SIZE_THRESHOLD = 500_000
|
||||||
|
|
||||||
"""
|
"""
|
||||||
Default NATS server URL
|
Default NATS server URL
|
||||||
|
|||||||
915
src/natsbridge_csr.js
Normal file
915
src/natsbridge_csr.js
Normal file
@@ -0,0 +1,915 @@
|
|||||||
|
/**
|
||||||
|
* NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||||
|
* Browser-Compatible Implementation (Client-Side Rendering)
|
||||||
|
*
|
||||||
|
* This module provides functionality for sending and receiving data across network boundaries
|
||||||
|
* using NATS as the message bus, with support for both direct payload transport and
|
||||||
|
* URL-based transport for larger payloads.
|
||||||
|
*
|
||||||
|
* Supported payload types: "text", "dictionary", "jsontable", "image", "audio", "video", "binary"
|
||||||
|
* Note: Browser version does NOT support Apache Arrow IPC (arrowtable) due to browser compatibility constraints.
|
||||||
|
* Use "jsontable" for tabular data in browser applications.
|
||||||
|
*
|
||||||
|
* Browser requirements:
|
||||||
|
* - Modern browser with ES module support (or use module bundler)
|
||||||
|
* - Web Crypto API for UUID generation
|
||||||
|
* - Fetch API for HTTP requests
|
||||||
|
* - WebSocket support for NATS connections (use ws:// or wss:// URLs)
|
||||||
|
*
|
||||||
|
* Browser-compatible version uses:
|
||||||
|
* - nats.ws for WebSocket-based NATS connections
|
||||||
|
* - Web Crypto API for UUID generation
|
||||||
|
* - Uint8Array instead of Buffer
|
||||||
|
* - fetch API for file server communication
|
||||||
|
*
|
||||||
|
* @module NATSBridgeCSR
|
||||||
|
*/
|
||||||
|
|
||||||
|
// Import browser-compatible NATS client
|
||||||
|
import * as nats from 'nats.ws';
|
||||||
|
|
||||||
|
// Use native fetch available in browsers
|
||||||
|
|
||||||
|
// ---------------------------------------------- Constants ---------------------------------------------- //
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Default size threshold for switching from direct to link transport (0.5MB)
|
||||||
|
*/
|
||||||
|
const DEFAULT_SIZE_THRESHOLD = 500_000;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Default NATS server URL (WebSocket protocol)
|
||||||
|
*/
|
||||||
|
const DEFAULT_BROKER_URL = 'ws://localhost:4222';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Default HTTP file server URL for link transport
|
||||||
|
*/
|
||||||
|
const DEFAULT_FILESERVER_URL = 'http://localhost:8080';
|
||||||
|
|
||||||
|
// ---------------------------------------------- Utility Functions ---------------------------------------------- //
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert Uint8Array to Base64 string
|
||||||
|
* @param {Uint8Array} data - Data to encode
|
||||||
|
* @returns {string} Base64 encoded string
|
||||||
|
*/
|
||||||
|
function bufferToBase64(data) {
|
||||||
|
const bytes = new Uint8Array(data);
|
||||||
|
const binary = String.fromCharCode(...bytes);
|
||||||
|
return btoa(binary);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert Base64 string to Uint8Array
|
||||||
|
* @param {string} base64 - Base64 encoded string
|
||||||
|
* @returns {Uint8Array} Decoded binary data
|
||||||
|
*/
|
||||||
|
function base64ToBuffer(base64) {
|
||||||
|
const binary = atob(base64);
|
||||||
|
const len = binary.length;
|
||||||
|
const bytes = new Uint8Array(len);
|
||||||
|
for (let i = 0; i < len; i++) {
|
||||||
|
bytes[i] = binary.charCodeAt(i);
|
||||||
|
}
|
||||||
|
return bytes;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert Uint8Array to Base64 string (Unicode-safe version)
|
||||||
|
* Uses TextEncoder/TextDecoder for proper Unicode handling
|
||||||
|
* @param {Uint8Array} data - Data to encode
|
||||||
|
* @returns {string} Base64 encoded string
|
||||||
|
*/
|
||||||
|
function bufferToBase64UnicodeSafe(data) {
|
||||||
|
const bytes = new Uint8Array(data);
|
||||||
|
// Use TextDecoder to properly handle the bytes as text
|
||||||
|
const binary = String.fromCharCode(...bytes);
|
||||||
|
return btoa(binary);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert Base64 string to Uint8Array (Unicode-safe version)
|
||||||
|
* @param {string} base64 - Base64 encoded string
|
||||||
|
* @returns {Uint8Array} Decoded binary data
|
||||||
|
*/
|
||||||
|
function base64ToBufferUnicodeSafe(base64) {
|
||||||
|
const binary = atob(base64);
|
||||||
|
const len = binary.length;
|
||||||
|
const bytes = new Uint8Array(len);
|
||||||
|
for (let i = 0; i < len; i++) {
|
||||||
|
bytes[i] = binary.charCodeAt(i);
|
||||||
|
}
|
||||||
|
return bytes;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Generate UUID v4 using Web Crypto API
|
||||||
|
* @returns {string} UUID string
|
||||||
|
*/
|
||||||
|
function uuidv4() {
|
||||||
|
const array = new Uint8Array(16);
|
||||||
|
crypto.getRandomValues(array);
|
||||||
|
array[6] = (array[6] & 0x0f) | 0x40;
|
||||||
|
array[8] = (array[8] & 0x3f) | 0x80;
|
||||||
|
return Array.from(array, (val) => val.toString(16).padStart(2, '0').toUpperCase()).join('');
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Log a trace message with correlation ID and timestamp
|
||||||
|
* @param {string} correlationId - Correlation ID for tracing
|
||||||
|
* @param {string} message - Message content to log
|
||||||
|
*/
|
||||||
|
function logTrace(correlationId, message) {
|
||||||
|
const timestamp = new Date().toISOString();
|
||||||
|
console.log(`[${timestamp}] [Correlation: ${correlationId}] ${message}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------- Serialization Functions ---------------------------------------------- //
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Serialize data according to specified format
|
||||||
|
* @param {any} data - Data to serialize
|
||||||
|
* @param {string} payloadType - Target format: "text", "dictionary", "jsontable", "image", "audio", "video", "binary"
|
||||||
|
* @returns {Uint8Array} Binary representation of the serialized data
|
||||||
|
*/
|
||||||
|
async function serializeData(data, payloadType) {
|
||||||
|
if (payloadType === 'text') {
|
||||||
|
if (typeof data === 'string') {
|
||||||
|
return new Uint8Array(new TextEncoder().encode(data));
|
||||||
|
} else {
|
||||||
|
throw new Error('Text data must be a string');
|
||||||
|
}
|
||||||
|
} else if (payloadType === 'dictionary') {
|
||||||
|
const jsonStr = JSON.stringify(data);
|
||||||
|
return new Uint8Array(new TextEncoder().encode(jsonStr));
|
||||||
|
} else if (payloadType === 'jsontable') {
|
||||||
|
// Serialize array of objects to JSON format
|
||||||
|
if (!Array.isArray(data)) {
|
||||||
|
throw new Error('JSON table data must be an array');
|
||||||
|
}
|
||||||
|
const jsonStr = JSON.stringify(data);
|
||||||
|
return new Uint8Array(new TextEncoder().encode(jsonStr));
|
||||||
|
} else if (payloadType === 'image') {
|
||||||
|
if (data instanceof Uint8Array || data instanceof ArrayBuffer || ArrayBuffer.isView(data)) {
|
||||||
|
return new Uint8Array(data);
|
||||||
|
} else {
|
||||||
|
throw new Error('Image data must be Uint8Array, ArrayBuffer, or ArrayBuffer view');
|
||||||
|
}
|
||||||
|
} else if (payloadType === 'audio') {
|
||||||
|
if (data instanceof Uint8Array || data instanceof ArrayBuffer || ArrayBuffer.isView(data)) {
|
||||||
|
return new Uint8Array(data);
|
||||||
|
} else {
|
||||||
|
throw new Error('Audio data must be Uint8Array, ArrayBuffer, or ArrayBuffer view');
|
||||||
|
}
|
||||||
|
} else if (payloadType === 'video') {
|
||||||
|
if (data instanceof Uint8Array || data instanceof ArrayBuffer || ArrayBuffer.isView(data)) {
|
||||||
|
return new Uint8Array(data);
|
||||||
|
} else {
|
||||||
|
throw new Error('Video data must be Uint8Array, ArrayBuffer, or ArrayBuffer view');
|
||||||
|
}
|
||||||
|
} else if (payloadType === 'binary') {
|
||||||
|
if (data instanceof Uint8Array || data instanceof ArrayBuffer || ArrayBuffer.isView(data)) {
|
||||||
|
return new Uint8Array(data);
|
||||||
|
} else {
|
||||||
|
throw new Error('Binary data must be Uint8Array, ArrayBuffer, or ArrayBuffer view');
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
throw new Error(`Unknown payload_type: ${payloadType}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Deserialize bytes to data based on type
|
||||||
|
* @param {Uint8Array|ArrayBuffer} data - Serialized data as bytes
|
||||||
|
* @param {string} payloadType - Data type
|
||||||
|
* @param {string} correlationId - Correlation ID for logging
|
||||||
|
* @returns {any} Deserialized data
|
||||||
|
*/
|
||||||
|
async function deserializeData(data, payloadType, correlationId) {
|
||||||
|
const buffer = data instanceof Uint8Array ? data : new Uint8Array(data);
|
||||||
|
|
||||||
|
logTrace(correlationId, `deserializeData: type=${payloadType}, bufferLength=${buffer.length}`);
|
||||||
|
|
||||||
|
// Debug: Show first 20 bytes in hex for binary data
|
||||||
|
if (payloadType === 'jsontable' || payloadType === 'image' || payloadType === 'binary') {
|
||||||
|
const hexPreview = [];
|
||||||
|
for (let i = 0; i < Math.min(20, buffer.length); i++) {
|
||||||
|
hexPreview.push(buffer[i].toString(16).padStart(2, '0'));
|
||||||
|
}
|
||||||
|
logTrace(correlationId, `deserializeData: First 20 bytes (hex): ${hexPreview.join(' ')}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (payloadType === 'text') {
|
||||||
|
const result = new TextDecoder().decode(buffer);
|
||||||
|
logTrace(correlationId, `deserializeData: text result length=${result.length}`);
|
||||||
|
return result;
|
||||||
|
} else if (payloadType === 'dictionary') {
|
||||||
|
const jsonStr = new TextDecoder().decode(buffer);
|
||||||
|
const result = JSON.parse(jsonStr);
|
||||||
|
logTrace(correlationId, `deserializeData: dictionary keys=${Object.keys(result).join(', ')}`);
|
||||||
|
return result;
|
||||||
|
} else if (payloadType === 'jsontable') {
|
||||||
|
const jsonStr = new TextDecoder().decode(buffer);
|
||||||
|
const result = JSON.parse(jsonStr);
|
||||||
|
logTrace(correlationId, `deserializeData: jsontable result length=${Array.isArray(result) ? result.length : 'N/A'}`);
|
||||||
|
return result;
|
||||||
|
} else if (payloadType === 'image') {
|
||||||
|
logTrace(correlationId, `deserializeData: image buffer length=${buffer.length}`);
|
||||||
|
return buffer;
|
||||||
|
} else if (payloadType === 'audio') {
|
||||||
|
logTrace(correlationId, `deserializeData: audio buffer length=${buffer.length}`);
|
||||||
|
return buffer;
|
||||||
|
} else if (payloadType === 'video') {
|
||||||
|
logTrace(correlationId, `deserializeData: video buffer length=${buffer.length}`);
|
||||||
|
return buffer;
|
||||||
|
} else if (payloadType === 'binary') {
|
||||||
|
logTrace(correlationId, `deserializeData: binary buffer length=${buffer.length}`);
|
||||||
|
return buffer;
|
||||||
|
} else {
|
||||||
|
throw new Error(`Unknown payload_type: ${payloadType}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------- File Server Handlers ---------------------------------------------- //
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Upload data to plik server in one-shot mode
|
||||||
|
* @param {string} fileServerUrl - Base URL of the plik server
|
||||||
|
* @param {string} dataname - Name of the file being uploaded
|
||||||
|
* @param {Uint8Array} data - Raw byte data of the file content
|
||||||
|
* @returns {Promise<{status: number, uploadid: string, fileid: string, url: string}>}
|
||||||
|
*/
|
||||||
|
async function plikOneshotUpload(fileServerUrl, dataname, data) {
|
||||||
|
const buffer = data instanceof Uint8Array ? data : new Uint8Array(data);
|
||||||
|
|
||||||
|
// Get upload id
|
||||||
|
const urlGetUploadID = `${fileServerUrl}/upload`;
|
||||||
|
const headers = { 'Content-Type': 'application/json' };
|
||||||
|
const body = JSON.stringify({ OneShot: true });
|
||||||
|
|
||||||
|
const httpResponse = await fetch(urlGetUploadID, {
|
||||||
|
method: 'POST',
|
||||||
|
headers,
|
||||||
|
body
|
||||||
|
});
|
||||||
|
|
||||||
|
const responseJson = await httpResponse.json();
|
||||||
|
const uploadid = responseJson.id;
|
||||||
|
const uploadtoken = responseJson.uploadToken;
|
||||||
|
|
||||||
|
// Upload file
|
||||||
|
const urlUpload = `${fileServerUrl}/file/${uploadid}`;
|
||||||
|
const form = new FormData();
|
||||||
|
const blob = new Blob([buffer], { type: 'application/octet-stream' });
|
||||||
|
form.append('file', blob, dataname);
|
||||||
|
|
||||||
|
const uploadHeaders = {
|
||||||
|
'X-UploadToken': uploadtoken
|
||||||
|
};
|
||||||
|
|
||||||
|
const uploadResponse = await fetch(urlUpload, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: uploadHeaders,
|
||||||
|
body: form
|
||||||
|
});
|
||||||
|
|
||||||
|
const uploadJson = await uploadResponse.json();
|
||||||
|
const fileid = uploadJson.id;
|
||||||
|
|
||||||
|
const url = `${fileServerUrl}/file/${uploadid}/${fileid}/${dataname}`;
|
||||||
|
|
||||||
|
return {
|
||||||
|
status: uploadResponse.status,
|
||||||
|
uploadid,
|
||||||
|
fileid,
|
||||||
|
url
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fetch data from URL with exponential backoff
|
||||||
|
* @param {string} url - URL to fetch from
|
||||||
|
* @param {number} maxRetries - Maximum number of retry attempts
|
||||||
|
* @param {number} baseDelay - Initial delay in milliseconds
|
||||||
|
* @param {number} maxDelay - Maximum delay in milliseconds
|
||||||
|
* @param {string} correlationId - Correlation ID for logging
|
||||||
|
* @returns {Promise<Uint8Array>} Fetched data as bytes
|
||||||
|
*/
|
||||||
|
async function fetchWithBackoff(url, maxRetries, baseDelay, maxDelay, correlationId) {
|
||||||
|
let delay = baseDelay;
|
||||||
|
|
||||||
|
for (let attempt = 1; attempt <= maxRetries; attempt++) {
|
||||||
|
try {
|
||||||
|
const response = await fetch(url);
|
||||||
|
|
||||||
|
if (response.status === 200) {
|
||||||
|
logTrace(correlationId, `Successfully fetched data from ${url} on attempt ${attempt}`);
|
||||||
|
const arrayBuffer = await response.arrayBuffer();
|
||||||
|
return new Uint8Array(arrayBuffer);
|
||||||
|
} else {
|
||||||
|
throw new Error(`Failed to fetch: ${response.status}`);
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
logTrace(correlationId, `Attempt ${attempt} failed: ${e.constructor.name} - ${e.message}`);
|
||||||
|
|
||||||
|
if (attempt < maxRetries) {
|
||||||
|
await new Promise(resolve => setTimeout(resolve, delay));
|
||||||
|
delay = Math.min(delay * 2, maxDelay);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new Error(`Failed to fetch data after ${maxRetries} attempts`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------- NATS Client ---------------------------------------------- //
|
||||||
|
|
||||||
|
/**
|
||||||
|
* NATS client wrapper for connection management
|
||||||
|
* Supports both single-use and persistent connection modes
|
||||||
|
*/
|
||||||
|
class NATSClient {
|
||||||
|
/**
|
||||||
|
* Create a new NATS client
|
||||||
|
* @param {string} url - NATS server URL (ws:// or wss://)
|
||||||
|
* @param {boolean} [keepAlive=false] - Keep connection open for multiple publishes
|
||||||
|
*/
|
||||||
|
constructor(url, keepAlive = false) {
|
||||||
|
this.url = url;
|
||||||
|
this.connection = null;
|
||||||
|
this.keepAlive = keepAlive;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Connect to NATS server
|
||||||
|
* @returns {Promise<NATS.Connection>}
|
||||||
|
*/
|
||||||
|
async connect() {
|
||||||
|
if (this.connection) {
|
||||||
|
return this.connection;
|
||||||
|
}
|
||||||
|
this.connection = await nats.connect({ servers: this.url });
|
||||||
|
return this.connection;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Publish message to NATS subject
|
||||||
|
* @param {string} subject - NATS subject to publish to
|
||||||
|
* @param {string} message - Message to publish
|
||||||
|
* @param {string} correlationId - Correlation ID for logging
|
||||||
|
*/
|
||||||
|
async publish(subject, message, correlationId) {
|
||||||
|
if (!this.connection) {
|
||||||
|
await this.connect();
|
||||||
|
}
|
||||||
|
await this.connection.publish(subject, message);
|
||||||
|
logTrace(correlationId, `Message published to ${subject}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Close the NATS connection
|
||||||
|
*/
|
||||||
|
async close() {
|
||||||
|
if (this.connection) {
|
||||||
|
this.connection.close();
|
||||||
|
this.connection = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get the current connection (for external use)
|
||||||
|
* @returns {NATS.Connection|null}
|
||||||
|
*/
|
||||||
|
getConnection() {
|
||||||
|
return this.connection;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if connected
|
||||||
|
* @returns {boolean}
|
||||||
|
*/
|
||||||
|
isConnected() {
|
||||||
|
return this.connection !== null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Connection pool for managing multiple NATS connections
|
||||||
|
* Useful for applications with multiple concurrent publishers
|
||||||
|
*/
|
||||||
|
class NATSConnectionPool {
|
||||||
|
/**
|
||||||
|
* Create a new connection pool
|
||||||
|
* @param {string} url - NATS server URL (ws:// or wss://)
|
||||||
|
* @param {number} [maxSize=10] - Maximum pool size
|
||||||
|
*/
|
||||||
|
constructor(url, maxSize = 10) {
|
||||||
|
this.url = url;
|
||||||
|
this.maxSize = maxSize;
|
||||||
|
this.connections = new Map();
|
||||||
|
this.idCounter = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get a connection from the pool (or create new)
|
||||||
|
* @returns {Promise<NATSClient>}
|
||||||
|
*/
|
||||||
|
async acquire() {
|
||||||
|
// Try to find an existing idle connection
|
||||||
|
for (const [id, client] of this.connections) {
|
||||||
|
if (client.isConnected()) {
|
||||||
|
return client;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create new connection if under limit
|
||||||
|
if (this.connections.size < this.maxSize) {
|
||||||
|
const id = `conn_${++this.idCounter}`;
|
||||||
|
const client = new NATSClient(this.url, true);
|
||||||
|
await client.connect();
|
||||||
|
this.connections.set(id, client);
|
||||||
|
return client;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Pool exhausted - create new connection (caller should close when done)
|
||||||
|
const client = new NATSClient(this.url, false);
|
||||||
|
await client.connect();
|
||||||
|
return client;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Return a connection to the pool
|
||||||
|
* @param {NATSClient} client - Connection to return
|
||||||
|
*/
|
||||||
|
release(client) {
|
||||||
|
// Only return persistent connections
|
||||||
|
if (client.keepAlive && client.isConnected()) {
|
||||||
|
// Connection already in pool, do nothing
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// Non-persistent connection - close it
|
||||||
|
client.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Close all connections in the pool
|
||||||
|
*/
|
||||||
|
async closeAll() {
|
||||||
|
for (const [id, client] of this.connections) {
|
||||||
|
await client.close();
|
||||||
|
}
|
||||||
|
this.connections.clear();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------- Core Functions ---------------------------------------------- //
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Publish message to NATS
|
||||||
|
* @param {string|NATSClient|NATS.Connection} brokerUrlOrClient - NATS URL, client, or connection
|
||||||
|
* @param {string} subject - NATS subject to publish to
|
||||||
|
* @param {string} message - JSON message to publish
|
||||||
|
* @param {string} correlationId - Correlation ID for tracing
|
||||||
|
* @param {boolean} [closeConnection=true] - Close connection after publish (set false for persistent connections)
|
||||||
|
*/
|
||||||
|
async function publishMessage(brokerUrlOrClient, subject, message, correlationId, closeConnection = true) {
|
||||||
|
let conn;
|
||||||
|
let shouldClose = false;
|
||||||
|
|
||||||
|
if (brokerUrlOrClient instanceof NATSClient) {
|
||||||
|
conn = brokerUrlOrClient;
|
||||||
|
} else if (brokerUrlOrClient && typeof brokerUrlOrClient.publish === 'function') {
|
||||||
|
// Create a wrapper for direct connection (duck-typing check for NATS connection)
|
||||||
|
conn = {
|
||||||
|
async publish(subj, msg) {
|
||||||
|
await brokerUrlOrClient.publish(subj, msg);
|
||||||
|
},
|
||||||
|
async close() {
|
||||||
|
await brokerUrlOrClient.close();
|
||||||
|
}
|
||||||
|
};
|
||||||
|
shouldClose = true;
|
||||||
|
} else {
|
||||||
|
// String URL - create new client
|
||||||
|
const client = new NATSClient(brokerUrlOrClient);
|
||||||
|
conn = client;
|
||||||
|
shouldClose = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
await conn.publish(subject, message, correlationId);
|
||||||
|
|
||||||
|
// Only close if explicitly requested and it's a short-lived client
|
||||||
|
if (shouldClose && closeConnection && conn instanceof NATSClient) {
|
||||||
|
await conn.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build message envelope from payloads and metadata
|
||||||
|
* @param {string} subject - NATS subject
|
||||||
|
* @param {Array} payloads - Array of payload objects
|
||||||
|
* @param {Object} options - Envelope metadata options
|
||||||
|
* @returns {Object} Envelope object
|
||||||
|
*/
|
||||||
|
function buildEnvelope(subject, payloads, options) {
|
||||||
|
return {
|
||||||
|
correlation_id: options.correlation_id,
|
||||||
|
msg_id: options.msg_id,
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
send_to: subject,
|
||||||
|
msg_purpose: options.msg_purpose,
|
||||||
|
sender_name: options.sender_name,
|
||||||
|
sender_id: options.sender_id,
|
||||||
|
receiver_name: options.receiver_name,
|
||||||
|
receiver_id: options.receiver_id,
|
||||||
|
reply_to: options.reply_to,
|
||||||
|
reply_to_msg_id: options.reply_to_msg_id,
|
||||||
|
broker_url: options.broker_url,
|
||||||
|
metadata: options.metadata || {},
|
||||||
|
payloads: payloads
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build payload object from serialized data
|
||||||
|
* @param {string} dataname - Name of the payload
|
||||||
|
* @param {string} payloadType - Type of the payload
|
||||||
|
* @param {Uint8Array} payloadBytes - Serialized payload bytes
|
||||||
|
* @param {string} transport - Transport type ("direct" or "link")
|
||||||
|
* @param {string} data - Data (base64 for direct, URL for link)
|
||||||
|
* @returns {Object} Payload object
|
||||||
|
*/
|
||||||
|
function buildPayload(dataname, payloadType, payloadBytes, transport, data) {
|
||||||
|
// Determine encoding based on payload type (matching Julia implementation)
|
||||||
|
let encoding = 'base64';
|
||||||
|
if (payloadType === 'jsontable') {
|
||||||
|
encoding = 'json';
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
id: uuidv4(),
|
||||||
|
dataname,
|
||||||
|
payload_type: payloadType,
|
||||||
|
transport,
|
||||||
|
encoding,
|
||||||
|
size: payloadBytes.byteLength,
|
||||||
|
data,
|
||||||
|
metadata: transport === 'direct' ? { payload_bytes: payloadBytes.byteLength } : {}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Send data via NATS with automatic transport selection
|
||||||
|
*
|
||||||
|
* This function intelligently routes data delivery based on payload size.
|
||||||
|
* If the serialized payload is smaller than size_threshold, it encodes the data as Base64
|
||||||
|
* and publishes directly over NATS. Otherwise, it uploads the data to a fileserver
|
||||||
|
* and publishes only the download URL over NATS.
|
||||||
|
*
|
||||||
|
* @param {string} subject - NATS subject to publish the message to
|
||||||
|
* @param {Array} data - List of [dataname, data, type] tuples to send
|
||||||
|
* - type: "text", "dictionary", "jsontable", "image", "audio", "video", "binary"
|
||||||
|
* - Note: "arrowtable" is NOT supported in browser (use "jsontable" for tabular data)
|
||||||
|
* @param {Object} options - Optional configuration
|
||||||
|
* @param {string} [options.broker_url=DEFAULT_BROKER_URL] - URL of the NATS server (WebSocket)
|
||||||
|
* @param {string} [options.fileserver_url=DEFAULT_FILESERVER_URL] - URL of the HTTP file server
|
||||||
|
* @param {Function} [options.fileserver_upload_handler=plikOneshotUpload] - Function to handle fileserver uploads
|
||||||
|
* @param {number} [options.size_threshold=DEFAULT_SIZE_THRESHOLD] - Threshold separating direct vs link transport
|
||||||
|
* @param {string} [options.correlation_id=uuidv4()] - Correlation ID for tracing
|
||||||
|
* @param {string} [options.msg_purpose="chat"] - Purpose of the message
|
||||||
|
* @param {string} [options.sender_name="NATSBridge"] - Name of the sender
|
||||||
|
* @param {string} [options.receiver_name=""] - Name of the receiver (empty means broadcast)
|
||||||
|
* @param {string} [options.receiver_id=""] - UUID of the receiver (empty means broadcast)
|
||||||
|
* @param {string} [options.reply_to=""] - Topic to reply to
|
||||||
|
* @param {string} [options.reply_to_msg_id=""] - Message ID this message is replying to
|
||||||
|
* @param {boolean} [options.is_publish=true] - Whether to automatically publish the message
|
||||||
|
* @param {NATSClient|NATS.Connection} [options.nats_connection=null] - Pre-existing NATS connection
|
||||||
|
* @param {string} [options.msg_id=uuidv4()] - Message ID
|
||||||
|
* @param {string} [options.sender_id=uuidv4()] - Sender ID
|
||||||
|
* @returns {Promise<[Object, string]>} Tuple of [env, env_json_str]
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* // Send a single payload
|
||||||
|
* const [env, envJsonStr] = await NATSBridgeCSR.smartsend(
|
||||||
|
* "/test",
|
||||||
|
* [["dataname1", data1, "dictionary"]],
|
||||||
|
* { broker_url: "wss://nats.example.com" }
|
||||||
|
* );
|
||||||
|
*
|
||||||
|
* // Send multiple payloads (use jsontable instead of arrowtable for browser)
|
||||||
|
* const [env, envJsonStr] = await NATSBridgeCSR.smartsend(
|
||||||
|
* "/test",
|
||||||
|
* [
|
||||||
|
* ["dataname1", data1, "dictionary"],
|
||||||
|
* ["dataname2", tableData, "jsontable"]
|
||||||
|
* ],
|
||||||
|
* { broker_url: "wss://nats.example.com" }
|
||||||
|
* );
|
||||||
|
*/
|
||||||
|
async function smartsend(subject, data, options = {}) {
|
||||||
|
const {
|
||||||
|
broker_url = DEFAULT_BROKER_URL,
|
||||||
|
fileserver_url = DEFAULT_FILESERVER_URL,
|
||||||
|
fileserver_upload_handler = plikOneshotUpload,
|
||||||
|
size_threshold = DEFAULT_SIZE_THRESHOLD,
|
||||||
|
correlation_id = uuidv4(),
|
||||||
|
msg_purpose = 'chat',
|
||||||
|
sender_name = 'NATSBridge',
|
||||||
|
receiver_name = '',
|
||||||
|
receiver_id = '',
|
||||||
|
reply_to = '',
|
||||||
|
reply_to_msg_id = '',
|
||||||
|
is_publish = true,
|
||||||
|
nats_connection = null,
|
||||||
|
msg_id = uuidv4(),
|
||||||
|
sender_id = uuidv4()
|
||||||
|
} = options;
|
||||||
|
|
||||||
|
logTrace(correlation_id, `Starting smartsend for subject: ${subject}`);
|
||||||
|
logTrace(correlation_id, `smartsend: data array length=${data.length}`);
|
||||||
|
|
||||||
|
// Debug: Log input data structure
|
||||||
|
for (let i = 0; i < data.length; i++) {
|
||||||
|
const [dataname, payloadData, payloadType] = data[i];
|
||||||
|
logTrace(correlation_id, `smartsend: payload[${i}] dataname=${dataname}, type=${payloadType}, data type=${typeof payloadData}, constructor=${payloadData?.constructor?.name}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Process payloads
|
||||||
|
const payloads = [];
|
||||||
|
for (const [dataname, payloadData, payloadType] of data) {
|
||||||
|
logTrace(correlation_id, `smartsend: Processing payload '${dataname}' type=${payloadType}`);
|
||||||
|
logTrace(correlation_id, `smartsend: payloadData type=${typeof payloadData}, constructor=${payloadData?.constructor?.name}`);
|
||||||
|
|
||||||
|
const payloadBytes = await serializeData(payloadData, payloadType);
|
||||||
|
const payloadSize = payloadBytes.byteLength;
|
||||||
|
|
||||||
|
logTrace(correlation_id, `Serialized payload '${dataname}' (type: ${payloadType}) size: ${payloadSize} bytes`);
|
||||||
|
|
||||||
|
// Debug: Show first 20 bytes of serialized data for table type
|
||||||
|
if (payloadType === 'table') {
|
||||||
|
const hexPreview = [];
|
||||||
|
for (let i = 0; i < Math.min(20, payloadBytes.length); i++) {
|
||||||
|
hexPreview.push(payloadBytes[i].toString(16).padStart(2, '0'));
|
||||||
|
}
|
||||||
|
logTrace(correlation_id, `Serialized table data first 20 bytes (hex): ${hexPreview.join(' ')}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (payloadSize < size_threshold) {
|
||||||
|
// Direct path
|
||||||
|
const payloadB64 = bufferToBase64(payloadBytes);
|
||||||
|
logTrace(correlation_id, `Using direct transport for ${payloadSize} bytes, base64 length=${payloadB64.length}`);
|
||||||
|
|
||||||
|
const payload = buildPayload(dataname, payloadType, payloadBytes, 'direct', payloadB64);
|
||||||
|
payloads.push(payload);
|
||||||
|
} else {
|
||||||
|
// Link path
|
||||||
|
logTrace(correlation_id, `Using link transport, uploading to fileserver`);
|
||||||
|
|
||||||
|
const response = await fileserver_upload_handler(fileserver_url, dataname, payloadBytes);
|
||||||
|
|
||||||
|
if (response.status !== 200) {
|
||||||
|
throw new Error(`Failed to upload data to fileserver: ${response.status}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
logTrace(correlation_id, `Uploaded to URL: ${response.url}`);
|
||||||
|
|
||||||
|
const payload = buildPayload(dataname, payloadType, payloadBytes, 'link', response.url);
|
||||||
|
payloads.push(payload);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build envelope
|
||||||
|
const env = buildEnvelope(subject, payloads, {
|
||||||
|
correlation_id,
|
||||||
|
msg_id,
|
||||||
|
msg_purpose,
|
||||||
|
sender_name,
|
||||||
|
sender_id,
|
||||||
|
receiver_name,
|
||||||
|
receiver_id,
|
||||||
|
reply_to,
|
||||||
|
reply_to_msg_id,
|
||||||
|
broker_url
|
||||||
|
});
|
||||||
|
|
||||||
|
const env_json_str = JSON.stringify(env);
|
||||||
|
|
||||||
|
if (is_publish) {
|
||||||
|
if (nats_connection) {
|
||||||
|
await publishMessage(nats_connection, subject, env_json_str, correlation_id);
|
||||||
|
} else {
|
||||||
|
await publishMessage(broker_url, subject, env_json_str, correlation_id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return [env, env_json_str];
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Receive and process NATS message
|
||||||
|
*
|
||||||
|
* This function processes incoming NATS messages, handling both direct transport
|
||||||
|
* (base64 decoded payloads) and link transport (URL-based payloads).
|
||||||
|
* It deserializes the data based on the transport type and returns the result.
|
||||||
|
*
|
||||||
|
* @param {Object} msg - NATS message object with payload property
|
||||||
|
* @param {Object} options - Optional configuration
|
||||||
|
* @param {Function} [options.fileserver_download_handler=fetchWithBackoff] - Function to handle fileserver downloads
|
||||||
|
* @param {number} [options.max_retries=5] - Maximum retry attempts for fetching URL
|
||||||
|
* @param {number} [options.base_delay=100] - Initial delay for exponential backoff in ms
|
||||||
|
* @param {number} [options.max_delay=5000] - Maximum delay for exponential backoff in ms
|
||||||
|
* @returns {Promise<Object>} Envelope object with processed payloads
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* // Receive and process message
|
||||||
|
* const env = await NATSBridgeCSR.smartreceive(msg, {
|
||||||
|
* fileserver_download_handler: NATSBridgeCSR.fetchWithBackoff,
|
||||||
|
* max_retries: 5,
|
||||||
|
* base_delay: 100,
|
||||||
|
* max_delay: 5000
|
||||||
|
* });
|
||||||
|
* // env.payloads is an Array of [dataname, data, type] arrays
|
||||||
|
* for (const [dataname, data, type] of env.payloads) {
|
||||||
|
* console.log(`${dataname}: ${data} (type: ${type})`);
|
||||||
|
* }
|
||||||
|
*/
|
||||||
|
async function smartreceive(msg, options = {}) {
|
||||||
|
const {
|
||||||
|
fileserver_download_handler = fetchWithBackoff,
|
||||||
|
max_retries = 5,
|
||||||
|
base_delay = 100,
|
||||||
|
max_delay = 5000
|
||||||
|
} = options;
|
||||||
|
|
||||||
|
// Debug: Log message object structure
|
||||||
|
logTrace('smartreceive', `smartreceive: msg object keys: ${Object.keys(msg).join(', ')}`);
|
||||||
|
logTrace('smartreceive', `smartreceive: msg.data type: ${typeof msg.data}, constructor: ${msg.data?.constructor?.name}`);
|
||||||
|
logTrace('smartreceive', `smartreceive: msg.payload type: ${typeof msg.payload}, constructor: ${msg.payload?.constructor?.name}`);
|
||||||
|
|
||||||
|
// Parse the JSON envelope
|
||||||
|
// NATS.js v2.x uses msg.data instead of msg.payload
|
||||||
|
let payload;
|
||||||
|
if (msg.data !== undefined) {
|
||||||
|
payload = typeof msg.data === 'string' ? msg.data : new TextDecoder().decode(msg.data);
|
||||||
|
} else if (msg.payload !== undefined) {
|
||||||
|
payload = typeof msg.payload === 'string' ? msg.payload : new TextDecoder().decode(msg.payload);
|
||||||
|
} else {
|
||||||
|
throw new Error('Message has neither data nor payload property');
|
||||||
|
}
|
||||||
|
|
||||||
|
logTrace('smartreceive', `smartreceive: raw payload length=${payload.length}`);
|
||||||
|
|
||||||
|
// Debug: Show first 200 chars of payload
|
||||||
|
const payloadPreview = payload.substring(0, 200);
|
||||||
|
logTrace('smartreceive', `smartreceive: payload preview: ${payloadPreview}`);
|
||||||
|
|
||||||
|
let envJsonObj;
|
||||||
|
try {
|
||||||
|
envJsonObj = JSON.parse(payload);
|
||||||
|
} catch (e) {
|
||||||
|
logTrace('smartreceive', `smartreceive: JSON parse failed: ${e.message}`);
|
||||||
|
throw e;
|
||||||
|
}
|
||||||
|
|
||||||
|
logTrace(envJsonObj.correlation_id, 'Processing received message');
|
||||||
|
logTrace(envJsonObj.correlation_id, `smartreceive: envelope has ${envJsonObj.payloads.length} payloads`);
|
||||||
|
|
||||||
|
// Process all payloads in the envelope
|
||||||
|
const payloadsList = [];
|
||||||
|
const numPayloads = envJsonObj.payloads.length;
|
||||||
|
|
||||||
|
logTrace(envJsonObj.correlation_id, `smartreceive: Processing ${numPayloads} payloads`);
|
||||||
|
|
||||||
|
for (let i = 0; i < numPayloads; i++) {
|
||||||
|
const payloadObj = envJsonObj.payloads[i];
|
||||||
|
const transport = payloadObj.transport;
|
||||||
|
const dataname = payloadObj.dataname;
|
||||||
|
const payloadType = payloadObj.payload_type;
|
||||||
|
|
||||||
|
logTrace(envJsonObj.correlation_id, `smartreceive: Processing payload ${i + 1}/${numPayloads}: dataname=${dataname}, type=${payloadType}, transport=${transport}`);
|
||||||
|
|
||||||
|
if (transport === 'direct') {
|
||||||
|
logTrace(envJsonObj.correlation_id, `Direct transport - decoding payload '${dataname}'`);
|
||||||
|
|
||||||
|
// Extract base64 payload from the payload
|
||||||
|
const payloadB64 = payloadObj.data;
|
||||||
|
logTrace(envJsonObj.correlation_id, `Direct transport: base64 length=${payloadB64?.length}`);
|
||||||
|
|
||||||
|
// Decode Base64 payload
|
||||||
|
const payloadBytes = base64ToBuffer(payloadB64);
|
||||||
|
logTrace(envJsonObj.correlation_id, `Direct transport: decoded bytes=${payloadBytes.length}`);
|
||||||
|
|
||||||
|
// Deserialize based on type
|
||||||
|
const dataType = payloadObj.payload_type;
|
||||||
|
const data = await deserializeData(payloadBytes, dataType, envJsonObj.correlation_id);
|
||||||
|
logTrace(envJsonObj.correlation_id, `Direct transport: deserialized data type=${typeof data}, constructor=${data?.constructor?.name}`);
|
||||||
|
|
||||||
|
payloadsList.push([dataname, data, dataType]);
|
||||||
|
} else if (transport === 'link') {
|
||||||
|
// Extract download URL from the payload
|
||||||
|
const url = payloadObj.data;
|
||||||
|
logTrace(envJsonObj.correlation_id, `Link transport - fetching '${dataname}' from URL: ${url}`);
|
||||||
|
|
||||||
|
// Fetch with exponential backoff using the download handler
|
||||||
|
const downloadedData = await fileserver_download_handler(
|
||||||
|
url,
|
||||||
|
max_retries,
|
||||||
|
base_delay,
|
||||||
|
max_delay,
|
||||||
|
envJsonObj.correlation_id
|
||||||
|
);
|
||||||
|
|
||||||
|
// Deserialize based on type
|
||||||
|
const dataType = payloadObj.payload_type;
|
||||||
|
const data = await deserializeData(downloadedData, dataType, envJsonObj.correlation_id);
|
||||||
|
|
||||||
|
payloadsList.push([dataname, data, dataType]);
|
||||||
|
} else {
|
||||||
|
throw new Error(`Unknown transport type for payload '${dataname}': ${transport}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
logTrace(envJsonObj.correlation_id, `smartreceive: Successfully processed all ${payloadsList.length} payloads`);
|
||||||
|
envJsonObj.payloads = payloadsList;
|
||||||
|
return envJsonObj;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------- Module Exports ---------------------------------------------- //
|
||||||
|
|
||||||
|
const NATSBridgeCSR = {
|
||||||
|
/**
|
||||||
|
* NATS client class for connection management
|
||||||
|
* Supports both single-use and persistent connection modes
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* // Single-use connection (closes after publish)
|
||||||
|
* const client = new NATSBridgeCSR.NATSClient("wss://nats.example.com");
|
||||||
|
* await NATSBridgeCSR.smartsend("/test", [["msg", "Hello", "text"]], { nats_connection: client });
|
||||||
|
* await client.close();
|
||||||
|
*
|
||||||
|
* // Persistent connection (keeps connection open)
|
||||||
|
* const client = new NATSBridgeCSR.NATSClient("wss://nats.example.com", true);
|
||||||
|
* await client.connect();
|
||||||
|
* await NATSBridgeCSR.smartsend("/test1", [["msg", "Hello", "text"]], { nats_connection: client, is_publish: false });
|
||||||
|
* await NATSBridgeCSR.publishMessage(client, "/test2", JSON.stringify({msg: "World"}), "trace-id");
|
||||||
|
* // Connection remains open for more publishes
|
||||||
|
* await client.close();
|
||||||
|
*/
|
||||||
|
NATSClient,
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Connection pool for managing multiple NATS connections
|
||||||
|
* Useful for applications with multiple concurrent publishers
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* const pool = new NATSBridgeCSR.NATSConnectionPool("wss://nats.example.com", 10);
|
||||||
|
* const client = await pool.acquire();
|
||||||
|
* await NATSBridgeCSR.smartsend("/test", [["msg", "Hello", "text"]], { nats_connection: client });
|
||||||
|
* pool.release(client);
|
||||||
|
* await pool.closeAll();
|
||||||
|
*/
|
||||||
|
NATSConnectionPool,
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Send data via NATS with automatic transport selection
|
||||||
|
*/
|
||||||
|
smartsend,
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Receive and process NATS message
|
||||||
|
*/
|
||||||
|
smartreceive,
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Publish message to NATS
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* // Using a persistent connection
|
||||||
|
* const client = new NATSBridgeCSR.NATSClient("wss://nats.example.com", true);
|
||||||
|
* await client.connect();
|
||||||
|
* await NATSBridgeCSR.publishMessage(client, "/subject", JSON.stringify({msg: "Hello"}), "trace-id", false);
|
||||||
|
* // Connection stays open for more publishes
|
||||||
|
* await client.close();
|
||||||
|
*/
|
||||||
|
publishMessage,
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Upload data to plik server in one-shot mode
|
||||||
|
*/
|
||||||
|
plikOneshotUpload,
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fetch data from URL with exponential backoff
|
||||||
|
*/
|
||||||
|
fetchWithBackoff,
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Default constants
|
||||||
|
*/
|
||||||
|
DEFAULT_SIZE_THRESHOLD,
|
||||||
|
DEFAULT_BROKER_URL,
|
||||||
|
DEFAULT_FILESERVER_URL
|
||||||
|
};
|
||||||
|
|
||||||
|
export default NATSBridgeCSR;
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
/**
|
/**
|
||||||
* NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
* NATSBridge - Cross-Platform Bi-Directional Data Bridge
|
||||||
* JavaScript/Node.js Implementation
|
* JavaScript/Node.js Implementation (Desktop/Server-Side)
|
||||||
*
|
*
|
||||||
* This module provides functionality for sending and receiving data across network boundaries
|
* This module provides functionality for sending and receiving data across network boundaries
|
||||||
* using NATS as the message bus, with support for both direct payload transport and
|
* using NATS as the message bus, with support for both direct payload transport and
|
||||||
@@ -8,6 +8,12 @@
|
|||||||
*
|
*
|
||||||
* Supported payload types: "text", "dictionary", "arrowtable", "jsontable", "image", "audio", "video", "binary"
|
* Supported payload types: "text", "dictionary", "arrowtable", "jsontable", "image", "audio", "video", "binary"
|
||||||
*
|
*
|
||||||
|
* Node.js-specific features:
|
||||||
|
* - Apache Arrow IPC support via apache-arrow
|
||||||
|
* - TCP NATS connections (nats:// or tls:// URLs)
|
||||||
|
* - Buffer for binary data handling
|
||||||
|
* - Connection pooling for high-throughput scenarios
|
||||||
|
*
|
||||||
* @module NATSBridge
|
* @module NATSBridge
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@@ -16,12 +22,22 @@ const crypto = require('crypto');
|
|||||||
// Use native fetch available in Node.js 18+
|
// Use native fetch available in Node.js 18+
|
||||||
const arrow = require('apache-arrow');
|
const arrow = require('apache-arrow');
|
||||||
|
|
||||||
|
// ---------------------------------------------- UUID Helper ---------------------------------------------- //
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Generate UUID v4 using crypto module (Node.js compatible)
|
||||||
|
* @returns {string} UUID string
|
||||||
|
*/
|
||||||
|
function uuidv4() {
|
||||||
|
return crypto.randomUUID();
|
||||||
|
}
|
||||||
|
|
||||||
// ---------------------------------------------- Constants ---------------------------------------------- //
|
// ---------------------------------------------- Constants ---------------------------------------------- //
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Default size threshold for switching from direct to link transport (1MB)
|
* Default size threshold for switching from direct to link transport (0.5MB)
|
||||||
*/
|
*/
|
||||||
const DEFAULT_SIZE_THRESHOLD = 1_000_000;
|
const DEFAULT_SIZE_THRESHOLD = 500_000;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Default NATS server URL
|
* Default NATS server URL
|
||||||
@@ -332,15 +348,18 @@ async function fetchWithBackoff(url, maxRetries, baseDelay, maxDelay, correlatio
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* NATS client wrapper for connection management
|
* NATS client wrapper for connection management
|
||||||
|
* Supports both single-use and persistent connection modes
|
||||||
*/
|
*/
|
||||||
class NATSClient {
|
class NATSClient {
|
||||||
/**
|
/**
|
||||||
* Create a new NATS client
|
* Create a new NATS client
|
||||||
* @param {string} url - NATS server URL
|
* @param {string} url - NATS server URL (nats:// or tls://)
|
||||||
|
* @param {boolean} [keepAlive=false] - Keep connection open for multiple publishes
|
||||||
*/
|
*/
|
||||||
constructor(url) {
|
constructor(url, keepAlive = false) {
|
||||||
this.url = url;
|
this.url = url;
|
||||||
this.connection = null;
|
this.connection = null;
|
||||||
|
this.keepAlive = keepAlive;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -348,6 +367,9 @@ class NATSClient {
|
|||||||
* @returns {Promise<NATS.Connection>}
|
* @returns {Promise<NATS.Connection>}
|
||||||
*/
|
*/
|
||||||
async connect() {
|
async connect() {
|
||||||
|
if (this.connection) {
|
||||||
|
return this.connection;
|
||||||
|
}
|
||||||
this.connection = await nats.connect({ servers: this.url });
|
this.connection = await nats.connect({ servers: this.url });
|
||||||
return this.connection;
|
return this.connection;
|
||||||
}
|
}
|
||||||
@@ -372,8 +394,94 @@ class NATSClient {
|
|||||||
async close() {
|
async close() {
|
||||||
if (this.connection) {
|
if (this.connection) {
|
||||||
this.connection.close();
|
this.connection.close();
|
||||||
|
this.connection = null;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get the current connection (for external use)
|
||||||
|
* @returns {NATS.Connection|null}
|
||||||
|
*/
|
||||||
|
getConnection() {
|
||||||
|
return this.connection;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if connected
|
||||||
|
* @returns {boolean}
|
||||||
|
*/
|
||||||
|
isConnected() {
|
||||||
|
return this.connection !== null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Connection pool for managing multiple NATS connections
|
||||||
|
* Useful for applications with multiple concurrent publishers
|
||||||
|
*/
|
||||||
|
class NATSConnectionPool {
|
||||||
|
/**
|
||||||
|
* Create a new connection pool
|
||||||
|
* @param {string} url - NATS server URL (nats:// or tls://)
|
||||||
|
* @param {number} [maxSize=10] - Maximum pool size
|
||||||
|
*/
|
||||||
|
constructor(url, maxSize = 10) {
|
||||||
|
this.url = url;
|
||||||
|
this.maxSize = maxSize;
|
||||||
|
this.connections = new Map();
|
||||||
|
this.idCounter = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get a connection from the pool (or create new)
|
||||||
|
* @returns {Promise<NATSClient>}
|
||||||
|
*/
|
||||||
|
async acquire() {
|
||||||
|
// Try to find an existing idle connection
|
||||||
|
for (const [id, client] of this.connections) {
|
||||||
|
if (client.isConnected()) {
|
||||||
|
return client;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create new connection if under limit
|
||||||
|
if (this.connections.size < this.maxSize) {
|
||||||
|
const id = `conn_${++this.idCounter}`;
|
||||||
|
const client = new NATSClient(this.url, true);
|
||||||
|
await client.connect();
|
||||||
|
this.connections.set(id, client);
|
||||||
|
return client;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Pool exhausted - create new connection (caller should close when done)
|
||||||
|
const client = new NATSClient(this.url, false);
|
||||||
|
await client.connect();
|
||||||
|
return client;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Return a connection to the pool
|
||||||
|
* @param {NATSClient} client - Connection to return
|
||||||
|
*/
|
||||||
|
release(client) {
|
||||||
|
// Only return persistent connections
|
||||||
|
if (client.keepAlive && client.isConnected()) {
|
||||||
|
// Connection already in pool, do nothing
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// Non-persistent connection - close it
|
||||||
|
client.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Close all connections in the pool
|
||||||
|
*/
|
||||||
|
async closeAll() {
|
||||||
|
for (const [id, client] of this.connections) {
|
||||||
|
await client.close();
|
||||||
|
}
|
||||||
|
this.connections.clear();
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------- Core Functions ---------------------------------------------- //
|
// ---------------------------------------------- Core Functions ---------------------------------------------- //
|
||||||
@@ -384,9 +492,11 @@ class NATSClient {
|
|||||||
* @param {string} subject - NATS subject to publish to
|
* @param {string} subject - NATS subject to publish to
|
||||||
* @param {string} message - JSON message to publish
|
* @param {string} message - JSON message to publish
|
||||||
* @param {string} correlationId - Correlation ID for tracing
|
* @param {string} correlationId - Correlation ID for tracing
|
||||||
|
* @param {boolean} [closeConnection=true] - Close connection after publish (set false for persistent connections)
|
||||||
*/
|
*/
|
||||||
async function publishMessage(brokerUrlOrClient, subject, message, correlationId) {
|
async function publishMessage(brokerUrlOrClient, subject, message, correlationId, closeConnection = true) {
|
||||||
let conn;
|
let conn;
|
||||||
|
let shouldClose = false;
|
||||||
|
|
||||||
if (brokerUrlOrClient instanceof NATSClient) {
|
if (brokerUrlOrClient instanceof NATSClient) {
|
||||||
conn = brokerUrlOrClient;
|
conn = brokerUrlOrClient;
|
||||||
@@ -400,15 +510,18 @@ async function publishMessage(brokerUrlOrClient, subject, message, correlationId
|
|||||||
await brokerUrlOrClient.close();
|
await brokerUrlOrClient.close();
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
shouldClose = true;
|
||||||
} else {
|
} else {
|
||||||
// String URL - create new client
|
// String URL - create new client
|
||||||
const client = new NATSClient(brokerUrlOrClient);
|
const client = new NATSClient(brokerUrlOrClient);
|
||||||
conn = client;
|
conn = client;
|
||||||
|
shouldClose = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
await conn.publish(subject, message, correlationId);
|
await conn.publish(subject, message, correlationId);
|
||||||
|
|
||||||
if (conn instanceof NATSClient) {
|
// Only close if explicitly requested and it's a short-lived client
|
||||||
|
if (shouldClose && closeConnection && conn instanceof NATSClient) {
|
||||||
await conn.close();
|
await conn.close();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -458,7 +571,7 @@ function buildPayload(dataname, payloadType, payloadBytes, transport, data) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
id: crypto.randomUUID(),
|
id: uuidv4(),
|
||||||
dataname,
|
dataname,
|
||||||
payload_type: payloadType,
|
payload_type: payloadType,
|
||||||
transport,
|
transport,
|
||||||
@@ -530,7 +643,7 @@ async function smartsend(subject, data, options = {}) {
|
|||||||
fileserver_url = DEFAULT_FILESERVER_URL,
|
fileserver_url = DEFAULT_FILESERVER_URL,
|
||||||
fileserver_upload_handler = plikOneshotUpload,
|
fileserver_upload_handler = plikOneshotUpload,
|
||||||
size_threshold = DEFAULT_SIZE_THRESHOLD,
|
size_threshold = DEFAULT_SIZE_THRESHOLD,
|
||||||
correlation_id = crypto.randomUUID(),
|
correlation_id = uuidv4(),
|
||||||
msg_purpose = 'chat',
|
msg_purpose = 'chat',
|
||||||
sender_name = 'NATSBridge',
|
sender_name = 'NATSBridge',
|
||||||
receiver_name = '',
|
receiver_name = '',
|
||||||
@@ -539,8 +652,8 @@ async function smartsend(subject, data, options = {}) {
|
|||||||
reply_to_msg_id = '',
|
reply_to_msg_id = '',
|
||||||
is_publish = true,
|
is_publish = true,
|
||||||
nats_connection = null,
|
nats_connection = null,
|
||||||
msg_id = crypto.randomUUID(),
|
msg_id = uuidv4(),
|
||||||
sender_id = crypto.randomUUID()
|
sender_id = uuidv4()
|
||||||
} = options;
|
} = options;
|
||||||
|
|
||||||
logTrace(correlation_id, `Starting smartsend for subject: ${subject}`);
|
logTrace(correlation_id, `Starting smartsend for subject: ${subject}`);
|
||||||
@@ -754,9 +867,37 @@ async function smartreceive(msg, options = {}) {
|
|||||||
const NATSBridge = {
|
const NATSBridge = {
|
||||||
/**
|
/**
|
||||||
* NATS client class for connection management
|
* NATS client class for connection management
|
||||||
|
* Supports both single-use and persistent connection modes
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* // Single-use connection (closes after publish)
|
||||||
|
* const client = new NATSBridge.NATSClient("nats://localhost:4222");
|
||||||
|
* await NATSBridge.smartsend("/test", [["msg", "Hello", "text"]], { nats_connection: client });
|
||||||
|
* await client.close();
|
||||||
|
*
|
||||||
|
* // Persistent connection (keeps connection open)
|
||||||
|
* const client = new NATSBridge.NATSClient("nats://localhost:4222", true);
|
||||||
|
* await client.connect();
|
||||||
|
* await NATSBridge.smartsend("/test1", [["msg", "Hello", "text"]], { nats_connection: client, is_publish: false });
|
||||||
|
* await NATSBridge.publishMessage(client, "/test2", JSON.stringify({msg: "World"}), "trace-id");
|
||||||
|
* // Connection remains open for more publishes
|
||||||
|
* await client.close();
|
||||||
*/
|
*/
|
||||||
NATSClient,
|
NATSClient,
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Connection pool for managing multiple NATS connections
|
||||||
|
* Useful for applications with multiple concurrent publishers
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* const pool = new NATSBridge.NATSConnectionPool("nats://localhost:4222", 10);
|
||||||
|
* const client = await pool.acquire();
|
||||||
|
* await NATSBridge.smartsend("/test", [["msg", "Hello", "text"]], { nats_connection: client });
|
||||||
|
* pool.release(client);
|
||||||
|
* await pool.closeAll();
|
||||||
|
*/
|
||||||
|
NATSConnectionPool,
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Send data via NATS with automatic transport selection
|
* Send data via NATS with automatic transport selection
|
||||||
*/
|
*/
|
||||||
@@ -767,6 +908,19 @@ const NATSBridge = {
|
|||||||
*/
|
*/
|
||||||
smartreceive,
|
smartreceive,
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Publish message to NATS
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* // Using a persistent connection
|
||||||
|
* const client = new NATSBridge.NATSClient("nats://localhost:4222", true);
|
||||||
|
* await client.connect();
|
||||||
|
* await NATSBridge.publishMessage(client, "/subject", JSON.stringify({msg: "Hello"}), "trace-id", false);
|
||||||
|
* // Connection stays open for more publishes
|
||||||
|
* await client.close();
|
||||||
|
*/
|
||||||
|
publishMessage,
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Upload data to plik server in one-shot mode
|
* Upload data to plik server in one-shot mode
|
||||||
*/
|
*/
|
||||||
74
test/test_dart_mix_payloads_receiver.dart
Normal file
74
test/test_dart_mix_payloads_receiver.dart
Normal file
@@ -0,0 +1,74 @@
|
|||||||
|
/// Dart Mix Payloads Receiver Test
|
||||||
|
/// Tests the smartreceive function with mixed payload types
|
||||||
|
///
|
||||||
|
/// This test mirrors test_julia_mix_payloads_receiver.jl and test_js_mix_payloads_receiver.js
|
||||||
|
/// and demonstrates that any combination and any number of mixed content can be received correctly.
|
||||||
|
|
||||||
|
import 'dart:io';
|
||||||
|
import 'package:http/http.dart' as http;
|
||||||
|
import 'package:uuid/uuid.dart';
|
||||||
|
|
||||||
|
// Add parent directory to path
|
||||||
|
import 'package:natsbridge/natsbridge.dart' as natsbridge;
|
||||||
|
|
||||||
|
const TEST_SUBJECT = '/natsbridge';
|
||||||
|
const TEST_BROKER_URL = String.fromEnvironment(
|
||||||
|
'NATS_URL',
|
||||||
|
defaultValue: 'nats.yiem.cc',
|
||||||
|
);
|
||||||
|
const TEST_FILESERVER_URL = String.fromEnvironment(
|
||||||
|
'FILESERVER_URL',
|
||||||
|
defaultValue: 'http://192.168.88.104:8080',
|
||||||
|
);
|
||||||
|
|
||||||
|
void logTrace(String message) {
|
||||||
|
final timestamp = DateTime.now().toUtc().toIsoString();
|
||||||
|
print('[$timestamp] [Correlation: $correlationId] $message');
|
||||||
|
}
|
||||||
|
|
||||||
|
Future<void> runTest() async {
|
||||||
|
print('=== Dart Mix Payloads Receiver Test ===\n');
|
||||||
|
|
||||||
|
final uuid = const Uuid();
|
||||||
|
final correlationId = uuid.v4();
|
||||||
|
print('Correlation ID: $correlationId');
|
||||||
|
print('Subject: $TEST_SUBJECT');
|
||||||
|
print('Broker URL: $TEST_BROKER_URL');
|
||||||
|
print('Fileserver URL: $TEST_FILESERVER_URL\n');
|
||||||
|
|
||||||
|
bool testPassed = true;
|
||||||
|
int messagesReceived = 0;
|
||||||
|
final receivedPayloads = [];
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Note: This is a receiver test that waits for messages
|
||||||
|
// You need to run the sender test first: dart test/test_dart_mix_payloads_sender.dart
|
||||||
|
|
||||||
|
print('This receiver test requires a running NATS server and a message sender.');
|
||||||
|
print('\nTo run this test:');
|
||||||
|
print('1. Start NATS server: nats-server');
|
||||||
|
print('2. Run sender: dart test/test_dart_mix_payloads_sender.dart');
|
||||||
|
print('3. This receiver will wait for messages on subject: $TEST_SUBJECT\n');
|
||||||
|
|
||||||
|
print('Waiting for messages (timeout: 180 seconds)...');
|
||||||
|
|
||||||
|
// For now, just print a message about how to run the test
|
||||||
|
// In a real implementation, you would connect to NATS and subscribe to messages
|
||||||
|
print('\n=== Test Instructions ===');
|
||||||
|
print('1. Start NATS server: nats-server');
|
||||||
|
print('2. Run sender: dart test/test_dart_mix_payloads_sender.dart');
|
||||||
|
print('3. This receiver will wait for messages\n');
|
||||||
|
|
||||||
|
print('Test completed. This is a receiver test that waits for messages from the sender.');
|
||||||
|
print('Run the sender test first to send messages to this receiver.');
|
||||||
|
|
||||||
|
} catch (error) {
|
||||||
|
print('\n❌ Test failed with error: $error');
|
||||||
|
print('$error');
|
||||||
|
exit(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void main() {
|
||||||
|
runTest();
|
||||||
|
}
|
||||||
230
test/test_dart_mix_payloads_sender.dart
Normal file
230
test/test_dart_mix_payloads_sender.dart
Normal file
@@ -0,0 +1,230 @@
|
|||||||
|
/// Dart Mix Payloads Sender Test
|
||||||
|
/// Tests the smartsend function with mixed payload types
|
||||||
|
///
|
||||||
|
/// This test mirrors test_julia_mix_payloads_sender.jl and test_js_mix_payloads_sender.js
|
||||||
|
/// and demonstrates that any combination and any number of mixed content can be sent correctly.
|
||||||
|
|
||||||
|
import 'dart:io';
|
||||||
|
import 'dart:typed_data';
|
||||||
|
import 'package:http/http.dart' as http;
|
||||||
|
import 'package:uuid/uuid.dart';
|
||||||
|
|
||||||
|
// Add parent directory to path
|
||||||
|
import 'package:natsbridge/natsbridge.dart' as natsbridge;
|
||||||
|
|
||||||
|
const TEST_SUBJECT = '/natsbridge';
|
||||||
|
const TEST_BROKER_URL = String.fromEnvironment(
|
||||||
|
'NATS_URL',
|
||||||
|
defaultValue: 'nats.yiem.cc',
|
||||||
|
);
|
||||||
|
const TEST_FILESERVER_URL = String.fromEnvironment(
|
||||||
|
'FILESERVER_URL',
|
||||||
|
defaultValue: 'http://192.168.88.104:8080',
|
||||||
|
);
|
||||||
|
const SIZE_THRESHOLD = 1000000; // 1MB threshold
|
||||||
|
|
||||||
|
void logTrace(String message) {
|
||||||
|
final timestamp = DateTime.now().toUtc().toIsoString();
|
||||||
|
print('[$timestamp] [Correlation: $correlationId] $message');
|
||||||
|
}
|
||||||
|
|
||||||
|
Future<void> runTest() async {
|
||||||
|
print('=== Dart Mix Payloads Sender Test ===\n');
|
||||||
|
|
||||||
|
final uuid = const Uuid();
|
||||||
|
final correlationId = uuid.v4();
|
||||||
|
print('Correlation ID: $correlationId');
|
||||||
|
print('Subject: $TEST_SUBJECT');
|
||||||
|
print('Broker URL: $TEST_BROKER_URL');
|
||||||
|
print('Fileserver URL: $TEST_FILESERVER_URL');
|
||||||
|
print('Size Threshold: $SIZE_THRESHOLD bytes (1MB)\n');
|
||||||
|
|
||||||
|
// Create sample data for each type (mirroring Julia test)
|
||||||
|
final textData = 'Hello! This is a test chat message. 🎉\nHow are you doing today? 😊';
|
||||||
|
|
||||||
|
final dictData = {
|
||||||
|
'type': 'chat',
|
||||||
|
'sender': 'serviceA',
|
||||||
|
'receiver': 'serviceB',
|
||||||
|
'metadata': {
|
||||||
|
'timestamp': DateTime.now().toUtc().toIsoString(),
|
||||||
|
'priority': 'high',
|
||||||
|
'tags': ['urgent', 'chat', 'test']
|
||||||
|
},
|
||||||
|
'content': {
|
||||||
|
'text': 'This is a JSON-formatted chat message with nested structure.',
|
||||||
|
'format': 'markdown',
|
||||||
|
'mentions': ['user1', 'user2']
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Arrow table data (small - direct transport)
|
||||||
|
final arrowTableSmall = [
|
||||||
|
{'id': 1, 'name': 'Alice', 'score': 95, 'active': true},
|
||||||
|
{'id': 2, 'name': 'Bob', 'score': 88, 'active': false},
|
||||||
|
{'id': 3, 'name': 'Charlie', 'score': 92, 'active': true},
|
||||||
|
{'id': 4, 'name': 'Diana', 'score': 78, 'active': true},
|
||||||
|
{'id': 5, 'name': 'Eve', 'score': 85, 'active': false},
|
||||||
|
{'id': 6, 'name': 'Frank', 'score': 91, 'active': true},
|
||||||
|
{'id': 7, 'name': 'Grace', 'score': 89, 'active': true},
|
||||||
|
{'id': 8, 'name': 'Henry', 'score': 76, 'active': false},
|
||||||
|
{'id': 9, 'name': 'Ivy', 'score': 94, 'active': true},
|
||||||
|
{'id': 10, 'name': 'Jack', 'score': 82, 'active': true}
|
||||||
|
];
|
||||||
|
|
||||||
|
// Json table data (small - direct transport)
|
||||||
|
final jsonTableSmall = [
|
||||||
|
{'id': 1, 'name': 'Alice', 'score': 95, 'active': true},
|
||||||
|
{'id': 2, 'name': 'Bob', 'score': 88, 'active': false},
|
||||||
|
{'id': 3, 'name': 'Charlie', 'score': 92, 'active': true},
|
||||||
|
{'id': 4, 'name': 'Diana', 'score': 78, 'active': true},
|
||||||
|
{'id': 5, 'name': 'Eve', 'score': 85, 'active': false},
|
||||||
|
{'id': 6, 'name': 'Frank', 'score': 91, 'active': true},
|
||||||
|
{'id': 7, 'name': 'Grace', 'score': 89, 'active': true},
|
||||||
|
{'id': 8, 'name': 'Henry', 'score': 76, 'active': false},
|
||||||
|
{'id': 9, 'name': 'Ivy', 'score': 94, 'active': true},
|
||||||
|
{'id': 10, 'name': 'Jack', 'score': 82, 'active': true}
|
||||||
|
];
|
||||||
|
|
||||||
|
// Audio data (small binary - direct transport)
|
||||||
|
final audioData = Uint8List(100);
|
||||||
|
for (int i = 0; i < 100; i++) {
|
||||||
|
audioData[i] = (Random().nextRange(1, 255)).toInt();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Video data (small binary - direct transport)
|
||||||
|
final videoData = Uint8List(150);
|
||||||
|
for (int i = 0; i < 150; i++) {
|
||||||
|
videoData[i] = (Random().nextRange(1, 255)).toInt();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Binary data (small - direct transport)
|
||||||
|
final binaryData = Uint8List(200);
|
||||||
|
for (int i = 0; i < 200; i++) {
|
||||||
|
binaryData[i] = (Random().nextRange(1, 255)).toInt();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Large data for link transport testing
|
||||||
|
final largeArrowTable = List.generate(20000, (i) => {
|
||||||
|
'id': i + 1,
|
||||||
|
'name': 'user_${i + 1}',
|
||||||
|
'score': (Random().nextRange(50, 100)).toInt(),
|
||||||
|
'active': Random().nextBool(),
|
||||||
|
'timestamp': DateTime.now().toUtc().toIsoString()
|
||||||
|
});
|
||||||
|
|
||||||
|
final largeJsonTable = List.generate(50000, (i) => {
|
||||||
|
'id': i + 1,
|
||||||
|
'name': 'user_${i + 1}',
|
||||||
|
'score': (Random().nextRange(50, 100)).toInt(),
|
||||||
|
'active': Random().nextBool()
|
||||||
|
});
|
||||||
|
|
||||||
|
final largeAudioData = Uint8List(1500000);
|
||||||
|
for (int i = 0; i < 1500000; i++) {
|
||||||
|
largeAudioData[i] = (Random().nextRange(1, 255)).toInt();
|
||||||
|
}
|
||||||
|
|
||||||
|
final largeVideoData = Uint8List(1500000);
|
||||||
|
for (int i = 0; i < 1500000; i++) {
|
||||||
|
largeVideoData[i] = (Random().nextRange(1, 255)).toInt();
|
||||||
|
}
|
||||||
|
|
||||||
|
final largeBinaryData = Uint8List(1500000);
|
||||||
|
for (int i = 0; i < 1500000; i++) {
|
||||||
|
largeBinaryData[i] = (Random().nextRange(1, 255)).toInt();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read image files from disk (following Julia test pattern)
|
||||||
|
final filePathSmallImage = './test/small_image.jpg';
|
||||||
|
final fileDataSmallImage = File(filePathSmallImage).readAsBytesSync();
|
||||||
|
final filenameSmallImage = filePathSmallImage.split('/').last;
|
||||||
|
|
||||||
|
final filePathLargeImage = './test/large_image.png';
|
||||||
|
final fileDataLargeImage = File(filePathLargeImage).readAsBytesSync();
|
||||||
|
final filenameLargeImage = filePathLargeImage.split('/').last;
|
||||||
|
|
||||||
|
logTrace('Creating payloads list with mixed content');
|
||||||
|
|
||||||
|
// Create payloads list - mixed content with both small and large data
|
||||||
|
// Small data uses direct transport, large data uses link transport
|
||||||
|
final payloads = [
|
||||||
|
// Small data (direct transport) - text, dictionary, arrowtable, jsontable, small image
|
||||||
|
['chat_text', textData, 'text'],
|
||||||
|
['chat_json', dictData, 'dictionary'],
|
||||||
|
// ['arrow_table_small', arrowTableSmall, 'arrowtable'],
|
||||||
|
['json_table_small', jsonTableSmall, 'jsontable'],
|
||||||
|
[filenameSmallImage, fileDataSmallImage, 'binary'],
|
||||||
|
|
||||||
|
// Large data (link transport) - large arrowtable, large jsontable, large image, large audio, large video, large binary
|
||||||
|
// ['arrow_table_large', largeArrowTable, 'arrowtable'],
|
||||||
|
['json_table_large', largeJsonTable, 'jsontable'],
|
||||||
|
[filenameLargeImage, fileDataLargeImage, 'binary'],
|
||||||
|
// ['audio_clip_large', largeAudioData, 'audio'],
|
||||||
|
// ['video_clip_large', largeVideoData, 'video'],
|
||||||
|
// ['binary_file_large', largeBinaryData, 'binary']
|
||||||
|
];
|
||||||
|
|
||||||
|
logTrace('Total payloads: ${payloads.length}');
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Send the message
|
||||||
|
print('Sending mixed payloads...\n');
|
||||||
|
final (env, envJsonStr) = await natsbridge.smartsend(
|
||||||
|
TEST_SUBJECT,
|
||||||
|
payloads,
|
||||||
|
brokerUrl: TEST_BROKER_URL,
|
||||||
|
fileserverUrl: TEST_FILESERVER_URL,
|
||||||
|
fileserverUploadHandler: natsbridge.plikOneshotUpload,
|
||||||
|
sizeThreshold: SIZE_THRESHOLD,
|
||||||
|
correlationId: correlationId,
|
||||||
|
msgPurpose: 'chat',
|
||||||
|
senderName: 'dart-mix-test',
|
||||||
|
receiverName: '',
|
||||||
|
receiverId: '',
|
||||||
|
replyTo: '',
|
||||||
|
replyToMsgId: '',
|
||||||
|
isPublish: true,
|
||||||
|
);
|
||||||
|
|
||||||
|
print('\n=== Envelope Created ===');
|
||||||
|
print('Correlation ID: ${env['correlation_id']}');
|
||||||
|
print('Message ID: ${env['msg_id']}');
|
||||||
|
print('Timestamp: ${env['timestamp']}');
|
||||||
|
print('Subject: ${env['send_to']}');
|
||||||
|
print('Purpose: ${env['msg_purpose']}');
|
||||||
|
print('Sender: ${env['sender_name']}');
|
||||||
|
print('Payloads: ${env['payloads'].length}\n');
|
||||||
|
|
||||||
|
// Log transport type for each payload
|
||||||
|
for (int i = 0; i < env['payloads'].length; i++) {
|
||||||
|
final payload = env['payloads'][i];
|
||||||
|
logTrace('Payload ${i + 1} (${payload['dataname']}):');
|
||||||
|
logTrace(' Transport: ${payload['transport']}');
|
||||||
|
logTrace(' Type: ${payload['payload_type']}');
|
||||||
|
logTrace(' Size: ${payload['size']} bytes');
|
||||||
|
logTrace(' Encoding: ${payload['encoding']}');
|
||||||
|
|
||||||
|
if (payload['transport'] == 'link') {
|
||||||
|
logTrace(' URL: ${payload['data']}');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Summary
|
||||||
|
print('\n--- Transport Summary ---');
|
||||||
|
final directCount = env['payloads'].where((p) => p['transport'] == 'direct').length;
|
||||||
|
final linkCount = env['payloads'].where((p) => p['transport'] == 'link').length;
|
||||||
|
logTrace('Direct transport: $directCount payloads');
|
||||||
|
logTrace('Link transport: $linkCount payloads');
|
||||||
|
|
||||||
|
print('\nTest completed.');
|
||||||
|
} catch (error) {
|
||||||
|
print('\n❌ Test failed with error: $error');
|
||||||
|
print('$error');
|
||||||
|
exit(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void main() {
|
||||||
|
runTest();
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user