417 lines
14 KiB
Markdown
417 lines
14 KiB
Markdown
# Implementation Plan: msghandler
|
|
|
|
**Version**: 1.3.0
|
|
**Date**: 2026-05-19
|
|
**Status**: Active
|
|
**Ground Truth**: [`src/msghandler.jl`](../src/msghandler.jl)
|
|
|
|
---
|
|
|
|
## 1. Implementation Phases and Timeline
|
|
|
|
### Phase 1: Core API Implementation (Week 1-2)
|
|
| Task | Priority | Estimated Effort | Status |
|
|
|------|----------|-----------------|--------|
|
|
| Core `smartpack()` implementation | P0 | 3 days | ✅ Complete |
|
|
| Core `smartunpack()` implementation | P0 | 3 days | ✅ Complete |
|
|
| Message envelope structure | P0 | 2 days | ✅ Complete |
|
|
| Payload type handling | P0 | 2 days | ✅ Complete |
|
|
| Transport adapter layer | P0 | 3 days | ✅ Complete |
|
|
|
|
**Deliverables**:
|
|
- Julia module: `src/msghandler.jl`
|
|
- Node.js module: `src/msghandler_ssr.js`
|
|
- Browser module: `src/msghandler_csr.js`
|
|
- Python module: `src/msghandler.py`
|
|
- MicroPython module: `src/msghandler_mpy.py`
|
|
|
|
### Phase 2: File Server Integration (Week 3)
|
|
| Task | Priority | Estimated Effort | Status |
|
|
|------|----------|-----------------|--------|
|
|
| File server upload handler | P1 | 2 days | ✅ Complete |
|
|
| File server download handler | P1 | 2 days | ✅ Complete |
|
|
| Exponential backoff logic | P1 | 1 day | ✅ Complete |
|
|
| Plik integration | P1 | 2 days | ✅ Complete |
|
|
|
|
**Deliverables**:
|
|
- Upload handler with plik_oneshot_upload
|
|
- Download handler with retry logic
|
|
- Configurable file server URL
|
|
|
|
### Phase 3: Platform-Specific Features (Week 4)
|
|
| Task | Priority | Estimated Effort | Status |
|
|
|------|----------|-----------------|--------|
|
|
| Arrow IPC support (Desktop) | P1 | 3 days | ✅ Complete |
|
|
| JSON table support (Browser) | P1 | 2 days | ✅ Complete |
|
|
| Browser WebSocket transport | P1 | 2 days | ✅ Complete |
|
|
| MicroPython optimizations | P2 | 2 days | ✅ Complete |
|
|
|
|
**Deliverables**:
|
|
- Arrow IPC serialization for tabular data
|
|
- JSON table format for browser compatibility
|
|
- Browser-specific transport layer
|
|
- Memory-optimized MicroPython implementation
|
|
|
|
### Phase 4: Cross-Platform Testing (Week 5)
|
|
| Task | Priority | Estimated Effort | Status |
|
|
|------|----------|-----------------|--------|
|
|
| Text message tests | P1 | 1 day | ✅ Complete |
|
|
| Dictionary tests | P1 | 1 day | ✅ Complete |
|
|
| Tabular data tests | P1 | 2 days | ✅ Complete |
|
|
| Mixed payload tests | P1 | 2 days | ✅ Complete |
|
|
| Large file tests | P1 | 2 days | ✅ Complete |
|
|
|
|
**Deliverables**:
|
|
- Platform-specific test suites
|
|
- Integration test scenarios
|
|
- Performance benchmarks
|
|
|
|
### Phase 5: Documentation & Examples (Week 6)
|
|
| Task | Priority | Estimated Effort | Status |
|
|
|------|----------|-----------------|--------|
|
|
| API documentation | P2 | 2 days | ✅ Complete |
|
|
| Walkthrough examples | P2 | 2 days | ✅ Complete |
|
|
| Architecture diagrams | P2 | 1 day | ✅ Complete |
|
|
| Deployment guides | P2 | 1 day | ✅ Complete |
|
|
|
|
**Deliverables**:
|
|
- Comprehensive documentation
|
|
- Code examples for all platforms
|
|
- Deployment runbooks
|
|
|
|
---
|
|
|
|
## 2. Module/Component Breakdown
|
|
|
|
### Core Modules
|
|
|
|
#### msghandler.jl (Julia)
|
|
```
|
|
src/
|
|
└── msghandler.jl
|
|
├── Constants (DEFAULT_SIZE_THRESHOLD, etc.)
|
|
├── msg_payload_v1 struct
|
|
├── msg_envelope_v1 struct
|
|
├── Serialization functions
|
|
│ ├── serialize_text()
|
|
│ ├── serialize_dictionary()
|
|
│ ├── serialize_arrowtable()
|
|
│ ├── serialize_jsontable()
|
|
│ └── serialize_binary()
|
|
├── Deserialization functions
|
|
│ ├── deserialize_text()
|
|
│ ├── deserialize_dictionary()
|
|
│ ├── deserialize_arrowtable()
|
|
│ ├── deserialize_jsontable()
|
|
│ └── deserialize_binary()
|
|
├── File server handlers
|
|
│ ├── plik_oneshot_upload()
|
|
│ └── _fetch_with_backoff()
|
|
├── smartpack() - Main sender function
|
|
└── smartunpack() - Main receiver function
|
|
```
|
|
|
|
**Dependencies**:
|
|
- JSON.jl (JSON serialization)
|
|
- Arrow.jl (Arrow IPC)
|
|
- HTTP.jl (File server)
|
|
- UUIDs.jl (IDs)
|
|
- DataFrames.jl (DataFrame support)
|
|
|
|
#### msghandler_ssr.js (Node.js)
|
|
```
|
|
src/
|
|
├── msghandler_ssr.js
|
|
│ ├── Constants
|
|
│ ├── msg_payload_v1 class
|
|
│ ├── msg_envelope_v1 class
|
|
│ ├── Serialization methods
|
|
│ ├── Deserialization methods
|
|
│ ├── File server handlers
|
|
│ ├── smartpack() function
|
|
│ └── smartunpack() function
|
|
└── nats/
|
|
├── NATSClient.js
|
|
└── NATSConnectionPool.js
|
|
```
|
|
|
|
**Dependencies**:
|
|
- nats (NATS client)
|
|
- node-fetch (HTTP file server)
|
|
|
|
#### msghandler_csr.js (Browser)
|
|
```
|
|
src/
|
|
└── msghandler_csr.js
|
|
├── Constants
|
|
├── msg_payload_v1 class
|
|
├── msg_envelope_v1 class
|
|
├── Serialization methods (JSON table only)
|
|
├── Deserialization methods
|
|
├── File server handlers (browser-compatible)
|
|
├── smartpack() function
|
|
└── smartunpack() function
|
|
```
|
|
|
|
**Dependencies**:
|
|
- nats.ws (Browser NATS client)
|
|
|
|
#### msghandler.py (Python)
|
|
```
|
|
src/
|
|
└── msghandler.py
|
|
├── Constants
|
|
├── msg_payload_v1 class
|
|
├── msg_envelope_v1 class
|
|
├── Serialization methods
|
|
├── Deserialization methods
|
|
├── File server handlers
|
|
├── smartpack() async function
|
|
└── smartunpack() async function
|
|
```
|
|
|
|
**Dependencies**:
|
|
- aiohttp (HTTP file server)
|
|
- pyarrow (Arrow IPC)
|
|
- uuid (IDs)
|
|
|
|
#### msghandler.rs (Rust)
|
|
```
|
|
src/
|
|
├── msghandler.rs
|
|
│ ├── Constants
|
|
│ ├── msg_payload_v1 struct
|
|
│ ├── msg_envelope_v1 struct
|
|
│ ├── Serialization traits
|
|
│ ├── Deserialization traits
|
|
│ ├── File server handlers
|
|
│ ├── smartpack() async function
|
|
│ └── smartunpack() async function
|
|
├── Payload enum
|
|
├── smartpackOptions struct
|
|
└── smartunpackOptions struct
|
|
```
|
|
|
|
**Dependencies**:
|
|
- tokio (Async runtime)
|
|
- serde (JSON serialization)
|
|
- reqwest (HTTP file server)
|
|
- arrow2 (Arrow IPC)
|
|
|
|
#### msghandler_mpy.py (MicroPython)
|
|
```
|
|
src/
|
|
└── msghandler_mpy.py
|
|
├── Constants (lower thresholds)
|
|
├── msg_payload_v1 class
|
|
├── msg_envelope_v1 class
|
|
├── serialize_text()
|
|
├── deserialize_text()
|
|
├── serialize_dictionary()
|
|
├── deserialize_dictionary()
|
|
└── smartpack()/smartunpack() functions
|
|
```
|
|
|
|
**Constraints**:
|
|
- Limited to text and dictionary types
|
|
- Direct transport only (no file server)
|
|
- 100KB threshold for memory constraints
|
|
|
|
---
|
|
|
|
## 3. Task List
|
|
|
|
### Core API Tasks
|
|
|
|
| Task ID | Description | Assignee | Priority | Status |
|
|
|---------|-------------|----------|----------|--------|
|
|
| T-001 | Implement `smartpack()` with tuple format | Developer A | P0 | ✅ Complete |
|
|
| T-002 | Implement `smartunpack()` with type handling | Developer A | P0 | ✅ Complete |
|
|
| T-003 | Create message envelope structure | Developer A | P0 | ✅ Complete |
|
|
| T-004 | Implement transport adapter | Developer B | P0 | ✅ Complete |
|
|
| T-005 | Add correlation ID support | Developer A | P0 | ✅ Complete |
|
|
|
|
### File Server Tasks
|
|
|
|
| Task ID | Description | Assignee | Priority | Status |
|
|
|---------|-------------|----------|----------|--------|
|
|
| T-006 | Implement Plik upload handler | Developer B | P1 | ✅ Complete |
|
|
| T-007 | Implement file download with retry | Developer B | P1 | ✅ Complete |
|
|
| T-008 | Add exponential backoff logic | Developer B | P1 | ✅ Complete |
|
|
|
|
### Platform Tasks
|
|
|
|
| Task ID | Description | Assignee | Priority | Status |
|
|
|---------|-------------|----------|----------|--------|
|
|
| T-009 | Implement Arrow IPC (Julia/Python/Node.js) | Developer A | P1 | ✅ Complete |
|
|
| T-010 | Implement JSON table (Browser) | Developer B | P1 | ✅ Complete |
|
|
| T-011 | Implement MicroPython optimizations | Developer C | P2 | ✅ Complete |
|
|
| T-012 | Browser WebSocket transport | Developer B | P1 | ✅ Complete |
|
|
|
|
### Testing Tasks
|
|
|
|
| Task ID | Description | Assignee | Priority | Status |
|
|
|---------|-------------|----------|------------------|
|
|
| T-013 | Text message tests | QA Team | P1 | ✅ Complete |
|
|
| T-014 | Dictionary tests | QA Team | P1 | ✅ Complete |
|
|
| T-015 | Tabular data tests | QA Team | P1 | ✅ Complete |
|
|
| T-016 | Mixed payload tests | QA Team | P1 | ✅ Complete |
|
|
| T-017 | Large file tests | QA Team | P1 | ✅ Complete |
|
|
|
|
---
|
|
|
|
## 4. Test Strategy
|
|
|
|
### Unit Tests
|
|
|
|
| Test Category | Coverage | Files | Requirements |
|
|
|---------------|----------|-------|--------------|
|
|
| Serialization | All payload types | `test/test_*_sender.*` | FR-001 through FR-012 |
|
|
| Deserialization | All payload types | `test/test_*_receiver.*` | FR-001 through FR-012 |
|
|
| Transport selection | Direct vs link | `test/test_*_mix_payloads.*` | FR-003, FR-004, FR-006 |
|
|
| File server upload | Plik integration | Platform-specific | FR-008, FR-009 |
|
|
| File server download | Exponential backoff | Platform-specific | FR-010, FR-011 |
|
|
|
|
### Integration Tests
|
|
|
|
| Scenario | Platforms | Payloads | Transport | Requirements |
|
|
|----------|-----------|----------|-----------|--------------|
|
|
| Single text (small) | All | text | direct | FR-001, FR-012 |
|
|
| Single dictionary (small) | All | dictionary | direct | FR-002, FR-012 |
|
|
| Single arrow table (small) | Desktop | arrowtable | direct | FR-002, FR-012 |
|
|
| Single JSON table (small) | All | jsontable | direct | FR-001, FR-002, FR-006 |
|
|
| Single image (small) | All | image | direct | FR-001, FR-006 |
|
|
| Single text (large) | All | text | link | FR-003, FR-008, FR-009 |
|
|
| Mixed payloads | All | text + dictionary + image | mixed | FR-006, FR-007 |
|
|
|
|
### Test Coverage Targets
|
|
|
|
| Phase | Coverage Target | Method |
|
|
|-------|----------------|--------|
|
|
| Phase 1 | 70% | Unit tests per platform |
|
|
| Phase 2 | 80% | Add integration tests |
|
|
| Phase 3 | 85% | Add edge case tests |
|
|
| Phase 4 | 90% | Add performance tests |
|
|
|
|
---
|
|
|
|
## 5. Build and Deployment Preparation
|
|
|
|
### Continuous Integration
|
|
|
|
| Check | Command | Purpose |
|
|
|-------|---------|---------|
|
|
| Linting | `npm run lint` | Code style enforcement |
|
|
| Type checking | `npx tsc --noEmit` | Type safety (JavaScript/TypeScript) |
|
|
| Unit tests | `npm test` | Functionality validation |
|
|
| Integration tests | `npm run test:integration` | Cross-platform validation |
|
|
| Coverage | `npm run coverage` | Test coverage tracking |
|
|
|
|
### Deployment Pipeline
|
|
|
|
```
|
|
GitHub Push
|
|
↓
|
|
CI/CD Pipeline
|
|
↓
|
|
├──→ Linting (all platforms)
|
|
├──→ Unit tests (all platforms)
|
|
├──→ Integration tests (cross-platform)
|
|
├──→ Coverage report
|
|
└──→ Build documentation
|
|
↓
|
|
Release (if all checks pass)
|
|
↓
|
|
├──→ GitHub Releases
|
|
├──→ Package registry (npm, PyPI)
|
|
└──→ Documentation site
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Risk Mitigation
|
|
|
|
### Known Blockers
|
|
|
|
| Risk | Mitigation Step | Owner |
|
|
|------|----------------|-------|
|
|
| **Browser Arrow IPC** | Use JSON table as fallback | Developer B |
|
|
| **MicroPython memory** | 100KB threshold, direct transport only | Developer C |
|
|
| **File server availability** | Exponential backoff with graceful degradation | Developer B |
|
|
|
|
### Known Unknowns
|
|
|
|
| Unknown | Monitoring Strategy | Response Plan |
|
|
|---------|-------------------|---------------|
|
|
| Platform-specific bugs | Comprehensive test coverage | Hotfix with platform-specific handling |
|
|
| Performance bottlenecks | Load testing and profiling | Optimized serialization/deserialization |
|
|
|
|
---
|
|
|
|
## 7. Requirements Traceability
|
|
|
|
### Functional Requirements
|
|
|
|
| Requirement ID | Implementation Location | Status |
|
|
|---------------|------------------------|--------|
|
|
| FR-001 | All platform modules | ✅ Complete |
|
|
| FR-002 | All platform modules | ✅ Complete |
|
|
| FR-003 | All platform modules (size_threshold logic) | ✅ Complete |
|
|
| FR-004 | All platform modules | ✅ Complete |
|
|
| FR-005 | MicroPython module | ✅ Complete |
|
|
| FR-006 | All platform modules | ✅ Complete |
|
|
| FR-007 | All platform modules | ✅ Complete |
|
|
| FR-008 | All platform modules | ✅ Complete |
|
|
| FR-009 | All platform modules | ✅ Complete |
|
|
| FR-010 | All platform modules | ✅ Complete |
|
|
| FR-011 | All platform modules | ✅ Complete |
|
|
| FR-012 | All platform modules | ✅ Complete |
|
|
| FR-013 | All platform modules | ✅ Complete |
|
|
| FR-014 | All platform modules | ✅ Complete |
|
|
|
|
### Non-Functional Requirements
|
|
|
|
| NFR ID | Implementation Location | Status |
|
|
|--------|------------------------|--------|
|
|
| NFR-101 | Serialization functions | ✅ Complete |
|
|
| NFR-102 | Deserialization functions | ✅ Complete |
|
|
| NFR-103 | Transport adapter | ✅ Complete |
|
|
| NFR-104 | File upload handler | ✅ Complete |
|
|
| NFR-105 | File download handler | ✅ Complete |
|
|
| NFR-106 | MicroPython module | ✅ Complete |
|
|
| NFR-107 | Performance benchmarks | ✅ Complete |
|
|
| NFR-201 | Transport adapter | ✅ Complete |
|
|
| NFR-202 | File download retry logic | ✅ Complete |
|
|
| NFR-203 | Transport adapter | ✅ Complete |
|
|
| NFR-401 | Message envelope | ✅ Complete |
|
|
| NFR-402 | Metrics instrumentation | ✅ Complete |
|
|
| NFR-403 | Correlation ID propagation | ✅ Complete |
|
|
|
|
---
|
|
|
|
## 8. Validation Gates
|
|
|
|
### Pre-Release Checklist
|
|
|
|
| Gate | Check | Pass Criteria |
|
|
|------|-------|--------------|
|
|
| **G-001** | All unit tests pass | 100% pass rate per platform |
|
|
| **G-002** | Integration tests pass | Cross-platform round-trip successful |
|
|
| **G-003** | Coverage threshold | ≥80% line coverage |
|
|
| **G-004** | Linting clean | No warnings or errors |
|
|
| **G-005** | Specification compliance | All spec rules validated |
|
|
| **G-006** | Documentation complete | All required docs present |
|
|
|
|
### CI/CD Validation
|
|
|
|
| Check | Command | Failure Action |
|
|
|-------|---------|---------------|
|
|
| Syntax | `julia --check-base` | Block PR |
|
|
| Unit tests | `julia test/runtests.jl` | Block PR |
|
|
| Integration | `npm run test:integration` | Block PR |
|
|
| Coverage | `codecov` | Report only |
|
|
|
|
---
|
|
|
|
*This implementation plan is versioned and maintained in git alongside the codebase. All implementations must adhere to this plan.*
|