update
This commit is contained in:
+38
-38
@@ -207,44 +207,36 @@ The walkthrough is organized as one document with a dedicated section per system
|
||||
|
||||
**Purpose**: The *enforcer* — defines tests **before implementation begins** to ensure spec compliance. Blocks drift and human error.
|
||||
|
||||
**Why It Matters**:
|
||||
- Ensures tests are written before implementation, so code can be tested as it's built
|
||||
- Prevents breaking changes from reaching production
|
||||
- Catches specification violations early in the CI pipeline
|
||||
- Maintains data integrity and API consistency
|
||||
- Reduces manual QA effort through automation
|
||||
- Provides immediate feedback to developers during implementation
|
||||
- Validates performance NFRs (latency, throughput) against measurable KPIs before deployment
|
||||
- Validates security requirements through adversarial testing before production
|
||||
Because implementation must satisfy specifications, Validation derives all test definitions from the preceding 5 documents:
|
||||
|
||||
**Content Guidelines**:
|
||||
- **Unit tests** — atomic tests for each specification rule, data schema field, and UI component state (cites specification.md sections, ui-specification.md component specs)
|
||||
- **Integration tests** — end-to-end flow validation covering workflows defined in 3-walkthrough.md (cites specification.md endpoint chains, 3-walkthrough.md participant sections)
|
||||
- **Performance tests** — load tests, stress tests, and benchmarking against NFR thresholds (cites requirements.md NFRs, solution-design.md performance decisions)
|
||||
- **Penetration tests** — adversarial testing of security requirements and auth flows (cites requirements.md security NFRs, specification.md auth endpoints)
|
||||
- Contract testing scripts (mapped to specification.md sections)
|
||||
- Linting rules and configurations (enforcing style guides from ui-specification.md)
|
||||
- Schema validation jobs (validating data shapes from specification.md)
|
||||
- Visual regression tests (verifying UI matches ui-specification.md)
|
||||
- Security scanning and audit jobs (SAST, dependency vulnerability checks)
|
||||
- CI/CD pipeline configurations for automated test execution
|
||||
- Each validation test must include explicit references to:
|
||||
- **specification.md** (spec IDs or paths — API contracts, data schemas, e.g., `/api/v1/endpoint` or `#/components/schemas/Request`)
|
||||
- **ui-specification.md** (components, interactions, visual states, e.g., `## Wine Table` or `### Sort States`)
|
||||
- **requirements.md** (e.g., FR-001, NFR-101, NFR-201) — business intent and NFR thresholds being validated
|
||||
- **solution-design.md** (e.g., SD-2.1, SD-3.3) — architectural decisions being tested
|
||||
| Source Document | What Validation Tests |
|
||||
|-----------------|----------------------|
|
||||
| **1. Requirements** | NFR thresholds (latency, throughput), security constraints, observability checks |
|
||||
| **2. Solution Design** | Architectural decisions, performance choices, risk mitigations |
|
||||
| **3. Walkthrough** | End-to-end flows for each participant section, cross-participant interactions |
|
||||
| **4. Specification** | API contracts, data schemas, error codes, validation rules |
|
||||
| **5. UI Specification** | Component states, visual patterns, accessibility compliance |
|
||||
|
||||
**Best Practices**:
|
||||
- Define tests **before** writing implementation code
|
||||
- Fail CI on specification violations
|
||||
- Run validation jobs on every commit and PR
|
||||
- Use automated code review tools
|
||||
- Maintain validation job health dashboard
|
||||
- Document validation failure remediation steps
|
||||
Every specification rule, walkthrough journey, and NFR threshold must have a corresponding test. Tests are written before code so implementation can be validated as it's built. If a spec rule, walkthrough journey, or NFR has no corresponding test, the Validation doc is incomplete.
|
||||
|
||||
---
|
||||
**Test Types**
|
||||
|
||||
#### Specification-to-Test Execution Trace
|
||||
Each validation test must include explicit references to its source document:
|
||||
|
||||
| Test Type | Source | Example Reference |
|
||||
|-----------|--------|-------------------|
|
||||
| **Unit tests** | specification.md, ui-specification.md | `specification.md:3.2.1` rejects camelCase headers; `ui-specification.md:## Wine Table` sort states |
|
||||
| **Integration tests** | 3-walkthrough.md, specification.md | Participant A section step 4.2 → endpoint chain in specification.md |
|
||||
| **Performance tests** | requirements.md NFRs, solution-design.md | `NFR-101` latency threshold; `SD-002` WebSocket performance decision |
|
||||
| **Penetration tests** | requirements.md security NFRs, specification.md auth endpoints | `NFR-301` authentication; `specification.md:auth/login` endpoint |
|
||||
| **Contract tests** | specification.md sections | Mapped to endpoint definitions and schema constraints |
|
||||
| **Linting rules** | ui-specification.md style guides | Enforcing design tokens from ui-specification.md |
|
||||
| **Schema validation** | specification.md data schemas | Validating Arrow stream headers (snake_case), JSON payloads |
|
||||
| **Visual regression** | ui-specification.md component states | Verifying hover, disabled, error states match spec |
|
||||
| **Security scanning** | requirements.md, specification.md | SAST, dependency vulnerability checks, OWASP rules |
|
||||
| **CI/CD pipeline** | All test types above | Automated execution on every commit and PR |
|
||||
|
||||
**Specification-to-Test Execution Trace**
|
||||
|
||||
Every specification rule must have a runnable, verifiable test. This trace maps each spec item to its corresponding test file, test function, and CI gate — proving that every requirement was tested, not just claimed.
|
||||
|
||||
@@ -252,12 +244,12 @@ Every specification rule must have a runnable, verifiable test. This trace maps
|
||||
|
||||
| Requirement ID | Source Path | Unit Test File | Unit Test Function | Additional Test File(s) | CI Gate | Status |
|
||||
|----------------|-------------|----------------|-------------------|------------------------|---------|--------|
|
||||
| FR-001 | `specification.md:3.2.1` | `tests/unit/schema-validation.test.ts` | `test_rejects_camelCase_headers` | — | `ci/contract-validation.yaml` | Pass |
|
||||
| FR-002 | `specification.md:4.1` | `tests/unit/api-routes.test.ts` | `test_post_wine_returns_201` | `tests/integration/wine-flow.test.ts` | `ci/contract-validation.yaml` | Pass |
|
||||
| FR-003 | `specification.md:3.2.3` | — | — | `tests/integration/error-handling.test.ts` | `ci/contract-validation.yaml` | Pass |
|
||||
| FR-001 | `specification.md#/components/schemas/Request` | `tests/unit/schema-validation.test.ts` | `test_rejects_camelCase_headers` | — | `ci/contract-validation.yaml` | Pass |
|
||||
| FR-002 | `specification.md#/paths/~1api~1v1~1endpoint` | `tests/unit/api-routes.test.ts` | `test_post_wine_returns_201` | `tests/integration/wine-flow.test.ts` | `ci/contract-validation.yaml` | Pass |
|
||||
| FR-003 | `specification.md#/paths/~1api~1v1~1endpoint` | — | — | `tests/integration/error-handling.test.ts` | `ci/contract-validation.yaml` | Pass |
|
||||
| NFR-101 | `requirements.md:4.1` | — | — | `tests/performance/load.test.ts` | `ci/performance.yaml` | Pass |
|
||||
| SD-001 | `solution-design.md:5.1` | — | — | `tests/integration/nats-async.test.ts` | `ci/integration.yaml` | Pass |
|
||||
| UI-15.3.2 | `ui-specification.md:15.3.2` | `tests/unit/wine-table.test.tsx` | `test_sort_by_name` | `tests/visual/wine-table.spec.js` | `ci/visual-regression.yaml` | Pass |
|
||||
| UI-WineTable | `ui-specification.md## Wine Table` | `tests/unit/wine-table.test.tsx` | `test_sort_by_name` | `tests/visual/wine-table.spec.js` | `ci/visual-regression.yaml` | Pass |
|
||||
|
||||
**Additional Test File(s)** column covers: integration tests (participant section flows), performance tests (NFR benchmarks), penetration tests (security NFRs), and visual regression tests (UI component states).
|
||||
|
||||
@@ -270,6 +262,14 @@ Every specification rule must have a runnable, verifiable test. This trace maps
|
||||
- A test may cover multiple spec items, but each spec item must be covered by **at least one** test
|
||||
- Status is tracked via CI gate results (Pass/Fail/Skipped) — not manual checks
|
||||
|
||||
**Best Practices**:
|
||||
- Define tests **before** writing implementation code
|
||||
- Fail CI on specification violations
|
||||
- Run validation jobs on every commit and PR
|
||||
- Use automated code review tools
|
||||
- Maintain validation job health dashboard
|
||||
- Document validation failure remediation steps
|
||||
|
||||
**How to Maintain**:
|
||||
1. When writing a new spec rule, add a trace row **before** implementation
|
||||
2. When writing the test, fill in the test file and function
|
||||
|
||||
Reference in New Issue
Block a user