From f5609513794b15a9990e2257ae8373229b96be11 Mon Sep 17 00:00:00 2001 From: narawat Date: Fri, 4 Sep 2026 09:45:40 +0700 Subject: [PATCH] update --- ASG_Framework.md | 76 ++++++++++++++++++++++++------------------------ 1 file changed, 38 insertions(+), 38 deletions(-) diff --git a/ASG_Framework.md b/ASG_Framework.md index 5d1c3ac..a6f56d5 100644 --- a/ASG_Framework.md +++ b/ASG_Framework.md @@ -207,44 +207,36 @@ The walkthrough is organized as one document with a dedicated section per system **Purpose**: The *enforcer* — defines tests **before implementation begins** to ensure spec compliance. Blocks drift and human error. -**Why It Matters**: -- Ensures tests are written before implementation, so code can be tested as it's built -- Prevents breaking changes from reaching production -- Catches specification violations early in the CI pipeline -- Maintains data integrity and API consistency -- Reduces manual QA effort through automation -- Provides immediate feedback to developers during implementation -- Validates performance NFRs (latency, throughput) against measurable KPIs before deployment -- Validates security requirements through adversarial testing before production +Because implementation must satisfy specifications, Validation derives all test definitions from the preceding 5 documents: -**Content Guidelines**: -- **Unit tests** — atomic tests for each specification rule, data schema field, and UI component state (cites specification.md sections, ui-specification.md component specs) -- **Integration tests** — end-to-end flow validation covering workflows defined in 3-walkthrough.md (cites specification.md endpoint chains, 3-walkthrough.md participant sections) -- **Performance tests** — load tests, stress tests, and benchmarking against NFR thresholds (cites requirements.md NFRs, solution-design.md performance decisions) -- **Penetration tests** — adversarial testing of security requirements and auth flows (cites requirements.md security NFRs, specification.md auth endpoints) -- Contract testing scripts (mapped to specification.md sections) -- Linting rules and configurations (enforcing style guides from ui-specification.md) -- Schema validation jobs (validating data shapes from specification.md) -- Visual regression tests (verifying UI matches ui-specification.md) -- Security scanning and audit jobs (SAST, dependency vulnerability checks) -- CI/CD pipeline configurations for automated test execution -- Each validation test must include explicit references to: - - **specification.md** (spec IDs or paths — API contracts, data schemas, e.g., `/api/v1/endpoint` or `#/components/schemas/Request`) - - **ui-specification.md** (components, interactions, visual states, e.g., `## Wine Table` or `### Sort States`) - - **requirements.md** (e.g., FR-001, NFR-101, NFR-201) — business intent and NFR thresholds being validated - - **solution-design.md** (e.g., SD-2.1, SD-3.3) — architectural decisions being tested +| Source Document | What Validation Tests | +|-----------------|----------------------| +| **1. Requirements** | NFR thresholds (latency, throughput), security constraints, observability checks | +| **2. Solution Design** | Architectural decisions, performance choices, risk mitigations | +| **3. Walkthrough** | End-to-end flows for each participant section, cross-participant interactions | +| **4. Specification** | API contracts, data schemas, error codes, validation rules | +| **5. UI Specification** | Component states, visual patterns, accessibility compliance | -**Best Practices**: -- Define tests **before** writing implementation code -- Fail CI on specification violations -- Run validation jobs on every commit and PR -- Use automated code review tools -- Maintain validation job health dashboard -- Document validation failure remediation steps +Every specification rule, walkthrough journey, and NFR threshold must have a corresponding test. Tests are written before code so implementation can be validated as it's built. If a spec rule, walkthrough journey, or NFR has no corresponding test, the Validation doc is incomplete. ---- +**Test Types** -#### Specification-to-Test Execution Trace +Each validation test must include explicit references to its source document: + +| Test Type | Source | Example Reference | +|-----------|--------|-------------------| +| **Unit tests** | specification.md, ui-specification.md | `specification.md:3.2.1` rejects camelCase headers; `ui-specification.md:## Wine Table` sort states | +| **Integration tests** | 3-walkthrough.md, specification.md | Participant A section step 4.2 → endpoint chain in specification.md | +| **Performance tests** | requirements.md NFRs, solution-design.md | `NFR-101` latency threshold; `SD-002` WebSocket performance decision | +| **Penetration tests** | requirements.md security NFRs, specification.md auth endpoints | `NFR-301` authentication; `specification.md:auth/login` endpoint | +| **Contract tests** | specification.md sections | Mapped to endpoint definitions and schema constraints | +| **Linting rules** | ui-specification.md style guides | Enforcing design tokens from ui-specification.md | +| **Schema validation** | specification.md data schemas | Validating Arrow stream headers (snake_case), JSON payloads | +| **Visual regression** | ui-specification.md component states | Verifying hover, disabled, error states match spec | +| **Security scanning** | requirements.md, specification.md | SAST, dependency vulnerability checks, OWASP rules | +| **CI/CD pipeline** | All test types above | Automated execution on every commit and PR | + +**Specification-to-Test Execution Trace** Every specification rule must have a runnable, verifiable test. This trace maps each spec item to its corresponding test file, test function, and CI gate — proving that every requirement was tested, not just claimed. @@ -252,12 +244,12 @@ Every specification rule must have a runnable, verifiable test. This trace maps | Requirement ID | Source Path | Unit Test File | Unit Test Function | Additional Test File(s) | CI Gate | Status | |----------------|-------------|----------------|-------------------|------------------------|---------|--------| -| FR-001 | `specification.md:3.2.1` | `tests/unit/schema-validation.test.ts` | `test_rejects_camelCase_headers` | — | `ci/contract-validation.yaml` | Pass | -| FR-002 | `specification.md:4.1` | `tests/unit/api-routes.test.ts` | `test_post_wine_returns_201` | `tests/integration/wine-flow.test.ts` | `ci/contract-validation.yaml` | Pass | -| FR-003 | `specification.md:3.2.3` | — | — | `tests/integration/error-handling.test.ts` | `ci/contract-validation.yaml` | Pass | +| FR-001 | `specification.md#/components/schemas/Request` | `tests/unit/schema-validation.test.ts` | `test_rejects_camelCase_headers` | — | `ci/contract-validation.yaml` | Pass | +| FR-002 | `specification.md#/paths/~1api~1v1~1endpoint` | `tests/unit/api-routes.test.ts` | `test_post_wine_returns_201` | `tests/integration/wine-flow.test.ts` | `ci/contract-validation.yaml` | Pass | +| FR-003 | `specification.md#/paths/~1api~1v1~1endpoint` | — | — | `tests/integration/error-handling.test.ts` | `ci/contract-validation.yaml` | Pass | | NFR-101 | `requirements.md:4.1` | — | — | `tests/performance/load.test.ts` | `ci/performance.yaml` | Pass | | SD-001 | `solution-design.md:5.1` | — | — | `tests/integration/nats-async.test.ts` | `ci/integration.yaml` | Pass | -| UI-15.3.2 | `ui-specification.md:15.3.2` | `tests/unit/wine-table.test.tsx` | `test_sort_by_name` | `tests/visual/wine-table.spec.js` | `ci/visual-regression.yaml` | Pass | +| UI-WineTable | `ui-specification.md## Wine Table` | `tests/unit/wine-table.test.tsx` | `test_sort_by_name` | `tests/visual/wine-table.spec.js` | `ci/visual-regression.yaml` | Pass | **Additional Test File(s)** column covers: integration tests (participant section flows), performance tests (NFR benchmarks), penetration tests (security NFRs), and visual regression tests (UI component states). @@ -270,6 +262,14 @@ Every specification rule must have a runnable, verifiable test. This trace maps - A test may cover multiple spec items, but each spec item must be covered by **at least one** test - Status is tracked via CI gate results (Pass/Fail/Skipped) — not manual checks +**Best Practices**: +- Define tests **before** writing implementation code +- Fail CI on specification violations +- Run validation jobs on every commit and PR +- Use automated code review tools +- Maintain validation job health dashboard +- Document validation failure remediation steps + **How to Maintain**: 1. When writing a new spec rule, add a trace row **before** implementation 2. When writing the test, fill in the test file and function