update_docs #10
279
SDD_FRAMEWORK.md
Normal file
279
SDD_FRAMEWORK.md
Normal file
@@ -0,0 +1,279 @@
|
||||
# SDD + GitOps Documentation Stack
|
||||
|
||||
A comprehensive documentation strategy for modern software development that aligns different types of documentation with their specific purposes, audiences, and tooling.
|
||||
|
||||
## The Big Picture
|
||||
|
||||
This framework ensures that every piece of documentation serves a clear purpose and reaches the right audience. It emphasizes:
|
||||
|
||||
- **Machine-readable truths** as the foundation for automation
|
||||
- **Separation of concerns** between human-facing docs and machine-consumable contracts
|
||||
- **GitOps integration** where deployment and configuration are version-controlled
|
||||
- **Multi-role audience targeting** from stakeholders to DevOps
|
||||
|
||||
---
|
||||
|
||||
## Documentation Matrix
|
||||
|
||||
| Document | Purpose ("The Why") | Primary Audience | Format / Tooling | Example (SaaS Context) |
|
||||
|----------|---------------------|------------------|------------------|------------------------|
|
||||
| **Requirements** | Define business goals & user needs | Stakeholders, PM, Lead Dev | GitHub Issues, Notion | "System must support 5-member teams with real-time sync." |
|
||||
| **The Spec** | The Contract. Machine-readable truth. | Developers, QA, Machines | OpenAPI, Protobuf, YAML | A `.yaml` file defining `user_id` as a UUID in snake_case. |
|
||||
| **Architecture** | High-level structural blueprint | Senior Devs, DevOps | Mermaid.js, IcePanel | Diagram of SvelteKit ↔ NATS ↔ Julia 6-node cluster. |
|
||||
| **Walkthrough** | The Intuition. The "Big Picture" narrative. | New Devs, The Team | Recorded Video, TOUR.md | "Why we use a Claim-Check pattern for large Arrow data." |
|
||||
| **Implementation** | The actual logic & generated code | Developers | SvelteKit, Julia, Node.js | Auto-generated TypeScript types from the OpenAPI spec. |
|
||||
| **Validation** | Automated "Contract" enforcement | CI/CD Pipelines, QA | GitHub Actions, Prism | A test that fails if the Julia API returns camelCase keys. |
|
||||
| **Runbook** | Deployment, Scaling, & Recovery | DevOps, SRE | K8s Manifests, Flux | `git push` to update the replica count from 3 to 6. |
|
||||
|
||||
---
|
||||
|
||||
## Detailed Explanations
|
||||
|
||||
### 1. Requirements
|
||||
|
||||
**Purpose**: Define business goals & user needs.
|
||||
|
||||
**Why it matters**: Before writing code, we need to understand *why* we're building something. Requirements capture the business context, user pain points, and success criteria.
|
||||
|
||||
**Primary Audience**:
|
||||
- **Stakeholders**: Business owners who need to approve the direction
|
||||
- **Product Managers**: Translate requirements into features
|
||||
- **Lead Developers**: Understand scope and technical constraints
|
||||
|
||||
**Format / Tooling**:
|
||||
- **GitHub Issues**: Simple, version-controlled, integrated with code
|
||||
- **Notion**: Rich text, collaborative, good for initial brainstorming
|
||||
|
||||
**Best Practices**:
|
||||
- Write in user story format: "As a [role], I want [feature] so that [benefit]"
|
||||
- Include acceptance criteria as checklist items
|
||||
- Link to related specs and architecture decisions
|
||||
|
||||
**Example**: "System must support 5-member teams with real-time sync."
|
||||
|
||||
---
|
||||
|
||||
### 2. The Spec (The Contract)
|
||||
|
||||
**Purpose**: Machine-readable truth that defines the API contract.
|
||||
|
||||
**Why it matters**: The spec is the single source of truth for how systems communicate. It enables code generation, automated testing, and ensures consistency across services.
|
||||
|
||||
**Primary Audience**:
|
||||
- **Developers**: Implement the API according to the spec
|
||||
- **QA Engineers**: Create test cases based on the spec
|
||||
- **Machines**: Used for code generation, validation, and documentation
|
||||
|
||||
**Format / Tooling**:
|
||||
- **OpenAPI (Swagger)**: REST API specifications
|
||||
- **Protobuf**: gRPC service definitions
|
||||
- **YAML/JSON**: Configuration and data schema definitions
|
||||
|
||||
**Best Practices**:
|
||||
- Use snake_case for consistency
|
||||
- Define all fields with types and constraints
|
||||
- Include examples for complex data structures
|
||||
- Keep specs versioned alongside code
|
||||
|
||||
**Example**: A `.yaml` file defining `user_id` as a UUID in snake_case.
|
||||
|
||||
---
|
||||
|
||||
### 3. Architecture
|
||||
|
||||
**Purpose**: High-level structural blueprint showing how components interact.
|
||||
|
||||
**Why it matters**: Architecture diagrams help everyone understand the system's structure without drowning in implementation details. They're crucial for onboarding, design reviews, and long-term maintainability.
|
||||
|
||||
**Primary Audience**:
|
||||
- **Senior Developers**: Design decisions and component responsibilities
|
||||
- **DevOps**: Understand deployment topology and service dependencies
|
||||
- **Technical Leads**: Evaluate trade-offs and scalability concerns
|
||||
|
||||
**Format / Tooling**:
|
||||
- **Mermaid.js**: Code-based diagrams that are version-controlled
|
||||
- **IcePanel**: Interactive, automated architecture visualization
|
||||
- **C4 Model**: Standardized approach to architectural diagrams
|
||||
|
||||
**Best Practices**:
|
||||
- Focus on *relationships* between components, not implementation details
|
||||
- Include technology choices (e.g., NATS vs WebSocket)
|
||||
- Show data flow direction with arrows
|
||||
- Update diagrams when architecture changes
|
||||
|
||||
**Example**: Diagram of SvelteKit ↔ NATS ↔ Julia 6-node cluster.
|
||||
|
||||
---
|
||||
|
||||
### 4. Walkthrough
|
||||
|
||||
**Purpose**: The intuition and "Big Picture" narrative.
|
||||
|
||||
**Why it matters**: Code alone doesn't explain *why* decisions were made. Walkthroughs provide context, historical decisions, and architectural intuition that helps new developers become productive quickly.
|
||||
|
||||
**Primary Audience**:
|
||||
- **New Developers**: Understand the system's philosophy and patterns
|
||||
- **The Team**: Share context and reasoning behind design choices
|
||||
- **Code Reviewers**: Evaluate design decisions alongside implementation
|
||||
|
||||
**Format / Tooling**:
|
||||
- **Recorded Video**: Personal, engaging, good for complex explanations
|
||||
- **TOUR.md**: Markdown file with narrative walk-through of the codebase
|
||||
- **Architecture Decision Records (ADRs)**: Formal documentation of key decisions
|
||||
|
||||
**Best Practices**:
|
||||
- Explain *why* more than *how*
|
||||
- Include anti-patterns to avoid
|
||||
- Link to related documentation
|
||||
- Keep walkthroughs updated with architecture changes
|
||||
|
||||
**Example**: "Why we use a Claim-Check pattern for large Arrow data."
|
||||
|
||||
---
|
||||
|
||||
### 5. Implementation
|
||||
|
||||
**Purpose**: The actual logic and generated code.
|
||||
|
||||
**Why it matters**: This is the executable truth of the system. Well-structured implementation code should be clear, maintainable, and follow established patterns.
|
||||
|
||||
**Primary Audience**:
|
||||
- **Developers**: Read, modify, and extend the code
|
||||
- **Reviewers**: Verify correctness and adherence to standards
|
||||
- **CI/CD**: Run tests and builds
|
||||
|
||||
**Format / Tooling**:
|
||||
- **SvelteKit**: Frontend framework with server-side rendering
|
||||
- **Julia**: High-performance numerical computing
|
||||
- **Node.js**: Backend services and tooling
|
||||
|
||||
**Best Practices**:
|
||||
- Generate code from specs to ensure consistency
|
||||
- Use consistent naming conventions (snake_case, camelCase appropriately)
|
||||
- Include unit tests alongside implementation
|
||||
- Document complex algorithms with inline comments
|
||||
|
||||
**Example**: Auto-generated TypeScript types from the OpenAPI spec.
|
||||
|
||||
---
|
||||
|
||||
### 6. Validation
|
||||
|
||||
**Purpose**: Automated "Contract" enforcement.
|
||||
|
||||
**Why it matters**: Automated tests ensure that the system behaves as specified and prevent regressions. Validation in CI/CD pipelines catches issues before they reach production.
|
||||
|
||||
**Primary Audience**:
|
||||
- **CI/CD Pipelines**: Run tests automatically on every commit
|
||||
- **QA Engineers**: Verify system behavior against requirements
|
||||
- **Developers**: Get immediate feedback on changes
|
||||
|
||||
**Format / Tooling**:
|
||||
- **GitHub Actions**: Automated testing and validation workflows
|
||||
- **Prism (ReadMe)**: OpenAPI spec validation in CI
|
||||
- **Jest/Vitest**: JavaScript testing framework
|
||||
- **Pytest**: Python testing framework
|
||||
|
||||
**Best Practices**:
|
||||
- Test the contract (spec) not just implementation details
|
||||
- Use contract testing (PACT) for service-to-service validation
|
||||
- Fail fast: tests should run quickly and provide clear error messages
|
||||
- Include negative test cases (invalid inputs, edge cases)
|
||||
|
||||
**Example**: A test that fails if the Julia API returns camelCase keys.
|
||||
|
||||
---
|
||||
|
||||
### 7. Runbook
|
||||
|
||||
**Purpose**: Deployment, scaling, and recovery procedures.
|
||||
|
||||
**Why it matters**: Runbooks ensure that deployments are consistent, repeatable, and recoverable. In GitOps, the runbook *is* the configuration, version-controlled alongside the code.
|
||||
|
||||
**Primary Audience**:
|
||||
- **DevOps Engineers**: Execute deployments and scaling operations
|
||||
- **SREs**: Manage system reliability and incident response
|
||||
- **Developers**: Deploy feature branches for testing
|
||||
|
||||
**Format / Tooling**:
|
||||
- **Kubernetes Manifests**: Declarative deployment configurations
|
||||
- **Flux**: GitOps operator for Kubernetes
|
||||
- **Helm Charts**: Package management for Kubernetes
|
||||
- **Docker Compose**: Local development environments
|
||||
|
||||
**Best Practices**:
|
||||
- Use Git as the source of truth (GitOps)
|
||||
- Make deployments idempotent (running twice has same effect)
|
||||
- Include rollback procedures
|
||||
- Document scaling procedures for different load levels
|
||||
|
||||
**Example**: `git push` to update the replica count from 3 to 6.
|
||||
|
||||
---
|
||||
|
||||
## How the Stack Fits Together
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Requirements │
|
||||
│ (Business goals, user needs) │
|
||||
└───────────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ The Spec │
|
||||
│ (Machine-readable contract: OpenAPI, Protobuf) │
|
||||
└───────────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Architecture │
|
||||
│ (Structural blueprint: Mermaid, IcePanel) │
|
||||
└───────────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Walkthrough │
|
||||
│ (Intuition, big picture narrative) │
|
||||
└───────────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Implementation │
|
||||
│ (Actual code: SvelteKit, Julia, Node.js) │
|
||||
└───────────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Validation │
|
||||
│ (Automated tests: GitHub Actions, Prism) │
|
||||
└───────────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Runbook │
|
||||
│ (Deployment, scaling: K8s, Flux) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Key Principles
|
||||
|
||||
1. **Machine-Readable Truth**: Specs and configurations should be machine-readable to enable automation
|
||||
2. **Separation of Concerns**: Different audiences need different types of information
|
||||
3. **Version Control**: All documentation should be in Git, just like code
|
||||
4. **Automation-First**: Validation should be automated and integrated into CI/CD
|
||||
5. **Living Documentation**: Documentation should evolve with the codebase
|
||||
|
||||
## Getting Started
|
||||
|
||||
To adopt this stack in your project:
|
||||
|
||||
1. Start with requirements in GitHub Issues or Notion
|
||||
2. Create a spec file (OpenAPI/Protobuf) as the contract
|
||||
3. Add architecture diagrams using Mermaid.js
|
||||
4. Write a walkthrough explaining the "why" behind decisions
|
||||
5. Implement code following the spec
|
||||
6. Add automated tests that validate the spec
|
||||
7. Create runbooks for deployment and scaling
|
||||
|
||||
This framework ensures that every piece of documentation serves a clear purpose and reaches the right audience.
|
||||
@@ -1,152 +0,0 @@
|
||||
# Software Design Document (SDD) Framework
|
||||
|
||||
A structured taxonomy of artifacts for comprehensive software documentation and design.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This framework categorizes Software Design Document artifacts by their purpose in the software development lifecycle. Each artifact type serves a specific role in defining **what** we build, **how** it works, and **how** we verify and document it.
|
||||
|
||||
---
|
||||
|
||||
## Artifact Taxonomy
|
||||
|
||||
| Document Type | "Artifact" Type | Purpose in SDD | Tooling Examples |
|
||||
|---------------|-----------------|----------------|------------------|
|
||||
| Requirements | User Stories / ADRs | Defines the "Business Contract" — why are we building this? | GitHub Issues, Jira, Architecture Decision Records (ADRs) |
|
||||
| The Spec | Schema Definition | The Source of Truth — defines data structures, types, and endpoints | OpenAPI (YAML), Protobuf, Apache Arrow Schemas |
|
||||
| Architecture | System Blueprint | Defines how the specs connect (e.g., Service A talks to B via NATS) | Mermaid.js (Diagrams-as-code), IcePanel, Structurizr |
|
||||
| Implementation | Generated Code | The code "fills in" the logic defined by the Spec | OpenAPI Generator, TypeSafe clients (Zod, TypeScript) |
|
||||
| Validation | Contract Tests | Automatically checks if the code matches the Spec | Prism (Mocking), Dredd, Schemathesis |
|
||||
| Tutorial | Interactive Sandbox | Allows others to "play" with the spec without writing code | Swagger UI, Redoc, Postman Collections |
|
||||
|
||||
---
|
||||
|
||||
## Purpose by Artifact Type
|
||||
|
||||
### 1. Requirements (User Stories / ADRs)
|
||||
|
||||
**Purpose:** Defines the "Business Contract" — why are we building this?
|
||||
|
||||
**Key Questions Answered:**
|
||||
- What problem are we solving?
|
||||
- Who are the stakeholders?
|
||||
- What are the success criteria?
|
||||
|
||||
**Examples:**
|
||||
- GitHub Issues
|
||||
- Jira tickets
|
||||
- Architecture Decision Records (ADRs)
|
||||
|
||||
---
|
||||
|
||||
### 2. The Spec (Schema Definition)
|
||||
|
||||
**Purpose:** The Source of Truth — defines data structures, types, and endpoints.
|
||||
|
||||
**Key Questions Answered:**
|
||||
- What data structures do we use?
|
||||
- What are the API endpoints?
|
||||
- What are the message formats?
|
||||
|
||||
**Examples:**
|
||||
- OpenAPI (YAML) specifications
|
||||
- Protocol Buffers (Protobuf)
|
||||
- Apache Arrow Schemas
|
||||
|
||||
---
|
||||
|
||||
### 3. Architecture (System Blueprint)
|
||||
|
||||
**Purpose:** Defines how the specs connect — the high-level structure and communication patterns.
|
||||
|
||||
**Key Questions Answered:**
|
||||
- How do services communicate?
|
||||
- What is the deployment topology?
|
||||
- What are the system boundaries?
|
||||
|
||||
**Examples:**
|
||||
- Mermaid.js diagrams (Diagrams-as-code)
|
||||
- IcePanel
|
||||
- Structurizr
|
||||
|
||||
---
|
||||
|
||||
### 4. Implementation (Generated Code)
|
||||
|
||||
**Purpose:** The code "fills in" the logic defined by the Spec.
|
||||
|
||||
**Key Questions Answered:**
|
||||
- How do we implement the spec?
|
||||
- What libraries/frameworks do we use?
|
||||
- How do we ensure type safety?
|
||||
|
||||
**Examples:**
|
||||
- OpenAPI Generator
|
||||
- TypeSafe clients (Zod, TypeScript)
|
||||
|
||||
---
|
||||
|
||||
### 5. Validation (Contract Tests)
|
||||
|
||||
**Purpose:** Automatically checks if the code matches the Spec.
|
||||
|
||||
**Key Questions Answered:**
|
||||
- Does the implementation match the spec?
|
||||
- Are there breaking changes?
|
||||
- Is the contract upheld?
|
||||
|
||||
**Examples:**
|
||||
- Prism (Mocking)
|
||||
- Dredd
|
||||
- Schemathesis
|
||||
|
||||
---
|
||||
|
||||
### 6. Tutorial (Interactive Sandbox)
|
||||
|
||||
**Purpose:** Allows others to "play" with the spec without writing code.
|
||||
|
||||
**Key Questions Answered:**
|
||||
- How do I use this API?
|
||||
- What are the expected inputs/outputs?
|
||||
- How can I experiment safely?
|
||||
|
||||
**Examples:**
|
||||
- Swagger UI
|
||||
- Redoc
|
||||
- Postman Collections
|
||||
|
||||
---
|
||||
|
||||
## Usage Guidelines
|
||||
|
||||
1. **Start with Requirements** — Define the business context and decision records
|
||||
2. **Define The Spec** — Create the source of truth for data and endpoints
|
||||
3. **Design Architecture** — Visualize how components connect
|
||||
4. **Implement** — Generate or write code based on the spec
|
||||
5. **Validate** — Run contract tests to ensure alignment
|
||||
6. **Tutorial** — Provide interactive documentation for users
|
||||
|
||||
---
|
||||
|
||||
## Relationship to NATSBridge
|
||||
|
||||
This NATSBridge project implements a NATS-based messaging bridge that can be used with all artifact types in this framework:
|
||||
|
||||
- **Architecture:** NATS as the communication backbone
|
||||
- **Spec:** Message schemas for bridge operations
|
||||
- **Implementation:** Generated code for NATS message handling
|
||||
- **Validation:** Contract tests for message format compliance
|
||||
- **Tutorial:** Interactive examples for bridge configuration
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Architecture Decision Records (ADRs): [http://adr.cascadely.com/](http://adr.cascadely.com/)
|
||||
- OpenAPI Specification: [https://spec.openapis.org/](https://spec.openapis.org/)
|
||||
- Protocol Buffers: [https://protobuf.dev/](https://protobuf.dev/)
|
||||
- Mermaid.js: [https://mermaid.js.org/](https://mermaid.js.org/)
|
||||
- Schemathesis: [https://schemathesis.io/](https://schemathesis.io/)
|
||||
Reference in New Issue
Block a user