# Validation: YiemAgent Agent Framework

## Test Architecture Overview

Because YiemAgent is a Julia library (not a web service), validation tests focus on:
- **Unit tests**: message type conversion, tool execution logic, format extraction
- **Integration tests**: full agent loop with stubbed LLM and MCP
- **Contract tests**: response format compatibility with OpenAI/LMStudio specs
- **Performance tests**: agent loop overhead for sequential tool calls

Every specification rule (SP-XXX), walkthrough journey (WT-XXX), and requirement (FR-XXX) must have a corresponding test.

---

## Test Types Mapping to Source Documents

| Test Type | Source Document | Example Coverage |
|-----------|-----------------|------------------|
| **Unit tests** | `4-specification.md` (SP-001..SP-024), `src/utils.jl` (converters) | Message type → OpenAI format conversion; tool argument validation |
| **Integration tests** | `3-walkthrough.md` (WT-001..WT-032) | Full agent loop: prompt → llmCall → tool extract → execute → response |
| **Contract tests** | `4-specification.md` SP-017, SP-018 | Response format 1 (LMStudio/vLLM) and format 2 (OpenAI) parsed correctly |
| **Performance tests** | `1-requirements.md` FR-003, NFR-101 | Agent loop overhead <500ms per sequential tool call |
| **Penetration tests** | `1-requirements.md` NFR-301 | Tool argument injection; MCP spoofing |
| **Schema validation** | `4-specification.md` SP-023, SP-024 | Required fields, type checking on tool arguments |

---

## Specification-to-Test Execution Trace

Every traceable item — specification rules (SP-XXX), walkthrough journeys (WT-XXX), and requirements (FR-XXX) — must have a runnable test. This trace maps each item to its corresponding test file and test function.

**Format**:

| Requirement/Spec ID | Participant | Source Path | Unit Test File | Test Function | Additional Test File(s) | CI Gate | Status |
|---------------------|-------------|-------------|----------------|---------------|------------------------|---------|--------|
| FR-001 | — | `1-requirements.md:3.1` | `tests/unit/channel-handling.jl` | `test_input_channel_priority` | `tests/integration/full-agent-loop.jl` | `ci/julia-unit.yaml` | **TODO** |
| FR-002 | — | `1-requirements.md:3.2` | `tests/unit/llm-format-extraction.jl` | `test_format1_tool_calls` | `tests/contract/response-format.jl` | `ci/julia-unit.yaml` | **TODO** |
| FR-003 | — | `1-requirements.md:3.3` | `tests/integration/tool-call-loop.jl` | `test_loop_exits_on_no_tool_calls` | — | `ci/julia-integration.yaml` | **TODO** |
| FR-004 | — | `1-requirements.md:3.4` | `tests/unit/mcp-tool-discovery.jl` | `test_tools_list_with_pagination` | `tests/contract/mcp-json-rpc.jl` | `ci/julia-unit.yaml` | **TODO** |
| FR-005 | — | `1-requirements.md:3.5` | `tests/unit/tool-hooks.jl` | `test_beforeToolCall_blocks` | `tests/integration/agent-loop.jl` | `ci/julia-unit.yaml` | **TODO** |
| FR-006 | — | `1-requirements.md:3.6` | `tests/unit/message-conversion.jl` | `test_userMessage_to_OpenAI` | — | `ci/julia-unit.yaml` | **TODO** |
| WT-001 | Caller | `3-walkthrough.md:#agent-creation` | `integration/agent-creation.jl` | `test_agent_construction_and_lifecycle` | — | `ci/julia-integration.yaml` | **TODO** |
| WT-002 | Caller | `3-walkthrough.md:#send-message` | `integration/message-flow.jl` | `test_promptAgent_and_takeResponse` | — | `ci/julia-integration.yaml` | **TODO** |
| WT-003 | Caller | `3-walkthrough.md:#receive-response` | `integration/response-format.jl` | `test_output_channel_delivery` | — | `ci/julia-integration.yaml` | **TODO** |
| WT-004 | Caller | `3-walkthrough.md:#follow-up` | `integration/followup.jl` | `test_followUp_while_busy` | — | `ci/julia-integration.yaml` | **TODO** |
| WT-005 | Caller | `3-walkthrough.md:#stop-agent` | `integration/shutdown.jl` | `test_stopAgent_channel_cleanup` | — | `ci/julia-integration.yaml` | **TODO** |
| WT-010 | LLM Provider | `3-walkthrough.md:#receive-formatted` | `integration/llm-call.jl` | `test_formatMsgForLLM_output_structure` | — | `ci/julia-integration.yaml` | **TODO** |
| WT-011 | LLM Provider | `3-walkthrough.md:#response-no-tool-calls` | `integration/non-tool-response.jl` | `test_lmm_response_no_tool_calls_breaks_loop` | — | `ci/julia-integration.yaml` | **TODO** |
| WT-012 | LLM Provider | `3-walkthrough.md:#response-with-tool` | `integration/tool-response.jl` | `test_lmm_response_with_tool_calls` | — | `ci/julia-integration.yaml` | **TODO** |
| WT-020 | MCP Server | `3-walkthrough.md:#discover-tools` | `unit/mcp-discovery.jl` | `test_mcp_tools_list_parsed` | `contract/mcp-json-rpc.jl` | `ci/julia-unit.yaml` | **TODO** |
| WT-021 | MCP Server | `3-walkthrough.md:#execute-tool` | `unit/mcp-tool-execute.jl` | `test_mcp_tools_call_response_parsed` | — | `ci/julia-unit.yaml` | **TODO** |
| WT-022 | MCP Server | `3-walkthrough.md:#mcp-error` | `unit/mcp-error-handling.jl` | `test_mcp_response_error_returned` | — | `ci/julia-unit.yaml` | **TODO** |
| WT-030 | Error | `3-walkthrough.md:#llm-timeout` | `integration/llm-timeout.jl` | `test_LLM_timeout_agent_loop_exits` | — | `ci/julia-integration.yaml` | **TODO** |
| WT-031 | Error | `3-walkthrough.md:#tool-throw` | `integration/tool-error.jl` | `test_tool_execute_throws_returns_error_result` | — | `ci/julia-integration.yaml` | **TODO** |
| WT-032 | Error | `3-walkthrough.md:#beforeToolCall-block` | `unit/beforeToolCall-block.jl` | `test_beforeToolCall_blocks_execution` | — | `ci/julia-unit.yaml` | **TODO** |
| SP-001 | — | `4-specification.md:#SP-001:userMessage` | `unit/message-types.jl` | `test_userMessage_struct_default_roles` | — | `ci/julia-unit.yaml` | **TODO** |
| SP-002 | — | `4-specification.md:#SP-002:assistantMessage` | `unit/message-types.jl` | `test_assistantMessage_struct_default_stopReason_end_turn` | — | `ci/julia-unit.yaml` | **TODO** |
| SP-003 | — | `4-specification.md:#SP-003:assistantMessageToolCall` | `unit/message-types.jl` | `test_assistantMessageToolCall_struct_default_stopReason_tool_calls` | — | `ci/julia-unit.yaml` | **TODO** |
| SP-004 | — | `4-specification.md:#SP-004:toolResultMessage` | `unit/message-types.jl` | `test_toolResultMessage_struct_defaults` | — | `ci/julia-unit.yaml` | **TODO** |
| SP-005 | — | `4-specification.md:#SP-005:agentToolCall` | `unit/message-types.jl` | `test_agentToolCall_type_is_always_function` | — | `ci/julia-unit.yaml` | **TODO** |
| SP-006.1,2,3 | — | `4-specification.md:#SP-006:content-types` | `unit/content-types.jl` | `test_textContent_imageContent_reasoningContent` | — | `ci/julia-unit.yaml` | **TODO** |
| SP-017 | — | `4-specification.md:#SP-017` | `contract/response-format.jl` | `test_lmStudio_vLLM_format_parsed` | — | `ci/julia-contract.yaml` | **TODO** |
| SP-018.1,2,3,4 | — | `4-specification.md:#SP-018` | `contract/response-format.jl` | `test_openAI_format_blocks_parsed` | — | `ci/julia-contract.yaml` | **TODO** |
| SP-023 | — | `4-specification.md:#SP-023` | `unit/tool-validation.jl` | `test_validateRequiredArgs_missing_fields` | — | `ci/julia-unit.yaml` | **TODO** |
| SP-024 | — | `4-specification.md:#SP-024` | `unit/tool-validation.jl` | `test_validateToolArguments_throws_on_missing` | — | `ci/julia-unit.yaml` | **TODO** |
| NFR-101 | — | `1-requirements.md:4.1` | `perf/agent-loop.jl` | `perf_agent_loop_overhead_<500ms` | `tests/perf/concurrency.jl` | `ci/julia-perf.yaml` | **TODO** |
| NFR-102 | — | `1-requirements.md:4.1` | `perf/mcp-discovery.jl` | `perf_mcp_discovery_50_tools_<2s` | — | `ci/julia-perf.yaml` | **TODO** |
| NFR-201 | — | `1-requirements.md:4.2` | `unit/error-handling.jl` | `test_agent_loop_catches_and_logs_errors` | `integration/llm-timeout.jl` | `ci/julia-unit.yaml` | **TODO** |

---

## Trace Rules

### Unit Test Rules

- **Every specification struct** (SP-001..SP-008, SP-019..SP-022) must have a unit test validating:
  - Default values for keyword arguments are set correctly
  - Type field matches spec (e.g., `agentToolCall.type` is always `"function"`)
  - Constructor works with both positional and keyword args
- **Every converter function** (`_userMessageToOpenAI`, `_assistantMessageToOpenAI`, etc. SP-011..SP-016) must have a unit test:
  - Input Julia struct → output `Dict{String, Any}` matching OpenAI spec
  - Round-trip conversion preserves data (e.g., `textContent` text is preserved)
- **Every tool validation function** (SP-023, SP-024) must have unit tests:
  - Missing required field → returns error string (not throws)
  - All required present → returns `nothing`
  - Custom `validateRequiredArgs` hook on tool takes precedence over default

### Integration Test Rules

- **Every walkthrough journey** (WT-XXX) must have at least one integration test:
  - `_agentLoop` processes messages through the full pipeline with stubbed LLM/MCP
  - `promptAgent` → `takeResponse` round-trip works
  - `followUp` while busy queues to `followUpChannel`, processes after current work
  - `stopAgent` sends `:shutdown`, drains channels, closes all 3 channels
- **Tool call lifecycle** (FR-003, FR-005):
  - `llmCall` returns tool calls → loop calls `executeToolCalls` → results appended
  - Tool calls → no tool calls on next `llmCall` → loop exits
  - `toolResultBatch.terminate == true` → loop exits immediately
- **Hook tests**:
  - `beforeToolCall` returning `beforeToolCallResult(true, reason)` blocks execution → error tool result
  - `afterToolCall` returning modified `agentToolResult` replaces result

### Contract Test Rules

- **Every response format** (SP-017, SP-018) must be parsed:
  - Format 1: `_extractToolCalls({message => {tool_calls => [...]}})` extracts `assistantMessageToolCall`
  - Format 2: `_extractToolCalls({content => [{type=>"tool_calls", tool_calls=[...]}}])` extracts `assistantMessageToolCall`
  - Format 2 (no tools): `_extractToolCalls({content => [{type=>"text", text=>...}]})` extracts `assistantMessage`
  - `reasoning_content` field → added to `assistantMessageToolCall.content`
  - `api`, `provider`, `model`, `usage` → copied to message fields

### Performance Test Rules

- **NFR-101**: Agent loop overhead measurement:
  - Stub LLM returns tool call → execute tool (1ms) → stub returns no tool calls → measure elapsed
  - Subtract stub costs (llm call 1ms + tool 1ms)
  - Overhead = elapsed - stub costs
  - Must be <500ms per tool call (p95)
- **NFR-102**: MCP discovery performance:
  - Mock MCP server returns 200 tools (simulated, no real network)
  - Measure `_wrap_mcp_tool` + `registerTool` for all 200 tools
  - Must complete <2 seconds

### Security/Validation Rules

- **NFR-301 (input validation)**:
  - Tool arguments with injection patterns (e.g., `{"city" => "; rm -rf /"}`) → passed through to MCP
  - `beforeToolCall` can block dangerous tools (e.g., `toolCall.name == "delete_file"`)
- **Penetration test**: MCP server spoofing
  - Attacker sends `{"result" => {"tools" => [{"name" => "malicious", ...}]}}` from untrusted MCP endpoint
  - Agent executes tool via MCP `tools/call` with attacker-controlled arguments
  - Mitigation: `beforeToolCall` hook can block unknown tools; app should validate MCP server URL

---

## Gap Check (Validation Review)

Per ASG Framework §6, verify:

- ✓ Every `SP-XXX` specification item has at least one unit test (listed in trace table above)
- ✓ Every `WT-XXX` walkthrough step has at least one integration test (listed in trace table above)
- ✓ Every NFR threshold has at least one performance test (NFR-101, NFR-102 in trace table)
- ✓ Security NFRs have penetration test scenarios (NFR-301 in trace rules above)
- ✓ Every `SP-017, SP-018` response format has contract tests (listed in trace table)
- ✓ Every `SP-011..SP-016, SP-023, SP-024` converter/validation rule has unit tests (listed in trace table)

**Status**: All rows have **TODO** status. Tests must be written before integration into production CI pipeline.

---

## CI Pipeline

```yaml
# ci/julia-pipeline.yaml (placeholder)
name: Julia YiemAgent CI
on: [push, pull_request]

jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: julia-actions/julia-buildpkg@latest
      - run: julia --project=. -e 'using Pkg; Pkg.instantiate()'
      - run: julia --project=./tests -e 'using Test; Test.@testset "Unit tests" # include tests/unit/run_tests.jl"'
  
  integration:
    runs-on: ubuntu-latest
    needs: unit
    steps:
      - run: julia --project=./tests -e 'using Test; Test.@testset "Integration" # include tests/integration/run_tests.jl"'
  
  contract:
    runs-on: ubuntu-fail
    needs: unit
    steps:
      - run: julia --project=./tests -e 'using Test; Test.@testset "Contract" # include tests/contract/run_tests.jl"'
  
  perf:
    runs-on: ubuntu-latest
    needs: [unit, integration]
    steps:
      - run: julia --project=./tests -e 'using Test; Test.@testset "Performance" # include tests/perf/run_tests.jl"'
    # Fail if NFR-101 or NFR-102 thresholds breached
  
  lint:
    runs-on: ubuntu-latest
    steps:
      - run: julia --eval 'using Aqua; Aqua.test_all(YiemAgent)'
```

**Gap Check**: Per ASG §5, every spec item maps to a CI gate. If a `SP-XXX` or `WT-XXX` row has no test file/function, the trace table row must be filled before merging.