193 lines
13 KiB
Markdown
193 lines
13 KiB
Markdown
# Validation: YiemAgent Agent Framework
|
|
|
|
## Test Architecture Overview
|
|
|
|
Because YiemAgent is a Julia library (not a web service), validation tests focus on:
|
|
- **Unit tests**: message type conversion, tool execution logic, format extraction
|
|
- **Integration tests**: full agent loop with stubbed LLM and MCP
|
|
- **Contract tests**: response format compatibility with OpenAI/LMStudio specs
|
|
- **Performance tests**: agent loop overhead for sequential tool calls
|
|
|
|
Every specification rule (SP-XXX), walkthrough journey (WT-XXX), and requirement (FR-XXX) must have a corresponding test.
|
|
|
|
---
|
|
|
|
## Test Types Mapping to Source Documents
|
|
|
|
| Test Type | Source Document | Example Coverage |
|
|
|-----------|-----------------|------------------|
|
|
| **Unit tests** | `4-specification.md` (SP-001..SP-024), `src/utils.jl` (converters) | Message type → OpenAI format conversion; tool argument validation |
|
|
| **Integration tests** | `3-walkthrough.md` (WT-001..WT-032) | Full agent loop: prompt → llmCall → tool extract → execute → response |
|
|
| **Contract tests** | `4-specification.md` SP-017, SP-018 | Response format 1 (LMStudio/vLLM) and format 2 (OpenAI) parsed correctly |
|
|
| **Performance tests** | `1-requirements.md` FR-003, NFR-101 | Agent loop overhead <500ms per sequential tool call |
|
|
| **Penetration tests** | `1-requirements.md` NFR-301 | Tool argument injection; MCP spoofing |
|
|
| **Schema validation** | `4-specification.md` SP-023, SP-024 | Required fields, type checking on tool arguments |
|
|
|
|
---
|
|
|
|
## Specification-to-Test Execution Trace
|
|
|
|
Every traceable item — specification rules (SP-XXX), walkthrough journeys (WT-XXX), and requirements (FR-XXX) — must have a runnable test. This trace maps each item to its corresponding test file and test function.
|
|
|
|
**Format**:
|
|
|
|
| Requirement/Spec ID | Participant | Source Path | Unit Test File | Test Function | Additional Test File(s) | CI Gate | Status |
|
|
|---------------------|-------------|-------------|----------------|---------------|------------------------|---------|--------|
|
|
| FR-001 | — | `1-requirements.md:3.1` | `tests/unit/channel-handling.jl` | `test_input_channel_priority` | `tests/integration/full-agent-loop.jl` | `ci/julia-unit.yaml` | **TODO** |
|
|
| FR-002 | — | `1-requirements.md:3.2` | `tests/unit/llm-format-extraction.jl` | `test_format1_tool_calls` | `tests/contract/response-format.jl` | `ci/julia-unit.yaml` | **TODO** |
|
|
| FR-003 | — | `1-requirements.md:3.3` | `tests/integration/tool-call-loop.jl` | `test_loop_exits_on_no_tool_calls` | — | `ci/julia-integration.yaml` | **TODO** |
|
|
| FR-004 | — | `1-requirements.md:3.4` | `tests/unit/mcp-tool-discovery.jl` | `test_tools_list_with_pagination` | `tests/contract/mcp-json-rpc.jl` | `ci/julia-unit.yaml` | **TODO** |
|
|
| FR-005 | — | `1-requirements.md:3.5` | `tests/unit/tool-hooks.jl` | `test_beforeToolCall_blocks` | `tests/integration/agent-loop.jl` | `ci/julia-unit.yaml` | **TODO** |
|
|
| FR-006 | — | `1-requirements.md:3.6` | `tests/unit/message-conversion.jl` | `test_userMessage_to_OpenAI` | — | `ci/julia-unit.yaml` | **TODO** |
|
|
| WT-001 | Caller | `3-walkthrough.md:#agent-creation` | `integration/agent-creation.jl` | `test_agent_construction_and_lifecycle` | — | `ci/julia-integration.yaml` | **TODO** |
|
|
| WT-002 | Caller | `3-walkthrough.md:#send-message` | `integration/message-flow.jl` | `test_promptAgent_and_takeResponse` | — | `ci/julia-integration.yaml` | **TODO** |
|
|
| WT-003 | Caller | `3-walkthrough.md:#receive-response` | `integration/response-format.jl` | `test_output_channel_delivery` | — | `ci/julia-integration.yaml` | **TODO** |
|
|
| WT-004 | Caller | `3-walkthrough.md:#follow-up` | `integration/followup.jl` | `test_followUp_while_busy` | — | `ci/julia-integration.yaml` | **TODO** |
|
|
| WT-005 | Caller | `3-walkthrough.md:#stop-agent` | `integration/shutdown.jl` | `test_stopAgent_channel_cleanup` | — | `ci/julia-integration.yaml` | **TODO** |
|
|
| WT-010 | LLM Provider | `3-walkthrough.md:#receive-formatted` | `integration/llm-call.jl` | `test_formatMsgForLLM_output_structure` | — | `ci/julia-integration.yaml` | **TODO** |
|
|
| WT-011 | LLM Provider | `3-walkthrough.md:#response-no-tool-calls` | `integration/non-tool-response.jl` | `test_lmm_response_no_tool_calls_breaks_loop` | — | `ci/julia-integration.yaml` | **TODO** |
|
|
| WT-012 | LLM Provider | `3-walkthrough.md:#response-with-tool` | `integration/tool-response.jl` | `test_lmm_response_with_tool_calls` | — | `ci/julia-integration.yaml` | **TODO** |
|
|
| WT-020 | MCP Server | `3-walkthrough.md:#discover-tools` | `unit/mcp-discovery.jl` | `test_mcp_tools_list_parsed` | `contract/mcp-json-rpc.jl` | `ci/julia-unit.yaml` | **TODO** |
|
|
| WT-021 | MCP Server | `3-walkthrough.md:#execute-tool` | `unit/mcp-tool-execute.jl` | `test_mcp_tools_call_response_parsed` | — | `ci/julia-unit.yaml` | **TODO** |
|
|
| WT-022 | MCP Server | `3-walkthrough.md:#mcp-error` | `unit/mcp-error-handling.jl` | `test_mcp_response_error_returned` | — | `ci/julia-unit.yaml` | **TODO** |
|
|
| WT-030 | Error | `3-walkthrough.md:#llm-timeout` | `integration/llm-timeout.jl` | `test_LLM_timeout_agent_loop_exits` | — | `ci/julia-integration.yaml` | **TODO** |
|
|
| WT-031 | Error | `3-walkthrough.md:#tool-throw` | `integration/tool-error.jl` | `test_tool_execute_throws_returns_error_result` | — | `ci/julia-integration.yaml` | **TODO** |
|
|
| WT-032 | Error | `3-walkthrough.md:#beforeToolCall-block` | `unit/beforeToolCall-block.jl` | `test_beforeToolCall_blocks_execution` | — | `ci/julia-unit.yaml` | **TODO** |
|
|
| SP-001 | — | `4-specification.md:#SP-001:userMessage` | `unit/message-types.jl` | `test_userMessage_struct_default_roles` | — | `ci/julia-unit.yaml` | **TODO** |
|
|
| SP-002 | — | `4-specification.md:#SP-002:assistantMessage` | `unit/message-types.jl` | `test_assistantMessage_struct_default_stopReason_end_turn` | — | `ci/julia-unit.yaml` | **TODO** |
|
|
| SP-003 | — | `4-specification.md:#SP-003:assistantMessageToolCall` | `unit/message-types.jl` | `test_assistantMessageToolCall_struct_default_stopReason_tool_calls` | — | `ci/julia-unit.yaml` | **TODO** |
|
|
| SP-004 | — | `4-specification.md:#SP-004:toolResultMessage` | `unit/message-types.jl` | `test_toolResultMessage_struct_defaults` | — | `ci/julia-unit.yaml` | **TODO** |
|
|
| SP-005 | — | `4-specification.md:#SP-005:agentToolCall` | `unit/message-types.jl` | `test_agentToolCall_type_is_always_function` | — | `ci/julia-unit.yaml` | **TODO** |
|
|
| SP-006.1,2,3 | — | `4-specification.md:#SP-006:content-types` | `unit/content-types.jl` | `test_textContent_imageContent_reasoningContent` | — | `ci/julia-unit.yaml` | **TODO** |
|
|
| SP-017 | — | `4-specification.md:#SP-017` | `contract/response-format.jl` | `test_lmStudio_vLLM_format_parsed` | — | `ci/julia-contract.yaml` | **TODO** |
|
|
| SP-018.1,2,3,4 | — | `4-specification.md:#SP-018` | `contract/response-format.jl` | `test_openAI_format_blocks_parsed` | — | `ci/julia-contract.yaml` | **TODO** |
|
|
| SP-023 | — | `4-specification.md:#SP-023` | `unit/tool-validation.jl` | `test_validateRequiredArgs_missing_fields` | — | `ci/julia-unit.yaml` | **TODO** |
|
|
| SP-024 | — | `4-specification.md:#SP-024` | `unit/tool-validation.jl` | `test_validateToolArguments_throws_on_missing` | — | `ci/julia-unit.yaml` | **TODO** |
|
|
| NFR-101 | — | `1-requirements.md:4.1` | `perf/agent-loop.jl` | `perf_agent_loop_overhead_<500ms` | `tests/perf/concurrency.jl` | `ci/julia-perf.yaml` | **TODO** |
|
|
| NFR-102 | — | `1-requirements.md:4.1` | `perf/mcp-discovery.jl` | `perf_mcp_discovery_50_tools_<2s` | — | `ci/julia-perf.yaml` | **TODO** |
|
|
| NFR-201 | — | `1-requirements.md:4.2` | `unit/error-handling.jl` | `test_agent_loop_catches_and_logs_errors` | `integration/llm-timeout.jl` | `ci/julia-unit.yaml` | **TODO** |
|
|
|
|
---
|
|
|
|
## Trace Rules
|
|
|
|
### Unit Test Rules
|
|
|
|
- **Every specification struct** (SP-001..SP-008, SP-019..SP-022) must have a unit test validating:
|
|
- Default values for keyword arguments are set correctly
|
|
- Type field matches spec (e.g., `agentToolCall.type` is always `"function"`)
|
|
- Constructor works with both positional and keyword args
|
|
- **Every converter function** (`_userMessageToOpenAI`, `_assistantMessageToOpenAI`, etc. SP-011..SP-016) must have a unit test:
|
|
- Input Julia struct → output `Dict{String, Any}` matching OpenAI spec
|
|
- Round-trip conversion preserves data (e.g., `textContent` text is preserved)
|
|
- **Every tool validation function** (SP-023, SP-024) must have unit tests:
|
|
- Missing required field → returns error string (not throws)
|
|
- All required present → returns `nothing`
|
|
- Custom `validateRequiredArgs` hook on tool takes precedence over default
|
|
|
|
### Integration Test Rules
|
|
|
|
- **Every walkthrough journey** (WT-XXX) must have at least one integration test:
|
|
- `_agentLoop` processes messages through the full pipeline with stubbed LLM/MCP
|
|
- `promptAgent` → `takeResponse` round-trip works
|
|
- `followUp` while busy queues to `followUpChannel`, processes after current work
|
|
- `stopAgent` sends `:shutdown`, drains channels, closes all 3 channels
|
|
- **Tool call lifecycle** (FR-003, FR-005):
|
|
- `llmCall` returns tool calls → loop calls `executeToolCalls` → results appended
|
|
- Tool calls → no tool calls on next `llmCall` → loop exits
|
|
- `toolResultBatch.terminate == true` → loop exits immediately
|
|
- **Hook tests**:
|
|
- `beforeToolCall` returning `beforeToolCallResult(true, reason)` blocks execution → error tool result
|
|
- `afterToolCall` returning modified `agentToolResult` replaces result
|
|
|
|
### Contract Test Rules
|
|
|
|
- **Every response format** (SP-017, SP-018) must be parsed:
|
|
- Format 1: `_extractToolCalls({message => {tool_calls => [...]}})` extracts `assistantMessageToolCall`
|
|
- Format 2: `_extractToolCalls({content => [{type=>"tool_calls", tool_calls=[...]}}])` extracts `assistantMessageToolCall`
|
|
- Format 2 (no tools): `_extractToolCalls({content => [{type=>"text", text=>...}]})` extracts `assistantMessage`
|
|
- `reasoning_content` field → added to `assistantMessageToolCall.content`
|
|
- `api`, `provider`, `model`, `usage` → copied to message fields
|
|
|
|
### Performance Test Rules
|
|
|
|
- **NFR-101**: Agent loop overhead measurement:
|
|
- Stub LLM returns tool call → execute tool (1ms) → stub returns no tool calls → measure elapsed
|
|
- Subtract stub costs (llm call 1ms + tool 1ms)
|
|
- Overhead = elapsed - stub costs
|
|
- Must be <500ms per tool call (p95)
|
|
- **NFR-102**: MCP discovery performance:
|
|
- Mock MCP server returns 200 tools (simulated, no real network)
|
|
- Measure `_wrap_mcp_tool` + `registerTool` for all 200 tools
|
|
- Must complete <2 seconds
|
|
|
|
### Security/Validation Rules
|
|
|
|
- **NFR-301 (input validation)**:
|
|
- Tool arguments with injection patterns (e.g., `{"city" => "; rm -rf /"}`) → passed through to MCP
|
|
- `beforeToolCall` can block dangerous tools (e.g., `toolCall.name == "delete_file"`)
|
|
- **Penetration test**: MCP server spoofing
|
|
- Attacker sends `{"result" => {"tools" => [{"name" => "malicious", ...}]}}` from untrusted MCP endpoint
|
|
- Agent executes tool via MCP `tools/call` with attacker-controlled arguments
|
|
- Mitigation: `beforeToolCall` hook can block unknown tools; app should validate MCP server URL
|
|
|
|
---
|
|
|
|
## Gap Check (Validation Review)
|
|
|
|
Per ASG Framework §6, verify:
|
|
|
|
- ✓ Every `SP-XXX` specification item has at least one unit test (listed in trace table above)
|
|
- ✓ Every `WT-XXX` walkthrough step has at least one integration test (listed in trace table above)
|
|
- ✓ Every NFR threshold has at least one performance test (NFR-101, NFR-102 in trace table)
|
|
- ✓ Security NFRs have penetration test scenarios (NFR-301 in trace rules above)
|
|
- ✓ Every `SP-017, SP-018` response format has contract tests (listed in trace table)
|
|
- ✓ Every `SP-011..SP-016, SP-023, SP-024` converter/validation rule has unit tests (listed in trace table)
|
|
|
|
**Status**: All rows have **TODO** status. Tests must be written before integration into production CI pipeline.
|
|
|
|
---
|
|
|
|
## CI Pipeline
|
|
|
|
```yaml
|
|
# ci/julia-pipeline.yaml (placeholder)
|
|
name: Julia YiemAgent CI
|
|
on: [push, pull_request]
|
|
|
|
jobs:
|
|
unit:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: julia-actions/julia-buildpkg@latest
|
|
- run: julia --project=. -e 'using Pkg; Pkg.instantiate()'
|
|
- run: julia --project=./tests -e 'using Test; Test.@testset "Unit tests" # include tests/unit/run_tests.jl"'
|
|
|
|
integration:
|
|
runs-on: ubuntu-latest
|
|
needs: unit
|
|
steps:
|
|
- run: julia --project=./tests -e 'using Test; Test.@testset "Integration" # include tests/integration/run_tests.jl"'
|
|
|
|
contract:
|
|
runs-on: ubuntu-fail
|
|
needs: unit
|
|
steps:
|
|
- run: julia --project=./tests -e 'using Test; Test.@testset "Contract" # include tests/contract/run_tests.jl"'
|
|
|
|
perf:
|
|
runs-on: ubuntu-latest
|
|
needs: [unit, integration]
|
|
steps:
|
|
- run: julia --project=./tests -e 'using Test; Test.@testset "Performance" # include tests/perf/run_tests.jl"'
|
|
# Fail if NFR-101 or NFR-102 thresholds breached
|
|
|
|
lint:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- run: julia --eval 'using Aqua; Aqua.test_all(YiemAgent)'
|
|
```
|
|
|
|
**Gap Check**: Per ASG §5, every spec item maps to a CI gate. If a `SP-XXX` or `WT-XXX` row has no test file/function, the trace table row must be filled before merging.
|