From e561b4b1aec0ac876e4f1660ebd5a750b5c54db9 Mon Sep 17 00:00:00 2001 From: narawat Date: Fri, 11 Sep 2026 08:28:49 +0700 Subject: [PATCH] update docs --- docs/1-requirements.md | 155 ++++++++++ docs/2-solution_design.md | 119 ++++++++ docs/3-walkthrough.md | 318 +++++++++++++++++++++ docs/4-specification.md | 518 ++++++++++++++++++++++++++++++++++ docs/6-validation.md | 192 +++++++++++++ docs/7-implementation-plan.md | 217 ++++++++++++++ docs/8-runbook.md | 445 +++++++++++++++++++++++++++++ 7 files changed, 1964 insertions(+) create mode 100644 docs/1-requirements.md create mode 100644 docs/2-solution_design.md create mode 100644 docs/3-walkthrough.md create mode 100644 docs/4-specification.md create mode 100644 docs/6-validation.md create mode 100644 docs/7-implementation-plan.md create mode 100644 docs/8-runbook.md diff --git a/docs/1-requirements.md b/docs/1-requirements.md new file mode 100644 index 0000000..4738814 --- /dev/null +++ b/docs/1-requirements.md @@ -0,0 +1,155 @@ +# YiemAgent: Agile Agent Framework + +## 1. Business Context & Success Metrics + +### Business Goal + +YiemAgent is a Julia-based AI agent framework that enables building chatbot assistants with tool-calling capabilities, model-agnostic LLM integration, and runtime tool discovery via MCP (Model Context Protocol). It provides an asynchronous agent loop with channel-based message passing, supporting both single-turn and multi-turn conversations with automatic tool call resolution. + +### User Stories + +- **US-001**: As a developer, I want to create an AI agent with a simple constructor so that I can quickly prototype chatbot functionality in Julia +- **US-002**: As a developer, I want the agent to discover tools automatically from an MCP server so that I don't need to manually register tool APIs +- **US-003**: As a developer, I want the agent to handle tool calls automatically in a loop so that the LLM can use tools without external orchestration +- **US-004**: As a developer, I want to intercept tool calls with before/after hooks so that I can add user approval, logging, or result sanitization +- **US-005**: As a developer, I want the framework to support multiple LLM models and providers through a pluggable `llmCall` interface so that I can swap models without changing agent logic + +### KPIs & Targets + +- **KPI-001**: Agent loop processes messages with <500ms overhead for sequential tool calls (validates US-003) +- **KPI-002**: MCP tool discovery completes within 2 seconds for 50+ tools (validates US-002) +- **KPI-003**: 100% of message types have OpenAI format conversion (validates US-001) + +## 2. Technical Boundaries + +### In Scope + +- Julia module implementing an AI agent with asynchronous message processing +- Channel-based input (user→agent), follow-up (async queue), and output (agent→user) interfaces +- LLM model abstraction with tool call extraction from multiple response formats (OpenAI content blocks, LMStudio/vLLM format) +- Tool registry with MCP server integration (tools/list, tools/call via JSON-RPC 2.0) +- Agent loop with before/after tool call hooks for lifecycle management +- Message type system with OpenAI format conversion utilities +- Abort signal and termination flag support via tool implementations + +### Out of Scope + +- GUI/CLI interface (exposed via channels only; UI layer is external) +- Persistent chat history or long-term memory (TODO comments in code) +- Multi-agent orchestration (single agent instance per `yiemAgent` struct) +- Embedding generation or retrieval-augmented generation (RAG) +- Natural language-to-code generation or execution sandboxes + +### Dependencies + +- **Julia Standard Library**: `Dates`, `UUIDs`, `Base.Threads` +- **External Packages**: `JSON`, `DataStructures`, `HTTP`, `PrettyPrinting`, `Serialization`, `DataFrames` +- **Internal Package**: `GeneralUtils` (project-specific utility library) +- **Optional**: `NATS` (MCP server communication), `LibPQ` (PostgreSQL) + +### Deployment Constraints + +- **NFR-001**: Library designed for embedded use in Docker containers (MicroK8s cluster, per `NFR-501` from template) +- **NFR-002**: Must support Julia 1.9+ for module include/export semantics compatibility +- **NFR-003**: Thread-safe channel operations; single-agent instance processes messages sequentially on background `@spawn` task + +## 3. Functional Requirements (FR) + +### FR-001: Agent Message Processing + +The agent shall process user messages through an asynchronous loop that: +- Accepts messages via `inputChannel` (capacity 16) and `followUpChannel` (capacity 32) +- Prioritizes `inputChannel` messages over `followUpChannel` messages +- Feeds responses to `outputChannel` (capacity 16) in OpenAI message format +- Exits cleanly on `:shutdown` signal + +**Traceability**: US-001, US-003 + +### FR-002: LLM Model Agnosticism + +The agent shall accept a pluggable `llmCall` function that: +- Receives formatted messages and returns an LLM response dict +- The response dict may contain `message.tool_calls` (LMStudio format) or `content` blocks (OpenAI format) +- The framework extracts tool calls regardless of response format + +**Traceability**: US-005 + +### FR-003: Tool Call Execution Loop + +The agent shall automatically execute tool calls returned by the LLM: +- Extract tool calls via `_extractToolCalls()` which supports both `message.tool_calls` and `content[{type="tool_calls"}]` formats +- Build `agentContext` via `prepareContext()` (pluggable hook) +- Format messages via `formatMsgForLLM()` to OpenAI format +- Extract tool calls → loop: execute tool calls → append results to history → call LLM again +- Break loop when LLM response has no tool calls +- Support `terminate` flag: if `toolResultBatch.terminate == true`, break loop and return final response + +**Traceability**: US-003 + +### FR-004: Tool Discovery and Registration + +The agent shall: +- Query MCP server via `mcpserver("tools/list")` (JSON-RPC 2.0) to discover tools +- Support pagination via `nextCursor` field +- Wrap each MCP tool definition as an `agentTool` with `execute` hook that calls `mcpserver("tools/call", name, args, sessionId)` +- Register a builtin `listTools` tool that triggers runtime discovery and lists all available tools +- Skip tools already registered (idempotent) + +**Traceability**: US-002 + +### FR-005: Tool Call Lifecycle Hooks + +The agent shall support two lifecycle hooks: +- `beforeToolCall(beforeToolCallContext, signal)` — can block execution by returning `beforeToolCallResult(true, reason)` +- `afterToolCall(afterToolCallContext)` — can transform result by returning modified `agentToolResult` +- Errors from hooks are caught and converted to `immediateOutcome`/error tool result messages (never throw) + +**Traceability**: US-004 + +### FR-006: Message Type Conversion + +The agent shall: +- Convert internal message types (`userMessage`, `assistantMessage`, `assistantMessageToolCall`, `toolResultMessage`) to OpenAI format via `_userMessageToOpenAI()`, `_assistantMessageToOpenAI()`, `_assistantMessageToolCallToOpenAI()`, `_toolResultMessageToOpenAI()` +- Convert internal `messageContent` types (`textContent`, `imageContent`, `reasoningContent`) to OpenAI content blocks via `_messageContentToBlocks()` +- Convert `agentTool` definitions to OpenAI function tool format via `_toolsToOpenAI()` + +**Traceability**: US-001 + +## 4. Non-Functional Requirements (NFRs) + +### 4.1 Performance & Scalability + +- **NFR-101**: Agent loop message processing overhead <500ms per sequential tool call (measured KPI: KPI-001) +- **NFR-102**: MCP tool discovery for 50+ tools completes within 2 seconds (measured KPI: KPI-002) +- **NFR-103**: Channel buffers prevent unbounded memory growth — `inputChannel` (16), `followUpChannel` (32), `outputChannel` (16) + +### 4.2 Availability & Reliability + +- **NFR-201**: Agent loop catches all errors via `try/catch` in `_agentLoop()` and calls `eventSink(errMsg)` before exit (measured by event sink logging) +- **NFR-202**: `prepareToolCall()` never throws errors — validation failures return `immediateOutcome` with error tool result + +### 4.3 Privacy & Security + +- **Data Classification**: Tool arguments and results may contain PII (user messages); MCP tool results treated as untrusted input +- **No built-in auth/authz**: Responsibility of `mcpServer` implementation and `beforeToolCall` hook +- **Input validation**: JSON schema validation via `validateToolArguments()` checks required fields only (tools need custom validation via `validateRequiredArgs` hook) + +### 4.4 Observability & Telemetry + +- **Required Logs**: `agentLoop` events (via `eventSink`) emit state machine steps (`_agentLoop 1` through `_agentLoop 6`), `_processMessage` steps (1 through 18), `prepareToolCall` steps (1 through 10) +- **Critical Metrics**: + - `agent._state.messages.length` — conversation history size (logged at each step) + - `toolExecutionStart`/`toolExecutionEnd` events (via `toolExecStartEvent`, `toolExecEndEvent` structs) +- **Retention**: No built-in log retention; agent is a library, does not manage external logging + +## 5. Acceptance Conditions + +- [ ] **FR-001**: Agent processes messages through channel system with priority ordering (inputChannel > followUpChannel) +- [ ] **FR-002**: `llmCall` is pluggable; response formats supported include `message.tool_calls` dict and `content` array blocks +- [ ] **FR-003**: Agent loops on tool calls until LLM returns no tool calls or terminate flag is set +- [ ] **FR-004**: MCP tools are discovered via `tools/list` with pagination; all registered tools are callable via `tools/call` +- [ ] **FR-005**: `beforeToolCall` hook can block; `afterToolCall` hook can transform result; errors return error tool result +- [ ] **FR-006**: All message types (userMessage, assistantMessage, assistantMessageToolCall, toolResultMessage) convert to OpenAI format +- [ ] **NFR-101**: Message processing overhead <500ms per sequential tool call +- [ ] **NFR-102**: MCP discovery for 50+ tools completes <2 seconds +- [ ] **NFR-201**: Agent loop catches errors and exits gracefully without panics diff --git a/docs/2-solution_design.md b/docs/2-solution_design.md new file mode 100644 index 0000000..047a1b5 --- /dev/null +++ b/docs/2-solution_design.md @@ -0,0 +1,119 @@ +# Solution Design: YiemAgent Agent Framework + +## 1. Problem Decomposition + +| Problem | Description | User Impact | +|---------|-------------|-------------| +| P-001 | Developers need a chatbot framework in Julia that handles LLM calls and tool calls without writing boilerplate | Must write separate request/response handling, tool call parsing, and loop management code for each agent | +| P-002 | Tool discovery is manual and repetitive | Must register every tool individually; no automatic discovery from MCP-compatible servers | +| P-003 | Multiple LLM providers have different response formats | Must handle `message.tool_calls` (LMStudio/vLLM) vs `content` blocks (OpenAI API) separately | +| P-004 | Tool execution needs lifecycle hooks for observability and control | No built-in before/after hooks for authorization, logging, or result sanitization | +| P-005 | Agent state management is error-prone | Must handle message history, tool registration, and pending tool call tracking manually | + +## 2. Solution Approach + +**Approach**: "Build a channel-based agent system where the LLM talks to tools automatically, tools talk to MCP servers automatically, and you compose agents by injecting `llmCall` and `mcpServer` functions — the framework owns the loop, the state, and the format conversion." + +**Key Principles**: +- **Channel-based async processing**: Each agent owns three channels (`inputChannel`, `followUpChannel`, `outputChannel`); a background `@spawn` task runs the agent loop, processing messages in priority order +- **Tool call loop**: The agent calls `llmCall()`, extracts tool calls, executes them (sequentially or in parallel), appends results, and calls `llmCall()` again — repeating until no tool calls remain +- **Pluggable hooks**: `prepareContext`, `formatMsgForLLM`, `beforeToolCall`, `afterToolCall`, and `llmCall` are all injectable functions — the framework provides sensible defaults, but you override what you need +- **MCP-native tool discovery**: Tools are discovered at startup via MCP `tools/list` and can be discovered at runtime via the `listTools` builtin tool (idempotent registration with pagination support) +- **Format-agnostic LLM responses**: `_extractToolCalls()` handles both OpenAI content blocks and LMStudio/vLLM nested message format, allowing the same agent to work with any model provider + +## 3. Decision Rationale + +| Decision ID | Decision | Rationale | Alternatives Rejected | +|-------------|----------|-----------|----------------------| +| SD-001 | Use channel-based async processing with `@spawn` task | Provides non-blocking message queues; `inputChannel` blocks if full (backpressure); `followUpChannel` allows async follow-ups while agent is busy | Threading with shared state — race conditions; callback-based async — hard to compose | +| SD-002 | Background `_agentLoop` task runs message dispatch loop | Isolates message processing from construction; agent can be created and used concurrently | Synchronous agent.run(message) — blocks caller; manual task management — boilerplate | +| SD-003 | `_processMessage` runs `llmCall()` in an inner loop (while loop) until no tool calls remain | Handles multi-turn tool use transparently — agent decides when to stop based on LLM response | External loop in caller code — duplicates agent loop logic; single-turn only — no multi-step tool use | +| SD-004 | MCP server integration via callable struct `mcpServer(method, args...)` | MCP is model-agnostic protocol; callable struct allows type-stable single dispatch; supports NATS transport (struct has `natsConn`, `topic`, `senderID` fields) | Ad-hoc HTTP client for MCP — breaks NATS-only transports; tight coupling to specific transport | +| SD-005 | Tool call lifecycle: `prepareToolCall()` → `executePreparedToolCall()` → `finalizeExecutedToolCall()` — three phases, never throw errors | Preparation validates args and runs `beforeToolCall` hook (can block); execution calls `tool.execute()`; finalization runs `afterToolCall` hook. Each phase returns a result type, not exceptions — agent loop catches all and builds error tool result | `try/catch` in loop — loses context; throwing errors breaks tool result message contract | +| SD-006 | Message types are Julia structs with keyword constructors (`userMessage(content=[...])`) | Type stable; keyword constructors provide defaults (`role="user"`, `timestamp=now()`); supports both positional and keyword usage | `Dict{String, Any}` for messages — lose type safety, docgen, IDE support | +| SD-007 | `eventSink` function passed to all deep lifecycle calls | Provides hooks for external monitoring (logging, metrics, NATS pub/sub) without modifying core logic | Built-in `println` — can't disable or redirect; no metrics integration | +| SD-008 | `abortSignal` struct with `.aborted::Bool` field passed to `prepareToolCall()` and hooks | Simple mutable state for interrupting long-running tool calls (no async cancellation — just flag check) | Threading with `Task.cancel()` — Julia thread model doesn't support it; `@async` spawn — complex cleanup | +| SD-009 | `_extractToolCalls()` supports both `message.tool_calls` and `content[{type="tool_calls"}]` formats | LMStudio.jl/vLLM vs OpenAI API have different response shapes; framework must work with both | Separate parser per provider — maintenance burden; require provider wrapper — adds indirection | + +## 4. High-Level Component Diagram + +```mermaid +flowchart TD + User[External Caller] -->|promptAgent / followUp| InputCh[inputChannel] + + subagent AgentLoop [Agent Loop Task] + InputCh -->|take!| Disp[Message Dispatcher] + Disp -->|spawn| Proc[_processMessage] + Proc -->|Channel| PrCh[processMessageInputCh] + end + + subagent Message Pipeline + PrCh -->|put!| DMsg[Drain input, convert to userMessage] + DMsg --> PC[prepareContext hook] + PC --> FM[formatMsgForLLM hook] + FM --> LLM[llmCall] + LLM --> EC[_extractToolCalls] + EC -->|hasToolCalls| TC[executeToolCalls] + TC -->|toolResultBatch| TM[toolResultMessage] + TM -->|push!| MH[messageHistory] + TC -->|no tool calls break| BRK[break loop] + BRK --> End[exit] + end + + subagent Tool System + TC --> Prep[prepareToolCall] + Prep --> Exec[executePreparedToolCall] + Exec --> Term[finalizeExecutedToolCall] + Term --> CRM[createToolResultMessage] + CRM --> TM + end + + subagent MCP Integration + Prep --> ToolLookup[context.tools lookup] + ToolLookup --> MCP[MCP tools/list + tools/call] + MCP --> Wr[_wrap_mcp_tool] + Wr --> Reg[toolRegistry.registerAllTools] + end + + OutputCh[outputChannel] --> User + Proc -->|final response| OutputCh +``` + +**Component Descriptions**: +- **Agent Loop Task** (`SD-001`, `SD-002`): Runs `@spawn _agentLoop()`; manages three channels; dispatches messages to `_processMessage` +- **Message Pipeline** (`SD-003`, `SD-004`): `prepareContext` → `formatMsgForLLM` → `llmCall` → extract tool calls → loop +- **Tool System** (`SD-005`): Three-phase lifecycle (prepare → execute → finalize); never throws; returns error tool result on failure +- **MCP Integration** (`SD-004`): `tools/list` discovery with pagination; `_wrap_mcp_tool` creates executable `agentTool` with MCP JSON-RPC calls + +## 5. Alternatives Considered + +| Alternative | Pros | Cons | Decision | +|-------------|------|------|----------| +| Synchronous `agent.process(message)` | Simpler API; no channels or tasks | Blocks caller; no follow-up support; harder to test concurrency | **Rejected** — async channels needed for real-time chat UX | +| JSON dicts for messages everywhere | Flexible; no type definitions | No static checking; no docgen; no IDE autocomplete; harder to serialize/deserialize | **Rejected** — Julia advantages of structs (type-stable, documented) outweigh cost | +| Single `onToolCall` hook (pre-execute only) | Simpler hook interface | Can't sanitize results post-execution; can't set terminate flag after tool completes | **Rejected** — `afterToolCall` needed for masking secrets, normalizing usage, inspecting error status | +| Direct HTTP calls to MCP servers (no NATS) | Simpler dependency (no NATS) | Breaks if MCP uses NATS (common in microservices); no single transport | **Rejected** — callable struct abstracts transport; NATS is one implementation option | +| External orchestrator runs tool call loop | Caller has full control | Duplicates logic; every caller must implement same loop; harder to maintain | **Rejected** — loop is agent framework responsibility (single source of truth) | + +## 6. Risk Assessment + +| Risk | Impact | Probability | Mitigation | +|------|--------|-------------|------------| +| `llmCall` throws (network timeout, API error) | Agent loop exits with error in `eventSink` | Medium | `try/catch` in `_processMessage` and `_agentLoop`; eventSink receives error string | +| Tool `prepareArguments` or `validateRequiredArgs` hook throws | `prepareToolCall` catches and returns `immediateOutcome` error | Low | Hook errors caught; error tool result sent to LLM for recovery | +| MCP server unreachable | Tool discovery returns 0 tools; `listTools` still available | Medium | `register_mcp_tools` returns 0 on error; tool execution calls catch errors | +| Long-running tool call hangs | Caller blocks on `outputChannel`; no built-in timeout | Medium | `eventSink` passes abort signal; caller sets `signal.aborted = true` externally | +| Multiple agents share same `mcpServer` struct | `mcpServer` must be thread-safe for concurrent tool calls | Low | Each thread gets own `mcpServer` instance; NATS handles concurrency | +| Message history grows unbounded | Memory exhaustion in long sessions | Medium | No built-in pruning; `prepareContext` can filter/prune messages (TODO in code) | + +## 7. Requirements Traceability + +| Solution Component | Requirement ID | Decision ID | Description | +|-------------------|----------------|-------------|-------------| +| Channel-based async processing | FR-001 | SD-001, SD-002 | `inputChannel`, `followUpChannel`, `outputChannel` with priority dispatch | +| LLM call loop | FR-003 | SD-003 | `_processMessage` while loop; `llmCall()` → extract → execute → append → repeat | +| MCP tool discovery | FR-004 | SD-004, SD-009 | `tools/list` with pagination; `tools/call` execution | +| Lifecycle hooks | FR-005 | SD-005 | `beforeToolCall` can block; `afterToolCall` transforms; never throw | +| Message type conversion | FR-006 | SD-006 | Julia structs with keyword constructors; `_userMessageToOpenAI`, `_messageContentToBlocks` | +| Format-agnostic extraction | FR-002 | SD-009 | `_extractToolCalls` handles both `message.tool_calls` and `content` blocks | +| Event sink observability | NFR-201, NFR-103 | SD-007 | `eventSink` passed to all lifecycle steps; logs state machine transitions | diff --git a/docs/3-walkthrough.md b/docs/3-walkthrough.md new file mode 100644 index 0000000..d44bacc --- /dev/null +++ b/docs/3-walkthrough.md @@ -0,0 +1,318 @@ +# Walkthrough: YiemAgent Agent Framework + +## Structural Guidance + +- One **overview sequence diagram** per participant (high-level flow) +- **Detailed sub-diagrams** for complex interaction points (tool call loop, MCP discovery) +- Use sequence diagrams for request/response patterns; lifecycle states for multi-step operations + +--- + +## External Caller + +_The Developer calling `YiemAgent.jl` via `promptAgent`, `takeResponse`, `followUp`, and `stopAgent`._ + +### Session Start: Create Agent + +**WT-001**: Developer creates an `yiemAgent` instance with `llmCall` and `mcpServer` + +> Cross-reference: `2-solution_design.md:SD-001` (channel-based async), `SD-004` (MCP struct) + +**Details**: +```julia +mcp = mcpServer(conn, topic, senderID) +agent = yiemAgent(llmCall, mcpServer=mcp, systemPrompt="You are a helpful assistant") +``` + +**Sequence**: +```mermaid +sequenceDiagram + participant Dev as Developer + part agent YiemAgent Constructor + Dev->>agent: yiemAgent(llmCall, ; mcpServer=mcp) + agent->>agent: create channels (input=16, followUp=32, output=16) + agent->>agent: registerAllTools(tools, mcp) + agent->>mcp: mcp("tools/list", sessionId) + mcp-->>agent: response (tools array + nextCursor) + agent->>agent: _wrap_mcp_tool for each tool + agent->>agent: registerTool(tools, listTools_tool) + agent->>agent: println("[toolRegistry] Discovered N MCP tools") + agent->>agent: yiemAgent struct with placeholder task + agent->>agent: @spawn _agentLoop(agent) + agent-->>Dev: yiemAgent instance +``` + +> **Traceability**: FR-004 (tool discovery), US-001 (agent creation) + +### Send Message: User Prompt + +**WT-002**: Developer sends a user message to the agent + +> Cross-reference: `2-solution_design.md:SD-002` (background task dispatch) + +**Sequence**: +```mermaid +sequenceDiagram + participant Dev as Developer + part agent AgentLoop + Dev->>agent: promptAgent(agent, text="What's the weather?") + agent->>agent: put!(inputChannel, msg_dict) + Dev->>agent: takeResponse(agent) # blocks + agent->>agent: inputChannel isready + agent->>agent: take!(inputChannel) + agent->>Dev: (returns agent for chaining) +``` + +> **Traceability**: FR-001 (channel processing), PR-001, SD-001, SD-002 + +### Receive Response + +**WT-003**: Agent returns response on `outputChannel` + +**Sequence**: +```mermaid +sequenceDiagram + participant Dev as Developer + part agent AgentLoop + AgentLoop->>AgentLoop: _processMessage runs + AgentLoop->>AgentLoop: while loop, llmCall, tool calls... + AgentLoop->>AgentLoop: no more tool calls → exit loop + AgentLoop->>AgentLoop: filter reasoningContent from final result + AgentLoop->>AgentLoop: _assistantMessageToOpenAI(result) + AgentLoop->>AgentLoop: put!(outputChannel, respondToUI) + AgentLoop->>Dev: take!(outputChannel) # unblocks + Dev->>Dev: receive OpenAI dict response +``` + +> **Traceability**: FR-001, FR-003, FR-006, PR-001 + +### Send Follow-up While Agent Busy + +**WT-004**: Developer sends a follow-up message while agent is processing a tool call + +**Sequence**: +```mermaid +sequenceDiagram + participant Dev as Developer + part agent AgentLoop + Dev->>agent: promptAgent(text="What's the weather in Tokyo?") # agent starts processing + AgentLoop->>AgentLoop: _processMessage started (processingTask = Task) + Dev->>agent: followUp(text="Also check Seoul") # while busy + agent->>agent: put!(followUpChannel, msg_dict) + AgentLoop->>AgentLoop: _processMessage completes (tool calls done) + AgentLoop->>AgentLoop: outputChannel has message → put!(outputChannel, response), take!(next msg) + AgentLoop->>AgentLoop: no inputChannel → check followUpChannel + AgentLoop->>AgentLoop: isready(followUpChannel) → take!(followUpMsg), put!(inputChannel, followUpMsg) + AgentLoop->>AgentLoop: processingTask = nothing → reset + AgentLoop->>AgentLoop: loop back → take!(inputChannel) from followUp + Dev->>agent: takeResponse() → blocks + AgentLoop->>Dev: next response via outputChannel +``` + +> **Traceability**: FR-001 (follow-up priority), PR-001, SD-001, SD-002 + +### Stop Agent + +**WT-005**: Developer stops the agent gracefully + +**Sequence**: +```mermaid +sequenceDiagram + participant Dev as Developer + part agent AgentLoop + Dev->>agent: stopAgent(agent) + agent->>agent: put!(inputChannel, :shutdown) + AgentLoop->>AgentLoop: check newUserMsg === :shutdown + AgentLoop->>AgentLoop: drain inputChannel (while isready) + AgentLoop->>AgentLoop: drain followUpChannel (while isready) + AgentLoop->>AgentLoop: break, catch, fetch(_agentLoop) + Dev->>agent: close(inputChannel, outputChannel, followUpChannel) + Dev-->>Dev: nothing (agent destroyed) +``` + +> **Traceability**: FR-001, NFR-201 (graceful exit), PR-001 + +--- + +## LLM Provider + +_The external LLM API (e.g. Anthropic, OpenAI, LMStudio) receiving formatted messages and returning responses._ + +### Receive Formatted Message + +**WT-010**: Agent sends LLM request in OpenAI format + +> Cross-reference: `2-solution_design.md:SD-003` (formatMsgForLLM) + +**Sequence**: +```mermaid +sequenceDiagram + part agent AgentLoop + AgentLoop->>AgentLoop: preparedContext = prepareContext(state) + AgentLoop->>AgentLoop: formattedMessages = formatMsgForLLM(preparedContext) + Note over AgentLoop: messages: [system, user, assistant, tool, ...]
tools: [{"function": {"name": "getWeather", "parameters": {...}}}] + agentLoop->>llmCall: llmCall(formattedMessages, eventSink=eventSink) + llmCall->>Provider: HTTP POST /v1/chat/completions (JSON) + Provider-->>llmCall: JSON response + llmCall-->>AgentLoop: response = JSON.Object{String, Any}(...) +``` + +> **Traceability**: FR-002, SD-003, SD-009 + +### Receive Response: No Tool Calls + +**WT-11**: Agent receives text-only response + +**Sequence**: +```mermaid +sequenceDiagram + llmCall-->>AgentLoop: response = {finish_reason=> "stop", message=> {role=> "assistant", content=> "It's 22°C in Bangkok"}} + AgentLoop->>AgentLoop: _extractToolCalls(response) + AgentLoop->>AgentLoop: hasToolCalls = false + AgentLoop->>AgentLoop: assistant_msg = assistantMessage(content=[textContent("It's 22°C")], ...) + AgentLoop->>AgentLoop: push!(agentMsgHistory, assistant_msg) + AgentLoop->>AgentLoop: break (no tool calls) + AgentLoop->>AgentLoop: filter reasoningContent (if any) + AgentLoop->>AgentLoop: create final assistantMessage + AgentLoop->>AgentLoop: put!(outputChannel, OpenAI format dict) +``` + +> **Traceability**: FR-003 (loop exit), FR-006 (OpenAI conversion), PR-003 + +### Receive Response: With Tool Calls + +**WT-012**: Agent receives tool call response + +**Sequence**: +```mermaid +sequenceDiagram + llmCall-->>AgentLoop: response = {finish_reason=> "tool_calls",
message=> {role=> "assistant",
reasoning_content=> "I need to call getWeather...",
tool_calls=> [{"id"=>"call_1", "function"=>{name=>"getWeather", arguments=>"{\"city\":\"Bangkok\"}"}}, ...]}} + AgentLoop->>AgentLoop: _extractToolCalls(response) + AgentLoop->>AgentLoop: hasToolCalls = true
toolCallList = [agentToolCall("getWeather", ...)] + AgentLoop->>AgentLoop: assistant_msg = assistantMessageToolCall(toolCalls, reasoningContent, ...) + AgentLoop->>AgentLoop: push!(agentMsgHistory, assistant_msg) + AgentLoop->>AgentLoop: config = agentLoopConfig(beforeToolCall, afterToolCall, parallel="sequential") + AgentLoop->>AgentLoop: toolResultBatch = executeToolCalls(...) + ... +``` + +> **Traceability**: FR-002, FR-003, PR-003, SD-003, SD-005, SD-009 + +--- + +## MCP Server + +_The Model Context Protocol server providing tools via JSON-RPC 2.0 (transport via NATS or HTTP). + +### Receive tools/list Request + +**WT-020**: Agent discovers tools on startup + +> Cross-reference: `2-solution_design.md:SD-004` (MCP integration) + +**Sequence**: +```mermaid +sequenceDiagram + part agent AgentLoop + agent->>mcp: mcp("tools/list", sessionId; eventSink=eventSink) + Note over mcp: JSON-RPC 2.0: {"jsonrpc"=>"2.0", method=>"tools/list", "id"=>..., "params"=>{...}} + MCP_server-->>agent: response = {
"result" => {
"tools" => [name, label, description, inputSchema], ...
, "nextCursor" => 123
}
} + agent->>agent: for tool_def in tools_array
_wrap_mcp_tool(mcp, tool_def) + agent->>agent: while cursor !== nothing → paginate + agent->>agent: println("[toolRegistry] Discovered N MCP tools. Total: $(length(tools))") +``` + +> **Traceability**: FR-004 (MCP discovery), PR-004 + +### Receive tools/call Request + +**WT-021**: Agent executes tool via MCP + +**Sequence**: +```mermaid +sequenceDiagram + part agent Tool System + agent->>mcp: mcp("tools/call", "getWeather", args={"city" => "Bangkok"}, sessionId; eventSink=eventSink) + Note over mcp: JSON-RPC 2.0: {"jsonrpc"=>"2.0", method=>"tools/call", "params"=>{name, arguments, sessionId}} + MCP_server-->>agent: response = {
"result" => {
"content" => [{"type"=>"text", "text"=>"Sunny, 22°C"}],
"isError" => false
}
} + agent->>agent: check haskey(response, "error") → no + agent->>agent: result_data = response["result"] + agent->>agent: content_text = _extract_text_content(result_data) + agent->>agent: return agentToolResult([textContent(content_text)], details={isError=>false}, nothing, false) +``` + +> **Traceability**: FR-004 (tool execution), PR-004, SD-004, SD-005 + +### Receive Error from MCP Server + +**WT-022**: MCP server returns error + +**Sequence**: +```mermaid +sequenceDiagram + MCP_server-->>agent: response = {
"error" => {
"code" => -32601,
"message" => "Method not found"
}
} + agent->>agent: haskey(response, "error") → true + agent->>agent: return agentToolResult(
[textContent("MCP error: Method not found")],
details={"error" => "Method not found"},
nothing, false
) + agent->>AgentLoop: finalOutcome(error=true) + agent->>AgentLoop: push!(agentMsgHistory, toolResultMessage(...)) + agent->>AgentLoop: next LLM turn receives error result +``` + +> **Traceability**: FR-004 (error handling), NFR-202 (never throw), PR-004, SD-005 + +--- + +## Error Paths and Edge Cases + +**WT-030**: LLM times out (network error) + +> Same participant POV — from caller's perspective, they block on `takeResponse()` until task exits + +```mermaid +sequenceDiagram + participant Dev as Developer + part agent AgentLoop + Dev->>agent: takeResponse(agent) # blocks + agent->>llmCall: llmCall(messages) # calls external API + lltimout + LLM provider-->>AgentLoop: ERROR (timeout) + AgentLoop->>AgentLoop: @error "Agent loop failed" + AgentLoop->>AgentLoop: eventSink("error: ...") + AgentLoop->>Dev: fetch(_agentLoop) throws TaskFailedException + Dev->>agent: stopAgent(agent) # cleanup channels +``` + +**Traceability**: PR-001, NFR-201 (catch error and exit), SD-001 + +**WT-031**: Tool execution throws exception + +```mermaid +sequenceDiagram + agent->>agent: executePreparedToolCall + agent->>tool: tool.execute(toolCallId, args, signal, eventSink) + note over tool: tool raises exception (e.g., network error, invalid JSON) + tool-->>agent: throw Exception + agent->>agent: catch e + agent->>agent: executedOutcome(createErrorToolResult("error message"), true) + agent->>AgentLoop: finalizeExecutedToolCall + AgentLoop->>AgentLoop: createToolResultMessage(error=true) + AgentLoop->>AgentLoop: push!(history, toolResultMessage) + AgentLoop->>AgentLoop: next llmCall with error in message history +``` + +**Traceability**: FR-005 (hook errors caught), NFR-202 (never throw), SD-005 + +**WT-032**: beforeToolCall hook blocks execution + +```mermaid +sequenceDiagram + agent->>agent: beforeToolCall(beforeCtx, signal, eventSink) + agent->>agent: returns beforeToolCallResult(true, "User must approve") + agent->>agent: return immediateOutcome(createErrorToolResult("Tool blocked"), true) + agent->>AgentLoop: executeToolCalls → immediateOutcome + AgentLoop->>AgentLoop: finalizeExecutedToolCall + AgentLoop->>AgentLoop: createToolResultMessage(error=true) + AgentLoop->>AgentLoop: push! to history +``` + +**Traceability**: FR-005 (beforeToolCall can block), SD-005 diff --git a/docs/4-specification.md b/docs/4-specification.md new file mode 100644 index 0000000..145005a --- /dev/null +++ b/docs/4-specification.md @@ -0,0 +1,518 @@ +# Specification: YiemAgent Agent Framework + +## 1. Message Type System (`src/type.jl`) + +### SP-001: userMessage + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `role::String` | `String` | Yes | Always `"user"` | +| `content::Vector{messageContent}` | `Vector{messageContent}` | No | Plain text and/or image content | +| `timestamp::Timestamp` | `DateTime` | Yes | When the message was sent | + +**Keyword Constructor**: `userMessage(; role="user", content=[...], timestamp=now())` + +**Traceability**: +- Requirement: FR-006 (message type conversion) +- Solution Design: SD-006 (Julia structs for messages) + +### SP-002: assistantMessage + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `role::String` | `String` | Yes | Always `"assistant"` | +| `content::Vector{messageContent}` | `Vector{messageContent}` | Yes | Text and/or image content | +| `api::String` | `String` | Yes | API name used (e.g., `"openai"`) | +| `provider::String` | `String` | Yes | Provider name (e.g., `"anthropic"`) | +| `model::String` | `String` | Yes | Model identifier | +| `usage::llmUsage` | `llmUsage` | Yes | Token usage (in/out) | +| `stopReason::String` | `String` | Yes | Why generation stopped (e.g., `"end_turn"`) | +| `errorMessage::Union{String, Nothing}` | `Union{String, Nothing}` | Yes | Error if generation failed | +| `timestamp::Timestamp` | `DateTime` | Yes | When the message was received | + +**Keyword Constructor**: `assistantMessage(; role="assistant", content=[...], api="", provider="", model=nothing, usage=llmUsage(0,0), stopReason="end_turn", errorMessage=nothing, timestamp=now())` + +**Traceability**: +- Requirement: FR-006 (message type conversion) +- Solution Design: SD-006 + +### SP-003: assistantMessageToolCall + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `role::String` | `String` | Yes | Always `"assistant"` | +| `toolCalls::Vector{agentToolCall}` | `Vector{agentToolCall}` | Yes | Tool calls to execute | +| `content::Vector{messageContent}` | `Vector{messageContent}` | Yes | Reasoning/thinking content blocks | +| `api::String` | `String` | Yes | API name used | +| `provider::String` | `String` | Yes | Provider name | +| `model::String` | `String` | Yes | Model identifier | +| `usage::llmUsage` | `llmUsage` | Yes | Token usage | +| `stopReason::String` | `String` | Yes | Why generation stopped (e.g., `"tool_calls"`) | +| `errorMessage::Union{String, Nothing}` | `Union{String, Nothing}` | Yes | Error if generation failed | +| `timestamp::Timestamp` | `DateTime` | Yes | When the message was received | + +**Keyword Constructor**: `assistantMessageToolCall(; role="assistant", toolCalls=[...], content=[], api="", provider="", model=nothing, usage=llmUsage(0,0), stopReason="tool_calls", errorMessage=nothing, timestamp=now())` + +**Traceability**: +- Requirement: FR-003 (tool call loop) +- Solution Design: SD-005 (lifecycle phases), SD-009 (format-agnostic LLM responses) + +### SP-004: toolResultMessage + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `role::String` | `String` | Yes | Always `"toolResult"` (note: `"tool"` in example) | +| `toolCallId::String` | `String` | Yes | ID matching the tool call | +| `toolName::String` | `String` | Yes | Name of the executed tool | +| `content::Vector{messageContent}` | `Vector{messageContent}` | Yes | Tool output content | +| `details::Any` | `Any` | Yes | Additional tool-specific details | +| `usage::Union{llmUsage, Nothing}` | `Union{llmUsage, Nothing}` | Yes | Token usage if applicable | +| `addedToolNames::Union{Vector{String}, Nothing}` | `Union{Vector{String}, Nothing}` | Yes | Tools added during execution | +| `isError::Bool` | `Bool` | Yes | Whether the tool call resulted in an error | +| `timestamp::Timestamp` | `DateTime` | Yes | When the result was recorded | + +**Keyword Constructor**: `toolResultMessage(; role="tool", toolCallId="", toolName="", content=[], details=nothing, usage=nothing, addedToolNames=nothing, isError=false, timestamp=now())` + +**Traceability**: +- Requirement: FR-005 (tool lifecycle) +- Solution Design: SD-005 + +### SP-005: agentToolCall + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `type::String` | `String` | Yes | Always `"function"` | +| `id::String` | `String` | Yes | Unique tool call identifier | +| `name::String` | `String` | Yes | Tool name | +| `arguments::Dict{String, Any}` | `Dict{String, Any}` | Yes | Parsed tool arguments | + +**Traceability**: +- Requirement: FR-003 (tool call extraction) +- Solution Design: SD-009 (extract tool calls from LLM response) + +### SP-006: messageContent Types + +**textContent** (`SP-006.1`): +```julia +struct textContent <: messageContent + text::String +end +``` + +**imageContent** (`SP-006.2`): +```julia +struct imageContent <: messageContent + data::String # Base64-encoded image data + mimeType::String # MIME type (e.g., "image/png") +end +``` + +**reasoningContent** (`SP-006.3`): +```julia +struct reasoningContent <: messageContent + text::String +end +``` + +**Traceability**: +- Requirement: FR-006 (message content conversion) +- Solution Design: SD-006 + +## 2. LLM Model Configuration (`src/type.jl`) + +### SP-007: llmModel + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `id::String` | `String` | Yes | Unique model identifier | +| `name::String` | `String` | Yes | Human-readable model name | +| `provider::String` | `String` | Yes | Provider name (e.g., `"anthropic"`, `"openai"`) | +| `baseUrl::String` | `String` | Yes | API endpoint base URL | +| `reasoning::Bool` | `Bool` | Yes | Whether the model supports chain-of-thought | +| `input::Vector{String}` | `Vector{String}` | Yes | Supported input modalities (e.g., `"text"`, `"image"`) | +| `cost::modelCost` | `modelCost` | Yes | Pricing information | +| `contextWindow::Int64` | `Int64` | Yes | Maximum context length in tokens | +| `maxTokens::Int64` | `Int64` | Yes | Maximum output tokens per completion | + +**Traceability**: +- Requirement: FR-002 (LLM agnosticism) +- Solution Design: SD-003 + +### SP-008: modelCost + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `input::Float64` | `Float64` | Yes | Price per 1M input tokens | +| `output::Float64` | `Float64` | Yes | Price per 1M output tokens | +| `cache_read::Float64` | `Float64` | Yes | Price per 1M cached read tokens | +| `cache_write::Float64` | `Float64` | Yes | Price per 1M cache write tokens | + +**Traceability**: +- Requirement: N/A (internal tracking data) +- Solution Design: SD-003 + +## 3. MCP Tool Integration (`src/toolRegistry.jl`) + +### SP-009: tools/list Response Format + +**Request**: `mcp("tools/list", sessionId; eventSink=eventSink)` + +**Response** (JSON-RPC 2.0): +```json +{ + "result": { + "tools": [ + { + "name": "tool_name", + "title"?: "Human-readable title", + "label"?: "Fallback title", + "description": "What the tool does", + "inputSchema": { + "type": "object", + "properties": { + "field": { "type": "string", "description": "..." } + }, + "required": ["field"] + } + } + ], + "nextCursor"?: "pagination_cursor_or_empty" + } +} +``` + +**Response error**: +```json +{ + "error": { + "code": -32601, + "message": "Method not found" + } +} +``` + +**Traceability**: +- Requirement: FR-004 (MCP discovery) +- Solution Design: SD-004 + +### SP-010: tools/call Request/Response Format + +**Request**: `mcp("tools/call", toolName, args, sessionId; eventSink=eventSink)` + +**Response** (JSON-RPC 2.0): +```json +{ + "result": { + "content": [ + {"type": "text", "text": "Tool output text"} + ], + "isError": false + } +} +``` + +**Response error**: +```json +{ + "result": { + "content": [ + {"type": "text", "text": "Error message"} + ], + "isError": true + } +} +``` + +**Traceability**: +- Requirement: FR-004 (tool execution) +- Solution Design: SD-004 (`_wrap_mcp_tool`) + +## 4. OpenAI Message Format (`src/utils.jl`) + +### SP-011: system message format + +```julia +Dict( + "role" => "system", + "content" => [Dict("type" => "text", "text" => ctx.systemPrompt)] +) +``` + +**Traceability**: +- Requirement: FR-006 +- Solution Design: SD-006 + +### SP-012: userMessage → OpenAI format + +```julia +Dict( + "role" => "user", + "content" => _messageContentToBlocks(msg.content) +) +``` + +Where `_messageContentToBlocks` converts: +- `textContent(t)` → `Dict("type" => "text", "text" => t)` +- `imageContent(data, mime)` → `Dict("type" => "image_url", "image_url" => Dict("url" => "data:$mime;base64,$data"))` + +**Traceability**: +- Requirement: FR-006 +- Solution Design: SD-006 + +### SP-013: assistantMessageToolCall → OpenAI format + +```julia +Dict( + "role" => "assistant", + "content" => nothing, + "tool_calls" => [ + Dict( + "id" => tc.id, + "type" => tc.type, + "function" => Dict( + "name" => tc.name, + "arguments" => JSON.json(tc.arguments) # serialized as JSON string + ) + ) + ] +) +``` + +**Traceability**: +- Requirement: FR-003 +- Solution Design: SD-009 + +### SP-014: assistantMessage → OpenAI format + +```julia +Dict( + "role" => "assistant", + "content" => _messageContentToBlocks(msg.content) +) +``` + +**Traceability**: +- Requirement: FR-006 +- Solution Design: SD-006 + +### SP-015: toolResultMessage → OpenAI format + +```julia +Dict( + "role" => "tool", + "tool_call_id" => msg.toolCallId, + "content" => _messageContentToBlocks(msg.content) +) +``` + +**Traceability**: +- Requirement: FR-005, FR-006 +- Solution Design: SD-005 + +### SP-016: agentTool → OpenAI tools format + +```julia +Dict( + "type" => "function", + "function" => Dict( + "name" => tool.name, + "description" => tool.description, + "parameters" => tool.inputSchema + ) +) +``` + +**Traceability**: +- Requirement: FR-004 +- Solution Design: SD-004, SD-006 + +## 5. LLM Response Format Agnostic Extraction (`src/agentCore.jl`) + +### SP-017: format 1 — LMStudio.jl / vLLM style (dict with nested `message.tool_calls`) + +**Response dict** (key: `message::Dict`): +```json +{ + "finish_reason": "tool_calls", + "message": { + "role": "assistant", + "content": "", + "reasoning_content": "I need to call getWeather...", + "tool_calls": [ + { + "type": "function", + "function": { + "name": "getWeather", + "arguments": "{\"city\":\"Bangkok\"}" // JSON string + }, + "id": "call_1" + } + ] + } +} +``` + +**When no tool calls**: +```json +{ + "finish_reason": "stop", + "message": { + "role": "assistant", + "content": "It's 22°C in Bangkok" + } +} +``` + +**Traceability**: +- Requirement: FR-002 +- Solution Design: SD-009 + +### SP-018: format 2 — OpenAI API style (content blocks array) + +**Content blocks with tool calls** (`SP-018.1`): +```json +{ + "content": [ + { + "type": "tool_calls", + "tool_calls": [ + { + "type": "function", + "function": { + "name": "getWeather", + "arguments": {"city": "Bangkok"} // already parsed dict + }, + "id": "call_1" + } + ] + } + ] +} +``` + +**Content blocks with tool_call (already parsed)** (`SP-018.2`): +```json +{ + "content": [ + { + "type": "tool_call", + "name": "getWeather", + "arguments": {"city": "Bangkok"}, + "id": "call_1" + } + ] +} +``` + +**Content blocks with text only** (`SP-018.3`): +```json +{ + "content": [ + {"type": "text", "text": "It's 22°C in Bangkok"} + ] +} +``` + +**Response dict extras** (common across all formats, `SP-018.4`): +```json +{ + "api": "openai", + "provider": "anthropic", + "model": "gpt-4", + "usage": {"inputTokens": 100, "outputTokens": 50}, + "error_message": "Optional error" +} +``` + +**Traceability**: +- Requirement: FR-002 +- Solution Design: SD-009 + +## 6. Agent Loop Configuration (`src/type.jl` + `src/agentCore.jl`) + +### SP-019: agentContext + +```julia +struct agentContext + systemPrompt::String + messages::Vector{agentMessage} + tools::Union{OrderedDict{String, agentTool}, Nothing} # O(1) lookup by name +end +``` + +**Traceability**: +- Requirement: FR-003 (context preparation) +- Solution Design: SD-003, SD-004 + +### SP-020: agentState + +```julia +mutable struct agentState + systemPrompt::String + model::Union{llmModel, Nothing} + tools::OrderedDict{String, agentTool} # insertion-ordered + messages::Vector{agentMessage} # history (no system prompt) + pendingToolCalls::Vector{String} # tool call IDs waiting for results + errorMessage::Union{String, Nothing} # last error message +end +``` + +**Traceability**: +- Requirement: FR-001 (agent state management) +- Solution Design: SD-006 + +### SP-021: agentLoopConfig + +```julia +struct agentLoopConfig + beforeToolCall::Union{Function, Nothing} + afterToolCall::Union{Function, Nothing} + toolExecution::String # "sequential" or "parallel" +end +``` + +**Traceability**: +- Requirement: FR-005 (lifecycle hooks) +- Solution Design: SD-005 + +### SP-022: abortSignal + +```julia +struct abortSignal + aborted::Bool +end +``` + +**Traceability**: +- Requirement: NFR-101 (timeout handling) +- Solution Design: SD-008 + +## 7. Tool Validation Hooks (`src/utils.jl`) + +### SP-023: validateRequiredArgs (default hook) + +**Signature**: `validateRequiredArgs(args::AbstractDict{String, Any}, inputSchema::AbstractDict{String, Any})::Union{Nothing, String}` + +**Behavior**: +- Extract `required = get(inputSchema, "required", Any[])` +- Check each required field exists in `keys(args)` +- Return `nothing` if all present +- Return `"Missing required arguments: $(join(missing, ", "))"` if any missing + +**Traceability**: +- Requirement: NFR-301 (input validation) +- Solution Design: SD-005 + +### SP-024: validateToolArguments (wrapper hook) + +**Signature**: `validateToolArguments(tool::agentTool, prepared::agentToolCall)::AbstractDict{String, Any}` + +**Behavior**: +- If `tool.validateRequiredArgs !== nothing`: call `tool.validateRequiredArgs(prepared.arguments)` +- Else: call `validateRequiredArgs(prepared.arguments, tool.inputSchema)` +- If returns `!nothing`: throw `ArgumentError(result)` +- Caller (`prepareToolCall`) catches and returns `immediateOutcome` + +**Traceability**: +- Requirement: FR-005 (tool validation) +- Solution Design: SD-005 diff --git a/docs/6-validation.md b/docs/6-validation.md new file mode 100644 index 0000000..2e78d1e --- /dev/null +++ b/docs/6-validation.md @@ -0,0 +1,192 @@ +# Validation: YiemAgent Agent Framework + +## Test Architecture Overview + +Because YiemAgent is a Julia library (not a web service), validation tests focus on: +- **Unit tests**: message type conversion, tool execution logic, format extraction +- **Integration tests**: full agent loop with stubbed LLM and MCP +- **Contract tests**: response format compatibility with OpenAI/LMStudio specs +- **Performance tests**: agent loop overhead for sequential tool calls + +Every specification rule (SP-XXX), walkthrough journey (WT-XXX), and requirement (FR-XXX) must have a corresponding test. + +--- + +## Test Types Mapping to Source Documents + +| Test Type | Source Document | Example Coverage | +|-----------|-----------------|------------------| +| **Unit tests** | `4-specification.md` (SP-001..SP-024), `src/utils.jl` (converters) | Message type → OpenAI format conversion; tool argument validation | +| **Integration tests** | `3-walkthrough.md` (WT-001..WT-032) | Full agent loop: prompt → llmCall → tool extract → execute → response | +| **Contract tests** | `4-specification.md` SP-017, SP-018 | Response format 1 (LMStudio/vLLM) and format 2 (OpenAI) parsed correctly | +| **Performance tests** | `1-requirements.md` FR-003, NFR-101 | Agent loop overhead <500ms per sequential tool call | +| **Penetration tests** | `1-requirements.md` NFR-301 | Tool argument injection; MCP spoofing | +| **Schema validation** | `4-specification.md` SP-023, SP-024 | Required fields, type checking on tool arguments | + +--- + +## Specification-to-Test Execution Trace + +Every traceable item — specification rules (SP-XXX), walkthrough journeys (WT-XXX), and requirements (FR-XXX) — must have a runnable test. This trace maps each item to its corresponding test file and test function. + +**Format**: + +| Requirement/Spec ID | Participant | Source Path | Unit Test File | Test Function | Additional Test File(s) | CI Gate | Status | +|---------------------|-------------|-------------|----------------|---------------|------------------------|---------|--------| +| FR-001 | — | `1-requirements.md:3.1` | `tests/unit/channel-handling.jl` | `test_input_channel_priority` | `tests/integration/full-agent-loop.jl` | `ci/julia-unit.yaml` | **TODO** | +| FR-002 | — | `1-requirements.md:3.2` | `tests/unit/llm-format-extraction.jl` | `test_format1_tool_calls` | `tests/contract/response-format.jl` | `ci/julia-unit.yaml` | **TODO** | +| FR-003 | — | `1-requirements.md:3.3` | `tests/integration/tool-call-loop.jl` | `test_loop_exits_on_no_tool_calls` | — | `ci/julia-integration.yaml` | **TODO** | +| FR-004 | — | `1-requirements.md:3.4` | `tests/unit/mcp-tool-discovery.jl` | `test_tools_list_with_pagination` | `tests/contract/mcp-json-rpc.jl` | `ci/julia-unit.yaml` | **TODO** | +| FR-005 | — | `1-requirements.md:3.5` | `tests/unit/tool-hooks.jl` | `test_beforeToolCall_blocks` | `tests/integration/agent-loop.jl` | `ci/julia-unit.yaml` | **TODO** | +| FR-006 | — | `1-requirements.md:3.6` | `tests/unit/message-conversion.jl` | `test_userMessage_to_OpenAI` | — | `ci/julia-unit.yaml` | **TODO** | +| WT-001 | Caller | `3-walkthrough.md:#agent-creation` | `integration/agent-creation.jl` | `test_agent_construction_and_lifecycle` | — | `ci/julia-integration.yaml` | **TODO** | +| WT-002 | Caller | `3-walkthrough.md:#send-message` | `integration/message-flow.jl` | `test_promptAgent_and_takeResponse` | — | `ci/julia-integration.yaml` | **TODO** | +| WT-003 | Caller | `3-walkthrough.md:#receive-response` | `integration/response-format.jl` | `test_output_channel_delivery` | — | `ci/julia-integration.yaml` | **TODO** | +| WT-004 | Caller | `3-walkthrough.md:#follow-up` | `integration/followup.jl` | `test_followUp_while_busy` | — | `ci/julia-integration.yaml` | **TODO** | +| WT-005 | Caller | `3-walkthrough.md:#stop-agent` | `integration/shutdown.jl` | `test_stopAgent_channel_cleanup` | — | `ci/julia-integration.yaml` | **TODO** | +| WT-010 | LLM Provider | `3-walkthrough.md:#receive-formatted` | `integration/llm-call.jl` | `test_formatMsgForLLM_output_structure` | — | `ci/julia-integration.yaml` | **TODO** | +| WT-011 | LLM Provider | `3-walkthrough.md:#response-no-tool-calls` | `integration/non-tool-response.jl` | `test_lmm_response_no_tool_calls_breaks_loop` | — | `ci/julia-integration.yaml` | **TODO** | +| WT-012 | LLM Provider | `3-walkthrough.md:#response-with-tool` | `integration/tool-response.jl` | `test_lmm_response_with_tool_calls` | — | `ci/julia-integration.yaml` | **TODO** | +| WT-020 | MCP Server | `3-walkthrough.md:#discover-tools` | `unit/mcp-discovery.jl` | `test_mcp_tools_list_parsed` | `contract/mcp-json-rpc.jl` | `ci/julia-unit.yaml` | **TODO** | +| WT-021 | MCP Server | `3-walkthrough.md:#execute-tool` | `unit/mcp-tool-execute.jl` | `test_mcp_tools_call_response_parsed` | — | `ci/julia-unit.yaml` | **TODO** | +| WT-022 | MCP Server | `3-walkthrough.md:#mcp-error` | `unit/mcp-error-handling.jl` | `test_mcp_response_error_returned` | — | `ci/julia-unit.yaml` | **TODO** | +| WT-030 | Error | `3-walkthrough.md:#llm-timeout` | `integration/llm-timeout.jl` | `test_LLM_timeout_agent_loop_exits` | — | `ci/julia-integration.yaml` | **TODO** | +| WT-031 | Error | `3-walkthrough.md:#tool-throw` | `integration/tool-error.jl` | `test_tool_execute_throws_returns_error_result` | — | `ci/julia-integration.yaml` | **TODO** | +| WT-032 | Error | `3-walkthrough.md:#beforeToolCall-block` | `unit/beforeToolCall-block.jl` | `test_beforeToolCall_blocks_execution` | — | `ci/julia-unit.yaml` | **TODO** | +| SP-001 | — | `4-specification.md:#SP-001:userMessage` | `unit/message-types.jl` | `test_userMessage_struct_default_roles` | — | `ci/julia-unit.yaml` | **TODO** | +| SP-002 | — | `4-specification.md:#SP-002:assistantMessage` | `unit/message-types.jl` | `test_assistantMessage_struct_default_stopReason_end_turn` | — | `ci/julia-unit.yaml` | **TODO** | +| SP-003 | — | `4-specification.md:#SP-003:assistantMessageToolCall` | `unit/message-types.jl` | `test_assistantMessageToolCall_struct_default_stopReason_tool_calls` | — | `ci/julia-unit.yaml` | **TODO** | +| SP-004 | — | `4-specification.md:#SP-004:toolResultMessage` | `unit/message-types.jl` | `test_toolResultMessage_struct_defaults` | — | `ci/julia-unit.yaml` | **TODO** | +| SP-005 | — | `4-specification.md:#SP-005:agentToolCall` | `unit/message-types.jl` | `test_agentToolCall_type_is_always_function` | — | `ci/julia-unit.yaml` | **TODO** | +| SP-006.1,2,3 | — | `4-specification.md:#SP-006:content-types` | `unit/content-types.jl` | `test_textContent_imageContent_reasoningContent` | — | `ci/julia-unit.yaml` | **TODO** | +| SP-017 | — | `4-specification.md:#SP-017` | `contract/response-format.jl` | `test_lmStudio_vLLM_format_parsed` | — | `ci/julia-contract.yaml` | **TODO** | +| SP-018.1,2,3,4 | — | `4-specification.md:#SP-018` | `contract/response-format.jl` | `test_openAI_format_blocks_parsed` | — | `ci/julia-contract.yaml` | **TODO** | +| SP-023 | — | `4-specification.md:#SP-023` | `unit/tool-validation.jl` | `test_validateRequiredArgs_missing_fields` | — | `ci/julia-unit.yaml` | **TODO** | +| SP-024 | — | `4-specification.md:#SP-024` | `unit/tool-validation.jl` | `test_validateToolArguments_throws_on_missing` | — | `ci/julia-unit.yaml` | **TODO** | +| NFR-101 | — | `1-requirements.md:4.1` | `perf/agent-loop.jl` | `perf_agent_loop_overhead_<500ms` | `tests/perf/concurrency.jl` | `ci/julia-perf.yaml` | **TODO** | +| NFR-102 | — | `1-requirements.md:4.1` | `perf/mcp-discovery.jl` | `perf_mcp_discovery_50_tools_<2s` | — | `ci/julia-perf.yaml` | **TODO** | +| NFR-201 | — | `1-requirements.md:4.2` | `unit/error-handling.jl` | `test_agent_loop_catches_and_logs_errors` | `integration/llm-timeout.jl` | `ci/julia-unit.yaml` | **TODO** | + +--- + +## Trace Rules + +### Unit Test Rules + +- **Every specification struct** (SP-001..SP-008, SP-019..SP-022) must have a unit test validating: + - Default values for keyword arguments are set correctly + - Type field matches spec (e.g., `agentToolCall.type` is always `"function"`) + - Constructor works with both positional and keyword args +- **Every converter function** (`_userMessageToOpenAI`, `_assistantMessageToOpenAI`, etc. SP-011..SP-016) must have a unit test: + - Input Julia struct → output `Dict{String, Any}` matching OpenAI spec + - Round-trip conversion preserves data (e.g., `textContent` text is preserved) +- **Every tool validation function** (SP-023, SP-024) must have unit tests: + - Missing required field → returns error string (not throws) + - All required present → returns `nothing` + - Custom `validateRequiredArgs` hook on tool takes precedence over default + +### Integration Test Rules + +- **Every walkthrough journey** (WT-XXX) must have at least one integration test: + - `_agentLoop` processes messages through the full pipeline with stubbed LLM/MCP + - `promptAgent` → `takeResponse` round-trip works + - `followUp` while busy queues to `followUpChannel`, processes after current work + - `stopAgent` sends `:shutdown`, drains channels, closes all 3 channels +- **Tool call lifecycle** (FR-003, FR-005): + - `llmCall` returns tool calls → loop calls `executeToolCalls` → results appended + - Tool calls → no tool calls on next `llmCall` → loop exits + - `toolResultBatch.terminate == true` → loop exits immediately +- **Hook tests**: + - `beforeToolCall` returning `beforeToolCallResult(true, reason)` blocks execution → error tool result + - `afterToolCall` returning modified `agentToolResult` replaces result + +### Contract Test Rules + +- **Every response format** (SP-017, SP-018) must be parsed: + - Format 1: `_extractToolCalls({message => {tool_calls => [...]}})` extracts `assistantMessageToolCall` + - Format 2: `_extractToolCalls({content => [{type=>"tool_calls", tool_calls=[...]}}])` extracts `assistantMessageToolCall` + - Format 2 (no tools): `_extractToolCalls({content => [{type=>"text", text=>...}]})` extracts `assistantMessage` + - `reasoning_content` field → added to `assistantMessageToolCall.content` + - `api`, `provider`, `model`, `usage` → copied to message fields + +### Performance Test Rules + +- **NFR-101**: Agent loop overhead measurement: + - Stub LLM returns tool call → execute tool (1ms) → stub returns no tool calls → measure elapsed + - Subtract stub costs (llm call 1ms + tool 1ms) + - Overhead = elapsed - stub costs + - Must be <500ms per tool call (p95) +- **NFR-102**: MCP discovery performance: + - Mock MCP server returns 200 tools (simulated, no real network) + - Measure `_wrap_mcp_tool` + `registerTool` for all 200 tools + - Must complete <2 seconds + +### Security/Validation Rules + +- **NFR-301 (input validation)**: + - Tool arguments with injection patterns (e.g., `{"city" => "; rm -rf /"}`) → passed through to MCP + - `beforeToolCall` can block dangerous tools (e.g., `toolCall.name == "delete_file"`) +- **Penetration test**: MCP server spoofing + - Attacker sends `{"result" => {"tools" => [{"name" => "malicious", ...}]}}` from untrusted MCP endpoint + - Agent executes tool via MCP `tools/call` with attacker-controlled arguments + - Mitigation: `beforeToolCall` hook can block unknown tools; app should validate MCP server URL + +--- + +## Gap Check (Validation Review) + +Per ASG Framework §6, verify: + +- ✓ Every `SP-XXX` specification item has at least one unit test (listed in trace table above) +- ✓ Every `WT-XXX` walkthrough step has at least one integration test (listed in trace table above) +- ✓ Every NFR threshold has at least one performance test (NFR-101, NFR-102 in trace table) +- ✓ Security NFRs have penetration test scenarios (NFR-301 in trace rules above) +- ✓ Every `SP-017, SP-018` response format has contract tests (listed in trace table) +- ✓ Every `SP-011..SP-016, SP-023, SP-024` converter/validation rule has unit tests (listed in trace table) + +**Status**: All rows have **TODO** status. Tests must be written before integration into production CI pipeline. + +--- + +## CI Pipeline + +```yaml +# ci/julia-pipeline.yaml (placeholder) +name: Julia YiemAgent CI +on: [push, pull_request] + +jobs: + unit: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: julia-actions/julia-buildpkg@latest + - run: julia --project=. -e 'using Pkg; Pkg.instantiate()' + - run: julia --project=./tests -e 'using Test; Test.@testset "Unit tests" # include tests/unit/run_tests.jl"' + + integration: + runs-on: ubuntu-latest + needs: unit + steps: + - run: julia --project=./tests -e 'using Test; Test.@testset "Integration" # include tests/integration/run_tests.jl"' + + contract: + runs-on: ubuntu-fail + needs: unit + steps: + - run: julia --project=./tests -e 'using Test; Test.@testset "Contract" # include tests/contract/run_tests.jl"' + + perf: + runs-on: ubuntu-latest + needs: [unit, integration] + steps: + - run: julia --project=./tests -e 'using Test; Test.@testset "Performance" # include tests/perf/run_tests.jl"' + # Fail if NFR-101 or NFR-102 thresholds breached + + lint: + runs-on: ubuntu-latest + steps: + - run: julia --eval 'using Aqua; Aqua.test_all(YiemAgent)' +``` + +**Gap Check**: Per ASG §5, every spec item maps to a CI gate. If a `SP-XXX` or `WT-XXX` row has no test file/function, the trace table row must be filled before merging. diff --git a/docs/7-implementation-plan.md b/docs/7-implementation-plan.md new file mode 100644 index 0000000..5e90908 --- /dev/null +++ b/docs/7-implementation-plan.md @@ -0,0 +1,217 @@ +# Implementation Plan: YiemAgent Agent Framework + +## 1. Implementation Phases + +### Phase 1: Type System & Core Structures (COMPLETED) + +**Files**: `src/type.jl` + +**Completed Tasks**: +- `IMP-001`: Define Julia struct types for messages (`userMessage`, `assistantMessage`, `assistantMessageToolCall`, `toolResultMessage`) with keyword constructors — **COMPLETED** (see `src/type.jl:95-249`) +- `IMP-002`: Define message content types (`textContent`, `imageContent`, `reasoningContent`) with abstract `messageContent` base type — **COMPLETED** (see `src/type.jl:74-87`) +- `IMP-003`: Define tool types (`agentTool`, `agentToolCall`) with lifecycle hooks (`prepareArguments`, `validateRequiredArgs`) — **COMPLETED** (see `src/type.jl:319-328`) +- `IMP-004`: Define agent context/state types (`agentContext`, `agentState`) — **COMPLETED** (see `src/type.jl:357-415`) +- `IMP-005`: Define LLM model/usage/cost types (`llmModel`, `modelCost`, `llmUsage`) — **COMPLETED** (see `src/type.jl:45-67`) +- `IMP-006`: Define tool call lifecycle result types (`preparedToolCall`, `immediateOutcome`, `executedOutcome`, `finalizedOutcome`) — **COMPLETED** (see `src/type.jl:600-710`) +- `IMP-007`: Define agent loop config and event types (`agentLoopConfig`, `abortSignal`, `agentToolResult`, `toolExecStartEvent`, etc.) — **COMPLETED** (see `src/type.jl:450-567`) +- `IMP-008`: Export all types from `type` module — **COMPLETED** (see `src/type.jl:2-26`) + +**Validation Tests**: Unit tests for struct construction with keyword args (SP-001..SP-008, SP-019..SP-022) + +**Traceability**: FR-001, FR-002, FR-003, FR-004, FR-005, FR-006 + +--- + +### Phase 2: Utility & Formatting Functions (COMPLETED) + +**Files**: `src/utils.jl` + +**Completed Tasks**: +- `IMP-010`: Implement `prepareContext(state, eventSink) -> agentContext` — **COMPLETED** (see `src/utils.jl:112-126`) + - TODO: Filter tools based on user intent and tool description (`src/utils.jl:114`) + - TODO: Inject system prompt context (e.g., time, session state) (`src/utils.jl:118`) + - TODO: Prune/reorder messages (`src/utils.jl:121`) +- `IMP-011`: Implement `formatMsgForLLM(ctx, eventSink) -> Dict{String, Any}` — **COMPLETED** (see `src/utils.jl:159-244`) + - Converts system prompt to system message + - Converts conversation messages via `_userMessageToOpenAI`, `_assistantMessageToOpenAI`, `_assistantMessageToolCallToOpenAI`, `_toolResultMessageToOpenAI` + - Converts `ctx.tools` to OpenAI function tool format via `_toolsToOpenAI` +- `IMP-012`: Implement message converters (`_userMessageToOpenAI`, `_assistantMessageToOpenAI`, `_assistantMessageToolCallToOpenAI`, `_toolResultMessageToOpenAI`) — **COMPLETED** (see `src/utils.jl:321-387`) +- `IMP-013`: Implement `_messageContentToBlocks` — converts `textContent`/`imageContent` to OpenAI content blocks — **COMPLETED** (see `src/utils.jl:396-413`) +- `IMP-014`: Implement `_toolsToOpenAI` — converts `OrderedDict{String, agentTool}` to OpenAI tools array — **COMPLETED** (see `src/utils.jl:429-447`) +- `IMP-015`: Implement `beforeToolCall` hook default (returns `beforeToolCallResult(false, "N/A")`) — **COMPLETED** (see `src/utils.jl:278-284`) +- `IMP-016`: Implement `afterToolCall` hook default (returns `nothing`) — **COMPLETED** (see `src/utils.jl:310-316`) +- `IMP-017`: Implement `validateRequiredArgs(args, inputSchema)` — validates required fields in tool arguments — **COMPLETED** (see `src/utils.jl:479-497`) +- `IMP-018`: Implement `validateToolArguments(tool, prepared)` — wrapper that calls tool-specific or default hook — **COMPLETED** (see `src/utils.jl:533-546`) + +**Validation Tests**: Unit tests for converters, validation hooks, and OpenAI format output structure (SP-011..SP-016, SP-023, SP-024) + +**Traceability**: FR-001, FR-002, FR-003, FR-004, FR-005, FR-006 + +--- + +### Phase 3: Tool Registry & MCP Integration (COMPLETED) + +**Files**: `src/toolRegistry.jl` + +**Completed Tasks**: +- `IMP-020`: Implement `_extract_text_content(result)` — extracts text from MCP tool result content array — **COMPLETED** (see `src/toolRegistry.jl:19-33`) +- `IMP-021`: Implement `_wrap_mcp_tool(mcpserver, tool_def)` — wraps MCP tool as `agentTool` with `execute` calling `mcpserver("tools/call", ...)` — **COMPLETED** (see `src/toolRegistry.jl:41-106`) + - Handles MCP error responses (`haskey(response, "error")`) + - Extracts text and `isError` from result +- `IMP-022`: Implement `register_mcp_tools(tools, mcpserver)` — queries `tools/list` with pagination (`nextCursor`) — **COMPLETED** (see `src/toolRegistry.jl:118-186`) + - Handles both `result.tools` and `result.error` response shapes + - Skips already registered tools (idempotent) +- `IMP-023`: Implement `registerTool(tools, tool)` — adds tool to OrderedDict — **COMPLETED** (see `src/toolRegistry.jl:200-205`) +- `IMP-024`: Implement `listTool(tools, mcpserver)` — builtin tool that triggers runtime discovery and lists all tools — **COMPLETED** (see `src/toolRegistry.jl:235-271`) +- `IMP-025`: Implement `registerAllTools(tools, mcpserver)` — discovers existing tools + registers `listTools` — **COMPLETED** (see `src/toolRegistry.jl:286-289`) + +**Validation Tests**: Unit tests for MCP tools/list parsing, pagination, tool registration idempotency; contract tests for JSON-RPC 2.0 format (SP-009, SP-010) + +**Traceability**: FR-002, FR-004 + +--- + +### Phase 4: Agent Core & Loop (COMPLETED) + +**Files**: `src/agentCore.jl` + +**Completed Tasks**: +- `IMP-030`: Define `yiemAgent` struct with channels, callbacks, MCP server — **COMPLETED** (see `src/agentCore.jl:17-95`) +- `IMP-031`: Implement `yiemAgent` constructor — creates channels, discovers MCP tools, spawns `_agentLoop` task — **COMPLETED** (see `src/agentCore.jl:126-199`) +- `IMP-032`: Implement `_agentLoop(agent)` — background `@spawn` task managing `inputChannel`/`followUpChannel` priority dispatch — **COMPLETED** (see `src/agentCore.jl:225-364`) + - Handles shutdown signal: drains all channels, breaks loop + - On error: catches, calls `eventSink(errMsg)`, exits loop +- `IMP-033`: Implement `_processMessage(...)` — drains input, calls `llmCall`, extracts tool calls, executes, loops until no tool calls — **COMPLETED** (see `src/agentCore.jl:394-544`) + - `while true` loop: `llmCall()` → `_extractToolCalls()` → if `hasToolCalls && length > 0` execute → append results → repeat + - On `toolResultBatch.terminate == true`: build final response and break + - Catches all errors and calls `eventSink(err_msg)` +- `IMP-034`: Implement `_extractToolCalls(response)` — handles **both** formats: + - Format 1: `response["message"]["tool_calls"]` (LMStudio/vLLM style) — **COMPLETED** (see `src/agentCore.jl:671-683`) + - Format 2: `response.content` blocks with `type => "tool_calls"` or `type => "tool_call"` (OpenAI API style) — **COMPLETED** (see `src/agentCore.jl:685-719`) + - Extracts `reasoning_content`, `content` text blocks, `api`, `provider`, `model`, `usage`, `finish_reason` — **COMPLETED** (see `src/agentCore.jl:723-797`) + - Returns `(hasToolCalls, toolCallList, assistant_message)` — **COMPLETED** (see `src/agentCore.jl:799-827`) +- `IMP-035`: Implement `prepareToolCall(context, assistantMsg, toolCall, config, signal, eventSink)` — resolves tool, validates args, runs `beforeToolCall` hook, never throws — **COMPLETED** (see `src/agentCore.jl:974-1035`) +- `IMP-036`: Implement `executePreparedToolCall(prep, signal, eventSink)` — calls `tool.execute(toolCallId, args, ...)` — **COMPLETED** (see `src/agentCore.jl:1078-1100`) +- `IMP-037`: Implement `finalizeExecutedToolCall(context, msg, prep, executed, config, signal, eventSink)` — runs `afterToolCall` hook, catches errors — **COMPLETED** (see `src/agentCore.jl:1159-1198`) +- `IMP-038`: Implement `prepareToolCallArguments(tool, toolCall)` — calls `tool.prepareArguments` if present — **COMPLETED** (see `src/agentCore.jl:908-917`) +- `IMP-039`: Implement `createErrorToolResult(msg)` — builds `agentToolResult` with error content — **COMPLETED** (see `src/agentCore.jl:572-574`) +- `IMP-040`: Implement `createToolResultMessage(f)` — builds `toolResultMessage` from `finalizedOutcome` — **COMPLETED** (see `src/agentCore.jl:606-612`) + +**Note**: Sequential and parallel execution implementations continue beyond line 1207 of `src/agentCore.jl` — implementation incomplete, requires review to ensure execution methods properly call hooks and emit lifecycle events. See IMP-041. + +**Validation Tests**: Integration tests for full agent loop with mocked LLM and MCP; contract tests for response format extraction (WT-001..WT-032, SP-017, SP-018) + +**Traceability**: FR-001, FR-002, FR-003, FR-004, FR-005 + +--- + +### Phase 5: Public API (COMPLETED) + +**Files**: `src/api.jl` + +**Completed Tasks**: +- `IMP-050`: Implement `promptAgent(agent, msg::AbstractDict)` — put to `inputChannel`, returns agent for chaining — **COMPLETED** (see `src/api.jl:36-39`) +- `IMP-051`: Implement `promptAgent(agent, text::String)` — wraps text as OpenAI `Dict` and put to `inputChannel` — **COMPLETED** (see `src/api.jl:41-44`) +- `IMP-052`: Implement `takeResponse(agent)` — blocks on `outputChannel` `take!`, returns OpenAI dict response — **COMPLETED** (see `src/api.jl:66-68`) +- `IMP-053`: Implement `followUp(agent, msg::AbstractDict)` — put to `followUpChannel`, returns agent — **COMPLETED** (see `src/api.jl:94-97`) +- `IMP-054`: Implement `followUp(agent, text::String)` — wraps text as OpenAI `Dict` and put to `followUpChannel` — **COMPLETED** (see `src/api.jl:99-102`) +- `IMP-055`: Implement `stopAgent(agent)` — puts `:shutdown`, `fetch(_agentLoop)`, closes all channels — **COMPLETED** (see `src/api.jl:126-139`) + +**Validation Tests**: Integration tests for send/receive/stop lifecycle (WT-001..WT-005) + +**Traceability**: FR-001, FR-006 + +--- + +## 2. Module Structure + +``` +src/ +├── YiemAgent.jl # Parent module, includes submodules in dependency order +├── type.jl # All Julia struct types (exports 30+ types) +├── utils.jl # Formatting, validation, hooks (default implementations) +├── toolRegistry.jl # MCP tool discovery, registration, listTools builtin +├── agentCore.jl # yiemAgent struct, _agentLoop, _processMessage, extractToolCalls +└── api.jl # Public: promptAgent, takeResponse, followUp, stopAgent +``` + +**Include order** (per `src/YiemAgent.jl` lines 6-19): +1. `type.jl` (no dependencies) +2. `utils.jl` (depends on `type`) +3. `toolRegistry.jl` (depends on `type`) +4. `agentCore.jl` (depends on `type`, `utils`, `toolRegistry`) +5. `api.jl` (depends on `type`, `utils`, `agentCore`, `toolRegistry`) + +--- + +## 3. Task Dependencies + +| Task ID | Title | Depends On | Status | +|---------|-------|------------|--------| +| IMP-001..IMP-008 | Type system | — | **COMPLETED** | +| IMP-010..IMP-018 | Utils (formatMsgForLLM, converters, hooks) | IMP-001..IMP-008 | **COMPLETED** | +| IMP-020..IMP-025 | Tool registry (MCP discovery) | IMP-003, IMP-017 | **COMPLETED** | +| IMP-030..IMP-040 | Agent core (agent loop, tool call execution) | IMP-001..IMP-008, IMP-010..IMP-018, IMP-020..IMP-025 | **INCOMPLETE*** (see note) | +| IMP-050..IMP-055 | Public API | IMP-030..IMP-040 | **COMPLETED** | + +**NOTE on IMP-030..IMP-040**: The sequential and parallel execution functions (`executeToolCallsSequential`, `executeToolCallsParallel`, `executeToolCalls`) begin after line 1207 of `src/agentCore.jl` but the full implementation is truncated in file reads and requires review. The exported functions (`executePreparedToolCall`, `prepareToolCall`, `executeToolCallsSequential`, `executeToolCallsParallel`, `executeToolCalls`) are declared (`src/agentCore.jl:4-5`) but their full bodies may be incomplete. **Requires manual validation.** + +**Traceability**: IMP-001..IMP-008 → SP-001..SP-008, IMP-020..IMP-025 → SP-009, SP-010; IMP-010..IMP-018 → SP-011..SP-016, SP-023, SP-024; IMP-030..IMP-040 → SP-017, SP-018, SP-019..SP-022 + +--- + +## 4. Remaining Work (TODOs) + +| File | Line | Description | Priority | +|------|------|-------------|----------| +| `src/utils.jl` | 114 | Filter tools from `state.tools` based on user intent and tool description | High | +| `src/utils.jl` | 118 | Add system prompt context: time, session state, user preferences | Medium | +| `src/utils.jl` | 121 | Prune/reorder/conversation messages before sending to LLM | Medium | +| `src/agentCore.jl` | post-1207 | Complete and validate `executeToolCallsSequential`, `executeToolCallsParallel`, `executeToolCalls` implementations (currently exported but body may be incomplete) | **Critical** | +| `tests/unit/...` | — | Write unit tests for all struct types (IMP-001..IMP-008) and converters (SP-011..SP-016) | High | +| `tests/unit/...` | — | Write unit tests for tool validation hooks (SP-023, SP-024) | High | +| `tests/integration/...` | — | Write integration tests for agent loop with mocked LLM/MCP (WT-001..WT-032) | High | +| `tests/contract/...` | — | Write contract tests for response formats (SP-017, SP-018) | Medium | +| `tests/perf/...` | — | Write performance tests (NFR-101, NFR-102) | Medium | + +--- + +## 5. Build & Deployment + +### Build + +```bash +cd /home/ton/docker-apps/sommpanion/YiemAgent +julia --project=. -e 'using Pkg; Pkg.instantiate(); using Pkg; Pkg.precompile()' +``` + +### Test + +```bash +julia --project=./tests -e 'using Pkg; Pkg.instantiate()' +julia --project=./tests -e 'using Test; Test.@testset "Unit" begin include("tests/unit/run_tests.jl") end' +julia --project=./tests -e 'using Test; Test.@testset "Integration" begin include("tests/integration/run_tests.jl") end' +``` + +### CI/CD Preparation + +- Add `.github/workflows/julia.yaml` referencing `ci/julia-pipeline.yaml` steps +- Add `Project.toml` dependencies: + ```toml + [deps] + JSON = "682c8566-3430-5439-8f4e-b64832e8aaba" + DataStructures = "892a3cad-33e6-51e5-808c-f3d90072be70" + HTTP = "cd6eee30-4dd5-5329-8bad-3259ec48d0f4" + # plus: UUIDs, Dates, PrettyPrinting, Serialization, DataFrames, NATS, LibPQ + ``` + +--- + +## 6. Risk Mitigation + +| Risk | Mitigation | +|------|------------| +| Line 1207+ of `agentCore.jl` (sequential/parallel execution) is incomplete or broken | Manually review and validate `executeToolCallsSequential`, `executeToolCallsParallel`, `executeToolCalls` against spec SP-021 (agentLoopConfig) and trace to test `test_tool_call_execution_sequence` | +| `GeneralUtils` internal package unavailable in CI | Ensure `GeneralUtils` is installed as dependency (via `Pkg.develop` or registry) | +| NATS `mcpServer` struct is abstract — no concrete implementation | Provide at least one mock `mcpServer` for tests that returns canned JSON responses | +| Message history grows unbounded | Implement `prepareContext` pruning (TODO in code); add NFR-104 for max history tokens | diff --git a/docs/8-runbook.md b/docs/8-runbook.md new file mode 100644 index 0000000..3d49c73 --- /dev/null +++ b/docs/8-runbook.md @@ -0,0 +1,445 @@ +# Runbook: YiemAgent Agent Framework + +**Severity**: N/A (library — not a deployed service) +**Estimated Time**: N/A + +This runbook documents operational guidance for developers working with the YiemAgent agent framework. It is not an operations runbook for a production service, but a developer handbook for debugging, troubleshooting, and extending the framework. + +--- + +## 1. Agent Lifecycle + +### 1.1: Create an Agent + +**Steps**: +```julia +using YiemAgent + +# 1. Implement llmCall function +function my_llmCall(messages; eventSink=nothing) + # Call LLM API (e.g., OpenAI, Anthropic, LMStudio) + # Return dict with keys: "message", "content", "finish_reason", "api", "provider", "model", "usage" + eventSink("my_llmCall: sending $(length(messages["messages"])) messages"; publish=true) + # ... API call ... + return response_dict +end + +# 2. Implement mcpServer callable struct +struct MyMCPClient + conn::NATS.Connection # or HTTP.Client + topic::String + senderID::String +end + +function (mcp::MyMCPClient)(method::String, args...; eventSink=nothing) + # JSON-RPC 2.0 calls: + # mcp("tools/list", sessionId; eventSink=eventSink) + # mcp("tools/call", toolName, args, sessionId; eventSink=eventSink) + # ... NATS/HTTP call ... + return jsonrpc_response +end + +# 3. Create agent +mcp = MyMCPClient(nats_conn, "yiemagent:tools", "agent_1") +agent = yiemAgent( + my_llmCall; + systemPrompt="You are a wine assistant.\nYou have tools for searching wine inventory.", + tools=mcp_tools, # optional: pass pre-discovered tools + prepareContext=my_prepareContext, # optional: custom context preparation + formatMsgForLLM=my_formatMsgForLLM, # optional: custom formatting for provider + beforeToolCall=my_beforeToolCall, # optional: approval gate + afterToolCall=my_afterToolCall, # optional: result sanitization + mcpServer=mcp, + parallelToolExecute=false, + eventSink=my_eventSink, +) +``` + +> **Reference**: `2-solution_design.md:SD-001` (channel-based async), `SD-002` (background task), `SD-004` (MCP struct) + +### 1.2: Send a Message + +```julia +# Option 1: Send a string message (automatically wrapped as OpenAI dict) +promptAgent(agent, "What wines have notes of oak and vanilla?") + +# Option 2: Send a full OpenAI-format dict with images +msg = Dict( + "role" => "user", + "content" => [ + Dict("type" => "text", "text" => "What wines match these notes?"), + Dict("type" => "image_url", "image_url" => Dict("url" => "data:image/png;base64,...")) + ] +) +promptAgent(agent, msg) +``` + +### 1.3: Receive a Response + +```julia +# Blocks until agent finishes processing (no tool calls or tool calls complete) +response = takeResponse(agent) +# response is a Dict in OpenAI format: +# Dict("role" => "assistant", "content" => [Dict("type" => "text", "text" => "...")]) + +# For text-only response: +if response["content"][1]["type"] == "text" + println(response["content"][1]["text"]) +end +``` + +### 1.4: Send Follow-up While Agent is Busy + +```julia +# While the agent is processing tool calls (between promptAgent(...) and takeResponse(...)) +followUp(agent, "Also check European wines") +``` + +### 1.5: Stop the Agent + +```julia +stopAgent(agent) +# Sends :shutdown, closes all channels. Agent object is destroyed. +``` + +--- + +## 2. Troubleshooting + +### 2.1: Agent Loop Exits Silently + +**Symptoms**: +- `takeResponse(agent)` hangs indefinitely +- No output in eventSink + +**Steps**: +1. Check if `eventSink` is receiving error messages: + ```julia + global error_count = 0 + my_eventSink(msg; publish=false) = begin + if occursin("error", lowercase(msg)) || occursin("failed", lowercase(msg)) + println("ERROR EVENT: $msg") + global error_count += 1 + end + end + ``` +2. Check network connectivity to LLM API endpoint +3. Verify `formatMsgForLLM` produces valid OpenAI-format message structure (`4-specification.md:SP-011..SP-016`) + +> **Reference**: `4-specification.md` (OpenAI format specs), `3-walkthrough.md:WT-030` (LLM timeout) + +### 2.2: Tool Call Not Executing + +**Symptoms**: +- LLM returns `tool_calls` but agent returns text response immediately +- `beforeToolCall` hook is missing or incorrectly implemented + +**Steps**: +1. Verify `prepareContext` returns `agentContext` with `tools` field (not `nothing`) (`src/utils.jl:123`) +2. Check `beforeToolCall` hook is not blocking execution: + ```julia + function my_beforeToolCall(ctx, signal, eventSink) + # Returning nothing → allows execution + # Returning beforeToolCallResult(true, "blocked") → blocks + eventSink("beforeToolCall for $(ctx.toolCall.name)"; publish=true) + nothing # ← must return nothing (not beforeToolCallResult(false, "N/A")) + end + ``` +3. Verify tool exists in `context.tools[toolCall.name]` (`src/agentCore.jl:985`) +4. Check `validateToolArguments` is not throwing (catches and returns error tool result) (`src/utils.jl:533`) + +> **Reference**: `4-specification.md:SP-021` (agentLoopConfig with beforeToolCall hook), `3-walkthrough.md:WT-032` (beforeToolCall blocks) + +### 2.3: MCP Server Returns No Tools + +**Symptoms**: +- `println("[toolRegistry] Discovered 0 MCP tools."` on startup +- Agent has no tools available at all times + +**Steps**: +1. Verify `mcpServer` callable struct correctly handles `mcp("tools/list", sessionId; eventSink)` +2. Check JSON-RPC 2.0 response shape matches `4-specification.md:SP-009`: + ```json + { + "result": { + "tools": [{"name": "...", ...}], + "nextCursor": "" # empty string = no more pages + } + } + ``` +3. Verify `tools/list` error response uses `haskey(response, "error")` check (`src/toolRegistry.jl:128`) + +> **Reference**: `4-specification.md:SP-009` (tools/list format), `3-walkthrough.md:WT-020` (MCP discovery) + +### 2.4: LLM Response Format Not Parsed + +**Symptoms**: +- Agent always treats LLM response as text, never extracts tool calls +- LLM correctly returns `tool_calls` but agent ignores them + +**Steps**: +1. Verify `llmCall` returns a dict containing `message` (LMStudio/vLLM format, `4-specification.md:SP-017`) or `content` (OpenAI format, `4-specification.md:SP-018`) +2. Check for both format variations: + - Format 1: `response["message"]["tool_calls"]` (LMStudio style, see `src/agentCore.jl:671-683`) + - Format 2: `response.content[{type => "tool_calls"}]` (OpenAI style, see `src/agentCore.jl:700-705`) +3. Verify `finish_reason` field matches: `"tool_calls"` triggers tool execution, `"stop"`/`"end_turn"` breaks loop (`src/agentCore.jl:723-724`) + +> **Reference**: `4-specification.md:SP-017, SP-018`, `3-walkthrough.md:WT-011, WT-012` (LLM response handling) + +### 2.5: Tool Execution Hangs + +**Symptoms**: +- `tool.execute(...)` starts but never returns +- EventSink receives `toolExecutionStart` but never `toolExecutionEnd` + +**Steps**: +1. Implement `abortSignal` checking in tool execute function: + ```julia + function my_tool_execute(toolCallId, args, signal, eventSink) + while !signal.aborted && !all_conditions_met + # Long-running operation with periodic abort checks + sleep(0.1) + end + if signal.aborted + return agentToolResult([textContent("Cancelled")], Dict(), nothing, false) + end + return agentToolResult([textContent("Done")], Dict(), nothing, false) + end + ``` +2. External caller sets `signal.aborted = true` (if `abortSignal` is shared via closure) +3. Add timeout logic to tool execute function: + ```julia + start_time = time() + timeout_seconds = 30.0 + while time() - start_time < timeout_seconds && !signal.aborted + # operation... + sleep(0.5) + end + ``` + +> **Reference**: `4-specification.md:SP-022` (abortSignal), `3-walkthrough.md:WT-030` (LLM timeout) + +### 2.6: Message History Grows Unbounded + +**Symptoms**: +- Memory usage grows linearly with conversation length +- LLM context window exceeded for long sessions + +**Steps**: +1. Implement pruning in `prepareContext` (`src/utils.jl:112-126`): + ```julia + function my_prepareContext(state::agentState, eventSink) + # Prune old messages, keep last N turns + max_turns = 20 + recent_msgs = isempty(state.messages) ? state.messages : + state.messages[max(1, end-2*max_turns):end] + + # Optionally inject system context (time, user prefs) + preparedMessages = recent_msgs + preparedSystemPrompt = state.systemPrompt * "\nCurrent time: $(now())" + + # Filter tools (optional: keep only relevant subset) + filteredTools = state.tools # or filter by intent... + + return agentContext(preparedSystemPrompt, preparedMessages, filteredTools) + end + ``` +2. Add hard limit in `_agentLoop`: + ```julia + if length(agent._state.messages) > 200 + # Truncate oldest messages + agent._state.messages = agent._state.messages[end-100:end] + end + ``` + +> **Reference**: `3-walkthrough.md` (agent message flow), `1-requirements.md:NFR-103` (channel buffer limits) + +--- + +## 3. Extending the Framework + +### 3.1: Add a Custom LLM Provider + +**Steps**: +1. Implement `llmCall` function that accepts `FormattedMessages::Dict{String, Any}` and returns response dict +2. Ensure response dict contains at least one of: + - `message.tool_calls` format (LMStudio/vLLM) — `4-specification.md:SP-017` + - `content[{type=>"tool_calls"}]` format (OpenAI) — `4-specification.md:SP-018` +3. Response dict may optionally contain: `api`, `provider`, `model`, `usage` (`llmUsage` or dict), `finish_reason`, `error_message` (`4-specification.md:SP-017`, `SP-018.4`) + +```julia +function my_openai_llmCall(messages::Dict{String, Any}; eventSink=nothing) + # messages["messages"] contains OpenAI-format messages + # messages["temperature"] (default 0.7) + # messages["tools"] (if tools are set) + + body = Dict( + "model" => "gpt-4", + "messages" => messages["messages"], + "tools" => get(messages, "tools", Any[]) + ) + + resp = HTTP.post("https://api.openai.com/v1/chat/completions", + headers, JSON.json(body)) + json_resp = JSON.parse(String(resp.body)) + + # Extract first completion + choice = json_resp["choices"][1]["message"] + + # Build uniform response + return Dict( + "api" => "openai", + "provider" => "openai", + "model" => "gpt-4", + "message" => choice, + "usage" => json_resp["usage"], + "finish_reason" => json_resp["choices"][1]["finish_reason"] + ) +end +``` + +> **Reference**: `1-requirements.md:FR-002`, `4-specification.md:SP-017, SP-018` + +### 3.2: Add a Custom Tool + +**Steps**: +1. Define tool schema in `inputSchema` (JSON Schema format): + ```julia + schema = Dict{String, Any}( + "type" => "object", + "properties" => Dict( + "query" => Dict("type" => "string", "description" => "Search query"), + "max_results" => Dict("type" => "integer", "default" => 5) + ), + "required" => ["query"] + ) + ``` +2. Implement `execute` function: + ```julia + tool = agentTool( + name="searchWine", + label="Search Wine Inventory", + description="Search for wines matching a query", + inputSchema=schema, + execute=(toolCallId, args, signal, eventSink) -> begin + query = args["query"] + max_results = get(args, "max_results", Dict{Any,Any}()["max_results"] => 5...) # handle default + + wines = run_search(query, max_results) + + result_text = join(["- $(w.name) (\$$(w.price))" for w in wines], "\n") + return agentToolResult( + [textContent("Found $(length(wines)) wines:\n$result_text")], + Dict("count" => length(wines)), + nothing, false # content, details, usage, terminate + ) + end, + prepareArguments = nothing, + validateRequiredArgs = nothing, # uses default validator + parallelToolExecute = false + ) + + # Register: push!(tools, tool.name => tool) + ``` +3. Optionally override validation: + ```julia + function my_validate(args) + if haskey(args, "query") && length(args["query"]) < 3 + return "Query must be at least 3 characters" + end + return nothing + end + + agentTool( + ..., + validateRequiredArgs = my_validate, # replaces default validator + ) + ``` + +> **Reference**: `1-requirements.md:FR-004`, `4-specification.md:SP-009, SP-010` + +### 3.3: Implement User Approval Gate (beforeToolCall Hook) + +**Steps**: +1. Implement `beforeToolCall` that blocks expensive/destructive operations: + ```julia + function my_approval_toolCall(ctx::beforeToolCallContext, signal::abortSignal, eventSink) + if ctx.toolCall.name in ("delete_wine", "update_price_bulk") + eventSink("Approval required for $(ctx.toolCall.name) with args $(ctx.args)") + # In production, send to UI and wait for user input + approved = ask_user_for_approval(ctx.toolCall.name, ctx.args) # blocking call + if !approved + return beforeToolCallResult(true, "User denied approval for $(ctx.toolCall.name)") + end + end + return nothing # allow execution + end + ``` + +> **Reference**: `2-solution_design.md:SD-005` (lifecycle hooks), `3-walkthrough.md:WT-032` (beforeToolCall blocks) + +### 3.4: Implement Result Sanitization (afterToolCall Hook) + +**Steps**: +1. Implement `afterToolCall` to mask sensitive data: + ```julia + function my_sanitize_toolCall(ctx::afterToolCallContext, signal::abortSignal, eventSink) + if ctx.toolCall.name == "getCustomerInfo" + # Mask sensitive fields before LLM sees result + result = ctx.result + for (i, block) in enumerate(result.content) + if block isa textContent + masked = replace(block.text, r"\d{12}" => "***-***-****") # mask SSN + result = agentToolResult( + [replace(block.text, r"\d{12}" => "***-***-****") for block in result.content], + result.details, + result.usage, + result.terminate + ) + end + end + return result + end + return nothing # pass through for other tools + end + ``` + +> **Reference**: `2-solution_design.md:SD-005` (lifecycle hooks) + +--- + +## 4. Common Error Codes + +| Error Condition | Response Code | Description | +|-----------------|---------------|-------------| +| Tool not found | `immediateOutcome("Tool not found", true)` | `context.tools[toolCall.name]` returns `nothing` | +| Missing required args | `immediateOutcome("Missing required arguments: ", true)` | `validateRequiredArgs` returns error string | +| Custom validation failed | `immediateOutcome("", true)` | Tool-specific validation hook returns error | +| beforeToolCall blocked | `immediateOutcome("Tool execution was blocked: ", true)` | `beforeToolCall` returns `beforeToolCallResult(true, reason)` | +| Operation aborted | `immediateOutcome("Operation aborted", true)` | `signal.aborted == true` after `beforeToolCall` | +| Tool execution crashed | `immediateOutcome("", true)` | `preptool.execute()` throws exception | +| MCP error | `agentToolResult(["MCP error: "], {..., "error"=>err_msg}, ...)` | Server returns `response["error"]` JSON-RPC envelope | +| MCP network error | `agentToolResult(["MCP call error: "], {..., "error"=>err_msg}, ...)` | Exception during `mcp("tools/call", ...)` | + +> **Reference**: `4-specification.md:SP-009, SP-010`, `src/agentCore.jl:989, 1013, 1019, 1033, 1097` + +--- + +## 5. Debugging Checklist + +Per ASG Framework §Pre-Code Debugging Habit: + +| Stage | Debug Question | +|-------|----------------| +| **Implementation** | If a tool is not found, does `prepareToolCall` return `immediateOutcome` with "Tool not found"? | +| **Testing** | If an LLM response has `tool_calls`, does `_extractToolCalls` return `(true, toolCallList, assistantMessageToolCall)`? | +| **Deploying** | If the agent loop exits, does `eventSink` contain the error message? | + +--- + +## 6. References + +- **API contract**: [`4-specification.md`](4-specification.md) — message types, MCP JSON-RPC format, response extraction formats +- **Solution decisions**: [`2-solution_design.md`](2-solution_design.md) — component architecture, lifecycle hooks, format-agnostic LLM responses +- **Walkthroughs**: [`3-walkthrough.md`](3-walkthrough.md) — end-to-end agent flow, error paths, hook sequences +- **Validation tests**: [`6-validation.md`](6-validation.md) — test traceability matrix mapping requirements/specs to test files