446 lines
16 KiB
Markdown
446 lines
16 KiB
Markdown
# Runbook: YiemAgent Agent Framework
|
|
|
|
**Severity**: N/A (library — not a deployed service)
|
|
**Estimated Time**: N/A
|
|
|
|
This runbook documents operational guidance for developers working with the YiemAgent agent framework. It is not an operations runbook for a production service, but a developer handbook for debugging, troubleshooting, and extending the framework.
|
|
|
|
---
|
|
|
|
## 1. Agent Lifecycle
|
|
|
|
### 1.1: Create an Agent
|
|
|
|
**Steps**:
|
|
```julia
|
|
using YiemAgent
|
|
|
|
# 1. Implement llmCall function
|
|
function my_llmCall(messages; eventSink=nothing)
|
|
# Call LLM API (e.g., OpenAI, Anthropic, LMStudio)
|
|
# Return dict with keys: "message", "content", "finish_reason", "api", "provider", "model", "usage"
|
|
eventSink("my_llmCall: sending $(length(messages["messages"])) messages"; publish=true)
|
|
# ... API call ...
|
|
return response_dict
|
|
end
|
|
|
|
# 2. Implement mcpServer callable struct
|
|
struct MyMCPClient
|
|
conn::NATS.Connection # or HTTP.Client
|
|
topic::String
|
|
senderID::String
|
|
end
|
|
|
|
function (mcp::MyMCPClient)(method::String, args...; eventSink=nothing)
|
|
# JSON-RPC 2.0 calls:
|
|
# mcp("tools/list", sessionId; eventSink=eventSink)
|
|
# mcp("tools/call", toolName, args, sessionId; eventSink=eventSink)
|
|
# ... NATS/HTTP call ...
|
|
return jsonrpc_response
|
|
end
|
|
|
|
# 3. Create agent
|
|
mcp = MyMCPClient(nats_conn, "yiemagent:tools", "agent_1")
|
|
agent = yiemAgent(
|
|
my_llmCall;
|
|
systemPrompt="You are a wine assistant.\nYou have tools for searching wine inventory.",
|
|
tools=mcp_tools, # optional: pass pre-discovered tools
|
|
prepareContext=my_prepareContext, # optional: custom context preparation
|
|
formatMsgForLLM=my_formatMsgForLLM, # optional: custom formatting for provider
|
|
beforeToolCall=my_beforeToolCall, # optional: approval gate
|
|
afterToolCall=my_afterToolCall, # optional: result sanitization
|
|
mcpServer=mcp,
|
|
parallelToolExecute=false,
|
|
eventSink=my_eventSink,
|
|
)
|
|
```
|
|
|
|
> **Reference**: `2-solution_design.md:SD-001` (channel-based async), `SD-002` (background task), `SD-004` (MCP struct)
|
|
|
|
### 1.2: Send a Message
|
|
|
|
```julia
|
|
# Option 1: Send a string message (automatically wrapped as OpenAI dict)
|
|
promptAgent(agent, "What wines have notes of oak and vanilla?")
|
|
|
|
# Option 2: Send a full OpenAI-format dict with images
|
|
msg = Dict(
|
|
"role" => "user",
|
|
"content" => [
|
|
Dict("type" => "text", "text" => "What wines match these notes?"),
|
|
Dict("type" => "image_url", "image_url" => Dict("url" => "data:image/png;base64,..."))
|
|
]
|
|
)
|
|
promptAgent(agent, msg)
|
|
```
|
|
|
|
### 1.3: Receive a Response
|
|
|
|
```julia
|
|
# Blocks until agent finishes processing (no tool calls or tool calls complete)
|
|
response = takeResponse(agent)
|
|
# response is a Dict in OpenAI format:
|
|
# Dict("role" => "assistant", "content" => [Dict("type" => "text", "text" => "...")])
|
|
|
|
# For text-only response:
|
|
if response["content"][1]["type"] == "text"
|
|
println(response["content"][1]["text"])
|
|
end
|
|
```
|
|
|
|
### 1.4: Send Follow-up While Agent is Busy
|
|
|
|
```julia
|
|
# While the agent is processing tool calls (between promptAgent(...) and takeResponse(...))
|
|
followUp(agent, "Also check European wines")
|
|
```
|
|
|
|
### 1.5: Stop the Agent
|
|
|
|
```julia
|
|
stopAgent(agent)
|
|
# Sends :shutdown, closes all channels. Agent object is destroyed.
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Troubleshooting
|
|
|
|
### 2.1: Agent Loop Exits Silently
|
|
|
|
**Symptoms**:
|
|
- `takeResponse(agent)` hangs indefinitely
|
|
- No output in eventSink
|
|
|
|
**Steps**:
|
|
1. Check if `eventSink` is receiving error messages:
|
|
```julia
|
|
global error_count = 0
|
|
my_eventSink(msg; publish=false) = begin
|
|
if occursin("error", lowercase(msg)) || occursin("failed", lowercase(msg))
|
|
println("ERROR EVENT: $msg")
|
|
global error_count += 1
|
|
end
|
|
end
|
|
```
|
|
2. Check network connectivity to LLM API endpoint
|
|
3. Verify `formatMsgForLLM` produces valid OpenAI-format message structure (`4-specification.md:SP-011..SP-016`)
|
|
|
|
> **Reference**: `4-specification.md` (OpenAI format specs), `3-walkthrough.md:WT-030` (LLM timeout)
|
|
|
|
### 2.2: Tool Call Not Executing
|
|
|
|
**Symptoms**:
|
|
- LLM returns `tool_calls` but agent returns text response immediately
|
|
- `beforeToolCall` hook is missing or incorrectly implemented
|
|
|
|
**Steps**:
|
|
1. Verify `prepareContext` returns `agentContext` with `tools` field (not `nothing`) (`src/utils.jl:123`)
|
|
2. Check `beforeToolCall` hook is not blocking execution:
|
|
```julia
|
|
function my_beforeToolCall(ctx, signal, eventSink)
|
|
# Returning nothing → allows execution
|
|
# Returning beforeToolCallResult(true, "blocked") → blocks
|
|
eventSink("beforeToolCall for $(ctx.toolCall.name)"; publish=true)
|
|
nothing # ← must return nothing (not beforeToolCallResult(false, "N/A"))
|
|
end
|
|
```
|
|
3. Verify tool exists in `context.tools[toolCall.name]` (`src/agentCore.jl:985`)
|
|
4. Check `validateToolArguments` is not throwing (catches and returns error tool result) (`src/utils.jl:533`)
|
|
|
|
> **Reference**: `4-specification.md:SP-021` (agentLoopConfig with beforeToolCall hook), `3-walkthrough.md:WT-032` (beforeToolCall blocks)
|
|
|
|
### 2.3: MCP Server Returns No Tools
|
|
|
|
**Symptoms**:
|
|
- `println("[toolRegistry] Discovered 0 MCP tools."` on startup
|
|
- Agent has no tools available at all times
|
|
|
|
**Steps**:
|
|
1. Verify `mcpServer` callable struct correctly handles `mcp("tools/list", sessionId; eventSink)`
|
|
2. Check JSON-RPC 2.0 response shape matches `4-specification.md:SP-009`:
|
|
```json
|
|
{
|
|
"result": {
|
|
"tools": [{"name": "...", ...}],
|
|
"nextCursor": "" # empty string = no more pages
|
|
}
|
|
}
|
|
```
|
|
3. Verify `tools/list` error response uses `haskey(response, "error")` check (`src/toolRegistry.jl:128`)
|
|
|
|
> **Reference**: `4-specification.md:SP-009` (tools/list format), `3-walkthrough.md:WT-020` (MCP discovery)
|
|
|
|
### 2.4: LLM Response Format Not Parsed
|
|
|
|
**Symptoms**:
|
|
- Agent always treats LLM response as text, never extracts tool calls
|
|
- LLM correctly returns `tool_calls` but agent ignores them
|
|
|
|
**Steps**:
|
|
1. Verify `llmCall` returns a dict containing `message` (LMStudio/vLLM format, `4-specification.md:SP-017`) or `content` (OpenAI format, `4-specification.md:SP-018`)
|
|
2. Check for both format variations:
|
|
- Format 1: `response["message"]["tool_calls"]` (LMStudio style, see `src/agentCore.jl:671-683`)
|
|
- Format 2: `response.content[{type => "tool_calls"}]` (OpenAI style, see `src/agentCore.jl:700-705`)
|
|
3. Verify `finish_reason` field matches: `"tool_calls"` triggers tool execution, `"stop"`/`"end_turn"` breaks loop (`src/agentCore.jl:723-724`)
|
|
|
|
> **Reference**: `4-specification.md:SP-017, SP-018`, `3-walkthrough.md:WT-011, WT-012` (LLM response handling)
|
|
|
|
### 2.5: Tool Execution Hangs
|
|
|
|
**Symptoms**:
|
|
- `tool.execute(...)` starts but never returns
|
|
- EventSink receives `toolExecutionStart` but never `toolExecutionEnd`
|
|
|
|
**Steps**:
|
|
1. Implement `abortSignal` checking in tool execute function:
|
|
```julia
|
|
function my_tool_execute(toolCallId, args, signal, eventSink)
|
|
while !signal.aborted && !all_conditions_met
|
|
# Long-running operation with periodic abort checks
|
|
sleep(0.1)
|
|
end
|
|
if signal.aborted
|
|
return agentToolResult([textContent("Cancelled")], Dict(), nothing, false)
|
|
end
|
|
return agentToolResult([textContent("Done")], Dict(), nothing, false)
|
|
end
|
|
```
|
|
2. External caller sets `signal.aborted = true` (if `abortSignal` is shared via closure)
|
|
3. Add timeout logic to tool execute function:
|
|
```julia
|
|
start_time = time()
|
|
timeout_seconds = 30.0
|
|
while time() - start_time < timeout_seconds && !signal.aborted
|
|
# operation...
|
|
sleep(0.5)
|
|
end
|
|
```
|
|
|
|
> **Reference**: `4-specification.md:SP-022` (abortSignal), `3-walkthrough.md:WT-030` (LLM timeout)
|
|
|
|
### 2.6: Message History Grows Unbounded
|
|
|
|
**Symptoms**:
|
|
- Memory usage grows linearly with conversation length
|
|
- LLM context window exceeded for long sessions
|
|
|
|
**Steps**:
|
|
1. Implement pruning in `prepareContext` (`src/utils.jl:112-126`):
|
|
```julia
|
|
function my_prepareContext(state::agentState, eventSink)
|
|
# Prune old messages, keep last N turns
|
|
max_turns = 20
|
|
recent_msgs = isempty(state.messages) ? state.messages :
|
|
state.messages[max(1, end-2*max_turns):end]
|
|
|
|
# Optionally inject system context (time, user prefs)
|
|
preparedMessages = recent_msgs
|
|
preparedSystemPrompt = state.systemPrompt * "\nCurrent time: $(now())"
|
|
|
|
# Filter tools (optional: keep only relevant subset)
|
|
filteredTools = state.tools # or filter by intent...
|
|
|
|
return agentContext(preparedSystemPrompt, preparedMessages, filteredTools)
|
|
end
|
|
```
|
|
2. Add hard limit in `_agentLoop`:
|
|
```julia
|
|
if length(agent._state.messages) > 200
|
|
# Truncate oldest messages
|
|
agent._state.messages = agent._state.messages[end-100:end]
|
|
end
|
|
```
|
|
|
|
> **Reference**: `3-walkthrough.md` (agent message flow), `1-requirements.md:NFR-103` (channel buffer limits)
|
|
|
|
---
|
|
|
|
## 3. Extending the Framework
|
|
|
|
### 3.1: Add a Custom LLM Provider
|
|
|
|
**Steps**:
|
|
1. Implement `llmCall` function that accepts `FormattedMessages::Dict{String, Any}` and returns response dict
|
|
2. Ensure response dict contains at least one of:
|
|
- `message.tool_calls` format (LMStudio/vLLM) — `4-specification.md:SP-017`
|
|
- `content[{type=>"tool_calls"}]` format (OpenAI) — `4-specification.md:SP-018`
|
|
3. Response dict may optionally contain: `api`, `provider`, `model`, `usage` (`llmUsage` or dict), `finish_reason`, `error_message` (`4-specification.md:SP-017`, `SP-018.4`)
|
|
|
|
```julia
|
|
function my_openai_llmCall(messages::Dict{String, Any}; eventSink=nothing)
|
|
# messages["messages"] contains OpenAI-format messages
|
|
# messages["temperature"] (default 0.7)
|
|
# messages["tools"] (if tools are set)
|
|
|
|
body = Dict(
|
|
"model" => "gpt-4",
|
|
"messages" => messages["messages"],
|
|
"tools" => get(messages, "tools", Any[])
|
|
)
|
|
|
|
resp = HTTP.post("https://api.openai.com/v1/chat/completions",
|
|
headers, JSON.json(body))
|
|
json_resp = JSON.parse(String(resp.body))
|
|
|
|
# Extract first completion
|
|
choice = json_resp["choices"][1]["message"]
|
|
|
|
# Build uniform response
|
|
return Dict(
|
|
"api" => "openai",
|
|
"provider" => "openai",
|
|
"model" => "gpt-4",
|
|
"message" => choice,
|
|
"usage" => json_resp["usage"],
|
|
"finish_reason" => json_resp["choices"][1]["finish_reason"]
|
|
)
|
|
end
|
|
```
|
|
|
|
> **Reference**: `1-requirements.md:FR-002`, `4-specification.md:SP-017, SP-018`
|
|
|
|
### 3.2: Add a Custom Tool
|
|
|
|
**Steps**:
|
|
1. Define tool schema in `inputSchema` (JSON Schema format):
|
|
```julia
|
|
schema = Dict{String, Any}(
|
|
"type" => "object",
|
|
"properties" => Dict(
|
|
"query" => Dict("type" => "string", "description" => "Search query"),
|
|
"max_results" => Dict("type" => "integer", "default" => 5)
|
|
),
|
|
"required" => ["query"]
|
|
)
|
|
```
|
|
2. Implement `execute` function:
|
|
```julia
|
|
tool = agentTool(
|
|
name="searchWine",
|
|
label="Search Wine Inventory",
|
|
description="Search for wines matching a query",
|
|
inputSchema=schema,
|
|
execute=(toolCallId, args, signal, eventSink) -> begin
|
|
query = args["query"]
|
|
max_results = get(args, "max_results", Dict{Any,Any}()["max_results"] => 5...) # handle default
|
|
|
|
wines = run_search(query, max_results)
|
|
|
|
result_text = join(["- $(w.name) (\$$(w.price))" for w in wines], "\n")
|
|
return agentToolResult(
|
|
[textContent("Found $(length(wines)) wines:\n$result_text")],
|
|
Dict("count" => length(wines)),
|
|
nothing, false # content, details, usage, terminate
|
|
)
|
|
end,
|
|
prepareArguments = nothing,
|
|
validateRequiredArgs = nothing, # uses default validator
|
|
parallelToolExecute = false
|
|
)
|
|
|
|
# Register: push!(tools, tool.name => tool)
|
|
```
|
|
3. Optionally override validation:
|
|
```julia
|
|
function my_validate(args)
|
|
if haskey(args, "query") && length(args["query"]) < 3
|
|
return "Query must be at least 3 characters"
|
|
end
|
|
return nothing
|
|
end
|
|
|
|
agentTool(
|
|
...,
|
|
validateRequiredArgs = my_validate, # replaces default validator
|
|
)
|
|
```
|
|
|
|
> **Reference**: `1-requirements.md:FR-004`, `4-specification.md:SP-009, SP-010`
|
|
|
|
### 3.3: Implement User Approval Gate (beforeToolCall Hook)
|
|
|
|
**Steps**:
|
|
1. Implement `beforeToolCall` that blocks expensive/destructive operations:
|
|
```julia
|
|
function my_approval_toolCall(ctx::beforeToolCallContext, signal::abortSignal, eventSink)
|
|
if ctx.toolCall.name in ("delete_wine", "update_price_bulk")
|
|
eventSink("Approval required for $(ctx.toolCall.name) with args $(ctx.args)")
|
|
# In production, send to UI and wait for user input
|
|
approved = ask_user_for_approval(ctx.toolCall.name, ctx.args) # blocking call
|
|
if !approved
|
|
return beforeToolCallResult(true, "User denied approval for $(ctx.toolCall.name)")
|
|
end
|
|
end
|
|
return nothing # allow execution
|
|
end
|
|
```
|
|
|
|
> **Reference**: `2-solution_design.md:SD-005` (lifecycle hooks), `3-walkthrough.md:WT-032` (beforeToolCall blocks)
|
|
|
|
### 3.4: Implement Result Sanitization (afterToolCall Hook)
|
|
|
|
**Steps**:
|
|
1. Implement `afterToolCall` to mask sensitive data:
|
|
```julia
|
|
function my_sanitize_toolCall(ctx::afterToolCallContext, signal::abortSignal, eventSink)
|
|
if ctx.toolCall.name == "getCustomerInfo"
|
|
# Mask sensitive fields before LLM sees result
|
|
result = ctx.result
|
|
for (i, block) in enumerate(result.content)
|
|
if block isa textContent
|
|
masked = replace(block.text, r"\d{12}" => "***-***-****") # mask SSN
|
|
result = agentToolResult(
|
|
[replace(block.text, r"\d{12}" => "***-***-****") for block in result.content],
|
|
result.details,
|
|
result.usage,
|
|
result.terminate
|
|
)
|
|
end
|
|
end
|
|
return result
|
|
end
|
|
return nothing # pass through for other tools
|
|
end
|
|
```
|
|
|
|
> **Reference**: `2-solution_design.md:SD-005` (lifecycle hooks)
|
|
|
|
---
|
|
|
|
## 4. Common Error Codes
|
|
|
|
| Error Condition | Response Code | Description |
|
|
|-----------------|---------------|-------------|
|
|
| Tool not found | `immediateOutcome("Tool <name> not found", true)` | `context.tools[toolCall.name]` returns `nothing` |
|
|
| Missing required args | `immediateOutcome("Missing required arguments: <fields>", true)` | `validateRequiredArgs` returns error string |
|
|
| Custom validation failed | `immediateOutcome("<tool.validateRequiredArgs result>", true)` | Tool-specific validation hook returns error |
|
|
| beforeToolCall blocked | `immediateOutcome("Tool execution was blocked: <reason>", true)` | `beforeToolCall` returns `beforeToolCallResult(true, reason)` |
|
|
| Operation aborted | `immediateOutcome("Operation aborted", true)` | `signal.aborted == true` after `beforeToolCall` |
|
|
| Tool execution crashed | `immediateOutcome("<exception message>", true)` | `preptool.execute()` throws exception |
|
|
| MCP error | `agentToolResult(["MCP error: <err_msg>"], {..., "error"=>err_msg}, ...)` | Server returns `response["error"]` JSON-RPC envelope |
|
|
| MCP network error | `agentToolResult(["MCP call error: <err_msg>"], {..., "error"=>err_msg}, ...)` | Exception during `mcp("tools/call", ...)` |
|
|
|
|
> **Reference**: `4-specification.md:SP-009, SP-010`, `src/agentCore.jl:989, 1013, 1019, 1033, 1097`
|
|
|
|
---
|
|
|
|
## 5. Debugging Checklist
|
|
|
|
Per ASG Framework §Pre-Code Debugging Habit:
|
|
|
|
| Stage | Debug Question |
|
|
|-------|----------------|
|
|
| **Implementation** | If a tool is not found, does `prepareToolCall` return `immediateOutcome` with "Tool not found"? |
|
|
| **Testing** | If an LLM response has `tool_calls`, does `_extractToolCalls` return `(true, toolCallList, assistantMessageToolCall)`? |
|
|
| **Deploying** | If the agent loop exits, does `eventSink` contain the error message? |
|
|
|
|
---
|
|
|
|
## 6. References
|
|
|
|
- **API contract**: [`4-specification.md`](4-specification.md) — message types, MCP JSON-RPC format, response extraction formats
|
|
- **Solution decisions**: [`2-solution_design.md`](2-solution_design.md) — component architecture, lifecycle hooks, format-agnostic LLM responses
|
|
- **Walkthroughs**: [`3-walkthrough.md`](3-walkthrough.md) — end-to-end agent flow, error paths, hook sequences
|
|
- **Validation tests**: [`6-validation.md`](6-validation.md) — test traceability matrix mapping requirements/specs to test files
|