# Runbook: YiemAgent Agent Framework

**Severity**: N/A (library — not a deployed service)
**Estimated Time**: N/A

This runbook documents operational guidance for developers working with the YiemAgent agent framework. It is not an operations runbook for a production service, but a developer handbook for debugging, troubleshooting, and extending the framework.

---

## 1. Agent Lifecycle

### 1.1: Create an Agent

**Steps**:
```julia
using YiemAgent

# 1. Implement llmCall function
function my_llmCall(messages; eventSink=nothing)
    # Call LLM API (e.g., OpenAI, Anthropic, LMStudio)
    # Return dict with keys: "message", "content", "finish_reason", "api", "provider", "model", "usage"
    eventSink("my_llmCall: sending $(length(messages["messages"])) messages"; publish=true)
    # ... API call ...
    return response_dict
end

# 2. Implement mcpServer callable struct
struct MyMCPClient
    conn::NATS.Connection   # or HTTP.Client
    topic::String
    senderID::String
end

function (mcp::MyMCPClient)(method::String, args...; eventSink=nothing)
    # JSON-RPC 2.0 calls:
    #   mcp("tools/list", sessionId; eventSink=eventSink)
    #   mcp("tools/call", toolName, args, sessionId; eventSink=eventSink)
    # ... NATS/HTTP call ...
    return jsonrpc_response
end

# 3. Create agent
mcp = MyMCPClient(nats_conn, "yiemagent:tools", "agent_1")
agent = yiemAgent(
    my_llmCall;
    systemPrompt="You are a wine assistant.\nYou have tools for searching wine inventory.",
    tools=mcp_tools,              # optional: pass pre-discovered tools
    prepareContext=my_prepareContext,   # optional: custom context preparation
    formatMsgForLLM=my_formatMsgForLLM, # optional: custom formatting for provider
    beforeToolCall=my_beforeToolCall,   # optional: approval gate
    afterToolCall=my_afterToolCall,     # optional: result sanitization
    mcpServer=mcp,
    parallelToolExecute=false,
    eventSink=my_eventSink,
)
```

> **Reference**: `2-solution_design.md:SD-001` (channel-based async), `SD-002` (background task), `SD-004` (MCP struct)

### 1.2: Send a Message

```julia
# Option 1: Send a string message (automatically wrapped as OpenAI dict)
promptAgent(agent, "What wines have notes of oak and vanilla?")

# Option 2: Send a full OpenAI-format dict with images
msg = Dict(
    "role" => "user",
    "content" => [
        Dict("type" => "text", "text" => "What wines match these notes?"),
        Dict("type" => "image_url", "image_url" => Dict("url" => "data:image/png;base64,..."))
    ]
)
promptAgent(agent, msg)
```

### 1.3: Receive a Response

```julia
# Blocks until agent finishes processing (no tool calls or tool calls complete)
response = takeResponse(agent)
# response is a Dict in OpenAI format:
#   Dict("role" => "assistant", "content" => [Dict("type" => "text", "text" => "...")])

# For text-only response:
if response["content"][1]["type"] == "text"
    println(response["content"][1]["text"])
end
```

### 1.4: Send Follow-up While Agent is Busy

```julia
# While the agent is processing tool calls (between promptAgent(...) and takeResponse(...))
followUp(agent, "Also check European wines")
```

### 1.5: Stop the Agent

```julia
stopAgent(agent)
# Sends :shutdown, closes all channels. Agent object is destroyed.
```

---

## 2. Troubleshooting

### 2.1: Agent Loop Exits Silently

**Symptoms**:
- `takeResponse(agent)` hangs indefinitely
- No output in eventSink

**Steps**:
1. Check if `eventSink` is receiving error messages:
   ```julia
   global error_count = 0
   my_eventSink(msg; publish=false) = begin
       if occursin("error", lowercase(msg)) || occursin("failed", lowercase(msg))
           println("ERROR EVENT: $msg")
           global error_count += 1
       end
   end
   ```
2. Check network connectivity to LLM API endpoint
3. Verify `formatMsgForLLM` produces valid OpenAI-format message structure (`4-specification.md:SP-011..SP-016`)

> **Reference**: `4-specification.md` (OpenAI format specs), `3-walkthrough.md:WT-030` (LLM timeout)

### 2.2: Tool Call Not Executing

**Symptoms**:
- LLM returns `tool_calls` but agent returns text response immediately
- `beforeToolCall` hook is missing or incorrectly implemented

**Steps**:
1. Verify `prepareContext` returns `agentContext` with `tools` field (not `nothing`) (`src/utils.jl:123`)
2. Check `beforeToolCall` hook is not blocking execution:
   ```julia
   function my_beforeToolCall(ctx, signal, eventSink)
       # Returning nothing → allows execution
       # Returning beforeToolCallResult(true, "blocked") → blocks
       eventSink("beforeToolCall for $(ctx.toolCall.name)"; publish=true)
       nothing  # ← must return nothing (not beforeToolCallResult(false, "N/A"))
   end
   ```
3. Verify tool exists in `context.tools[toolCall.name]` (`src/agentCore.jl:985`)
4. Check `validateToolArguments` is not throwing (catches and returns error tool result) (`src/utils.jl:533`)

> **Reference**: `4-specification.md:SP-021` (agentLoopConfig with beforeToolCall hook), `3-walkthrough.md:WT-032` (beforeToolCall blocks)

### 2.3: MCP Server Returns No Tools

**Symptoms**:
- `println("[toolRegistry] Discovered 0 MCP tools."` on startup
- Agent has no tools available at all times

**Steps**:
1. Verify `mcpServer` callable struct correctly handles `mcp("tools/list", sessionId; eventSink)`
2. Check JSON-RPC 2.0 response shape matches `4-specification.md:SP-009`:
   ```json
   {
     "result": {
       "tools": [{"name": "...", ...}],
       "nextCursor": ""  # empty string = no more pages
     }
   }
   ```
3. Verify `tools/list` error response uses `haskey(response, "error")` check (`src/toolRegistry.jl:128`)

> **Reference**: `4-specification.md:SP-009` (tools/list format), `3-walkthrough.md:WT-020` (MCP discovery)

### 2.4: LLM Response Format Not Parsed

**Symptoms**:
- Agent always treats LLM response as text, never extracts tool calls
- LLM correctly returns `tool_calls` but agent ignores them

**Steps**:
1. Verify `llmCall` returns a dict containing `message` (LMStudio/vLLM format, `4-specification.md:SP-017`) or `content` (OpenAI format, `4-specification.md:SP-018`)
2. Check for both format variations:
   - Format 1: `response["message"]["tool_calls"]` (LMStudio style, see `src/agentCore.jl:671-683`)
   - Format 2: `response.content[{type => "tool_calls"}]` (OpenAI style, see `src/agentCore.jl:700-705`)
3. Verify `finish_reason` field matches: `"tool_calls"` triggers tool execution, `"stop"`/`"end_turn"` breaks loop (`src/agentCore.jl:723-724`)

> **Reference**: `4-specification.md:SP-017, SP-018`, `3-walkthrough.md:WT-011, WT-012` (LLM response handling)

### 2.5: Tool Execution Hangs

**Symptoms**:
- `tool.execute(...)` starts but never returns
- EventSink receives `toolExecutionStart` but never `toolExecutionEnd`

**Steps**:
1. Implement `abortSignal` checking in tool execute function:
   ```julia
   function my_tool_execute(toolCallId, args, signal, eventSink)
       while !signal.aborted && !all_conditions_met
           # Long-running operation with periodic abort checks
           sleep(0.1)
       end
       if signal.aborted
           return agentToolResult([textContent("Cancelled")], Dict(), nothing, false)
       end
       return agentToolResult([textContent("Done")], Dict(), nothing, false)
   end
   ```
2. External caller sets `signal.aborted = true` (if `abortSignal` is shared via closure)
3. Add timeout logic to tool execute function:
   ```julia
   start_time = time()
   timeout_seconds = 30.0
   while time() - start_time < timeout_seconds && !signal.aborted
       # operation...
       sleep(0.5)
   end
   ```

> **Reference**: `4-specification.md:SP-022` (abortSignal), `3-walkthrough.md:WT-030` (LLM timeout)

### 2.6: Message History Grows Unbounded

**Symptoms**:
- Memory usage grows linearly with conversation length
- LLM context window exceeded for long sessions

**Steps**:
1. Implement pruning in `prepareContext` (`src/utils.jl:112-126`):
   ```julia
   function my_prepareContext(state::agentState, eventSink)
       # Prune old messages, keep last N turns
       max_turns = 20
       recent_msgs = isempty(state.messages) ? state.messages :
                     state.messages[max(1, end-2*max_turns):end]
       
       # Optionally inject system context (time, user prefs)
       preparedMessages = recent_msgs
       preparedSystemPrompt = state.systemPrompt * "\nCurrent time: $(now())"
       
       # Filter tools (optional: keep only relevant subset)
       filteredTools = state.tools  # or filter by intent...
       
       return agentContext(preparedSystemPrompt, preparedMessages, filteredTools)
   end
   ```
2. Add hard limit in `_agentLoop`:
   ```julia
   if length(agent._state.messages) > 200
       # Truncate oldest messages
       agent._state.messages = agent._state.messages[end-100:end]
   end
   ```

> **Reference**: `3-walkthrough.md` (agent message flow), `1-requirements.md:NFR-103` (channel buffer limits)

---

## 3. Extending the Framework

### 3.1: Add a Custom LLM Provider

**Steps**:
1. Implement `llmCall` function that accepts `FormattedMessages::Dict{String, Any}` and returns response dict
2. Ensure response dict contains at least one of:
   - `message.tool_calls` format (LMStudio/vLLM) — `4-specification.md:SP-017`
   - `content[{type=>"tool_calls"}]` format (OpenAI) — `4-specification.md:SP-018`
3. Response dict may optionally contain: `api`, `provider`, `model`, `usage` (`llmUsage` or dict), `finish_reason`, `error_message` (`4-specification.md:SP-017`, `SP-018.4`)

```julia
function my_openai_llmCall(messages::Dict{String, Any}; eventSink=nothing)
    # messages["messages"] contains OpenAI-format messages
    # messages["temperature"] (default 0.7)
    # messages["tools"] (if tools are set)
    
    body = Dict(
        "model" => "gpt-4",
        "messages" => messages["messages"],
        "tools" => get(messages, "tools", Any[])
    )
    
    resp = HTTP.post("https://api.openai.com/v1/chat/completions", 
                     headers, JSON.json(body))
    json_resp = JSON.parse(String(resp.body))
    
    # Extract first completion
    choice = json_resp["choices"][1]["message"]
    
    # Build uniform response
    return Dict(
        "api" => "openai",
        "provider" => "openai",
        "model" => "gpt-4",
        "message" => choice,
        "usage" => json_resp["usage"],
        "finish_reason" => json_resp["choices"][1]["finish_reason"]
    )
end
```

> **Reference**: `1-requirements.md:FR-002`, `4-specification.md:SP-017, SP-018`

### 3.2: Add a Custom Tool

**Steps**:
1. Define tool schema in `inputSchema` (JSON Schema format):
   ```julia
   schema = Dict{String, Any}(
       "type" => "object",
       "properties" => Dict(
           "query" => Dict("type" => "string", "description" => "Search query"),
           "max_results" => Dict("type" => "integer", "default" => 5)
       ),
       "required" => ["query"]
   )
   ```
2. Implement `execute` function:
   ```julia
   tool = agentTool(
       name="searchWine",
       label="Search Wine Inventory",
       description="Search for wines matching a query",
       inputSchema=schema,
       execute=(toolCallId, args, signal, eventSink) -> begin
           query = args["query"]
           max_results = get(args, "max_results", Dict{Any,Any}()["max_results"] => 5...)  # handle default
           
           wines = run_search(query, max_results)
           
           result_text = join(["- $(w.name) (\$$(w.price))" for w in wines], "\n")
           return agentToolResult(
               [textContent("Found $(length(wines)) wines:\n$result_text")],
               Dict("count" => length(wines)),
               nothing, false  # content, details, usage, terminate
           )
       end,
       prepareArguments = nothing,
       validateRequiredArgs = nothing,  # uses default validator
       parallelToolExecute = false
   )
   
   # Register: push!(tools, tool.name => tool)
   ```
3. Optionally override validation:
   ```julia
   function my_validate(args)
       if haskey(args, "query") && length(args["query"]) < 3
           return "Query must be at least 3 characters"
       end
       return nothing
   end
   
   agentTool(
       ...,
       validateRequiredArgs = my_validate,  # replaces default validator
   )
   ```

> **Reference**: `1-requirements.md:FR-004`, `4-specification.md:SP-009, SP-010`

### 3.3: Implement User Approval Gate (beforeToolCall Hook)

**Steps**:
1. Implement `beforeToolCall` that blocks expensive/destructive operations:
   ```julia
   function my_approval_toolCall(ctx::beforeToolCallContext, signal::abortSignal, eventSink)
       if ctx.toolCall.name in ("delete_wine", "update_price_bulk")
           eventSink("Approval required for $(ctx.toolCall.name) with args $(ctx.args)")
           # In production, send to UI and wait for user input
           approved = ask_user_for_approval(ctx.toolCall.name, ctx.args)  # blocking call
           if !approved
               return beforeToolCallResult(true, "User denied approval for $(ctx.toolCall.name)")
           end
       end
       return nothing  # allow execution
   end
   ```

> **Reference**: `2-solution_design.md:SD-005` (lifecycle hooks), `3-walkthrough.md:WT-032` (beforeToolCall blocks)

### 3.4: Implement Result Sanitization (afterToolCall Hook)

**Steps**:
1. Implement `afterToolCall` to mask sensitive data:
   ```julia
   function my_sanitize_toolCall(ctx::afterToolCallContext, signal::abortSignal, eventSink)
       if ctx.toolCall.name == "getCustomerInfo"
           # Mask sensitive fields before LLM sees result
           result = ctx.result
           for (i, block) in enumerate(result.content)
               if block isa textContent
                   masked = replace(block.text, r"\d{12}" => "***-***-****")  # mask SSN
                   result = agentToolResult(
                       [replace(block.text, r"\d{12}" => "***-***-****") for block in result.content],
                       result.details,
                       result.usage,
                       result.terminate
                   )
               end
           end
           return result
       end
       return nothing  # pass through for other tools
   end
   ```

> **Reference**: `2-solution_design.md:SD-005` (lifecycle hooks)

---

## 4. Common Error Codes

| Error Condition | Response Code | Description |
|-----------------|---------------|-------------|
| Tool not found | `immediateOutcome("Tool <name> not found", true)` | `context.tools[toolCall.name]` returns `nothing` |
| Missing required args | `immediateOutcome("Missing required arguments: <fields>", true)` | `validateRequiredArgs` returns error string |
| Custom validation failed | `immediateOutcome("<tool.validateRequiredArgs result>", true)` | Tool-specific validation hook returns error |
| beforeToolCall blocked | `immediateOutcome("Tool execution was blocked: <reason>", true)` | `beforeToolCall` returns `beforeToolCallResult(true, reason)` |
| Operation aborted | `immediateOutcome("Operation aborted", true)` | `signal.aborted == true` after `beforeToolCall` |
| Tool execution crashed | `immediateOutcome("<exception message>", true)` | `preptool.execute()` throws exception |
| MCP error | `agentToolResult(["MCP error: <err_msg>"], {..., "error"=>err_msg}, ...)` | Server returns `response["error"]` JSON-RPC envelope |
| MCP network error | `agentToolResult(["MCP call error: <err_msg>"], {..., "error"=>err_msg}, ...)` | Exception during `mcp("tools/call", ...)` |

> **Reference**: `4-specification.md:SP-009, SP-010`, `src/agentCore.jl:989, 1013, 1019, 1033, 1097`

---

## 5. Debugging Checklist

Per ASG Framework §Pre-Code Debugging Habit:

| Stage | Debug Question |
|-------|----------------|
| **Implementation** | If a tool is not found, does `prepareToolCall` return `immediateOutcome` with "Tool not found"? |
| **Testing** | If an LLM response has `tool_calls`, does `_extractToolCalls` return `(true, toolCallList, assistantMessageToolCall)`? |
| **Deploying** | If the agent loop exits, does `eventSink` contain the error message? |

---

## 6. References

- **API contract**: [`4-specification.md`](4-specification.md) — message types, MCP JSON-RPC format, response extraction formats
- **Solution decisions**: [`2-solution_design.md`](2-solution_design.md) — component architecture, lifecycle hooks, format-agnostic LLM responses
- **Walkthroughs**: [`3-walkthrough.md`](3-walkthrough.md) — end-to-end agent flow, error paths, hook sequences
- **Validation tests**: [`6-validation.md`](6-validation.md) — test traceability matrix mapping requirements/specs to test files