YiemAgent/docs/8-runbook.md

Runbook: YiemAgent Agent Framework

Severity: N/A (library — not a deployed service) Estimated Time: N/A

This runbook documents operational guidance for developers working with the YiemAgent agent framework. It is not an operations runbook for a production service, but a developer handbook for debugging, troubleshooting, and extending the framework.

---

1. Agent Lifecycle

1.1: Create an Agent

Steps:

using YiemAgent

# 1. Implement llmCall function
function my_llmCall(messages; eventSink=nothing)
    # Call LLM API (e.g., OpenAI, Anthropic, LMStudio)
    # Return dict with keys: "message", "content", "finish_reason", "api", "provider", "model", "usage"
    eventSink("my_llmCall: sending $(length(messages["messages"])) messages"; publish=true)
    # ... API call ...
    return response_dict
end

# 2. Implement mcpServer callable struct
struct MyMCPClient
    conn::NATS.Connection   # or HTTP.Client
    topic::String
    senderID::String
end

function (mcp::MyMCPClient)(method::String, args...; eventSink=nothing)
    # JSON-RPC 2.0 calls:
    #   mcp("tools/list", sessionId; eventSink=eventSink)
    #   mcp("tools/call", toolName, args, sessionId; eventSink=eventSink)
    # ... NATS/HTTP call ...
    return jsonrpc_response
end

# 3. Create agent
mcp = MyMCPClient(nats_conn, "yiemagent:tools", "agent_1")
agent = yiemAgent(
    my_llmCall;
    systemPrompt="You are a wine assistant.\nYou have tools for searching wine inventory.",
    tools=mcp_tools,              # optional: pass pre-discovered tools
    prepareContext=my_prepareContext,   # optional: custom context preparation
    formatMsgForLLM=my_formatMsgForLLM, # optional: custom formatting for provider
    beforeToolCall=my_beforeToolCall,   # optional: approval gate
    afterToolCall=my_afterToolCall,     # optional: result sanitization
    mcpServer=mcp,
    parallelToolExecute=false,
    eventSink=my_eventSink,
)

Reference: 2-solution_design.md:SD-001 (channel-based async), SD-002 (background task), SD-004 (MCP struct)

1.2: Send a Message

# Option 1: Send a string message (automatically wrapped as OpenAI dict)
promptAgent(agent, "What wines have notes of oak and vanilla?")

# Option 2: Send a full OpenAI-format dict with images
msg = Dict(
    "role" => "user",
    "content" => [
        Dict("type" => "text", "text" => "What wines match these notes?"),
        Dict("type" => "image_url", "image_url" => Dict("url" => "data:image/png;base64,..."))
    ]
)
promptAgent(agent, msg)

1.3: Receive a Response

# Blocks until agent finishes processing (no tool calls or tool calls complete)
response = takeResponse(agent)
# response is a Dict in OpenAI format:
#   Dict("role" => "assistant", "content" => [Dict("type" => "text", "text" => "...")])

# For text-only response:
if response["content"][1]["type"] == "text"
    println(response["content"][1]["text"])
end

1.4: Send Follow-up While Agent is Busy

# While the agent is processing tool calls (between promptAgent(...) and takeResponse(...))
followUp(agent, "Also check European wines")

1.5: Stop the Agent

stopAgent(agent)
# Sends :shutdown, closes all channels. Agent object is destroyed.

---

2. Troubleshooting

2.1: Agent Loop Exits Silently

Symptoms:

• takeResponse(agent) hangs indefinitely
• No output in eventSink

Steps:

1. Check if eventSink is receiving error messages:

   global error_count = 0
   my_eventSink(msg; publish=false) = begin
       if occursin("error", lowercase(msg)) || occursin("failed", lowercase(msg))
           println("ERROR EVENT: $msg")
           global error_count += 1
       end
   end
   

2. Check network connectivity to LLM API endpoint
3. Verify formatMsgForLLM produces valid OpenAI-format message structure (4-specification.md:SP-011..SP-016)

Reference: 4-specification.md (OpenAI format specs), 3-walkthrough.md:WT-030 (LLM timeout)

2.2: Tool Call Not Executing

Symptoms:

• LLM returns tool_calls but agent returns text response immediately
• beforeToolCall hook is missing or incorrectly implemented

Steps:

1. Verify prepareContext returns agentContext with tools field (not nothing) (src/utils.jl:123)
2. Check beforeToolCall hook is not blocking execution:

   function my_beforeToolCall(ctx, signal, eventSink)
       # Returning nothing → allows execution
       # Returning beforeToolCallResult(true, "blocked") → blocks
       eventSink("beforeToolCall for $(ctx.toolCall.name)"; publish=true)
       nothing  # ← must return nothing (not beforeToolCallResult(false, "N/A"))
   end
   

3. Verify tool exists in context.tools[toolCall.name] (src/agentCore.jl:985)
4. Check validateToolArguments is not throwing (catches and returns error tool result) (src/utils.jl:533)

Reference: 4-specification.md:SP-021 (agentLoopConfig with beforeToolCall hook), 3-walkthrough.md:WT-032 (beforeToolCall blocks)

2.3: MCP Server Returns No Tools

Symptoms:

• println("[toolRegistry] Discovered 0 MCP tools." on startup
• Agent has no tools available at all times

Steps:

1. Verify mcpServer callable struct correctly handles mcp("tools/list", sessionId; eventSink)
2. Check JSON-RPC 2.0 response shape matches 4-specification.md:SP-009:

   {
     "result": {
       "tools": [{"name": "...", ...}],
       "nextCursor": ""  # empty string = no more pages
     }
   }
   

3. Verify tools/list error response uses haskey(response, "error") check (src/toolRegistry.jl:128)

Reference: 4-specification.md:SP-009 (tools/list format), 3-walkthrough.md:WT-020 (MCP discovery)

2.4: LLM Response Format Not Parsed

Symptoms:

• Agent always treats LLM response as text, never extracts tool calls
• LLM correctly returns tool_calls but agent ignores them

Steps:

1. Verify llmCall returns a dict containing message (LMStudio/vLLM format, 4-specification.md:SP-017) or content (OpenAI format, 4-specification.md:SP-018)
2. Check for both format variations:
   • Format 1: response["message"]["tool_calls"] (LMStudio style, see src/agentCore.jl:671-683)
   • Format 2: response.content[{type => "tool_calls"}] (OpenAI style, see src/agentCore.jl:700-705)
3. Verify finish_reason field matches: "tool_calls" triggers tool execution, "stop"/"end_turn" breaks loop (src/agentCore.jl:723-724)

Reference: 4-specification.md:SP-017, SP-018, 3-walkthrough.md:WT-011, WT-012 (LLM response handling)

2.5: Tool Execution Hangs

Symptoms:

• tool.execute(...) starts but never returns
• EventSink receives toolExecutionStart but never toolExecutionEnd

Steps:

1. Implement abortSignal checking in tool execute function:

   function my_tool_execute(toolCallId, args, signal, eventSink)
       while !signal.aborted && !all_conditions_met
           # Long-running operation with periodic abort checks
           sleep(0.1)
       end
       if signal.aborted
           return agentToolResult([textContent("Cancelled")], Dict(), nothing, false)
       end
       return agentToolResult([textContent("Done")], Dict(), nothing, false)
   end
   

2. External caller sets signal.aborted = true (if abortSignal is shared via closure)
3. Add timeout logic to tool execute function:

   start_time = time()
   timeout_seconds = 30.0
   while time() - start_time < timeout_seconds && !signal.aborted
       # operation...
       sleep(0.5)
   end
   

Reference: 4-specification.md:SP-022 (abortSignal), 3-walkthrough.md:WT-030 (LLM timeout)

2.6: Message History Grows Unbounded

Symptoms:

• Memory usage grows linearly with conversation length
• LLM context window exceeded for long sessions

Steps:

1. Implement pruning in prepareContext (src/utils.jl:112-126):

   function my_prepareContext(state::agentState, eventSink)
       # Prune old messages, keep last N turns
       max_turns = 20
       recent_msgs = isempty(state.messages) ? state.messages :
                     state.messages[max(1, end-2*max_turns):end]
   
       # Optionally inject system context (time, user prefs)
       preparedMessages = recent_msgs
       preparedSystemPrompt = state.systemPrompt * "\nCurrent time: $(now())"
   
       # Filter tools (optional: keep only relevant subset)
       filteredTools = state.tools  # or filter by intent...
   
       return agentContext(preparedSystemPrompt, preparedMessages, filteredTools)
   end
   

2. Add hard limit in _agentLoop:

   if length(agent._state.messages) > 200
       # Truncate oldest messages
       agent._state.messages = agent._state.messages[end-100:end]
   end
   

Reference: 3-walkthrough.md (agent message flow), 1-requirements.md:NFR-103 (channel buffer limits)

---

3. Extending the Framework

3.1: Add a Custom LLM Provider

Steps:

1. Implement llmCall function that accepts FormattedMessages::Dict{String, Any} and returns response dict
2. Ensure response dict contains at least one of:
   • message.tool_calls format (LMStudio/vLLM) — 4-specification.md:SP-017
   • content[{type=>"tool_calls"}] format (OpenAI) — 4-specification.md:SP-018
3. Response dict may optionally contain: api, provider, model, usage (llmUsage or dict), finish_reason, error_message (4-specification.md:SP-017, SP-018.4)

function my_openai_llmCall(messages::Dict{String, Any}; eventSink=nothing)
    # messages["messages"] contains OpenAI-format messages
    # messages["temperature"] (default 0.7)
    # messages["tools"] (if tools are set)
    
    body = Dict(
        "model" => "gpt-4",
        "messages" => messages["messages"],
        "tools" => get(messages, "tools", Any[])
    )
    
    resp = HTTP.post("https://api.openai.com/v1/chat/completions", 
                     headers, JSON.json(body))
    json_resp = JSON.parse(String(resp.body))
    
    # Extract first completion
    choice = json_resp["choices"][1]["message"]
    
    # Build uniform response
    return Dict(
        "api" => "openai",
        "provider" => "openai",
        "model" => "gpt-4",
        "message" => choice,
        "usage" => json_resp["usage"],
        "finish_reason" => json_resp["choices"][1]["finish_reason"]
    )
end

Reference: 1-requirements.md:FR-002, 4-specification.md:SP-017, SP-018

3.2: Add a Custom Tool

Steps:

1. Define tool schema in inputSchema (JSON Schema format):

   schema = Dict{String, Any}(
       "type" => "object",
       "properties" => Dict(
           "query" => Dict("type" => "string", "description" => "Search query"),
           "max_results" => Dict("type" => "integer", "default" => 5)
       ),
       "required" => ["query"]
   )
   

2. Implement execute function:

   tool = agentTool(
       name="searchWine",
       label="Search Wine Inventory",
       description="Search for wines matching a query",
       inputSchema=schema,
       execute=(toolCallId, args, signal, eventSink) -> begin
           query = args["query"]
           max_results = get(args, "max_results", Dict{Any,Any}()["max_results"] => 5...)  # handle default
   
           wines = run_search(query, max_results)
   
           result_text = join(["- $(w.name) (\$$(w.price))" for w in wines], "\n")
           return agentToolResult(
               [textContent("Found $(length(wines)) wines:\n$result_text")],
               Dict("count" => length(wines)),
               nothing, false  # content, details, usage, terminate
           )
       end,
       prepareArguments = nothing,
       validateRequiredArgs = nothing,  # uses default validator
       parallelToolExecute = false
   )
   
   # Register: push!(tools, tool.name => tool)
   

3. Optionally override validation:

   function my_validate(args)
       if haskey(args, "query") && length(args["query"]) < 3
           return "Query must be at least 3 characters"
       end
       return nothing
   end
   
   agentTool(
       ...,
       validateRequiredArgs = my_validate,  # replaces default validator
   )
   

Reference: 1-requirements.md:FR-004, 4-specification.md:SP-009, SP-010

3.3: Implement User Approval Gate (beforeToolCall Hook)

Steps:

1. Implement beforeToolCall that blocks expensive/destructive operations:

   function my_approval_toolCall(ctx::beforeToolCallContext, signal::abortSignal, eventSink)
       if ctx.toolCall.name in ("delete_wine", "update_price_bulk")
           eventSink("Approval required for $(ctx.toolCall.name) with args $(ctx.args)")
           # In production, send to UI and wait for user input
           approved = ask_user_for_approval(ctx.toolCall.name, ctx.args)  # blocking call
           if !approved
               return beforeToolCallResult(true, "User denied approval for $(ctx.toolCall.name)")
           end
       end
       return nothing  # allow execution
   end
   

Reference: 2-solution_design.md:SD-005 (lifecycle hooks), 3-walkthrough.md:WT-032 (beforeToolCall blocks)

3.4: Implement Result Sanitization (afterToolCall Hook)

Steps:

1. Implement afterToolCall to mask sensitive data:

   function my_sanitize_toolCall(ctx::afterToolCallContext, signal::abortSignal, eventSink)
       if ctx.toolCall.name == "getCustomerInfo"
           # Mask sensitive fields before LLM sees result
           result = ctx.result
           for (i, block) in enumerate(result.content)
               if block isa textContent
                   masked = replace(block.text, r"\d{12}" => "***-***-****")  # mask SSN
                   result = agentToolResult(
                       [replace(block.text, r"\d{12}" => "***-***-****") for block in result.content],
                       result.details,
                       result.usage,
                       result.terminate
                   )
               end
           end
           return result
       end
       return nothing  # pass through for other tools
   end
   

Reference: 2-solution_design.md:SD-005 (lifecycle hooks)

---

4. Common Error Codes

Error Condition	Response Code	Description
Tool not found	immediateOutcome("Tool <name> not found", true)	context.tools[toolCall.name] returns nothing
Missing required args	immediateOutcome("Missing required arguments: <fields>", true)	validateRequiredArgs returns error string
Custom validation failed	immediateOutcome("<tool.validateRequiredArgs result>", true)	Tool-specific validation hook returns error
beforeToolCall blocked	immediateOutcome("Tool execution was blocked: <reason>", true)	beforeToolCall returns beforeToolCallResult(true, reason)
Operation aborted	immediateOutcome("Operation aborted", true)	signal.aborted == true after beforeToolCall
Tool execution crashed	immediateOutcome("<exception message>", true)	preptool.execute() throws exception
MCP error	agentToolResult(["MCP error: <err_msg>"], {..., "error"=>err_msg}, ...)	Server returns response["error"] JSON-RPC envelope
MCP network error	agentToolResult(["MCP call error: <err_msg>"], {..., "error"=>err_msg}, ...)	Exception during mcp("tools/call", ...)

Reference: 4-specification.md:SP-009, SP-010, src/agentCore.jl:989, 1013, 1019, 1033, 1097

---

5. Debugging Checklist

Per ASG Framework §Pre-Code Debugging Habit:

Stage	Debug Question
Implementation	If a tool is not found, does prepareToolCall return immediateOutcome with "Tool not found"?
Testing	If an LLM response has tool_calls, does _extractToolCalls return (true, toolCallList, assistantMessageToolCall)?
Deploying	If the agent loop exits, does eventSink contain the error message?

---

6. References

• API contract: 4-specification.md — message types, MCP JSON-RPC format, response extraction formats
• Solution decisions: 2-solution_design.md — component architecture, lifecycle hooks, format-agnostic LLM responses
• Walkthroughs: 3-walkthrough.md — end-to-end agent flow, error paths, hook sequences
• Validation tests: 6-validation.md — test traceability matrix mapping requirements/specs to test files