add tracing

This commit is contained in:
2026-07-30 16:23:08 +07:00
parent 244cfc4b96
commit 8d5c661562
+552
View File
@@ -0,0 +1,552 @@
# Agent Loop Tracing
This document traces the agent loop through two example interactions.
## Architecture Overview
```
Agent (src/agent.jl)
|
v
AgentLoop (src/agent_loop.jl) -- runLoop() is the core while(true) loop
|
v
StreamFn (src/stream_fn.jl) -- LLM streaming function (user-provided)
|
v
Tools (src/tools/*.jl) -- bash, read, write, edit
```
Key types:
- `Agent` (agent.jl:85) -- high-level wrapper with state, queues, listeners
- `agentLoop()` (agent_loop.jl:23) -- entry point, spawns thread, returns `EventStream`
- `runLoop()` (agent_loop.jl:169) -- the core `while(true)` loop
- `streamAssistantResponse()` (agent_loop.jl:361) -- calls LLM, streams events, returns `AssistantMessage`
- `executeToolCalls()` (agent_loop.jl:476) -- runs tool calls (sequential or parallel)
- `AgentContext` (types.jl:186) -- system_prompt + messages + tools
- `AgentLoopConfig` -- model, thinking_level, callbacks for steering/follow-up/tool execution
---
## Scenario 1: User asks "what is the content of text.txt file", agent responds
### Step 1: User invokes `prompt(agent, "what is the content of text.txt file")`
**File: agent.jl:284-292**
```julia
prompt(agent, "what is the content of text.txt file")
-> normalizePromptInput(agent, "what is the content of text.txt file", [])
-> [UserMessage("user", [TextContent("what is the content of text.txt file")], timestamp)]
-> runPromptMessages(agent, messages)
```
The string is normalized into a single `UserMessage`.
### Step 2: `runPromptMessages` calls `agentLoop()`
**File: agent.jl:310-313** (TODO stub, but conceptually):
```julia
runPromptMessages(agent, messages)
-> AgentLoop.agentLoop(
prompts = [UserMessage(...)],
context = createContextSnapshot(agent), # AgentContext with system_prompt, messages, tools
config = createLoopConfig(agent),
signal = nothing,
stream_fn = agent.stream_function,
)
```
### Step 3: `agentLoop()` spawns thread and calls `runAgentLoop()`
**File: agent_loop.jl:23-45**
```julia
agentLoop(prompts, context, config, signal, stream_fn)
-> createAgentStream() # creates EventStream
-> Threads.@spawn begin
runAgentLoop(prompts, context, config, emit, signal, stream_fn)
end(stream, messages)
end
-> return stream
```
### Step 4: `runAgentLoop()` initializes and enters `runLoop()`
**File: agent_loop.jl:85-116**
```julia
runAgentLoop(prompts, context, config, emit, signal, stream_fn)
-> new_messages = copy(prompts) # [UserMessage(...)]
-> current_context = AgentContext(context.system_prompt, [UserMessage(...)], context.tools)
-> emit(AgentStartEvent())
-> emit(TurnStartEvent())
-> for prompt in prompts: emit(MessageStartEvent(prompt)); emit(MessageEndEvent(prompt)) end
-> runLoop(current_context, new_messages, config, signal, emit, stream_fn)
```
Events emitted so far:
1. `AgentStartEvent`
2. `TurnStartEvent`
3. `MessageStartEvent(UserMessage)`
4. `MessageEndEvent(UserMessage)`
### Step 5: `runLoop()` -- first iteration
**File: agent_loop.jl:169-310**
```julia
runLoop(initial_context, new_messages, initial_config, signal, emit, stream_function)
-> current_context = AgentContext(system_prompt, [UserMessage(...)], tools)
-> first_turn = true
-> pending_messages = []
-> while true:
has_more_tool_calls = true # reset each outer iteration
# Inner loop: has_more_tool_calls || !isempty(pending_messages)
while has_more_tool_calls || !isempty(pending_messages)
first_turn = false # TurnStartEvent NOT emitted (already done)
# no pending_messages
# === STEP 5a: Call LLM ===
message = streamAssistantResponse(current_context, config, signal, emit, stream_function)
```
### Step 5a: `streamAssistantResponse()` -- LLM call
**File: agent_loop.jl:361-435**
```julia
streamAssistantResponse(context, config, signal, emit, stream_function)
-> messages = context.messages # [UserMessage(...)]
-> llm_messages = config.convert_to_llm(messages) # filter to user/assistant/toolResult roles
-> llm_context = Context(context.system_prompt, llm_messages, context.tools)
-> response = stream_function(config.model, llm_context, merged_config)
```
The `stream_function` (user-provided via StreamFn) calls the LLM API. It yields events:
```
StartEvent(partial=AssistantMessage(role="assistant", content=[]))
-> push!(context.messages, partial_message)
-> emit(MessageStartEvent(partial_message))
TextDeltaEvent(partial=AssistantMessage with ToolCall for "read")
-> context.messages[end] = partial_message
-> emit(MessageUpdateEvent(partial_message, event))
TextDeltaEvent(...) -- streaming continues
ToolCallEvent -- tool call detected: read(file="text.txt")
DoneEvent(reason="tool_calls", ...)
-> final_message = AssistantMessage(role="assistant", content=[ToolCall(...)])
-> context.messages[end] = final_message
-> emit(MessageEndEvent(final_message))
-> return final_message
```
Back in `runLoop`:
- `message` = `AssistantMessage` with `stop_reason = "tool_calls"`
- `push!(new_messages, message)`
### Step 5b: Tool call detection
**File: agent_loop.jl:219-244**
```julia
tool_calls = filter(c -> c isa ToolCall, message.content)
# tool_calls = [ToolCall(type="tool_call", id="call_1", name="read", arguments={file="text.txt"}, ...)]
tool_results = []
has_more_tool_calls = true
if !isempty(tool_calls)
executed_tool_batch = executeToolCalls(
current_context, message, config, signal, emit,
)
append!(tool_results, executed_tool_batch.messages)
has_more_tool_calls = !executed_tool_batch.terminate
```
### Step 5c: `executeToolCalls()` -- sequential or parallel
**File: agent_loop.jl:476-514**
Since there's only one tool call and no sequential mode forced, it uses `executeToolCallsParallel()` (or sequential -- both paths converge for a single tool call).
```julia
executeToolCalls(context, assistant_message, tool_calls, config, signal, emit)
-> tool = findfirst(t -> t.name == "read", context.tools)
-> preparation = prepareToolCall(...)
-> validated_args = {file="text.txt"}
-> return PreparedToolCall("prepared", tool_call, tool, validated_args)
executed = executePreparedToolCall(preparation, signal, emit)
-> result = prepared.tool.execute("call_1", {file="text.txt"}, signal, on_update, context)
# This invokes the read tool's execute function (src/tools/read.jl:26)
# TODO: in the current code, it returns a placeholder
-> return ExecutedToolCallOutcome(result, false)
finalized = finalizeExecutedToolCall(...)
# Creates FinalizedToolCallOutcome
emitToolExecutionEnd(finalized, emit)
# emits ToolExecutionEndEvent
tool_result_message = createToolResultMessage(finalized)
# creates ToolResultMessage(role="toolResult", tool_call_id="call_1", tool_name="read", content=[TextContent(...)])
emitToolResultMessage(tool_result_message, emit)
# emits MessageStartEvent(tool_result_message), MessageEndEvent(tool_result_message)
```
Events emitted during tool execution:
5. `MessageStartEvent(assistant_message)` (from LLM)
6. `MessageEndEvent(assistant_message)` (from LLM done)
7. `ToolExecutionStartEvent`
8. `ToolExecutionEndEvent`
9. `MessageStartEvent(tool_result_message)`
10. `MessageEndEvent(tool_result_message)`
### Step 5d: Back in inner loop
**File: agent_loop.jl:240-294**
```julia
push!(current_context.messages, tool_result_message)
push!(new_messages, tool_result_message)
emit(TurnEndEvent(message, tool_results))
next_turn_snapshot = prepare_next_turn(config, PrepareNextTurnContext(...))
# Returns nothing by default (no custom prepare_next_turn)
if !isnothing(next_turn_snapshot) ... end # skipped
if should_stop_after_turn(config, ...) ... end # returns false by default
pending_messages = get_steering_messages(config) # returns []
# inner while continues: has_more_tool_calls = true, pending_messages = []
# === SECOND LLM CALL ===
message = streamAssistantResponse(current_context, config, signal, emit, stream_function)
# context.messages now = [UserMessage(...), AssistantMessage(read tool call), ToolResultMessage(file contents)]
```
### Step 5e: Second LLM call -- agent responds
**File: agent_loop.jl:361-435**
```julia
streamAssistantResponse(context, config, signal, emit, stream_function)
-> llm_messages = [UserMessage(...), AssistantMessage(...), ToolResultMessage(...)]
-> response = stream_function(model, Context(system_prompt, llm_messages, tools), config)
```
The LLM receives the user's question + its own tool call + the file contents as a tool result. It generates a text response.
Events:
```
StartEvent -> MessageStartEvent
TextDeltaEvent -> MessageUpdateEvent (text streaming)
...
DoneEvent(reason="end_turn") -> MessageEndEvent
```
### Step 5f: No more tool calls -- loop exits
**File: agent_loop.jl:219-244**
```julia
tool_calls = filter(c -> c isa ToolCall, message.content)
# tool_calls = [] (no tool calls in the final response)
has_more_tool_calls = false # stays false
emit(TurnEndEvent(message, ToolResultMessage[]))
next_turn_snapshot = prepare_next_turn(...) # nothing
should_stop_after_turn(...) # false
pending_messages = get_steering_messages(...) # []
# inner while: has_more_tool_calls=false, pending_messages=[] -> exits inner loop
follow_up_messages = get_follow_up_messages(...) # []
# exits outer while
emit(AgentEndEvent(new_messages))
```
Events emitted at end:
11. `MessageStartEvent(assistant_response)`
12. `MessageUpdateEvent(...)` (text deltas)
13. `MessageEndEvent(assistant_response)`
14. `TurnEndEvent(response, [])`
15. `AgentEndEvent([UserMessage, AssistantMessage, ToolResultMessage, AssistantResponse])`
### Summary of Scenario 1 event sequence:
| # | Event | Source |
|---|-------|--------|
| 1 | `AgentStartEvent` | runLoop() |
| 2 | `TurnStartEvent` | runLoop() |
| 3 | `MessageStartEvent(UserMessage)` | runAgentLoop() |
| 4 | `MessageEndEvent(UserMessage)` | runAgentLoop() |
| 5 | `MessageStartEvent(AssistantMessage)` | streamAssistantResponse() |
| 6 | `MessageUpdateEvent(AssistantMessage)` | streamAssistantResponse() (streaming) |
| 7 | `MessageEndEvent(AssistantMessage)` | streamAssistantResponse() |
| 8 | `ToolExecutionStartEvent` | executeToolCalls() |
| 9 | `ToolExecutionEndEvent` | executeToolCalls() |
| 10 | `MessageStartEvent(ToolResultMessage)` | emitToolResultMessage() |
| 11 | `MessageEndEvent(ToolResultMessage)` | emitToolResultMessage() |
| 12 | `MessageStartEvent(AssistantMessage)` | streamAssistantResponse() (2nd call) |
| 13 | `MessageUpdateEvent(AssistantMessage)` | streamAssistantResponse() (text streaming) |
| 14 | `MessageEndEvent(AssistantMessage)` | streamAssistantResponse() |
| 15 | `TurnEndEvent(AssistantResponse, [])` | runLoop() |
| 16 | `AgentEndEvent([all messages])` | runLoop() |
---
## Scenario 2: User asks "copy text.txt to text.md", agent responds
### Step 1-4: Same as Scenario 1
User invokes `prompt(agent, "copy text.txt to text.md")`, which flows through `agentLoop()` -> `runAgentLoop()` -> `runLoop()`.
Events 1-4 are identical (AgentStart, TurnStart, UserMessage start/end).
### Step 5: First LLM call -- agent decides to use tools
The LLM receives:
```
System: <system_prompt>
User: "copy text.txt to text.md"
```
The LLM decides it needs to:
1. Read text.txt (to get its contents), then
2. Write those contents to text.md
The LLM may emit a single `AssistantMessage` with **two** `ToolCall` objects:
```
AssistantMessage(content=[
ToolCall(id="call_1", name="read", arguments={file="text.txt"}),
ToolCall(id="call_2", name="write", arguments={file="text.md", content="...contents of text.txt..."}),
])
```
Or it may emit one tool call at a time (sequential), which is also supported.
### Step 5b: Tool execution
**File: agent_loop.jl:219-244**
```julia
tool_calls = filter(c -> c isa ToolCall, message.content)
# tool_calls = [ToolCall(read), ToolCall(write)]
executed_tool_batch = executeToolCalls(context, message, config, signal, emit)
```
If `tool_execution == EXECUTION_PARALLEL` (default) and no tool forces sequential mode:
**File: agent_loop.jl:568-633 (executeToolCallsParallel)**
```julia
executeToolCallsParallel(...)
-> for tool_call in tool_calls:
# call_1: read
emit(ToolExecutionStartEvent("call_1", "read", {file="text.txt"}))
preparation = prepareToolCall(...) # validated
push!(finalized_calls, () -> executed_read()) # closure for deferred execution
# call_2: write
emit(ToolExecutionStartEvent("call_2", "write", {file="text.md", content="..."}))
preparation = prepareToolCall(...)
push!(finalized_calls, () -> executed_write()) # closure
# Execute in order
ordered_finalized_calls = map(entry -> entry(), finalized_calls)
for finalized in ordered_finalized_calls:
tool_result_message = createToolResultMessage(finalized)
emitToolResultMessage(tool_result_message, emit)
```
Events for parallel execution:
```
ToolExecutionStartEvent(call_1, "read", ...)
ToolExecutionStartEvent(call_2, "write", ...)
MessageStartEvent(ToolResultMessage[read result])
MessageEndEvent(ToolResultMessage[read result])
MessageStartEvent(ToolResultMessage[write result])
MessageEndEvent(ToolResultMessage[write result])
```
If `tool_execution == EXECUTION_SEQUENTIAL` or any tool is marked sequential:
**File: agent_loop.jl:520-562 (executeToolCallsSequential)**
```julia
for tool_call in tool_calls:
emit(ToolExecutionStartEvent(...))
# execute, finalize, emit result
# THEN proceed to next
```
Events for sequential execution:
```
ToolExecutionStartEvent(call_1, "read", ...)
MessageStartEvent(ToolResultMessage[read result])
MessageEndEvent(ToolResultMessage[read result])
ToolExecutionStartEvent(call_2, "write", ...)
MessageStartEvent(ToolResultMessage[write result])
MessageEndEvent(ToolResultMessage[write result])
```
### Step 5d: Second LLM call
```julia
has_more_tool_calls = !executed_tool_batch.terminate # false (unless terminate=true)
# inner loop continues since pending_messages is still empty
# Actually: has_more_tool_calls = false, pending_messages = []
# -> exits inner loop
# follow_up_messages = []
# -> exits outer loop
emit(TurnEndEvent(message, tool_results))
```
Wait -- this depends on whether the LLM's first response included only tool calls (no text answer). If the LLM only returned tool calls and the tool results were processed, the agent may need a **third** LLM call to generate the final user-facing response.
**Revised flow for two tool calls:**
After tool results are added to context:
```
context.messages = [
UserMessage("copy text.txt to text.md"),
AssistantMessage([ToolCall(read), ToolCall(write)]),
ToolResultMessage(read result),
ToolResultMessage(write result),
]
```
The agent needs another LLM call to generate a response. Let's trace it:
### Step 5e: Second LLM call -- final response
```julia
message = streamAssistantResponse(current_context, ...)
```
LLM receives:
```
System: <system_prompt>
User: "copy text.txt to text.md"
Assistant: [ToolCall(read), ToolCall(write)]
ToolResult: (contents of text.txt)
ToolResult: (write confirmation)
```
LLM generates: "I've copied text.txt to text.md."
Events:
```
MessageStartEvent(AssistantMessage)
MessageUpdateEvent(... text deltas ...)
MessageEndEvent(AssistantMessage)
```
### Step 5f: No tool calls, loop exits
```julia
tool_calls = [] # no ToolCalls in response
has_more_tool_calls = false
emit(TurnEndEvent(message, []))
pending_messages = []
follow_up_messages = []
emit(AgentEndEvent(new_messages))
```
### Summary of Scenario 2 event sequence (parallel tool execution):
| # | Event | Source |
|---|-------|--------|
| 1 | `AgentStartEvent` | runLoop() |
| 2 | `TurnStartEvent` | runLoop() |
| 3 | `MessageStartEvent(UserMessage)` | runAgentLoop() |
| 4 | `MessageEndEvent(UserMessage)` | runAgentLoop() |
| 5 | `MessageStartEvent(AssistantMessage)` | streamAssistantResponse() (1st LLM call) |
| 6 | `MessageEndEvent(AssistantMessage)` | streamAssistantResponse() |
| 7 | `ToolExecutionStartEvent(call_1, "read")` | executeToolCallsParallel() |
| 8 | `ToolExecutionStartEvent(call_2, "write")` | executeToolCallsParallel() |
| 9 | `MessageStartEvent(ToolResultMessage[read])` | emitToolResultMessage() |
| 10 | `MessageEndEvent(ToolResultMessage[read])` | emitToolResultMessage() |
| 11 | `MessageStartEvent(ToolResultMessage[write])` | emitToolResultMessage() |
| 12 | `MessageEndEvent(ToolResultMessage[write])` | emitToolResultMessage() |
| 13 | `TurnEndEvent(AssistantToolCalls, [read_result, write_result])` | runLoop() |
| 14 | `MessageStartEvent(AssistantMessage)` | streamAssistantResponse() (2nd LLM call) |
| 15 | `MessageUpdateEvent(AssistantMessage)` | streamAssistantResponse() (text streaming) |
| 16 | `MessageEndEvent(AssistantMessage)` | streamAssistantResponse() |
| 17 | `TurnEndEvent(AssistantResponse, [])` | runLoop() |
| 18 | `AgentEndEvent([all messages])` | runLoop() |
---
## Key Design Patterns
### 1. Event Stream Architecture
Events flow through `emit::AgentEventSink` (a function) into an `EventStream`. Consumers subscribe to the stream and receive events as they occur. The stream terminates when `AgentEndEvent` is emitted.
### 2. Context Accumulation
`AgentContext.messages` grows across turns:
```
[UserMessage, AssistantMessage, ToolResultMessage, AssistantMessage, ToolResultMessage, ...]
```
### 3. LLM Conversion
Before each LLM call, `config.convert_to_llm()` filters the agent messages to only include user/assistant/toolResult roles (src/agent.jl:18-23):
```julia
filter(m -> m.role in ("user", "assistant", "toolResult"), messages)
```
### 4. Tool Execution Modes
- `EXECUTION_PARALLEL` (default): tool calls are prepared as closures and executed in sequence after all are prepared
- `EXECUTION_SEQUENTIAL`: each tool is prepared, executed, and finalized before the next begins
### 5. Turn Continuation
The inner `while has_more_tool_calls` loop handles:
- Multiple tool calls from a single assistant response
- Pending steering/follow-up messages injected between turns
The outer `while true` loop handles:
- Full turns (LLM call + tool execution)
- Switching between tool-result turns and response turns
### 6. Message Types
| Type | Role | Created By |
|------|------|------------|
| `UserMessage` | "user" | User via `prompt()` |
| `AssistantMessage` | "assistant" | LLM via `streamAssistantResponse()` |
| `ToolResultMessage` | "toolResult" | `createToolResultMessage()` after tool execution |
| `BashExecutionMessage` | "user" | Bash tool (excluded from context by default) |
| `CompactionSummaryMessage` | "user" | Compaction process |
| `BranchSummaryMessage` | "user" | Branch summarization |
### 7. Tool Call Lifecycle
```
ToolCall (from LLM)
-> prepareToolCall() (validate args, before_tool_call hook)
-> executePreparedToolCall() (invoke tool.execute)
-> finalizeExecutedToolCall() (after_tool_call hook)
-> createToolResultMessage() (wrap result in ToolResultMessage)
-> emitToolResultMessage() (emit MessageStart/MessageEnd)
```