diff --git a/docs/agent_loop_diagram.md b/docs/agent_loop_diagram.md index 396cc6f..0b43789 100644 --- a/docs/agent_loop_diagram.md +++ b/docs/agent_loop_diagram.md @@ -5,12 +5,11 @@ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ 1. INITIALIZATION │ -├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ │ │ Agent.prompt(user_input) │ │ │ │ │ ▼ │ -│ normalizePrompt() ← Convert input (String/Message/Vector) to AgentMessage[] │ +│ normalizePrompt() ← Convert input to AgentMessage[] │ │ │ │ │ ▼ │ │ runPromptMessages() │ @@ -21,10 +20,12 @@ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ 2. AGENT LOOP START (runAgentLoop) │ -├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ │ -│ new_messages = copy(prompts) ← User messages copied to new_messages │ -│ current_context.messages = vcat(context.messages, copy(prompts)) ← User messages added to context │ +│ new_messages = copy(prompts) │ +│ current_context.messages = vcat(context.messages, copy(prompts)) │ +│ │ │ +│ └─→ User messages are IMMEDIATELY added to context.messages │ +│ (They are NOT in the steering queue!) │ │ │ │ emit(AgentStartEvent) │ │ emit(TurnStartEvent) │ @@ -32,33 +33,35 @@ │ for prompt in prompts: │ │ emit(MessageStartEvent(prompt)) │ │ emit(MessageEndEvent(prompt)) │ -│ │ │ -│ ├─→ push to current_context.messages (for LLM) │ -│ └─→ push to new_messages (track what we've added) │ │ │ └─────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ 3. MAIN LOOP (runLoop - while true) │ -├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ │ -│ pending_messages = get_steering_messages() ← Check steering queue (empty on first turn) │ +│ pending_messages = get_steering_messages() │ +│ │ │ +│ └─→ Steering queue: messages from agent.steer() │ +│ These are for CONTINUING conversation (NOT new user prompts) │ │ │ │ ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ │ │ While has pending_messages OR has_tool_calls: │ │ │ │ │ │ │ │ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ │ -│ │ │ 4. PENDING MESSAGE HANDLING │ │ │ -│ │ │ (Handles steering messages queued via agent.steer() AFTER previous turn) │ │ │ +│ │ │ 4. PENDING MESSAGE HANDLING (steering messages only) │ │ │ │ │ │ │ │ │ +│ │ │ pending_messages = get_steering() │ │ │ │ │ │ if !isempty(pending_messages): │ │ │ │ │ │ for msg in pending_messages: │ │ │ │ │ │ emit(MessageStartEvent(msg)) │ │ │ │ │ │ emit(MessageEndEvent(msg)) │ │ │ -│ │ │ push to current_context.messages │ │ │ -│ │ │ push to new_messages │ │ │ +│ │ │ push to current_context.messages ← Steering messages go HERE │ │ │ +│ │ │ push to new_messages │ │ │ │ │ │ pending_messages = [] │ │ │ +│ │ │ │ │ │ +│ │ │ Note: User messages from Agent.prompt() are ALREADY in context.messages │ │ │ +│ │ │ (They were added in runAgentLoop via vcat(), not via this queue) │ │ │ │ │ └─────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ │ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ │ @@ -163,12 +166,14 @@ │ │ │ 2. runAgentLoop() │ │ new_messages = [UserMessage("What is Julia?")] │ -│ current_context.messages = [...existing..., UserMessage("What is Julia?")] │ +│ current_context.messages = vcat([...existing...], [UserMessage("What is Julia?")]) │ +│ │ │ +│ └─→ User message IMMEDIATELY added to context.messages (NOT via steering queue!) │ │ emit(AgentStartEvent), emit(TurnStartEvent) │ │ emit(MessageStart/End) for user message │ │ │ │ 3. runLoop() │ -│ pending_messages = get_steering() = [] ← Steering queue is empty │ +│ pending_messages = get_steering() = [] ← Steering queue is empty (no agent.steer() yet) │ │ │ │ 4. streamAssistantResponse() │ │ convert_to_llm([UserMessage]) → Message[] │ @@ -202,6 +207,11 @@ │ │ follow_up_queue: [] │ │ │ └───────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ +│ LLM SEES (convert_to_llm() filters): │ │ +│ ┌─────────────────────────────────────────────────────────────────────────────────────────────────┐ │ +│ │ Messages passed to LLM API: │ │ +│ │ [UserMessage("What is Julia?"), AssistantMessage("Julia is...")] │ │ +│ └─────────────────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ │ TURN #2: User asks "How does it work?" │ │ ───────────────────────────────────────── │ @@ -211,7 +221,9 @@ │ │ │ 2. runAgentLoop() │ │ new_messages = [UserMessage("How does it work?")] │ -│ current_context.messages = [...previous..., UserMessage("How does it work?")] │ +│ current_context.messages = vcat([...previous..., UserMessage("How does it work?")]) │ +│ │ │ +│ └─→ User message added (context preserved from Turn #1) │ │ emit(AgentStartEvent), emit(TurnStartEvent) │ │ emit(MessageStart/End) for user message │ │ │ @@ -232,25 +244,122 @@ │ │ [UserMsg1, AssistantMsg1, UserMsg2, AssistantMsg2] │ │ │ └───────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ │ │ +│ LLM SEES: │ +│ ┌─────────────────────────────────────────────────────────────────────────────────────────────────┐ │ +│ │ Messages passed to LLM API: │ │ +│ │ [UserMessage("What is Julia?"), │ │ +│ │ AssistantMessage("Julia is..."), │ │ +│ │ UserMessage("How does it work?"), │ │ +│ │ AssistantMessage("It works by...")] │ │ +│ └─────────────────────────────────────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ +│ STEERING MESSAGES │ +├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ What is a steering message? │ +│ • A message (any AgentMessage type) injected via: `agent.steer(message)` │ +│ • Goes into the steering queue, not immediately to context.messages │ +│ │ +│ How is it created? │ +│ • User code calls: agent.steer(UserMessage("...")) │ +│ • Or: agent.steer(AssistantMessage("...")) │ +│ • Or any other AgentMessage subtype │ +│ │ +│ When is it processed? │ +│ • At the START of the next loop iteration (line 194-202 in agent_loop.jl) │ +│ • AFTER the previous assistant turn completes │ +│ • BEFORE the next assistant response is streamed │ +│ │ +│ Why use steering? │ +│ Use case 1: Tool execution result injection │ +│ - Agent calls a tool (e.g., read_file, bash) │ +│ - Tool returns result │ +│ - You want to inject a follow-up question based on the result │ +│ - agent.steer(UserMessage("Based on the file, what should we do next?")) │ +│ │ +│ Use case 2: Multi-turn conversation without user input │ +│ - Agent responds to user │ +│ - Before user types again, you want to inject a system message │ +│ - agent.steer(BashExecutionMessage(...)) or custom message │ +│ - This continues the conversation automatically │ +│ │ +│ Use case 3: Branch navigation recovery │ +│ - User navigates between conversation branches │ +│ - After switching branches, you want to inject a context message │ +│ - agent.steer(BranchSummaryMessage(...)) │ +│ - The agent can then continue from the new branch context │ +│ │ +│ Use case 4: Compaction summary injection │ +│ - Conversation history is compacted │ +│ - After compaction, inject summary message │ +│ - agent.steer(CompactionSummaryMessage(...)) │ +│ - Agent knows old history was summarized │ +│ │ +│ Example: │ +│ agent.steer(UserMessage("Follow-up question here")) │ +│ # This will be processed in the next loop iteration, │ +│ # appearing in context.messages before the next LLM call │ +│ │ +│ The LLM sees: │ +│ ┌─────────────────────────────────────────────────────────────────────────────────────────────────┐ │ +│ │ All messages become Message[] via convert_to_llm(): │ │ +│ │ [UserMessage(...), AssistantMessage(...), UserMessage(from_steer), ...] │ │ +│ │ │ │ +│ │ The LLM cannot tell which came from Agent.prompt() vs agent.steer() │ │ +│ └─────────────────────────────────────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ +│ LLM PROCESSING: How LLM sees messages │ +├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ The LLM NEVER sees "user message" vs "steering message" - it only sees Message types: │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────────────────────────────────┐ │ +│ │ convert_to_llm() transforms ALL AgentMessages to Message[]: │ │ +│ │ │ │ +│ │ UserMessage("user") → UserMessage (for LLM) │ │ +│ │ Steering UserMessage("user") → UserMessage (for LLM) ← Same! │ │ +│ │ AssistantMessage("assistant") → AssistantMessage (for LLM) │ │ +│ │ ToolResultMessage("toolResult") → ToolResultMessage (for LLM) │ │ +│ │ │ │ +│ │ BranchSummaryMessage → UserMessage (wrapped in summary tags) │ │ +│ │ CompactionSummaryMessage → UserMessage (wrapped in summary tags) │ │ +│ │ BashExecutionMessage → UserMessage (if not excluded) │ │ +│ │ CustomMessage → UserMessage │ │ +│ └─────────────────────────────────────────────────────────────────────────────────────────────────┘ │ +│ │ +│ The difference is ONLY in HOW messages enter the system: │ +│ • User messages: Agent.prompt() → vcat() → context.messages (direct) │ +│ • Steering: agent.steer() → queue → loop → context.messages (indirect) │ +│ │ +│ At LLM level: BOTH become UserMessage in the conversation! │ +│ │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ KEY INSIGHTS │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ │ -│ 1. User prompts are NOT added to steering queue │ -│ They go directly into context.messages via vcat() in runAgentLoop() │ +│ 1. User prompts go DIRECTLY to context.messages via vcat() in runAgentLoop() │ │ │ -│ 2. Steering queue is for messages injected AFTER a turn │ -│ Via agent.steer(message) - used for continuation without new prompt │ +│ 2. Steering queue is for messages injected via agent.steer() AFTER a turn finishes │ +│ This allows continuing conversation without calling Agent.prompt() again │ │ │ -│ 3. Context is preserved across turns │ -│ Each turn appends to context.messages, so LLM sees full history │ +│ 3. Context is preserved across turns - context.messages grows with each turn │ +│ LLM sees the full conversation history │ │ │ -│ 4. New turn = New prompt OR steering/follow-up messages │ -│ - New Agent.prompt() call starts new turn with new messages │ -│ - Steering messages continue from current state │ -│ - Follow-up messages run when agent would stop │ +│ 4. At LLM level, ALL messages become Message types (UserMessage/AssistantMessage/ToolResultMessage) │ +│ The "steering" vs "user" distinction is just a control mechanism, not a message type │ +│ │ +│ 5. New turn is triggered by: │ +│ - New Agent.prompt() call (adds user messages) │ +│ - Steering messages (adds steering messages) │ +│ - Follow-up messages (adds follow-up messages) │ │ │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ -``` \ No newline at end of file +``` diff --git a/etc.jl b/etc.jl index a35aae6..5d75a01 100644 --- a/etc.jl +++ b/etc.jl @@ -1,10 +1,110 @@ -# check if this column has vector embedding. if there is one, seach vector version instead - column_name_embedding = column_name * "_embedding" - if occursin(column_name_embedding, tables_schema[column_name_embedding]) - vector_column = Dict( - "table_name"=> table_name, - "column_name"=> column_name_embedding, - "operator"=> "vector_similarity", - "value"=> column_obj["value"] - ) - end \ No newline at end of file + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +read codebase. +I need to understand this agent concept deeply. +Can you write related documents (.md files) that will help me understand the agent +and save in "/home/ton/docker-apps/sommpanion/YiemAgent/learning" folder? +I'm learning best in **Top-Down** style so I know how each component are synchonized. + +P.S. use diagram to show how process flow and relationship + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/learning/01-ARCHITECTURE_OVERVIEW.md b/learning/01-ARCHITECTURE_OVERVIEW.md new file mode 100644 index 0000000..e8fb755 --- /dev/null +++ b/learning/01-ARCHITECTURE_OVERVIEW.md @@ -0,0 +1,498 @@ +# AgentCore.jl - Architecture Overview + +## Top-Down Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ AgentCore.jl Layers │ +└─────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────┐ +│ Level 1: AgentHarness (Session Management & Persistence) │ +│ - Session persistence with JSONL storage │ +│ - Resource management (skills, prompt templates) │ +│ - Extension hooks system │ +│ - Branch navigation and compaction │ +└─────────────────────────────────────────────────────────────────────────┘ + │ + │ orchestrates + ▼ +┌─────────────────────────────────────────────────────────────────────────┐ +│ Level 2: Agent (State Management & Event Streaming) │ +│ - Conversation state (messages, tools, system prompt) │ +│ - Event streaming and lifecycle management │ +│ - Steering and follow-up message queues │ +│ - Abort handling │ +└─────────────────────────────────────────────────────────────────────────┘ + │ + │ delegates to + ▼ +┌─────────────────────────────────────────────────────────────────────────┐ +│ Level 3: AgentLoop (Core LLM Interaction Loop) │ +│ - Stateful LLM interactions │ +│ - Tool execution (parallel or sequential) │ +│ - Event emission lifecycle │ +│ - Steering/follow-up message handling │ +└─────────────────────────────────────────────────────────────────────────┘ + │ + │ transforms to + ▼ +┌─────────────────────────────────────────────────────────────────────────┐ +│ Level 4: Session (Conversation History Management) │ +│ - Tree-based conversation history │ +│ - Branch support with compaction │ +│ - Message and metadata persistence │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +## Process Flow + +### 1. Agent Lifecycle + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ Agent Lifecycle │ +└─────────────────────────────────────────────────────────────────────────┘ + + User Code + │ + │ 1. Create Agent + ▼ + ┌──────────────┐ + │ Agent() │ ──► Initialize state, queues, listeners + └──────────────┘ + │ + │ 2. Subscribe to events + ▼ + ┌──────────────────┐ + │ subscribe() │ ──► Register event handlers + └──────────────────┘ + │ + │ 3. Run prompt + ▼ + ┌──────────────────┐ + │ prompt() │ ──► Validate input, normalize messages + └──────────────────┘ + │ + │ 4. Start AgentLoop + ▼ + ┌──────────────────┐ + │ runPromptMessages│ ──► Create ActiveRun, spawn loop + └──────────────────┘ + │ + ▼ + ┌──────────────────────────────────────────────────────────────┐ + │ AgentLoop (runs in separate thread) │ + │ │ + │ ┌────────────────────────────────────────────────────────┐ │ + │ │ 1. Emit AgentStartEvent │ │ + │ │ 2. Emit TurnStartEvent │ │ + │ │ 3. Process prompts (emit MessageStart/End) │ │ + │ │ 4. ┌──────────────────────────────────────────────┐ │ │ + │ │ │ while true: │ │ │ + │ │ │ │ Process steering/follow-up messages │ │ │ + │ │ │ │ Stream assistant response (LLM call) │ │ │ + │ │ │ │ Execute tool calls (parallel/sequential) │ │ │ + │ │ │ │ Emit TurnEndEvent │ │ │ + │ │ │ │ Check if should stop │ │ │ + │ │ │ │ Get next steering messages │ │ │ + │ │ └───┴────────────────────────────────────────────┘ │ │ + │ └────────────────────────────────────────────────────────┘ │ + └──────────────────────────────────────────────────────────────┘ + │ + │ 5. Event streaming + ▼ + ┌──────────────────┐ + │ Event Handlers │ ──► User-defined listeners receive events + └──────────────────┘ + │ + │ 6. Wait for completion + ▼ + ┌──────────────────┐ + │ waitForIdle() │ ──► Resolve when all events processed + └──────────────────┘ +``` + +### 2. AgentLoop Flow Diagram + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ AgentLoop Process Flow │ +└─────────────────────────────────────────────────────────────────────────┘ + + ┌────────────────────────────────────────────────────────────────────┐ + │ AgentLoop Entrypoint │ + │ ┌────────────────────────────────────────────────────────────┐ │ + │ │ agentLoop(prompts, context, config, signal, stream_fn) │ │ + │ └────────────────────────────────────────────────────────────┘ │ + │ │ │ + │ ▼ │ + │ ┌────────────────────────────────────────────────────────────┐ │ + │ │ runAgentLoop(prompts, context, config, emit, signal) │ │ + │ └────────────────────────────────────────────────────────────┘ │ + │ │ │ + │ ▼ │ + │ ┌────────────────────────────────────────────────────────────┐ │ + │ │ runLoop() - Main Event Loop │ │ + │ └────────────────────────────────────────────────────────────┘ │ + │ │ │ + └──────────────────────────────┼───────────────────────────────────────┘ + │ + │ Loop Iteration + ▼ + ┌────────────────────────────────────────────────────────────────────┐ + │ Main Processing Loop │ + │ │ + │ ┌────────────────────────────────────────────────────────────┐ │ + │ │ 1. Get Steering/Follow-up Messages │ │ + │ │ ┌────────────────────┐ ┌──────────────────────┐ │ │ + │ │ │ steering_queue │ │ follow_up_queue │ │ │ + │ │ │ (after assistant) │ │ (after stop) │ │ │ + │ │ └────────────────────┘ └──────────────────────┘ │ │ + │ └────────────────────────────────────────────────────────────┘ │ + │ │ │ + │ ▼ │ + │ ┌────────────────────────────────────────────────────────────┐ │ + │ │ 2. Stream Assistant Response │ │ + │ │ ┌────────────────────────────────────────────────────┐ │ │ + │ │ │ transform_context() │ │ │ + │ │ │ convert_to_llm(messages) -> Message[] │ │ │ + │ │ │ stream_fn(model, context, config) -> Response │ │ │ + │ │ │ - Text deltas │ │ │ + │ │ │ - Tool call deltas │ │ │ + │ │ └────────────────────────────────────────────────────┘ │ │ + │ │ │ │ │ + │ │ ▼ │ │ + │ │ ┌────────────────────────────────────────────────────┐ │ │ + │ │ │ Emit: MessageStartEvent, MessageUpdateEvent, │ │ │ + │ │ │ MessageEndEvent │ │ │ + │ │ └────────────────────────────────────────────────────┘ │ │ + │ └────────────────────────────────────────────────────────────┘ │ + │ │ │ + │ ▼ │ + │ ┌────────────────────────────────────────────────────────────┐ │ + │ │ 3. Execute Tool Calls │ │ + │ │ ┌────────────────────────────────────────────────────┐ │ │ + │ │ │ extract ToolCall from assistant content │ │ │ + │ │ │ │ │ │ + │ │ │ if EXECUTION_SEQUENTIAL || has_sequential_tool: │ │ │ + │ │ │ executeToolCallsSequential() │ │ │ + │ │ │ else: │ │ │ + │ │ │ executeToolCallsParallel() │ │ │ + │ │ └────────────────────────────────────────────────────┘ │ │ + │ │ │ │ │ + │ │ ▼ │ │ + │ │ ┌────────────────────────────────────────────────────┐ │ │ + │ │ │ For each tool call: │ │ │ + │ │ │ 1. before_tool_call hook │ │ │ + │ │ │ 2. prepareToolCall() │ │ │ + │ │ │ 3. execute() │ │ │ + │ │ │ 4. after_tool_call hook │ │ │ + │ │ │ 5. Emit ToolExecutionStart/Update/EndEvent │ │ │ + │ │ │ 6. Emit ToolResultMessage │ │ │ + │ │ └────────────────────────────────────────────────────┘ │ │ + │ └────────────────────────────────────────────────────────────┘ │ + │ │ │ + │ ▼ │ + │ ┌────────────────────────────────────────────────────────────┐ │ + │ │ 4. Prepare Next Turn │ │ + │ │ ┌────────────────────────────────────────────────────┐ │ │ + │ │ │ prepare_next_turn(context) -> next_turn_snapshot │ │ │ + │ │ │ - Optional: Update model/thinking_level │ │ │ + │ │ │ - Optional: Update context │ │ │ + │ │ └────────────────────────────────────────────────────┘ │ │ + │ └────────────────────────────────────────────────────────────┘ │ + │ │ │ + │ ▼ │ + │ ┌────────────────────────────────────────────────────────────┐ │ + │ │ 5. Check Termination Conditions │ │ + │ │ ┌────────────────────────────────────────────────────┐ │ │ + │ │ │ should_stop_after_turn(context) -> bool │ │ │ + │ │ │ - Max turns reached? │ │ │ + │ │ │ - Tool returned terminate=true? │ │ │ + │ │ │ - Steering queue empty and follow-up empty? │ │ │ + │ │ └────────────────────────────────────────────────────┘ │ │ + │ └────────────────────────────────────────────────────────────┘ │ + │ │ │ + │ ▼ │ + │ ┌────────────────────────────────────────────────────────────┐ │ + │ │ 6. Emit TurnEndEvent (message, tool_results) │ │ + │ └────────────────────────────────────────────────────────────┘ │ + │ │ │ + │ ▼ │ + │ ┌────────────────────────────────────────────────────────────┐ │ + │ │ Loop continues until termination condition met │ │ + │ └────────────────────────────────────────────────────────────┘ │ + │ │ │ + └──────────────────────────────┼───────────────────────────────────────┘ + │ + ▼ + ┌────────────────────────────────────────────────────────────────────┐ + │ AgentEndEvent with final messages │ + └────────────────────────────────────────────────────────────────────┘ +``` + +### 3. Tool Execution Flow + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ Tool Execution Flow │ +└─────────────────────────────────────────────────────────────────────────┘ + + ┌───────────────────────────────────────────────────────────────────┐ + │ Assistant Message with Tool Calls │ + │ ┌─────────────────────────────────────────────────────────────┐ │ + │ │ AssistantMessage: │ │ + │ │ content: [ │ │ + │ │ TextContent("I'll help you"), │ │ + │ │ ToolCall(id="tc1", name="bash", args={...}), │ │ + │ │ ToolCall(id="tc2", name="read", args={...}) │ │ + │ │ ] │ │ + │ └─────────────────────────────────────────────────────────────┘ │ + │ │ │ + │ ▼ │ + └───────────────────────────────────────────────────────────────────┘ + │ + │ executeToolCalls() + ▼ + ┌───────────────────────────────────────────────────────────────────┐ + │ Determine Execution Mode │ + │ ┌─────────────────────────────────────────────────────────────┐ │ + │ │ config.tool_execution == EXECUTION_SEQUENTIAL? │ │ + │ │ OR any tool has execution_mode == EXECUTION_SEQUENTIAL? │ │ + │ └─────────────────────────────────────────────────────────────┘ │ + │ │ │ + │ ┌───────────────┴───────────────┐ │ + │ ▼ ▼ │ + │ ┌────────────────────────┐ ┌────────────────────────┐ │ + │ │ executeSequential() │ │ executeParallel() │ │ + │ └────────────────────────┘ └────────────────────────┘ │ + │ │ │ │ + └──────────────┼───────────────────────────────┼────────────────────┘ + │ │ + │ │ + ▼ ▼ + ┌──────────────────────┐ ┌──────────────────────┐ + │ Sequential Execution │ │ Parallel Execution │ + │ │ │ │ + │ for tool_call in: │ │ for tool_call in: │ + │ prepareToolCall() │ │ prepareToolCall() │ + │ execute() │ │ execute() (async) │ + │ finalize() │ │ │ + │ │ │ wait all results │ + │ │ └──────────────────────┘ + └──────────────────────┘ + │ + ▼ + ┌───────────────────────────────────────────────────────────────────┐ + │ For Each Tool Call │ + │ ┌─────────────────────────────────────────────────────────────┐ │ + │ │ 1. before_tool_call hook (optional) │ │ + │ │ - Can block execution │ │ + │ │ 2. prepareToolCall() │ │ + │ │ - validateToolArguments() │ │ + │ │ - prepareToolCallArguments() (optional) │ │ + │ │ 3. Execute Tool: │ │ + │ │ tool.execute(tool_call_id, args, signal, on_update) │ │ + │ │ 4. after_tool_call hook (optional) │ │ + │ │ - Can modify result content │ │ + │ │ 5. Emit events: │ │ + │ │ - ToolExecutionStartEvent │ │ + │ │ - ToolExecutionUpdateEvent (optional) │ │ + │ │ - ToolExecutionEndEvent │ │ + │ │ 6. Create ToolResultMessage │ │ + │ └─────────────────────────────────────────────────────────────┘ │ + └───────────────────────────────────────────────────────────────────┘ + │ + ▼ + ┌───────────────────────────────────────────────────────────────────┐ + │ Tool Result Messages │ + │ ┌─────────────────────────────────────────────────────────────┐ │ + │ │ ToolResultMessage: │ │ + │ │ role: "toolResult" │ │ + │ │ tool_call_id: "tc1" │ │ + │ │ tool_name: "bash" │ │ + │ │ content: [TextContent("command output")] │ │ + │ │ is_error: false │ │ + │ └─────────────────────────────────────────────────────────────┘ │ + └───────────────────────────────────────────────────────────────────┘ +``` + +### 4. Session & Tree Structure + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ Session Tree Structure │ +└─────────────────────────────────────────────────────────────────────────┘ + + Session = Linked List of Entries (tree structure) + + ┌───────────────────────────────────────────────────────────────────┐ + │ Branch Navigation │ + │ │ + │ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │ + │ │ E1 │────▶│ E2 │────▶│ E3 │────▶│ E4 │────▶│ E5 │ (leaf) │ + │ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ │ + │ │ │ │ │ │ │ + │ ▼ ▼ ▼ ▼ ▼ │ + │ Message Message Compaction Message BranchSummary │ + │ │ + │ E3 is a Compaction Entry: │ + │ - Summary of E1, E2 │ + │ - first_kept_entry_id: reference to first retained message │ + │ - tokens_before: context size before compaction │ + │ │ + │ E5 is a BranchSummary Entry: │ + │ - Summary of branch from from_id │ + │ - Represents a fork point in conversation history │ + │ │ + └───────────────────────────────────────────────────────────────────┘ + │ + │ Session.moveTo() + ▼ + ┌───────────────────────────────────────────────────────────────────┐ + │ Forking & Branching │ + │ │ + │ Current branch: │ + │ ┌─────┐ ┌─────┐ ┌─────┐ │ + │ │ E1 │────▶│ E2 │────▶│ E3 │ │ + │ └─────┘ └─────┘ └─────┘ │ + │ │ │ + │ │ moveTo(E2) │ + │ ▼ │ + │ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │ + │ │ E1 │────▶│ E2 │────▶│ E3' │────▶│ E4' │ (new branch) │ + │ └─────┘ └─────┘ └─────┘ └─────┘ │ + │ │ │ + │ │ create BranchSummary │ + │ ▼ │ + │ ┌─────┐ │ + │ │ E5 │ (branch summary) │ + │ └─────┘ │ + │ │ + └───────────────────────────────────────────────────────────────────┘ +``` + +## Component Relationships + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Component Relationships │ +└─────────────────────────────────────────────────────────────────────────────┘ + + User Code + │ + ├── Creates ──► Agent + │ │ + │ ├── Uses ──► AgentLoop + │ │ │ + │ │ ├── Uses ──► StreamFn (LLM API) + │ │ │ + │ │ └── Uses ──► Session + │ │ + │ ├── Manages ──► AgentState + │ │ + │ ├── Queues ──► SteeringQueue + │ │ + │ └── Queues ──► FollowUpQueue + │ + └── Interacts With ──► AgentHarness (optional, higher level) + │ + ├── Manages ──► SessionRepo + │ + ├── Manages ──► Skills + │ + └── Manages ──► PromptTemplates +``` + +## Data Flow + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Data Flow Between Layers │ +└─────────────────────────────────────────────────────────────────────────────┘ + + User Input (String/Message) + │ + ▼ + ┌──────────────────────┐ + │ Agent.prompt() │ + │ - normalizeInput() │ + └──────────────────────┘ + │ + ▼ + ┌──────────────────────┐ + │ AgentState.messages │ ──► AgentMessage[] + └──────────────────────┘ + │ + ▼ + ┌──────────────────────┐ + │ AgentLoop │ + │ - transform_context │ + └──────────────────────┘ + │ + ▼ + ┌──────────────────────┐ + │ convertToLlm() │ ──► Transforms AgentMessage[] to Message[] + └──────────────────────┘ + │ + ▼ + ┌──────────────────────┐ + │ LLM API (StreamFn) │ + │ - Context: Message[] │ + └──────────────────────┘ + │ + ▼ + ┌──────────────────────┐ + │ Response (Streaming) │ + │ - Text deltas │ + │ - Tool call deltas │ + └──────────────────────┘ + │ + ▼ + ┌──────────────────────┐ + │ AssistantMessage │ + │ - content: Message[] │ + └──────────────────────┘ + │ + ▼ + ┌──────────────────────┐ + │ AgentState.messages │ ──► Appended to conversation + └──────────────────────┘ + │ + ▼ + ┌──────────────────────┐ + │ Tool Execution │ + │ - Extract ToolCalls │ + │ - Execute tools │ + └──────────────────────┘ + │ + ▼ + ┌──────────────────────┐ + │ ToolResultMessage[] │ + └──────────────────────┘ + │ + ▼ + ┌──────────────────────┐ + │ AgentState.messages │ ──► Tool results appended + └──────────────────────┘ + │ + │ (Loop back to LLM or end) + ▼ + ┌──────────────────────┐ + │ Session Storage │ + │ - JSONL format │ + │ - Tree entries │ + └──────────────────────┘ +``` + +## Summary + +The AgentCore.jl architecture follows a clean separation of concerns: + +1. **AgentHarness** - Highest level, handles persistence and resources +2. **Agent** - State management and event streaming +3. **AgentLoop** - Core LLM interaction loop +4. **Session** - Conversation history management + +Each layer transforms data and passes it to the next layer, with clear interfaces and event hooks for customization. diff --git a/learning/02-AGENT_COMPONENT.md b/learning/02-AGENT_COMPONENT.md new file mode 100644 index 0000000..24f942d --- /dev/null +++ b/learning/02-AGENT_COMPONENT.md @@ -0,0 +1,465 @@ +# AgentCore.jl - Agent Component Deep Dive + +## Agent Structure + +```julia +mutable struct Agent + _state::AgentState + listeners::Set{Tuple{Function, Ref{Bool}}} + steering_queue::PendingMessageQueue + follow_up_queue::PendingMessageQueue + + convert_to_llm::Function + transform_context::Union{Function, Nothing} + stream_function::StreamFn + get_api_key::Union{Function, Nothing} + on_payload::Union{Function, Nothing} + on_response::Union{Function, Nothing} + before_tool_call::Union{Function, Nothing} + after_tool_call::Union{Function, Nothing} + prepare_next_turn::Union{Function, Nothing} + prepare_next_turn_with_context::Union{Function, Nothing} + active_run::Union{ActiveRun, Nothing} + session_id::Union{String, Nothing} + thinking_budgets::Union{Dict{String, Int64}, Nothing} + transport::String + max_retry_delay_ms::Union{Int64, Nothing} + tool_execution::ToolExecutionMode +end +``` + +## Agent Lifecycle + +### 1. Initialization + +```julia +# Create agent with options +agent = Agent(Dict{Symbol, Any}( + :systemPrompt => "You are a helpful assistant", + :model => Model(...), + :thinkingLevel => THINKING_MEDIUM, + :tools => [bash_tool, read_tool], + :steeringMode => QUEUE_ONE_AT_A_TIME, + :followUpMode => QUEUE_ONE_AT_A_TIME, + :toolExecution => EXECUTION_PARALLEL, +)) + +# Subscribe to events +unsubscribe = subscribe(agent) do event, signal + if event isa MessageEndEvent + println("Message: $(event.message)") + elseif event isa ToolExecutionEndEvent + println("Tool completed: $(event.tool_name)") + end +end +``` + +### 2. Message Queues + +#### Steering Queue +- Messages injected **after** the current assistant turn finishes +- Used to correct or redirect the agent's behavior +- Example: "Actually, let's do X instead" + +#### Follow-Up Queue +- Messages run **only after** the agent would otherwise stop +- Used to continue conversation when agent thinks it's done +- Example: "Wait, there's one more thing" + +#### Queue Modes +- `QUEUE_ALL` - Drain all messages at once +- `QUEUE_ONE_AT_A_TIME` - Process one message at a time + +```julia +# Queue a steering message +steer(agent, UserMessage(...)) + +# Queue a follow-up message +followUp(agent, UserMessage(...)) + +# Check if queues have items +hasQueuedMessages(agent) # Returns Bool + +# Clear queues +clearSteeringQueue(agent) +clearFollowUpQueue(agent) +clearAllQueues(agent) +``` + +### 3. Event System + +#### Agent Events + +```julia +abstract type AgentEvent end + +# Lifecycle events +struct AgentStartEvent <: AgentEvent end +struct AgentEndEvent <: AgentEvent + messages::Vector{AgentMessage} +end + +# Turn events +struct TurnStartEvent <: AgentEvent end +struct TurnEndEvent <: AgentEvent + message::AgentMessage + tool_results::Vector{ToolResultMessage} +end + +# Message events +struct MessageStartEvent <: AgentEvent + message::AgentMessage +end +struct MessageUpdateEvent <: AgentEvent + message::AgentMessage + assistant_message_event::Any +end +struct MessageEndEvent <: AgentEvent + message::AgentMessage +end + +# Tool execution events +struct ToolExecutionStartEvent <: AgentEvent + tool_call_id::String + tool_name::String + args::Any +end +struct ToolExecutionUpdateEvent <: AgentEvent + tool_call_id::String + tool_name::String + args::Any + partial_result::Any +end +struct ToolExecutionEndEvent <: AgentEvent + tool_call_id::String + tool_name::String + result::Any + is_error::Bool +end +``` + +#### Event Flow Diagram + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ Event Timeline │ +└─────────────────────────────────────────────────────────────────────────┘ + +AgentStartEvent + │ + ├─ TurnStartEvent + │ │ + │ ├─ MessageStartEvent (user prompt) + │ ├─ MessageEndEvent (user prompt) + │ │ + │ ├─ [Loop starts] + │ │ │ + │ │ ├─ MessageStartEvent (assistant response) + │ │ ├─ MessageUpdateEvent (text delta 1) + │ │ ├─ MessageUpdateEvent (text delta 2) + │ │ ├─ MessageUpdateEvent (tool call delta) + │ │ ├─ MessageEndEvent (assistant complete) + │ │ │ + │ │ ├─ ToolExecutionStartEvent (tc1) + │ │ ├─ ToolExecutionUpdateEvent (partial result) + │ │ ├─ ToolExecutionEndEvent (tc1 done) + │ │ │ + │ │ ├─ ToolExecutionStartEvent (tc2) + │ │ ├─ ToolExecutionEndEvent (tc2 done) + │ │ │ + │ │ └─ TurnEndEvent (assistant + tools) + │ │ + │ └─ [Next turn if needed] + │ + └─ AgentEndEvent (final messages) +``` + +### 4. State Management + +```julia +mutable struct AgentState + system_prompt::String + model::Model + thinking_level::ThinkingLevel + tools::Vector{AgentTool} + messages::Vector{AgentMessage} + is_streaming::Bool + streaming_message::Union{AgentMessage, Nothing} + pending_tool_calls::Set{String} + error_message::Union{String, Nothing} +end +``` + +#### State Access + +```julia +# Get current state +state = get_state(agent) + +# Reset state +reset!(agent) # Clears messages, queues, and runtime state +``` + +### 5. Main Methods + +#### prompt() + +```julia +# Start a new conversation +prompt(agent, "Hello, how are you?") + +# With multiple messages +prompt(agent, [ + UserMessage(...), + AssistantMessage(...), + UserMessage(...) +]) + +# With images +prompt(agent, "Analyze this image", [ImageContent(data, "image/png")]) +``` + +#### continue!() + +```julia +# Continue from current transcript +# Last message must be user or tool-result +continue!(agent) +``` + +#### steer() and followUp() + +```julia +# Steering: Redirect after next assistant turn +steer(agent, UserMessage(...)) + +# Follow-up: Continue after agent would stop +followUp(agent, UserMessage(...)) +``` + +### 6. Hooks + +#### convert_to_llm + +```julia +# Transform messages before sending to LLM +function myConvertToLlm(messages::Vector{AgentMessage}) + return filter( + m -> m.role in ["user", "assistant", "toolResult"], + messages + ) +end + +agent = Agent(Dict(:convertToLlm => myConvertToLlm)) +``` + +#### transform_context + +```julia +# Transform context before LLM call +function myTransformContext(messages, signal) + # Can truncate, filter, or modify messages + return messages +end + +agent = Agent(Dict(:transformContext => myTransformContext)) +``` + +#### before_tool_call + +```julia +# Hook before tool execution +function myBeforeToolCall(context, signal) + println("About to execute: $(context.tool_call.name)") + return nothing # Return block=true to prevent execution +end + +agent = Agent(Dict(:beforeToolCall => myBeforeToolCall)) +``` + +#### after_tool_call + +```julia +# Hook after tool execution +function myAfterToolCall(context, signal) + # Can modify tool result + return AfterToolCallResult( + content = context.result.content, + terminate = context.result.terminate + ) +end + +agent = Agent(Dict(:afterToolCall => myAfterToolCall)) +``` + +#### prepare_next_turn + +```julia +# Modify context/model/thinking level between turns +function myPrepareNextTurn(context, signal) + # context: PrepareNextTurnContext + # Returns AgentLoopTurnUpdate or nothing + return AgentLoopTurnUpdate( + context = context.context, + model = context.context.model, # Can change model + thinking_level = THINKING_HIGH # Can change thinking level + ) +end + +agent = Agent(Dict(:prepareNextTurn => myPrepareNextTurn)) +``` + +### 7. Active Run Management + +```julia +# Check if agent is busy +if !isnothing(agent.active_run) + # Agent is processing + abort(agent) # Abort current run +end + +# Wait for completion +wait_for_idle(agent) # Returns Promise +``` + +## Complete Example + +```julia +using AgentCore + +# 1. Create agent +agent = Agent(Dict( + :systemPrompt => "You are a helpful assistant.", + :model => Model(...), + :tools => [bash_tool, read_tool], +)) + +# 2. Subscribe to events +events_received = [] +unsubscribe = subscribe(agent) do event, signal + push!(events_received, event) + + if event isa MessageEndEvent + println("Message: $(event.message)") + end +end + +# 3. Start conversation +prompt(agent, "What's in the current directory?") + +# 4. Wait for completion +wait_for_idle(agent) + +# 5. Check final state +state = get_state(agent) +println("Total messages: $(length(state.messages))") + +# 6. Continue with steering +steer(agent, UserMessage(...)) +wait_for_idle(agent) + +# 7. Clean up +unsubscribe() # Stop listening +reset!(agent) # Clear state +``` + +## Key Concepts + +### Message Queueing + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ Message Queue Behavior │ +└─────────────────────────────────────────────────────────────────────────┘ + +Scenario: User sends message, agent responds with tool calls + +┌────────────────────────────────────────────────────────────┐ +│ Time 0: User sends message │ +│ ┌──────────────┐ │ +│ │ prompt(msg) │ │ +│ └──────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────┐ │ +│ │ AgentLoop │ │ +│ │ processes │ │ +│ │ msg │ │ +│ └─────────────┘ │ +└────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────┐ +│ Time 1: Agent responds with tool calls │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ AssistantMessage: │ │ +│ │ content: [Text("I'll check..."), │ │ +│ │ ToolCall("bash", {...}), │ │ +│ │ ToolCall("read", {...})] │ │ +│ └──────────────────────────────────────────────────────┘ │ +└────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────┐ +│ Time 2: User queues steering message │ +│ ┌──────────────────┐ │ +│ │ steer(msg2) │ ──► steering_queue.push(msg2) │ +│ └──────────────────┘ │ +│ │ +│ (msg2 not processed yet!) │ +└────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────┐ +│ Time 3: Tool execution │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ Execute bash tool... │ │ +│ │ Execute read tool... │ │ +│ │ Emit ToolResultMessage[] │ │ +│ └──────────────────────────────────────────────────────┘ │ +└────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────┐ +│ Time 4: Agent responds to tool results │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ AssistantMessage (2nd turn): │ │ +│ │ content: [Text("The results are...")] │ │ +│ └──────────────────────────────────────────────────────┘ │ +└────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────┐ +│ Time 5: Steering message processed │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ steering_queue.drain() → [msg2] │ │ +│ │ Emit msg2 as UserMessage │ │ +│ └──────────────────────────────────────────────────────┘ │ +└────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────┐ +│ Time 6: Next turn (agent responds to steering) │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ AssistantMessage (3rd turn): │ │ +│ │ content: [Text("Okay, I'll do X instead...")] │ │ +│ └──────────────────────────────────────────────────────┘ │ +└────────────────────────────────────────────────────────────┘ +``` + +### Queue Behavior Summary + +| Action | Queue | When Processed | +|--------|-------|----------------| +| `prompt()` | N/A | Immediate | +| `steer()` | steering_queue | After assistant turn completes | +| `followUp()` | follow_up_queue | After agent would normally stop | +| `continue!()` | N/A | Immediately if last message is user/tool | + +## Best Practices + +1. **Use steering for redirects**: When user wants to change direction mid-conversation +2. **Use follow-up for continuation**: When agent thinks it's done but user wants more +3. **Subscribe to events**: Monitor agent behavior and debug issues +4. **Clear queues**: Use `clearAllQueues()` when resetting conversation +5. **Check active run**: Don't call `prompt()` while agent is busy diff --git a/learning/03-AGENTLOOP_COMPONENT.md b/learning/03-AGENTLOOP_COMPONENT.md new file mode 100644 index 0000000..8c5c2d2 --- /dev/null +++ b/learning/03-AGENTLOOP_COMPONENT.md @@ -0,0 +1,758 @@ +# AgentCore.jl - AgentLoop Component Deep Dive + +## AgentLoop Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ AgentLoop Layer │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Public API │ +└─────────────────────────────────────────────────────────────────────────────┘ + +agentLoop() + ├─ prompts: Vector{AgentMessage} + ├─ context: AgentContext + ├─ config: AgentLoopConfig + ├─ signal: Union{Nothing, AbortSignal} + └─ stream_fn: StreamFn + └─ Returns: EventStream + +agentLoopContinue() + ├─ context: AgentContext + ├─ config: AgentLoopConfig + ├─ signal: Union{Nothing, AbortSignal} + └─ stream_fn: StreamFn + └─ Returns: EventStream + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Internal Flow │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ 1. runAgentLoop() ── Entry point for new conversation │ +│ - Creates copy of prompts │ +│ - Appends prompts to context.messages │ +│ - Emits AgentStartEvent │ +│ - Emits TurnStartEvent │ +│ - Emits MessageStart/End for each prompt │ +│ - Calls runLoop() │ +└─────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────────────┐ +│ 2. runLoop() ── Main event loop │ +│ ┌────────────────────────────────────────────────────────────────────┐ │ +│ │ while true: │ │ +│ │ 1. Get steering/follow-up messages (if any) │ │ +│ │ 2. Emit messages as UserMessage │ │ +│ │ 3. streamAssistantResponse() │ │ +│ │ 4. Execute tool calls (sequential or parallel) │ │ +│ │ 5. Emit TurnEndEvent │ │ +│ │ 6. prepare_next_turn (optional) │ │ +│ │ 7. should_stop_after_turn? (check termination) │ │ +│ │ 8. Loop continues if not terminated │ │ +│ └────────────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────────────┐ +│ 3. streamAssistantResponse() ── LLM interaction │ +│ - transform_context (optional) │ +│ - convert_to_llm (transform to Message[]) │ +│ - Call stream_fn (LLM API) │ +│ - Stream response deltas │ +│ - Emit MessageStart/Update/End events │ +└─────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────────────┐ +│ 4. executeToolCalls() ── Tool execution │ +│ ┌────────────────────────────────────────────────────────────────────┐ │ +│ │ if EXECUTION_SEQUENTIAL || has_sequential_tool: │ │ +│ │ executeToolCallsSequential() │ │ +│ │ else: │ │ +│ │ executeToolCallsParallel() │ │ +│ └────────────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────────────┐ +│ 5. AgentEndEvent ── Final event with all messages │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +## AgentLoopConfig + +```julia +struct AgentLoopConfig + model::Model + reasoning::Union{ThinkingLevel, Nothing} + session_id::Union{String, Nothing} + on_payload::Union{Function, Nothing} + on_response::Union{Function, Nothing} + transport::String + thinking_budgets::Union{Dict{String, Int64}, Nothing} + max_retry_delay_ms::Union{Int64, Nothing} + tool_execution::ToolExecutionMode + before_tool_call::Union{Function, Nothing} + after_tool_call::Union{Function, Nothing} + prepare_next_turn::Union{Function, Nothing} + convert_to_llm::Function + transform_context::Union{Function, Nothing} + get_api_key::Union{Function, Nothing} + get_steering_messages::Union{Function, Nothing} + get_follow_up_messages::Union{Function, Nothing} +end +``` + +## Main Functions + +### agentLoop() + +```julia +function agentLoop( + prompts::Vector{AgentMessage}, + context::AgentContext, + config::AgentLoopConfig, + signal::Union{Nothing, AbortSignal}, + stream_fn::StreamFn, +)::EventStream +``` + +**Purpose**: Start a new conversation with initial prompts + +**Flow**: +1. Create event stream +2. Spawn thread to run agent loop +3. Return stream for event consumption + +```julia +stream = agentLoop( + [UserMessage("user", [TextContent("Hello")], timestamp)], + AgentContext(system_prompt, messages, tools), + config, + nothing, + stream_fn, +) + +# Consume events +for event in stream + if event isa MessageEndEvent + println("Received: $(event.message)") + end +end +``` + +### runAgentLoop() + +```julia +function runAgentLoop( + prompts::Vector{AgentMessage}, + context::AgentContext, + config::AgentLoopConfig, + emit::AgentEventSink, + signal::Union{Nothing, AbortSignal}, + stream_fn::StreamFn, +)::Vector{AgentMessage} +``` + +**Purpose**: Execute agent loop with initial prompts + +**Flow**: +1. Copy prompts to new_messages +2. Append prompts to context.messages +3. Emit AgentStartEvent +4. For each prompt: emit MessageStartEvent, MessageEndEvent +5. Call runLoop() + +### runLoop() - The Heart of AgentLoop + +```julia +function runLoop( + initial_context::AgentContext, + new_messages::Vector{AgentMessage}, + initial_config::AgentLoopConfig, + signal::Union{Nothing, AbortSignal}, + emit::AgentEventSink, + stream_function::StreamFn, +)::Nothing +``` + +**Main Loop**: +```julia +current_context = initial_context +config = initial_config +first_turn = true +pending_messages = get_steering_messages() + +while true + # Process steering/follow-up messages + while !isempty(pending_messages) + if !first_turn + emit(TurnStartEvent()) + else + first_turn = false + end + + # Emit pending messages + for message in pending_messages + emit(MessageStartEvent(message)) + emit(MessageEndEvent(message)) + push!(current_context.messages, message) + push!(new_messages, message) + end + + pending_messages = [] + end + + # Stream assistant response + message = streamAssistantResponse( + current_context, + config, + signal, + emit, + stream_function, + ) + push!(new_messages, message) + + # Check for errors + if message.stop_reason in ("error", "aborted") + emit(TurnEndEvent(message, [])) + emit(AgentEndEvent(new_messages)) + return + end + + # Execute tool calls + tool_calls = filter(c -> c isa ToolCall, message.content) + tool_results = [] + has_more_tool_calls = false + + if !isempty(tool_calls) + executed_batch = if message.stop_reason == "length" + failToolCallsFromTruncatedMessage(tool_calls, emit) + else + executeToolCalls( + current_context, + message, + config, + signal, + emit, + ) + end + append!(tool_results, executed_batch.messages) + has_more_tool_calls = !executed_batch.terminate + + for result in tool_results + push!(current_context.messages, result) + push!(new_messages, result) + end + end + + emit(TurnEndEvent(message, tool_results)) + + # Prepare next turn (optional) + next_turn_context = PrepareNextTurnContext( + message, tool_results, current_context, new_messages + ) + next_turn_snapshot = prepare_next_turn(config, next_turn_context) + + if !isnothing(next_turn_snapshot) + current_context = next_turn_snapshot.context + config = AgentLoopConfig( + model = next_turn_snapshot.model, + reasoning = next_turn_snapshot.thinking_level, + # ... other config fields + ) + end + + # Check if should stop + if should_stop_after_turn(config, next_turn_context) + emit(AgentEndEvent(new_messages)) + return + end + + # Get next pending messages + pending_messages = get_steering_messages() + + # Check follow-up messages + follow_up_messages = get_follow_up_messages() + if !isempty(follow_up_messages) + pending_messages = follow_up_messages + continue + end + + break +end + +emit(AgentEndEvent(new_messages)) +``` + +### streamAssistantResponse() + +```julia +function streamAssistantResponse( + context::AgentContext, + config::AgentLoopConfig, + signal::Union{Nothing, AbortSignal}, + emit::AgentEventSink, + stream_function::StreamFn, +)::AssistantMessage +``` + +**Flow**: +1. Get messages from context +2. Apply transform_context (optional) +3. Convert to LLM messages with convert_to_llm +4. Create Context object +5. Resolve API key +6. Call stream_fn with model, context, and config +7. Stream events: + - "start" → MessageStartEvent + - "text_start", "text_delta", "text_end" → MessageUpdateEvent + - "done", "error" → MessageEndEvent + +### executeToolCalls() + +```julia +function executeToolCalls( + current_context::AgentContext, + assistant_message::AssistantMessage, + config::AgentLoopConfig, + signal::Union{Nothing, AbortSignal}, + emit::AgentEventSink, +)::ExecutedToolCallBatch +``` + +**Logic**: +```julia +tool_calls = filter(c -> c isa ToolCall, assistant_message.content) + +# Check if any tool requires sequential execution +has_sequential = any(tc -> begin + tool = findfirst(t -> t.name == tc.name, current_context.tools) + !isnothing(tool) && tool.execution_mode == EXECUTION_SEQUENTIAL +end, tool_calls) + +# Determine execution mode +if config.tool_execution == EXECUTION_SEQUENTIAL || has_sequential + executeToolCallsSequential(...) +else + executeToolCallsParallel(...) +end +``` + +### executeToolCallsSequential() + +```julia +function executeToolCallsSequential( + current_context::AgentContext, + assistant_message::AssistantMessage, + tool_calls::Vector{ToolCall}, + config::AgentLoopConfig, + signal::Union{Nothing, AbortSignal}, + emit::AgentEventSink, +)::ExecutedToolCallBatch +``` + +**Flow** (for each tool call): +1. Emit ToolExecutionStartEvent +2. prepareToolCall() → PreparedToolCall or ImmediateToolCallOutcome +3. If prepared: executePreparedToolCall() +4. finalizeExecutedToolCall() +5. Emit ToolExecutionEndEvent +6. Emit ToolResultMessage +7. Check if signal.aborted → break + +### executeToolCallsParallel() + +```julia +function executeToolCallsParallel( + current_context::AgentContext, + assistant_message::AssistantMessage, + tool_calls::Vector{ToolCall}, + config::AgentLoopConfig, + signal::Union{Nothing, AbortSignal}, + emit::AgentEventSink, +)::ExecutedToolCallBatch +``` + +**Flow**: +1. For each tool call: + - If immediate: execute and add to finalized_calls + - If prepared: create closure, add to finalized_calls +2. For each entry in finalized_calls: + - If closure: execute closure + - If finalized: use as-is +3. Collect all tool results +4. Return batch + +### prepareToolCall() + +```julia +function prepareToolCall( + current_context::AgentContext, + assistant_message::AssistantMessage, + tool_call::ToolCall, + config::AgentLoopConfig, + signal::Union{Nothing, AbortSignal}, +)::Union{PreparedToolCall, ImmediateToolCallOutcome} +``` + +**Flow**: +1. Find tool by name +2. If not found → ImmediateToolCallOutcome (error) +3. before_tool_call hook (optional) +4. prepareToolCallArguments() (optional) +5. validateToolArguments() +6. Return PreparedToolCall + +### executePreparedToolCall() + +```julia +function executePreparedToolCall( + prepared::PreparedToolCall, + signal::Union{Nothing, AbortSignal}, + emit::AgentEventSink, +)::ExecutedToolCallOutcome +``` + +**Flow**: +1. Call tool.execute(id, args, signal, on_update) +2. Collect update events (if any) +3. Wait for all update events +4. Return ExecutedToolCallOutcome(result) + +### finalizeExecutedToolCall() + +```julia +function finalizeExecutedToolCall( + current_context::AgentContext, + assistant_message::AssistantMessage, + prepared::PreparedToolCall, + executed::ExecutedToolCallOutcome, + config::AgentLoopConfig, + signal::Union{Nothing, AbortSignal}, +)::FinalizedToolCallOutcome +``` + +**Flow**: +1. after_tool_call hook (optional) +2. Return FinalizedToolCallOutcome + +### createToolResultMessage() + +```julia +function createToolResultMessage( + finalized::FinalizedToolCallOutcome, +)::ToolResultMessage +``` + +**Creates**: +```julia +ToolResultMessage( + "toolResult", + finalized.tool_call.id, + finalized.tool_call.name, + finalized.result.content, + finalized.result.details, + finalized.result.usage, + finalized.result.added_tool_names, + finalized.is_error, + timestamp, +) +``` + +## Execution Modes + +### Sequential Execution + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ Sequential Execution Flow │ +└─────────────────────────────────────────────────────────────────────────┘ + +┌──────┐ +│ TC1 │ ──► prepareToolCall() +└──────┘ │ + ▼ + ┌──────────────┐ + │ execute() │ ──► Wait for completion + └──────────────┘ │ + │ ▼ + ├───────────── createToolResultMessage() + │ │ + ▼ ▼ + ┌──────────────┐ ┌──────────┐ + │ TC2 │ ──► │ │ Result1 │ + └──────┘ └──────────┘ + │ + ▼ + ┌──────────────┐ + │ execute() │ + └──────────────┘ + │ + ▼ + ┌──────────────┐ + │ TC3 │ ──► │ + └──────┘ │ + │ ▼ + ├───── createToolResultMessage() + │ │ + ▼ ▼ + ┌──────────────┐ ┌──────────┐ + │ execute() │ │ │ Result2 │ + └──────────────┘ └──────────┘ + │ + ▼ + ┌──────────┐ + │ Result3 │ + └──────────┘ +``` + +### Parallel Execution + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ Parallel Execution Flow │ +└─────────────────────────────────────────────────────────────────────────┘ + +┌──────┐ +│ TC1 │ ──► prepareToolCall() ──► create closure ──► ┐ +└──────┘ │ + │ +┌──────┐ │ +│ TC2 │ ──► prepareToolCall() ──► create closure ──► ├─► All closures queued +└──────┘ │ + │ +┌──────┐ │ +│ TC3 │ ──► prepareToolCall() ──► create closure ──► ┘ +└──────┘ + + │ + ▼ + ┌───────────────────────┐ + │ for closure in closures│ + │ execute_closure() │ + └───────────────────────┘ + │ + ▼ + ┌───────────────────────┐ + │ Collect all results │ + └───────────────────────┘ + │ + ▼ + ┌───────────────────────┐ + │ createToolResult() │ + └───────────────────────┘ +``` + +## Helper Types + +### ExecutedToolCallBatch + +```julia +struct ExecutedToolCallBatch + messages::Vector{ToolResultMessage} + terminate::Bool +end +``` + +- `messages`: All tool result messages +- `terminate`: If true, stop agent after this batch + +### PrepareNextTurnContext + +```julia +struct PrepareNextTurnContext + message::AssistantMessage + tool_results::Vector{ToolResultMessage} + context::AgentContext + new_messages::Vector{AgentMessage} +end +``` + +Used by prepare_next_turn hook to decide next steps. + +### Before/After Tool Call Contexts + +```julia +struct BeforeToolCallContext + assistant_message::AssistantMessage + tool_call::ToolCall + args::Any + context::AgentContext +end + +struct BeforeToolCallResult + block::Union{Bool, Nothing} + reason::Union{String, Nothing} +end + +struct AfterToolCallContext + assistant_message::AssistantMessage + tool_call::ToolCall + args::Any + result::AgentToolResult + is_error::Bool + context::AgentContext +end + +struct AfterToolCallResult + content::Union{Vector{MessageContent}, Nothing} + details::Union{Any, Nothing} + is_error::Union{Bool, Nothing} + usage::Union{Usage, Nothing} + terminate::Union{Bool, Nothing} +end +``` + +## Event Emission Timeline + +``` +AgentStartEvent + │ + ├─ TurnStartEvent (turn 1) + │ │ + │ ├─ MessageStartEvent (user prompt) + │ ├─ MessageEndEvent (user prompt) + │ │ + │ ├─ MessageStartEvent (assistant) + │ ├─ MessageUpdateEvent (text delta) + │ ├─ MessageUpdateEvent (tool call delta) + │ ├─ MessageEndEvent (assistant) + │ │ + │ ├─ ToolExecutionStartEvent (tc1) + │ ├─ ToolExecutionEndEvent (tc1) + │ │ + │ ├─ ToolExecutionStartEvent (tc2) + │ ├─ ToolExecutionEndEvent (tc2) + │ │ + │ └─ TurnEndEvent (assistant, tool_results) + │ + ├─ TurnStartEvent (turn 2 - if needed) + │ │ + │ ├─ MessageStartEvent (steering/follow-up) + │ ├─ MessageEndEvent (steering/follow-up) + │ │ + │ ├─ MessageStartEvent (assistant) + │ ├─ MessageUpdateEvent (text) + │ ├─ MessageEndEvent (assistant) + │ │ + │ └─ TurnEndEvent (assistant, []) + │ + └─ AgentEndEvent (final messages) +``` + +## Key Concepts + +### 1. Message Transformation Pipeline + +``` +AgentMessage[] (internal) + │ + │ transform_context() + ▼ +AgentMessage[] (transformed) + │ + │ convert_to_llm() + ▼ +Message[] (LLM API) +``` + +### 2. Tool Call Lifecycle + +``` +ToolCall (in assistant message) + │ + ├─ before_tool_call (hook) + │ + ├─ prepareToolCall() + │ ├─ validate arguments + │ └─ prepare arguments (optional) + │ + ├─ execute() + │ ├─ Immediate: return result + │ └─ Prepared: async execution + │ + ├─ after_tool_call (hook) + │ + └─ createToolResultMessage() +``` + +### 3. Turn Termination + +```julia +# Turn ends when: +# 1. No more pending messages +# 2. No more tool calls to execute +# 3. should_stop_after_turn() returns true + +# Reasons to stop: +# - Max turns reached +# - Tool returned terminate=true +# - Error or abort +# - Steering/follow-up queues empty +``` + +## Best Practices + +1. **Use sequential execution** for tools that modify shared state +2. **Use parallel execution** for independent tool calls (better performance) +3. **Implement prepare_next_turn** for dynamic model/thinking level changes +4. **Use before_tool_call** for logging or blocking sensitive operations +5. **Use after_tool_call** for modifying results or collecting metrics + +## Complete Example + +```julia +using AgentCore + +# Create config +config = AgentLoopConfig( + model = my_model, + reasoning = THINKING_MEDIUM, + tool_execution = EXECUTION_PARALLEL, + before_tool_call = myBeforeToolCallHook, + after_tool_call = myAfterToolCallHook, + prepare_next_turn = myPrepareNextTurnHook, + convert_to_llm = myConvertToLlm, + transform_context = myTransformContext, + get_api_key = myGetApiKey, + get_steering_messages = myGetSteeringMessages, + get_follow_up_messages = myGetFollowUpMessages, +) + +# Start agent loop +stream = agentLoop( + [UserMessage("user", [TextContent("Hello")], timestamp)], + AgentContext(system_prompt, messages, tools), + config, + nothing, + stream_fn, +) + +# Consume events +final_messages = [] +for event in stream + if event isa MessageEndEvent + push!(final_messages, event.message) + end +end + +# Or use event sink +messages = [] +emit(event) = push!(messages, event) + +messages = runAgentLoop( + [UserMessage(...)], + context, + config, + emit, + nothing, + stream_fn, +) +``` + +This documentation provides a comprehensive understanding of the AgentLoop component, including its architecture, main functions, execution modes, and best practices for building AI agents with AgentCore.jl. diff --git a/learning/04-TYPES_MESSAGES.md b/learning/04-TYPES_MESSAGES.md new file mode 100644 index 0000000..0ee20cc --- /dev/null +++ b/learning/04-TYPES_MESSAGES.md @@ -0,0 +1,589 @@ +# AgentCore.jl - Types and Messages Deep Dive + +## Core Type Hierarchy + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Type Hierarchy │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ ThinkingLevel (Enum) │ +│ - THINKING_OFF │ +│ - THINKING_MINIMAL │ +│ - THINKING_LOW │ +│ - THINKING_MEDIUM │ +│ - THINKING_HIGH │ +│ - THINKING_XHIGH │ +│ - THINKING_MAX │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ ToolExecutionMode (Enum) │ +│ - EXECUTION_SEQUENTIAL (Tools run one at a time) │ +│ - EXECUTION_PARALLEL (Tools run concurrently) │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ QueueMode (Enum) │ +│ - QUEUE_ALL (Drain all messages at once) │ +│ - QUEUE_ONE_AT_A_TIME (Process one message at a time) │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ MessageContent (Abstract Type) │ +│ ├── TextContent (String) │ +│ └── ImageContent (data::String, mime_type::String) │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Message (Abstract Type) │ +│ ├── UserMessage │ +│ │ └─ role: "user", content: Message[], timestamp: Int64 │ +│ ├── AssistantMessage │ +│ │ └─ role: "assistant", content: Message[], api, provider, model, │ +│ │ usage: Usage, stop_reason, error_message, timestamp │ +│ └── ToolResultMessage │ +│ └─ role: "toolResult", tool_call_id, tool_name, content, details, │ +│ usage, added_tool_names, is_error, timestamp │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ AgentMessage (Abstract Type) │ +│ └─ Union of all message types above + custom types │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ AgentTool │ +│ - name: String │ +│ - label: String │ +│ - description: String │ +│ - parameters: Any │ +│ - execute: Function │ +│ - prepare_arguments: Union{Function, Nothing} │ +│ - execution_mode: Union{ToolExecutionMode, Nothing} │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ AgentContext │ +│ - system_prompt: String │ +│ - messages: Vector{AgentMessage} │ +│ - tools: Union{Vector{AgentTool}, Nothing} │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ AgentEvent (Abstract Type) │ +│ ├── AgentStartEvent / AgentEndEvent │ +│ ├── TurnStartEvent / TurnEndEvent │ +│ ├── MessageStartEvent / MessageEndEvent │ +│ ├── MessageUpdateEvent │ +│ ├── ToolExecutionStartEvent / ToolExecutionEndEvent │ +│ └── ToolExecutionUpdateEvent │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Usage & ModelCost │ +│ Usage: input, output, cache_read, cache_write, total_tokens, cost │ +│ ModelCost: input, output, cache_read, cache_write (all Float64) │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Model │ +│ - id, name, api, provider, base_url, reasoning: Bool │ +│ - input: Vector{String} │ +│ - cost: ModelCost │ +│ - context_window, max_tokens: Int64 │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +## Message Types + +### UserMessage + +```julia +struct UserMessage <: Message + role::String # "user" + content::Vector{MessageContent} + timestamp::Timestamp # Int64 (Unix timestamp) +end +``` + +**Usage**: +```julia +# Simple text message +UserMessage( + "user", + [TextContent("Hello, how are you?")], + Int64(Dates.now(Dates.UTC).datetime) +) + +# With multiple content types +UserMessage( + "user", + [ + TextContent("Analyze this image"), + ImageContent(data_base64, "image/png") + ], + timestamp +) +``` + +### AssistantMessage + +```julia +struct AssistantMessage <: Message + role::String # "assistant" + content::Vector{MessageContent} + api::String # API identifier + provider::String # Provider name + model::String # Model ID + usage::Usage + stop_reason::String # "done", "error", "aborted", "length", etc. + error_message::Union{String, Nothing} + timestamp::Timestamp +end +``` + +**Content can include**: +- TextContent +- ToolCall + +```julia +AssistantMessage( + "assistant", + [ + TextContent("I'll check the directory for you."), + ToolCall( + "tool", + "tc_123", + "bash", + Dict("command" => "ls -la"), + nothing + ), + ToolCall( + "tool", + "tc_456", + "read", + Dict("path" => "README.md"), + nothing + ) + ], + "openai", + "openai", + "gpt-4", + Usage(100, 50, 0, 0, 150, UsageCost(0.001, 0.002, 0.0, 0.0, 0.003)), + "done", + nothing, + timestamp +) +``` + +### ToolResultMessage + +```julia +struct ToolResultMessage <: Message + role::String # "toolResult" + tool_call_id::String # Reference to original ToolCall + tool_name::String # Name of tool that executed + content::Vector{MessageContent} + details::Any # Additional tool-specific details + usage::Union{Usage, Nothing} + added_tool_names::Union{Vector{String}, Nothing} + is_error::Bool # True if tool execution failed + timestamp::Timestamp +end +``` + +**Usage**: +```julia +ToolResultMessage( + "toolResult", + "tc_123", + "bash", + [TextContent("file1.md\nfile2.md\n")], + BashToolDetails(...), + nothing, + nothing, + false, + timestamp +) +``` + +## AgentTool Structure + +```julia +struct AgentTool{TParameters, TDetails} + name::String + label::String + description::String + parameters::TParameters + execute::Function + prepare_arguments::Union{Function, Nothing} + execution_mode::Union{ToolExecutionMode, Nothing} +end +``` + +**Parameters**: +- `name`: Unique identifier for the tool +- `label`: Display name +- `description`: What the tool does +- `parameters`: JSON schema for tool arguments +- `execute`: Main execution function +- `prepare_arguments`: Optional preprocessing +- `execution_mode`: Sequential or parallel + +### Tool Execution Function Signature + +```julia +execute::Function( + tool_call_id::String, + params::Dict{String, Any}, + signal::Union{Any, Nothing}, # Abort signal + on_update::Function, # Callback for streaming updates + context::Any, # Tool context +)::AgentToolResult +``` + +**Returns**: +```julia +AgentToolResult( + content::Vector{MessageContent}, # Result content + details::T, # Tool-specific details + usage::Union{Usage, Nothing}, # Usage statistics + added_tool_names::Union{Vector{String}, Nothing}, + terminate::Union{Bool, Nothing}, # If true, stop agent after this +) +``` + +## AgentContext + +```julia +struct AgentContext + system_prompt::String + messages::Vector{AgentMessage} + tools::Union{Vector{AgentTool}, Nothing} +end +``` + +**Purpose**: Read-only snapshot of agent state for LLM calls + +**Usage in AgentLoop**: +```julia +function streamAssistantResponse( + context::AgentContext, # Contains messages, tools, system prompt + config::AgentLoopConfig, + ... +)::AssistantMessage + # Convert to LLM format + llm_messages = config.convert_to_llm(context.messages) + + # Create context for API + llm_context = Context( + context.system_prompt, + llm_messages, + context.tools, + ) + + # Call LLM + return stream_function(context.model, llm_context, config) +end +``` + +## Event Types + +### Agent Lifecycle Events + +```julia +struct AgentStartEvent <: AgentEvent end +struct AgentEndEvent <: AgentEvent + messages::Vector{AgentMessage} +end +``` + +### Turn Events + +```julia +struct TurnStartEvent <: AgentEvent end +struct TurnEndEvent <: AgentEvent + message::AgentMessage + tool_results::Vector{ToolResultMessage} +end +``` + +### Message Events + +```julia +struct MessageStartEvent <: AgentEvent + message::AgentMessage +end +struct MessageUpdateEvent <: AgentEvent + message::AgentMessage + assistant_message_event::Any # Partial message event +end +struct MessageEndEvent <: AgentEvent + message::AgentMessage +end +``` + +### Tool Execution Events + +```julia +struct ToolExecutionStartEvent <: AgentEvent + tool_call_id::String + tool_name::String + args::Any +end +struct ToolExecutionUpdateEvent <: AgentEvent + tool_call_id::String + tool_name::String + args::Any + partial_result::Any +end +struct ToolExecutionEndEvent <: AgentEvent + tool_call_id::String + tool_name::String + result::Any + is_error::Bool +end +``` + +## Usage Statistics + +```julia +struct Usage + input::Int64 # Input tokens + output::Int64 # Output tokens + cache_read::Int64 # Cache read tokens + cache_write::Int64 # Cache write tokens + total_tokens::Int64 # Total tokens + cost::UsageCost +end + +struct UsageCost + input::Float64 + output::Float64 + cache_read::Float64 + cache_write::Float64 + total::Float64 +end +``` + +**Example**: +```julia +Usage( + 1000, # input tokens + 200, # output tokens + 500, # cache read tokens + 0, # cache write tokens + 1700, # total tokens + UsageCost( + 0.0005, # input cost ($0.50 per 1M tokens) + 0.0015, # output cost ($1.50 per 1M tokens) + 0.00025, # cache read cost + 0.0, # cache write cost + 0.0035 # total cost + ) +) +``` + +## Model Type + +```julia +struct Model{Api} + id::String # Model identifier (e.g., "gpt-4") + name::String # Model name (e.g., "GPT-4") + api::Api # API type (String, Symbol, or custom type) + provider::String # Provider name (e.g., "openai") + base_url::String # API base URL + reasoning::Bool # Whether model supports reasoning + input::Vector{String} # Input modes (e.g., ["text", "image"]) + cost::ModelCost + context_window::Int64 # Max context window (e.g., 128000) + max_tokens::Int64 # Max output tokens +end + +struct ModelCost + input::Float64 + output::Float64 + cache_read::Float64 + cache_write::Float64 +end +``` + +## ToolCall Type + +```julia +struct ToolCall + type::String # "tool" + id::String # Unique ID for this tool call + name::String # Tool name to call + arguments::Dict{String, Any} # Tool arguments as JSON-like Dict + partial_json::Union{String, Nothing} # Partial JSON string +end +``` + +**Example**: +```julia +ToolCall( + "tool", + "call_abc123", + "bash", + Dict( + "command" => "ls -la", + "timeout" => 30 + ), + nothing +) +``` + +## Custom Message Types + +### BashExecutionMessage + +```julia +mutable struct BashExecutionMessage + role::String # "custom" + command::String + output::String + exit_code::Union{Int64, Nothing} + cancelled::Bool + truncated::Bool + full_output_path::Union{String, Nothing} + timestamp::Timestamp + exclude_from_context::Bool +end +``` + +### CompactionSummaryMessage + +```julia +mutable struct CompactionSummaryMessage + role::String # "compactionSummary" + summary::String # Summary of compacted history + tokens_before::Int64 # Context size before compaction + timestamp::Timestamp +end +``` + +### BranchSummaryMessage + +```julia +mutable struct BranchSummaryMessage + role::String # "branchSummary" + summary::String # Summary of branch history + from_id::String # Branch point ID + timestamp::Timestamp +end +``` + +## AgentState + +```julia +mutable struct AgentState + system_prompt::String + model::Model + thinking_level::ThinkingLevel + tools::Vector{AgentTool} + messages::Vector{AgentMessage} + is_streaming::Bool + streaming_message::Union{AgentMessage, Nothing} + pending_tool_calls::Set{String} + error_message::Union{String, Nothing} +end +``` + +**Purpose**: Runtime state of the Agent + +**Note**: AgentState is mutable and used internally by Agent + +## Key Conversion Functions + +### convertToLlm() + +```julia +function convertToLlm(messages::Vector{AgentMessage})::Vector{Message} + result::Vector{Message} = Message[] + + for m in messages + converted = convertToLlmMessage(m) + if !isnothing(converted) + push!(result, converted) + end + end + + return result +end +``` + +**Purpose**: Transform AgentMessage[] to Message[] for LLM API + +**Example**: +```julia +# Input: AgentMessage[] +[ + UserMessage(...), + AssistantMessage(...), + ToolResultMessage(...), + BashExecutionMessage(...), # Will be converted to UserMessage + CompactionSummaryMessage(...), # Will be converted to UserMessage +] + +# Output: Message[] +[ + UserMessage(...), + AssistantMessage(...), + ToolResultMessage(...), + UserMessage(...), # Converted from BashExecutionMessage + UserMessage(...), # Converted from CompactionSummaryMessage +] +``` + +### Default convertToLlmMessage Implementations + +```julia +function convertToLlmMessage(m::BashExecutionMessage) + if m.exclude_from_context + return nothing + end + return UserMessage("user", [TextContent(bashExecutionToText(m))], m.timestamp) +end + +function convertToLlmMessage(m::CompactionSummaryMessage) + text = COMPACTION_SUMMARY_PREFIX * m.summary * COMPACTION_SUMMARY_SUFFIX + return UserMessage("user", [TextContent(text)], m.timestamp) +end + +function convertToLlmMessage(m::BranchSummaryMessage) + text = BRANCH_SUMMARY_PREFIX * m.summary * BRANCH_SUMMARY_SUFFIX + return UserMessage("user", [TextContent(text)], m.timestamp) +end + +function convertToLlmMessage(m::UserMessage) + return m # Pass through +end + +function convertToLlmMessage(m::AssistantMessage) + return m # Pass through +end + +function convertToLlmMessage(m::ToolResultMessage) + return m # Pass through +end +``` + +## Summary + +The type system in AgentCore.jl provides: + +1. **Strong typing** for different message types +2. **Extensibility** through abstract types and multiple dispatch +3. **Clear separation** between internal (AgentMessage) and external (Message) formats +4. **Rich metadata** in Usage and Model types for cost tracking +5. **Event-driven architecture** through Event types +6. **Tool execution flexibility** through Tool types with hooks + +All types are designed for: +- **Interoperability** with LLM APIs +- **Extensibility** for custom message types +- **Performance** with immutable structs where possible +- **Debuggability** through rich event system diff --git a/learning/05-SESSION_MANAGEMENT.md b/learning/05-SESSION_MANAGEMENT.md new file mode 100644 index 0000000..bac0b4f --- /dev/null +++ b/learning/05-SESSION_MANAGEMENT.md @@ -0,0 +1,763 @@ +# AgentCore.jl - Session Management Deep Dive + +## Session Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Session Layer │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Session = Tree of Entries │ +│ │ +│ Each entry represents a change in conversation state │ +│ │ +│ Branch Navigation: │ +│ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │ +│ │ E1 │────▶│ E2 │────▶│ E3 │────▶│ E4 │────▶│ E5 │ (current leaf) │ +│ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ │ +│ │ │ │ │ │ │ +│ ▼ ▼ ▼ ▼ ▼ │ +│ Message Message Compaction Message BranchSummary │ +│ │ +│ To navigate to E2 (fork point): │ +│ Session.moveTo(E2) │ +│ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │ +│ │ E1 │────▶│ E2 │────▶│ E3' │────▶│ E4' │────▶│ E5' │ (new branch) │ +│ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ │ +│ │ │ │ +│ │ ▼ create BranchSummary │ +│ │ ┌─────┐ │ +│ └──────│ E6 │ (branch summary) │ +│ └─────┘ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +## Entry Types + +```julia +abstract type SessionTreeEntry end +``` + +### 1. MessageEntry + +```julia +struct MessageEntry <: SessionTreeEntry + type::String # "message" + id::String # Unique entry ID + parent_id::Union{String, Nothing} + timestamp::String # ISO 8601 timestamp + message::AgentMessage # The actual message +end +``` + +**Represents**: A user, assistant, or tool message + +### 2. ThinkingLevelChangeEntry + +```julia +struct ThinkingLevelChangeEntry <: SessionTreeEntry + type::String # "thinking_level_change" + id::String + parent_id::Union{String, Nothing} + timestamp::String + thinking_level::String # "off", "minimal", "low", "medium", etc. +end +``` + +**Represents**: Change in model thinking level + +### 3. ModelChangeEntry + +```julia +struct ModelChangeEntry <: SessionTreeEntry + type::String # "model_change" + id::String + parent_id::Union{String, Nothing} + timestamp::String + provider::String # "openai", "anthropic", etc. + model_id::String # Model identifier +end +``` + +**Represents**: Change in model + +### 4. ActiveToolsChangeEntry + +```julia +struct ActiveToolsChangeEntry <: SessionTreeEntry + type::String # "active_tools_change" + id::String + parent_id::Union{String, Nothing} + timestamp::String + active_tool_names::Vector{String} +end +``` + +**Represents**: Change in active tools + +### 5. CompactionEntry + +```julia +struct CompactionEntry <: SessionTreeEntry + type::String # "compaction" + id::String + parent_id::Union{String, Nothing} + timestamp::String + summary::String # Summary of compacted history + first_kept_entry_id::Union{String, Nothing} + tokens_before::Int64 # Context size before compaction + retained_tail::Union{Vector{AgentMessage}, Nothing} + details::Union{Any, Nothing} + usage::Union{Usage, Nothing} + from_hook::Bool # Whether triggered by hook +end +``` + +**Represents**: Context window compression + +**Key fields**: +- `summary`: Summary of removed messages +- `first_kept_entry_id`: First entry that was kept +- `tokens_before`: Context size before compaction +- `retained_tail`: Messages kept after compaction point + +### 6. BranchSummaryEntry + +```julia +struct BranchSummaryEntry <: SessionTreeEntry + type::String # "branch_summary" + id::String + parent_id::Union{String, Nothing} + timestamp::String + from_id::String # Branch point entry ID + summary::String # Summary of branch history + details::Union{Any, Nothing} + usage::Union{Usage, Nothing} + from_hook::Bool +end +``` + +**Represents**: Branch point with summary + +### 7. CustomEntry + +```julia +struct CustomEntry <: SessionTreeEntry + type::String # Custom type + id::String + parent_id::Union{String, Nothing} + timestamp::String + custom_type::String + data::Union{Any, Nothing} +end +``` + +**Represents**: Custom application-specific data + +### 8. CustomMessageEntry + +```julia +struct CustomMessageEntry <: SessionTreeEntry + type::String + id::String + parent_id::Union{String, Nothing} + timestamp::String + custom_type::String + content::String + details::Union{Any, Nothing} + display::Bool +end +``` + +**Represents**: Custom message to display to user + +### 9. LabelEntry + +```julia +struct LabelEntry <: SessionTreeEntry + type::String + id::String + parent_id::Union{String, Nothing} + timestamp::String + target_id::String # Entry being labeled + label::Union{String, Nothing} +end +``` + +**Represents**: Label/note on an entry + +### 10. SessionInfoEntry + +```julia +struct SessionInfoEntry <: SessionTreeEntry + type::String + id::String + parent_id::Union{String, Nothing} + timestamp::String + name::Union{String, Nothing} +end +``` + +**Represents**: Session metadata (name, etc.) + +### 11. LeafEntry + +```julia +struct LeafEntry <: SessionTreeEntry + type::String + id::String + parent_id::Union{String, Nothing} + timestamp::String + target_id::Union{String, Nothing} +end +``` + +**Represents**: Change in current leaf (branch pointer) + +## Session Storage Interface + +```julia +abstract type SessionStorage{T<:SessionMetadata} end +``` + +### Storage Methods + +```julia +# Metadata +getMetadata(storage::SessionStorage)::Promise{T} + +# Leaf management +getLeafId(storage::SessionStorage)::Promise{Union{String, Nothing}} +setLeafId(storage::SessionStorage, leaf_id::String)::Promise{Nothing} + +# Entry management +createEntryId(storage::SessionStorage)::Promise{String} +appendEntry(storage::SessionStorage, entry::SessionTreeEntry)::Promise{Nothing} +getEntry(storage::SessionStorage, id::String)::Promise{Union{SessionTreeEntry, Nothing}} + +# Query +findEntries(storage::SessionStorage, type::String)::Promise{Vector{SessionTreeEntry}} +getLabel(storage::SessionStorage, id::String)::Promise{Union{String, Nothing}} +getSessionName(storage::SessionStorage)::Promise{Union{String, Nothing}} + +# Branch navigation +getPathToRootOrCompaction( + storage::SessionStorage, + leaf_id::String, +)::Promise{Vector{SessionTreeEntry}} + +getEntries(storage::SessionStorage, options::Dict{String, Any})::Promise{Vector{SessionTreeEntry}} + +# Stats +getSessionStats(storage::SessionStorage)::Promise{SessionStats} +``` + +## JsonlSessionStorage + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ JSONL Storage Format │ +└─────────────────────────────────────────────────────────────────────────────┘ + +File: session.jsonl + +Entry 1 (Metadata): +{"type":"session","id":"meta_1","created_at":"2024-01-01T00:00:00Z","cwd":"/path","path":"/path/session.jsonl"} + +Entry 2 (Message): +{"type":"message","id":"msg_1","parent_id":null,"timestamp":"2024-01-01T00:00:01Z","message":{"role":"user","content":[{"type":"text","text":"Hello"}]}} + +Entry 3 (Thinking Level): +{"type":"thinking_level_change","id":"tl_1","parent_id":"msg_1","timestamp":"2024-01-01T00:00:02Z","thinking_level":"medium"} + +Entry 4 (Model Change): +{"type":"model_change","id":"mc_1","parent_id":"tl_1","timestamp":"2024-01-01T00:00:03Z","provider":"openai","model_id":"gpt-4"} + +Entry 5 (Compaction): +{"type":"compaction","id":"comp_1","parent_id":"mc_1","timestamp":"2024-01-01T00:00:04Z","summary":"Previous messages summarized...","first_kept_entry_id":"msg_3","tokens_before":100000,"tokens_after":50000} + +Entry 6 (Branch Summary): +{"type":"branch_summary","id":"branch_1","parent_id":"comp_1","timestamp":"2024-01-01T00:00:05Z","from_id":"msg_3","summary":"Branch from message 3"} + +Entry 7 (Active Tools): +{"type":"active_tools_change","id":"tools_1","parent_id":"branch_1","timestamp":"2024-01-01T00:00:06Z","active_tool_names":["bash","read"]} + +Entry 8 (Leaf): +{"type":"leaf","id":"leaf_1","parent_id":"tools_1","timestamp":"2024-01-01T00:00:07Z","target_id":"msg_5"} + +Notes: +- Each line is a JSON object (JSONL format) +- parent_id references previous entry (linked list structure) +- Leaf entry points to current position in tree +- To fork, create new branch from any entry +``` + +## InMemorySessionStorage + +```julia +mutable struct InMemorySessionStorage + metadata::SessionMetadata + leaf_id::Union{String, Nothing} + entries::Dict{String, SessionTreeEntry} + labels::Dict{String, String} +end +``` + +**Purpose**: Testing and temporary sessions + +**Advantages**: +- Fast (no I/O) +- Easy to inspect +- Perfect for tests + +## Session Class + +```julia +mutable struct Session{T<:SessionMetadata} + storage::SessionStorage{T} + context_build_options::SessionContextBuildOptions +end +``` + +### Session Methods + +#### appendMessage() + +```julia +function appendMessage(session::Session, message::AgentMessage)::String + entry = MessageEntry( + "message", + createEntryId(session.storage), + getLeafId(session.storage), + create_timestamp(), + message, + ) + return appendTypedEntry(session, entry) +end +``` + +**Usage**: +```julia +session = Session(storage) + +# Add user message +user_id = appendMessage(session, UserMessage("user", [TextContent("Hello")], timestamp)) + +# Add assistant message +assistant_id = appendMessage(session, AssistantMessage(...)) + +# Add tool result +tool_id = appendMessage(session, ToolResultMessage(...)) +``` + +#### appendThinkingLevelChange() + +```julia +function appendThinkingLevelChange( + session::Session, + thinking_level::String, +)::String + entry = ThinkingLevelChangeEntry( + "thinking_level_change", + createEntryId(session.storage), + getLeafId(session.storage), + create_timestamp(), + thinking_level, + ) + return appendTypedEntry(session, entry) +end +``` + +#### appendCompaction() + +```julia +function appendCompaction( + session::Session, + summary::String, + first_kept_entry_id::Union{String, Nothing}, + tokens_before::Int64, + details::Union{Any, Nothing}=nothing, + from_hook::Bool=false, + usage::Union{Usage, Nothing}=nothing, + retained_tail::Union{Vector{AgentMessage}, Nothing}=nothing, +)::String + entry = CompactionEntry( + "compaction", + createEntryId(session.storage), + getLeafId(session.storage), + create_timestamp(), + summary, + first_kept_entry_id, + tokens_before, + retained_tail, + details, + usage, + from_hook, + ) + return appendTypedEntry(session, entry) +end +``` + +#### moveTo() + +```julia +function moveTo( + session::Session, + entry_id::Union{String, Nothing}, + summary::Union{Dict{String, Any}, Nothing}=nothing, +)::Union{String, Nothing} + # Set new leaf + setLeafId(session.storage, entry_id) + + # Optionally create branch summary + if !isnothing(summary) + return appendTypedEntry(session, BranchSummaryEntry( + "branch_summary", + createEntryId(session.storage), + entry_id, + create_timestamp(), + entry_id, + summary["summary"], + get(summary, "details", nothing), + get(summary, "usage", nothing), + get(summary, "from_hook", false), + )) + end + + return nothing +end +``` + +**Usage**: +```julia +# Fork from a specific point +session.moveTo(msg_3_id) + +# Branch with summary +session.moveTo( + msg_3_id, + Dict( + "summary" => "User wanted to focus on file operations", + "details" => Dict("focus" => "files"), + ) +) +``` + +## Build Session Context + +```julia +function buildSessionContext( + path_entries::Vector{SessionTreeEntry}, + options::SessionContextBuildOptions=SessionContextBuildOptions(), +)::SessionContext + state = deriveSessionContextState(path_entries) + context_entries = buildContextEntries(path_entries, options) + messages = SessionTreeEntry[] + for (i, entry) in enumerate(context_entries) + append!(messages, sessionEntryToContextMessages(entry, i, context_entries, options)) + end + return SessionContext(messages, state.thinking_level, state.model, state.active_tool_names) +end +``` + +### Context Entry Transform + +```julia +function defaultContextEntryTransform( + path_entries::Vector{SessionTreeEntry}, +)::Vector{SessionTreeEntry} + compaction = nothing + for entry in path_entries + if entry isa CompactionEntry + compaction = entry + break + end + end + + if isnothing(compaction) + return copy(path_entries) + end + + # Include compaction entry + entries = [compaction] + + # Include retained tail if present + if !isnothing(compaction.retained_tail) + compaction_idx = findfirst(e -> e.id == compaction.id, path_entries) + append!(entries, path_entries[compaction_idx+1:end]) + return entries + end + + # Otherwise include entries after first_kept_entry_id + if !isnothing(compaction.first_kept_entry_id) + found_first_kept = false + compaction_idx = findfirst(e -> e.id == compaction.id, path_entries) + for i in 1:compaction_idx-1 + entry = path_entries[i] + if entry.id == compaction.first_kept_entry_id + found_first_kept = true + end + if found_first_kept + push!(entries, entry) + end + end + end + + # Include entries after compaction + compaction_idx = findfirst(e -> e.id == compaction.id, path_entries) + append!(entries, path_entries[compaction_idx+1:end]) + + return entries +end +``` + +### Session Entry to Context Messages + +```julia +function sessionEntryToContextMessages( + entry::SessionTreeEntry, + index::Int64, + entries::Vector{SessionTreeEntry}, + options::SessionContextBuildOptions=SessionContextBuildOptions(), +)::Vector{AgentMessage} + if entry isa MessageEntry + return [entry.message] + end + + if entry isa CustomMessageEntry + return [createCustomMessage( + entry.custom_type, + entry.content, + entry.display, + entry.details, + entry.timestamp, + )] + end + + if entry isa CompactionEntry + messages = [createCompactionSummaryMessage( + entry.summary, + entry.tokens_before, + entry.timestamp, + )] + if !isnothing(entry.retained_tail) + append!(messages, entry.retained_tail) + end + return messages + end + + if entry isa BranchSummaryEntry + return [createBranchSummaryMessage( + entry.summary, + entry.from_id, + entry.timestamp, + )] + end + + if entry isa CustomEntry + # Custom projectors can transform custom entries + if !isnothing(options.entry_projectors) && haskey(options.entry_projectors, entry.custom_type) + projector = options.entry_projectors[entry.custom_type] + return projector(entry, index, entries) + end + return AgentMessage[] + end + + return AgentMessage[] +end +``` + +## Branch Navigation + +``` +Scenario: User wants to explore a different path + +Initial Branch (current path): +┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ +│ E1 │────▶│ E2 │────▶│ E3 │────▶│ E4 │ (leaf) +└─────┘ └─────┘ └─────┘ └─────┘ + │ │ │ │ +Message Message Compaction Message + +Step 1: Fork from E2 +┌─────┐ ┌─────┐ ┌─────┐ +│ E1 │────▶│ E2 │─────────────────┐ +└─────┘ └─────┘ │ + │ │ │ + │ ▼ create BranchSummary│ + │ ┌─────┐ │ + │ │ E5 │ (branch summary) │ + │ └─────┘ │ + └──────────────────────────────────┘ + (new branch from E2) + +Step 2: Continue on new branch +┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ +│ E1 │────▶│ E2 │────▶│ E3' │────▶│ E4' │────▶│ E5' │ (new leaf) +└─────┘ └─────┘ └─────┘ └─────┘ └─────┘ + +Current branch now is: +[ E1, E2, E3', E4', E5' ] + +Original branch is: +[ E1, E2, E5 ] (E3, E4 are now separate branch) + +Key Points: +- Shared entries: E1, E2 +- Branch point: E2 +- Branch summary: E5 (points to E2) +- Each branch has independent tail +``` + +## Compaction Strategy + +### Why Compaction? + +LLM context windows have limits: +- GPT-4: 128K tokens +- Claude 2: 100K tokens +- Llama 2: 4K tokens + +**Problem**: Conversations grow unbounded +**Solution**: Compaction - summarize old messages + +### Compaction Process + +```julia +# 1. Identify messages to compact +# - Keep recent N messages (e.g., last 2 turns) +# - Summarize everything before + +# 2. Generate summary +# - Use LLM to summarize +# - Include key facts, decisions, user preferences + +# 3. Create CompactionEntry +# - summary: The summary text +# - first_kept_entry_id: First entry that was NOT compacted +# - tokens_before: Context size before compaction +# - retained_tail: Messages kept after compaction point + +# 4. Update storage +# - Append CompactionEntry +# - Update leaf to CompactionEntry +``` + +### Compaction Example + +```julia +# Before compaction (100K tokens): +[ + msg_1, # User: "I need to set up a project" + msg_2, # Assistant: "Sure, what language?" + msg_3, # User: "Python" + msg_4, # Assistant: "I'll create a Python project" + msg_5, # User: "With FastAPI" + msg_6, # Assistant: "Creating FastAPI project..." + msg_7, # Tool: bash("mkdir myapp") + msg_8, # Tool: write("myapp/main.py", ...) + msg_9, # Assistant: "Project created!" + msg_10, # User: "Can you add auth?" + msg_11, # Assistant: "Adding auth..." + msg_12, # User: "Use JWT" + msg_13, # Assistant: "Implementing JWT..." + msg_14, # Tool: bash("pip install jwt") + msg_15, # Tool: write("myapp/auth.py", ...) + msg_16, # Assistant: "Auth implemented!" +] + +# After compaction (20K tokens): +[ + compaction_entry, # Summary of msg_1 to msg_10 + msg_11, # Keep recent messages + msg_12, + msg_13, + msg_14, + msg_15, + msg_16, +] + +# Compaction summary: +""" +Previous conversation summary: +- User wanted to create a Python project +- Chose FastAPI framework +- Assistant created project structure in myapp/ +- User requested authentication +- Chose JWT for auth +- Assistant implemented JWT auth in myapp/auth.py +""" +``` + +## Complete Session Example + +```julia +using AgentCore + +# 1. Create storage +storage = JsonlSessionStorage( + SessionMetadata("session_1", "2024-01-01T00:00:00Z"), + "/path/to/session.jsonl", +) + +# 2. Create session +session = Session(storage) + +# 3. Add messages +msg1_id = appendMessage(session, UserMessage("user", [TextContent("Hello")], timestamp)) +msg2_id = appendMessage(session, AssistantMessage("assistant", [TextContent("Hi!")], ...)) + +# 4. Change thinking level +tl_id = appendThinkingLevelChange(session, "medium") + +# 5. Change model +mc_id = appendModelChange(session, "openai", "gpt-4") + +# 6. Add more messages +msg3_id = appendMessage(session, UserMessage("user", [TextContent("What can you do?")], timestamp)) +msg4_id = appendMessage(session, AssistantMessage("assistant", [TextContent("I can...")], ...)) + +# 7. Compact context (100K tokens → 20K) +compact_id = appendCompaction( + session, + "User asked about capabilities and assistant explained", + msg2_id, + 100000, + Dict("summary_length" => 50), + false, + usage, + [msg3, msg4], # Retained tail +) + +# 8. Fork and branch +session.moveTo(msg2_id) # Go back to msg2 + +# 9. Create new branch +branch_id = appendBranchSummary( + session, + "User changed direction to focus on file operations", + msg2_id, + Dict("focus" => "files"), +) + +# 10. Continue on new branch +msg5_id = appendMessage(session, UserMessage("user", [TextContent("Let's work with files")], timestamp)) + +# 11. Query session context +context = buildSessionContext(session) + +# 12. Get stats +stats = getSessionStats(session) +println("Messages: $(stats.message_count)") +println("Total tokens: $(stats.total_tokens)") +println("Cost: $$(stats.cost_total)") +``` + +## Best Practices + +1. **Use compaction** for long conversations to stay within context limits +2. **Create branch summaries** when forking to document divergent paths +3. **Retain tail messages** after compaction for context +4. **Track token usage** to optimize compaction timing +5. **Use InMemorySessionStorage** for testing diff --git a/learning/06-TOOLS.md b/learning/06-TOOLS.md new file mode 100644 index 0000000..d367911 --- /dev/null +++ b/learning/06-TOOLS.md @@ -0,0 +1,767 @@ +# AgentCore.jl - Tools Deep Dive + +## Tool Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Tool Layer │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ AgentTool │ +│ - name: String (identifier) │ +│ - label: String (display name) │ +│ - description: String (what it does) │ +│ - parameters: JSON schema │ +│ - execute::Function (main logic) │ +│ - prepare_arguments::Union{Function, Nothing} │ +│ - execution_mode::Union{ToolExecutionMode, Nothing} │ +└─────────────────────────────────────────────────────────────────────────────┘ + │ + ┌───────────────┼───────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ + │ BashTool │ │ ReadTool │ │ WriteTool │ + │ - bash() │ │ - read() │ │ - write() │ + └─────────────┘ └─────────────┘ └─────────────┘ + ┌─────────────┐ + │ EditTool │ + │ - edit() │ + └─────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ Tool Execution Flow │ +└─────────────────────────────────────────────────────────────────────────────┘ + + Assistant Message + ┌────────────────────────────────────────────────────────┐ + │ AssistantMessage: │ + │ content: [ │ + │ TextContent("I'll check the files..."), │ + │ ToolCall("bash", {command: "ls -la"}), │ + │ ToolCall("read", {path: "README.md"}) │ + │ ] │ + └────────────────────────────────────────────────────────┘ + │ + ▼ + ┌────────────────────────────────────────────────────────┐ + │ AgentLoop.executeToolCalls() │ + │ - Extract ToolCalls from message content │ + │ - Determine execution mode (sequential/parallel) │ + └────────────────────────────────────────────────────────┘ + │ + ├─► executeToolCallsSequential() + │ (for tools that require order) + │ + └─► executeToolCallsParallel() + (for independent tools) + + │ + ├─► prepareToolCall() + │ - before_tool_call hook (optional) + │ - validate arguments + │ - prepare arguments (optional) + │ + ├─► execute() + │ - Tool-specific logic + │ - Return AgentToolResult + │ + ├─► finalizeExecutedToolCall() + │ - after_tool_call hook (optional) + │ + └─► createToolResultMessage() + - Emit ToolResultMessage + + │ + ▼ + ┌────────────────────────────────────────────────────────┐ + │ ToolResultMessage │ + │ - tool_call_id: "ref to original ToolCall" │ + │ - tool_name: "bash" │ + │ - content: [TextContent("file1.md\nfile2.md\n")] │ + │ - is_error: false │ + └────────────────────────────────────────────────────────┘ + │ + ▼ + ┌────────────────────────────────────────────────────────┐ + │ AgentState.messages.append(tool_result) │ + │ - Next turn: LLM sees tool results │ + └────────────────────────────────────────────────────────┘ +``` + +## Built-in Tools + +### 1. BashTool + +```julia +struct BashToolOptions{TContext} + command_prefix::Union{String, Nothing} + prepare::Union{BashPrepare{TContext}, Nothing} +end + +struct BashPrepare{TContext} + function::Function + context::TContext + signal::Union{Any, Nothing} +end + +struct BashToolDetails + truncation::Union{Any, Nothing} + full_output_path::Union{String, Nothing} +end +``` + +#### createBashTool() + +```julia +function createBashTool{TContext}(options::Union{BashToolOptions{TContext}, Nothing}=nothing) + return AgentTool( + "bash", + "bash", + "Execute a bash command in the current working directory.", + Dict{String, Any}(), + (tool_call_id, params, signal, on_update, context) -> begin + # Execute command + result = executeBashCommand(params, signal, on_update) + + # Return result + return AgentToolResult( + [TextContent(result.output)], + BashToolDetails(result.truncation, result.full_path), + nothing, + nothing, + result.terminate, + ) + end, + nothing, # prepare_arguments + nothing, # execution_mode (default: use config) + ) +end +``` + +**Parameters Schema**: +```json +{ + "command": "string", + "timeout": "number (optional)", + "cwd": "string (optional)", + "env": "object (optional)" +} +``` + +**Example**: +```julia +# Create tool +bash_tool = createBashTool() + +# Agent receives command +tool_call = ToolCall("tool", "tc1", "bash", Dict( + "command" => "ls -la", + "timeout" => 30 +), nothing) + +# Execute +result = bash_tool.execute( + "tc1", + Dict("command" => "ls -la", "timeout" => 30), + nothing, + on_update, # Callback for streaming output + nothing, +) + +# Result +AgentToolResult( + [TextContent("total 12\n-rw-r--r-- 1 user user 100 Jan 1 file1.md\n-rw-r--r-- 1 user user 200 Jan 2 file2.md\n")], + BashToolDetails(truncation_info, nothing), + nothing, + nothing, + nothing, +) +``` + +### 2. ReadTool + +```julia +struct ReadToolOptions{TContext} + max_size::Union{Int64, Nothing} + max_lines::Union{Int64, Nothing} + image_processor::Union{ReadImageProcessor, Nothing} + prepare::Union{ReadPrepare{TContext}, Nothing} +end + +struct ReadImageProcessor + function::Function + context::Any +end + +struct ReadImageProcessorResult + content::Vector{MessageContent} + usage::Union{Usage, Nothing} +end +``` + +#### createReadTool() + +```julia +function createReadTool{TContext}(options::Union{ReadToolOptions{TContext}, Nothing}=nothing) + return AgentTool( + "read", + "read", + "Read a file from the file system.", + Dict{String, Any}(), + (tool_call_id, params, signal, on_update, context) -> begin + # Read file + result = readFileSystem(params, signal, options) + + # Process content + content = if isImage(params.path) + # Image processing + image_result = options.image_processor.function(result.path, context) + image_result.content + else + # Text content + [TextContent(result.content)] + end + + return AgentToolResult( + content, + ReadToolDetails(result.size, result.truncated, result.full_path), + nothing, + nothing, + nothing, + ) + end, + nothing, + nothing, + ) +end +``` + +**Parameters Schema**: +```json +{ + "path": "string" +} +``` + +**Example**: +```julia +# Create tool +read_tool = createReadTool() + +# Agent requests to read file +tool_call = ToolCall("tool", "tc2", "read", Dict( + "path" => "src/main.jl" +), nothing) + +# Execute +result = read_tool.execute("tc2", Dict("path" => "src/main.jl"), nothing, nothing, nothing) + +# Result +AgentToolResult( + [TextContent("module Main\nfunction main()\n println(\"Hello\")\nend\nend\n")], + ReadToolDetails(1234, false, "/path/to/src/main.jl"), + nothing, + nothing, + nothing, +) +``` + +### 3. WriteTool + +```julia +struct WriteToolInput + path::String + content::String +end +``` + +#### createWriteTool() + +```julia +function createWriteTool{TContext}(options::Union{WriteToolOptions{TContext}, Nothing}=nothing) + return AgentTool( + "write", + "write", + "Write content to a file.", + Dict{String, Any}(), + (tool_call_id, params, signal, on_update, context) -> begin + # Write file + result = writeToFile(params, signal) + + return AgentToolResult( + [TextContent(result.message)], + nothing, + nothing, + nothing, + nothing, + ) + end, + nothing, + nothing, + ) +end +``` + +**Parameters Schema**: +```json +{ + "path": "string", + "content": "string" +} +``` + +**Example**: +```julia +# Create tool +write_tool = createWriteTool() + +# Agent wants to write file +tool_call = ToolCall("tool", "tc3", "write", Dict( + "path" => "output.txt", + "content" => "Hello World" +), nothing) + +# Execute +result = write_tool.execute("tc3", Dict( + "path" => "output.txt", + "content" => "Hello World" +), nothing, nothing, nothing) + +# Result +AgentToolResult( + [TextContent("File written: output.txt (11 bytes)")], + nothing, + nothing, + nothing, + nothing, +) +``` + +### 4. EditTool + +```julia +struct EditToolInput + path::String + find::String + replacement::String +end + +struct EditToolDetails + edits::Vector{Edit} + before_content::String + after_content::String +end +``` + +#### createEditTool() + +```julia +function createEditTool{TContext}(options::Union{EditToolOptions{TContext}, Nothing}=nothing) + return AgentTool( + "edit", + "edit", + "Edit a file by finding and replacing text.", + Dict{String, Any}(), + (tool_call_id, params, signal, on_update, context) -> begin + # Read file + before_content = read(params.path) + + # Apply edit + after_content = replace(before_content, params.find => params.replacement) + + # Write file + write(params.path, after_content) + + return AgentToolResult( + [TextContent("Edit applied successfully")], + EditToolDetails([Edit(params.find, params.replacement)], before_content, after_content), + nothing, + nothing, + nothing, + ) + end, + nothing, + nothing, + ) +end +``` + +**Parameters Schema**: +```json +{ + "path": "string", + "find": "string", + "replacement": "string" +} +``` + +**Example**: +```julia +# Create tool +edit_tool = createEditTool() + +# Agent wants to replace text +tool_call = ToolCall("tool", "tc4", "edit", Dict( + "path" => "README.md", + "find" => "v1.0.0", + "replacement" => "v2.0.0" +), nothing) + +# Execute +result = edit_tool.execute("tc4", Dict( + "path" => "README.md", + "find" => "v1.0.0", + "replacement" => "v2.0.0" +), nothing, nothing, nothing) + +# Result +AgentToolResult( + [TextContent("Edit applied: README.md")], + EditToolDetails([Edit("v1.0.0", "v2.0.0")], "Version 1.0.0", "Version 2.0.0"), + nothing, + nothing, + nothing, +) +``` + +## Tool Execution Hooks + +### before_tool_call + +```julia +struct BeforeToolCallContext + assistant_message::AssistantMessage + tool_call::ToolCall + args::Any + context::AgentContext +end + +struct BeforeToolCallResult + block::Union{Bool, Nothing} + reason::Union{String, Nothing} +end +``` + +**Usage**: +```julia +function myBeforeToolCall(context, signal) + tool_name = context.tool_call.name + + # Block dangerous commands + if tool_name == "bash" && contains(context.args["command"], "rm -rf /") + return BeforeToolCallResult( + true, + "Blocking dangerous command: rm -rf /" + ) + end + + # Log tool execution + println("Executing tool: $tool_name") + + return nothing # Allow execution +end + +# Configure agent +agent = Agent(Dict( + :beforeToolCall => myBeforeToolCall, +)) +``` + +### after_tool_call + +```julia +struct AfterToolCallContext + assistant_message::AssistantMessage + tool_call::ToolCall + args::Any + result::AgentToolResult + is_error::Bool + context::AgentContext +end + +struct AfterToolCallResult + content::Union{Vector{MessageContent}, Nothing} + details::Union{Any, Nothing} + is_error::Union{Bool, Nothing} + usage::Union{Usage, Nothing} + terminate::Union{Bool, Nothing} +end +``` + +**Usage**: +```julia +function myAfterToolCall(context, signal) + tool_name = context.tool_call.name + + # Modify bash output + if tool_name == "bash" + # Add timestamp to output + new_content = [ + TextContent("[Executed at $(Dates.now())]\n"), + context.result.content[1], + ] + return AfterToolCallResult( + content = new_content, + details = context.result.details, + is_error = context.is_error, + usage = context.result.usage, + terminate = context.result.terminate, + ) + end + + return nothing # Use original result +end + +# Configure agent +agent = Agent(Dict( + :afterToolCall => myAfterToolCall, +)) +``` + +### prepare_next_turn + +```julia +struct PrepareNextTurnContext + message::AssistantMessage + tool_results::Vector{ToolResultMessage} + context::AgentContext + new_messages::Vector{AgentMessage} +end + +struct AgentLoopTurnUpdate + context::Union{AgentContext, Nothing} + model::Union{Model, Nothing} + thinking_level::Union{ThinkingLevel, Nothing} +end +``` + +**Usage**: +```julia +function myPrepareNextTurn(context, signal) + # Check if we should use a different model + last_message = context.message + tool_results = context.tool_results + + # If tool execution had errors, use more capable model + has_errors = any(r -> r.is_error, tool_results) + if has_errors + return AgentLoopTurnUpdate( + context = context.context, + model = Model("gpt-4", "GPT-4", "openai", "openai", "", ...), + thinking_level = THINKING_HIGH, + ) + end + + return nothing # Keep current settings +end + +# Configure agent +agent = Agent(Dict( + :prepareNextTurn => myPrepareNextTurn, +)) +``` + +## Tool Execution Modes + +### Sequential Execution + +```julia +# Tools run one at a time, in order +# Use case: Tools that modify shared state + +# Configure tool +bash_tool = AgentTool( + "bash", + "bash", + "Execute bash command", + ..., + execute, + nothing, + EXECUTION_SEQUENTIAL, # Force sequential +) + +# Or configure globally +agent = Agent(Dict( + :toolExecution => EXECUTION_SEQUENTIAL, +)) +``` + +**Example Scenario**: +```julia +# Sequential execution (correct order) + +1. Tool 1: create_directory("build/") + └─ Creates build/ directory + +2. Tool 2: write("build/app.js", "...") + └─ Writes file to build/ + +(If parallel: might fail because build/ doesn't exist yet) +``` + +### Parallel Execution + +```julia +# Tools run concurrently +# Use case: Independent operations + +# Default behavior +agent = Agent(Dict( + :toolExecution => EXECUTION_PARALLEL, # Default +)) +``` + +**Example Scenario**: +```julia +# Parallel execution (independent operations) + +1. Tool 1: read("README.md") ─────┐ +2. Tool 2: read("CHANGELOG.md") ─┼─► Run simultaneously +3. Tool 3: read("LICENSE") ──────┘ + +(Parallel: All three read operations can happen at once) +(Sequential: Would wait for each read to complete) +``` + +## Custom Tools + +### Example: Database Tool + +```julia +function createDatabaseTool() + return AgentTool( + "database", + "database", + "Execute SQL queries against the database.", + Dict{String, Any}( + "type" => "object", + "properties" => Dict( + "query" => Dict("type" => "string"), + "params" => Dict("type" => "array", "items" => Dict("type" => "string")), + ), + "required" => ["query"], + ), + (tool_call_id, params, signal, on_update, context) -> begin + # Execute query + query = params["query"] + result = executeQuery(query) + + # Format output + output = formatQueryResult(result) + + return AgentToolResult( + [TextContent(output)], + Dict("rows_affected" => result.rows_affected), + nothing, + nothing, + nothing, + ) + end, + nothing, + EXECUTION_SEQUENTIAL, + ) +end + +# Usage +db_tool = createDatabaseTool() +agent = Agent(Dict(:tools => [db_tool])) +``` + +### Example: HTTP Request Tool + +```julia +function createHTTPTool() + return AgentTool( + "http", + "http", + "Make HTTP requests.", + Dict{String, Any}( + "type" => "object", + "properties" => Dict( + "url" => Dict("type" => "string"), + "method" => Dict("type" => "string", "enum" => ["GET", "POST", "PUT", "DELETE"]), + "body" => Dict("type" => "string"), + "headers" => Dict("type" => "object"), + ), + "required" => ["url", "method"], + ), + (tool_call_id, params, signal, on_update, context) -> begin + # Make request + url = params["url"] + method = params["method"] + body = get(params, "body", nothing) + headers = get(params, "headers", Dict()) + + response = makeHTTPRequest(method, url, body, headers) + + return AgentToolResult( + [TextContent(response.body)], + Dict( + "status_code" => response.status_code, + "headers" => response.headers, + ), + nothing, + nothing, + nothing, + ) + end, + nothing, + EXECUTION_PARALLEL, + ) +end +``` + +## Complete Example + +```julia +using AgentCore + +# 1. Create tools +bash_tool = createBashTool() +read_tool = createReadTool() +write_tool = createWriteTool() + +# 2. Configure hooks +before_hook = (context, signal) -> begin + println("About to execute: $(context.tool_call.name)") + return nothing +end + +after_hook = (context, signal) -> begin + if context.is_error + println("Tool failed: $(context.tool_call.name)") + else + println("Tool completed: $(context.tool_call.name)") + end + return nothing +end + +# 3. Create agent +agent = Agent(Dict( + :systemPrompt => "You are a helpful assistant with file system access.", + :tools => [bash_tool, read_tool, write_tool], + :beforeToolCall => before_hook, + :afterToolCall => after_hook, +)) + +# 4. Run conversation +prompt(agent, "List files in current directory and read the first one") + +# 5. Agent will: +# - Execute bash("ls -la") tool +# - Parse output to find first file +# - Execute read("path/to/file") tool +# - Return content to user +``` + +## Best Practices + +1. **Use sequential execution** for tools that depend on shared state +2. **Use parallel execution** for independent operations +3. **Implement before_tool_call hook** for logging and validation +4. **Implement after_tool_call hook** for result modification +5. **Use prepare_next_turn hook** for dynamic model/thinking level changes +6. **Return terminate=true** from tool when agent should stop +7. **Include usage statistics** in tool results when possible diff --git a/learning/07-AGENTHARNESS.md b/learning/07-AGENTHARNESS.md new file mode 100644 index 0000000..78e5ec2 --- /dev/null +++ b/learning/07-AGENTHARNESS.md @@ -0,0 +1,754 @@ +# AgentCore.jl - AgentHarness Deep Dive + +## AgentHarness Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ AgentHarness Layer │ +└─────────────────────────────────────────────────────────────────────────────┐ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ AgentHarness = Agent + Session + Resources │ +│ │ +│ ┌───────────────────────────────────────────────────────────────────────┐ │ +│ │ AgentHarness │ │ +│ │ - Manages Agent instances │ │ +│ │ - Provides Session persistence │ │ +│ │ - Manages resources (skills, prompt templates) │ │ +│ │ - Handles extension hooks │ │ +│ │ - Coordinates tool execution with context │ │ +│ └───────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌─────────────────────┼─────────────────────┐ │ +│ ▼ ▼ ▼ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ Agent │ │ SessionRepo │ │ Resources │ │ +│ │ (state, │ │ (create, │ │ (skills, │ │ +│ │ events) │ │ open, │ │ templates) │ │ +│ └──────────────┘ │ list) │ └──────────────┘ │ +│ └──────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌──────────────┐ │ +│ │ Session │ │ +│ │ (history, │ │ +│ │ branching) │ │ +│ └──────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ AgentHarnessEvent System │ +└─────────────────────────────────────────────────────────────────────────────┘ + +AgentEvent (from Agent) +├─ AgentHarnessOwnEvent +│ ├─ BeforeAgentStartEvent +│ ├─ ContextEvent +│ ├─ BeforeProviderRequestEvent +│ ├─ BeforeProviderPayloadEvent +│ ├─ AfterProviderResponseEvent +│ ├─ ToolCallEvent +│ ├─ ToolResultEvent +│ ├─ SessionBeforeCompactEvent +│ ├─ SessionCompactEvent +│ ├─ SessionBeforeTreeEvent +│ ├─ SessionTreeEvent +│ ├─ ModelUpdateEvent +│ ├─ ThinkingLevelUpdateEvent +│ ├─ ToolsUpdateEvent +│ ├─ ResourcesUpdateEvent +│ └─ ... (other session events) + +└─ AgentEvent (from AgentLoop) + ├─ AgentStartEvent / AgentEndEvent + ├─ TurnStartEvent / TurnEndEvent + ├─ MessageStartEvent / MessageEndEvent + └─ ToolExecutionStartEvent / ToolExecutionEndEvent +``` + +## AgentHarness Components + +### 1. AgentHarnessOptions + +```julia +mutable struct AgentHarnessOptions{ + TC, TSkill<:Skill, TPromptTemplate<:PromptTemplate, TTool<:AgentHarnessTool +} + session::Session + models::Any + tools::Union{Vector{TTool}, Nothing} + resources::Union{AgentHarnessResources{TSkill, TPromptTemplate}, Nothing} + system_prompt::Union{AgentHarnessSystemPrompt{TC, TSkill, TPromptTemplate, TTool}, Nothing} + stream_options::Union{AgentHarnessStreamOptions, Nothing} + retry::Union{Any, Nothing} + model::Model + thinking_level::Union{ThinkingLevel, Nothing} + active_tool_names::Union{Vector{String}, Nothing} + steering_mode::Union{QueueMode, Nothing} + follow_up_mode::Union{QueueMode, Nothing} + tool_context::Union{AgentHarnessToolContextSource{TC}, Nothing} +end +``` + +**Purpose**: Configure AgentHarness with all necessary options + +**Key fields**: +- `session`: Session instance for persistence +- `models`: Available models +- `tools`: Agent tools +- `resources`: Skills and prompt templates +- `system_prompt`: System prompt (string or function) +- `stream_options`: LLM streaming options +- `model`: Default model +- `thinking_level`: Default thinking level +- `active_tool_names`: Active tools +- `tool_context`: Context source for tools + +### 2. AgentHarnessResources + +```julia +mutable struct AgentHarnessResources{TSkill<:Skill, TPromptTemplate<:PromptTemplate} + promptTemplates::Union{Vector{TPromptTemplate}, Nothing} + skills::Union{Vector{TSkill}, Nothing} +end +``` + +**Purpose**: Load and manage skills and prompt templates + +### 3. Skill + +```julia +mutable struct Skill + name::String + description::String + content::String + filePath::String + disableModelInvocation::Bool +end +``` + +**Purpose**: Define specialized instructions for specific tasks + +**Format**: +```markdown + +{ + "name": "File Operations", + "description": "Handle file system operations", + "disable-model-invocation": false +} +--- + +# File Operations Skill + +This skill provides instructions for working with files... +``` + +### 4. PromptTemplate + +```julia +mutable struct PromptTemplate + name::String + description::Union{String, Nothing} + content::String +end +``` + +**Purpose**: Reusable prompt snippets with arguments + +**Format**: +```markdown + +{ + "description": "Generate commit message" +} +--- + +Generate a git commit message for: +$1 +$ARGUMENTS +``` + +### 5. AgentHarnessStreamOptions + +```julia +mutable struct AgentHarnessStreamOptions + transport::Union{String, Nothing} + timeout_ms::Union{Int64, Nothing} + max_retries::Union{Int64, Nothing} + max_retry_delay_ms::Union{Int64, Nothing} + headers::Union{Dict{String, String}, Nothing} + metadata::Union{Dict{String, Any}, Nothing} + cache_retention::Union{String, Nothing} +end +``` + +**Purpose**: Configure LLM API call options + +## SessionRepo Interface + +```julia +abstract type SessionRepo< + TMetadata<:SessionMetadata, + TCreateOptions, + TListOptions +> end +``` + +### Repo Methods + +```julia +# Create new session +create(repo::SessionRepo, options::TCreateOptions)::Promise{Session} + +# Open existing session +open(repo::SessionRepo, metadata::TMetadata)::Promise{Session} + +# List sessions +list(repo::SessionRepo, options::TListOptions)::Promise{Vector{TMetadata}} + +# Delete session +delete(repo::SessionRepo, metadata::TMetadata)::Promise{Nothing} + +# Fork session (create branch) +fork(repo::SessionRepo, source::TMetadata, options::Dict{String, Any})::Promise{Session} +``` + +### JsonlSessionRepo + +```julia +# JSONL-based session repository +# - Sessions stored as JSONL files +# - Supports create, open, list, delete, fork +# - Branch navigation via session tree +``` + +## Extension Hooks + +### Hook Types + +```julia +# Before agent starts +BeforeAgentStartEvent +├─ prompt: String +├─ images: Union{Vector{ImageContent}, Nothing} +├─ system_prompt: String +└─ resources: AgentHarnessResources + +BeforeAgentStartResult +├─ messages: Union{Vector{AgentMessage}, Nothing} +└─ system_prompt: Union{String, Nothing} + +# Context event +ContextEvent +└─ messages: Vector{AgentMessage} + +ContextResult +└─ messages: Vector{AgentMessage} + +# Before LLM request +BeforeProviderRequestEvent +├─ model: Model +├─ session_id: String +└─ stream_options: AgentHarnessStreamOptions + +BeforeProviderRequestResult +└─ stream_options: Union{AgentHarnessStreamOptionsPatch, Nothing} + +# Before LLM payload +BeforeProviderPayloadEvent +├─ model: Model +└─ payload: Any + +BeforeProviderPayloadResult +└─ payload: Any + +# After LLM response +AfterProviderResponseEvent +├─ status: Int64 +└─ headers: Dict{String, String} + +# Tool call +ToolCallEvent +├─ tool_call_id: String +├─ tool_name: String +└─ input: Dict{String, Any} + +ToolCallResult +├─ block: Union{Bool, Nothing} +└─ reason: Union{String, Nothing} + +# Tool result +ToolResultEvent +├─ tool_call_id: String +├─ tool_name: String +├─ input: Dict{String, Any} +├─ content: Vector{MessageContent} +├─ details: Any +├─ is_error: Bool +└─ usage: Union{Usage, Nothing} + +ToolResultPatch +├─ content: Union{Vector{MessageContent}, Nothing} +├─ details: Union{Any, Nothing} +├─ is_error: Union{Bool, Nothing} +├─ usage: Union{Usage, Nothing} +└─ terminate: Union{Bool, Nothing} + +# Session compaction +SessionBeforeCompactEvent +├─ preparation: Any +├─ branch_entries: Vector{SessionTreeEntry} +├─ custom_instructions: Union{String, Nothing} +└─ signal: Any + +SessionBeforeCompactResult +├─ cancel: Union{Bool, Nothing} +└─ compaction: Union{CompactResult, Nothing} + +SessionCompactEvent +├─ compaction_entry: CompactionEntry +└─ from_hook: Bool + +# Session tree (branching) +SessionBeforeTreeEvent +├─ preparation: Any +└─ signal: Any + +SessionBeforeTreeResult +├─ cancel: Union{Bool, Nothing} +├─ summary: Union{Dict{String, Any}, Nothing} +├─ custom_instructions: Union{String, Nothing} +├─ replace_instructions: Union{Bool, Nothing} +└─ label: Union{String, Nothing} + +SessionTreeEvent +├─ new_leaf_id: Union{String, Nothing} +├─ old_leaf_id: Union{String, Nothing} +├─ summary_entry: Union{BranchSummaryEntry, Nothing} +└─ from_hook: Union{Bool, Nothing} +``` + +### Hook Usage Examples + +#### BeforeAgentStartHook + +```julia +function beforeAgentStart(event, signal) + # Modify system prompt based on context + new_system_prompt = "$(event.system_prompt)\n\nUser prefers concise responses." + + # Prepend initial messages + initial_messages = [ + UserMessage("user", [TextContent("Context: $(event.prompt)")], timestamp), + ] + + return BeforeAgentStartResult( + initial_messages, + new_system_prompt, + ) +end + +# Configure harness +harness = AgentHarness(Dict( + :beforeAgentStart => beforeAgentStart, +)) +``` + +#### BeforeProviderPayloadHook + +```julia +function beforeProviderPayload(event, signal) + # Modify LLM payload before sending + payload = event.payload + + # Add custom metadata + payload.metadata = merge(payload.metadata, Dict( + "session_id" => event.session_id, + "timestamp" => Dates.now(), + )) + + return BeforeProviderPayloadResult(payload) +end +``` + +#### ToolCallHook + +```julia +function toolCall(event, signal) + # Block dangerous tool calls + if event.tool_name == "bash" && contains(event.input["command"], "rm -rf /") + return ToolCallResult(true, "Blocking dangerous command") + end + + # Log tool execution + println("Tool call: $(event.tool_name)") + + return nothing # Allow execution +end +``` + +#### BeforeCompactHook + +```julia +function beforeCompact(event, signal) + # Add custom instructions for compaction + custom_instructions = """ + Focus on retaining user preferences and key decisions. + Omit verbose tool outputs that don't add value. + """ + + return SessionBeforeCompactResult( + false, # Don't cancel + Dict( + "summary" => "Custom compaction with focus on user intent", + "custom_instructions" => custom_instructions, + ), + ) +end +``` + +## Tool Context + +### AgentHarnessToolContextSource + +```julia +mutable struct AgentHarnessToolContextSource{TContext} + context::Union{TContext, Function} +end +``` + +**Purpose**: Provide context to tools during execution + +### Tool Execution Context + +```julia +# Tools receive context from AgentHarness +tool.execute( + tool_call_id, + params, + signal, + on_update, + context, # From AgentHarnessToolContextSource +) + +# Context can be: +# - Static value +# - Function that returns value +``` + +## Complete Example + +```julia +using AgentCore + +# 1. Create skills +skills, skill_diagnostics = loadSkills( + execution_env, + "/path/to/skills", +) + +# 2. Create prompt templates +templates, template_diagnostics = loadPromptTemplates( + execution_env, + "/path/to/templates", +) + +# 3. Create resources +resources = AgentHarnessResources( + templates, + skills, +) + +# 4. Create session repo +repo = JsonlSessionRepo( + "/path/to/sessions", +) + +# 5. Create session +session = create(repo, Dict( + "cwd" => "/path/to/project", + "metadata" => Dict("project" => "my-project"), +)) + +# 6. Configure tools +bash_tool = createBashTool() +read_tool = createReadTool() + +tools = [bash_tool, read_tool] + +# 7. Configure hooks +hooks = Dict( + :beforeAgentStart => beforeAgentStartHook, + :beforeProviderPayload => beforePayloadHook, + :toolCall => toolCallHook, +) + +# 8. Create harness +harness = AgentHarness(Dict( + :session => session, + :models => models, + :tools => tools, + :resources => resources, + :system_prompt => "You are a helpful assistant.", + :model => Model(...), + :thinking_level => THINKING_MEDIUM, + :active_tool_names => ["bash", "read"], + :steering_mode => QUEUE_ONE_AT_A_TIME, + :follow_up_mode => QUEUE_ONE_AT_A_TIME, + :tool_context => AgentHarnessToolContextSource(context), + :stream_options => AgentHarnessStreamOptions( + transport = "auto", + timeout_ms = 30000, + max_retries = 3, + ), +)) + +# 9. Subscribe to events +subscribe(harness) do event, signal + if event isa BeforeAgentStartEvent + println("Agent starting...") + elseif event isa MessageEndEvent + println("Message: $(event.message)") + end +end + +# 10. Run conversation +harness.prompt("What files are in the current directory?") + +# 11. Wait for completion +wait_for_idle(harness) + +# 12. Manage branches +session.moveTo(some_entry_id) # Fork from entry +``` + +## Hook Execution Flow + +``` +User Code + │ + ├─► AgentHarness.prompt() + │ + ▼ +┌────────────────────────────────────────────────────────────────────────┐ +│ BeforeAgentStartEvent │ +│ ├─ User prompt │ +│ ├─ System prompt │ +│ └─ Resources │ +│ │ │ +│ └─► beforeAgentStart hook (optional) │ +│ └─► BeforeAgentStartResult (optional modifications) │ +└────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────────────────┐ +│ Agent.createLoopConfig() │ +│ └─► Merge options with hooks │ +└────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────────────────┐ +│ Agent.prompt() │ +│ └─► Start AgentLoop │ +└────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────────────────┐ +│ AgentLoop.agentLoop() │ +│ │ │ +│ ├─► transform_context hook (optional) │ +│ └─► convert_to_llm() │ +│ └─► Message[] for LLM API │ +└────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────────────────┐ +│ BeforeProviderRequestEvent │ +│ ├─ Model │ +│ ├─ Session ID │ +│ └─ Stream Options │ +│ │ │ +│ └─► beforeProviderRequest hook (optional) │ +│ └─► BeforeProviderRequestResult (optional modifications) │ +└────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────────────────┐ +│ StreamFn (LLM API call) │ +└────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────────────────┐ +│ AfterProviderResponseEvent │ +│ ├─ Status code │ +│ └─ Response headers │ +└────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────────────────┐ +│ BeforeProviderPayloadEvent │ +│ ├─ Model │ +│ └─ Payload (before sending) │ +│ │ │ +│ └─► beforeProviderPayload hook (optional) │ +│ └─► BeforeProviderPayloadResult (optional modifications) │ +└────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────────────────┐ +│ LLM API Request │ +└────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────────────────┐ +│ Assistant Message (streaming) │ +│ │ │ +│ ├─► Text deltas │ +│ └─► Tool calls │ +└────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────────────────┐ +│ Tool Execution (for each tool call) │ +│ │ │ +│ ├─► before_tool_call hook (Agent) │ +│ ├─► toolCall hook (Harness - optional) │ +│ │ └─► ToolCallResult (can block execution) │ +│ ├─► prepareToolCall() │ +│ ├─► execute() │ +│ │ └─► Tool execution with context │ +│ ├─► after_tool_call hook (Agent) │ +│ └─► toolResult hook (Harness - optional) │ +│ └─► ToolResultPatch (can modify result) │ +└────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────────────────┐ +│ AgentLoop continues with tool results │ +│ │ │ +│ ├─► Next LLM call with tool results │ +│ └─► Or end of conversation │ +└────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────────────────────┐ +│ AgentEndEvent │ +│ └─► Final messages in session │ +└────────────────────────────────────────────────────────────────────────┘ +``` + +## Session Management with Harness + +```julia +# Create harness with session repo +repo = JsonlSessionRepo("/path/to/sessions") + +# Create session +session = create(repo, Dict( + "cwd" => "/path/to/project", + "metadata" => Dict("name" => "my-session"), +)) + +# Or open existing session +metadata = JsonlSessionMetadata(...) +session = open(repo, metadata) + +# List sessions +sessions = list(repo, Dict()) +for meta in sessions + println("Session: $(meta.id)") +end + +# Delete session +delete(repo, metadata) + +# Fork session (branch) +forked_session = fork(repo, source_metadata, Dict( + "summary" => "Branch for feature X", +)) +``` + +## Resources Management + +```julia +# Load skills from directory +skills, diagnostics = loadSkills( + execution_env, + "/path/to/skills", +) + +# Load prompt templates from directory +templates, diagnostics = loadPromptTemplates( + execution_env, + "/path/to/templates", +) + +# Create resources +resources = AgentHarnessResources( + templates, + skills, +) + +# Use in harness +harness = AgentHarness(Dict( + :resources => resources, +)) +``` + +## Best Practices + +1. **Use hooks for logging and validation** + - `beforeAgentStart` for initialization + - `beforeProviderPayload` for custom metadata + - `toolCall` for blocking dangerous operations + +2. **Organize skills by domain** + - File operations + - Database queries + - HTTP requests + - Git operations + +3. **Use templates for common patterns** + - Commit message generation + - Code review instructions + - Testing prompts + +4. **Manage sessions carefully** + - Compact periodically + - Use branches for exploration + - Clean up old sessions + +5. **Monitor resource usage** + - Track token counts + - Watch API costs + - Optimize tool execution + +## Troubleshooting + +### Hook not being called + +```julia +# Check hook is registered +if isnothing(harness.beforeAgentStart) + println("Hook not registered") +end +``` + +### Session not persisting + +```julia +# Check repo is configured +if isnothing(harness.repo) + println("No repo configured") +end +``` + +### Resources not loading + +```julia +# Check diagnostics +for diag in skill_diagnostics + println("Skill warning: $(diag.message)") +end +``` diff --git a/learning/08-EXAMPLES.md b/learning/08-EXAMPLES.md new file mode 100644 index 0000000..0e5a63c --- /dev/null +++ b/learning/08-EXAMPLES.md @@ -0,0 +1,893 @@ +# AgentCore.jl - Examples and Patterns + +## Quick Start Examples + +### Example 1: Basic Conversation + +```julia +using AgentCore + +# Create model +model = Model( + "gpt-4", + "GPT-4", + "openai", + "openai", + "https://api.openai.com/v1", + true, + ["text"], + ModelCost(0.00003, 0.00006, 0.0, 0.0), + 128000, + 4096, +) + +# Create tools +bash_tool = createBashTool() + +# Create agent +agent = Agent(Dict( + :systemPrompt => "You are a helpful assistant.", + :model => model, + :tools => [bash_tool], + :thinkingLevel => THINKING_MEDIUM, + :toolExecution => EXECUTION_PARALLEL, +)) + +# Subscribe to events +subscribe(agent) do event, signal + if event isa MessageEndEvent + println("Agent: $(event.message)") + end +end + +# Start conversation +prompt(agent, "What's in the current directory?") + +# Wait for completion +wait_for_idle(agent) + +# Get final state +state = get_state(agent) +println("Total messages: $(length(state.messages))") +``` + +### Example 2: Conversation with Memory + +```julia +# Create session storage +storage = JsonlSessionStorage( + JsonlSessionMetadata( + "session_1", + "2024-01-01T00:00:00Z", + "/path/to/project", + "/path/to/session.jsonl", + nothing, + Dict("project" => "my-project"), + ), + "/path/to/session.jsonl", +) + +# Create session +session = Session(storage) + +# Create agent with session +agent = Agent(Dict( + :systemPrompt => "You are a helpful assistant.", + :model => model, + :tools => [bash_tool], + :sessionId => session.getMetadata().id, +)) + +# Add messages to session +function addToSession(session, message) + appendMessage(session, message) +end + +# Start conversation +prompt(agent, "Hello, my name is Alice.") + +# Continue conversation (messages persist in session) +prompt(agent, "What's the weather like today?") + +# Check session stats +stats = getSessionStats(session) +println("Messages: $(stats.message_count)") +println("Total tokens: $(stats.total_tokens)") +``` + +### Example 3: Steering and Follow-Up + +```julia +# Start conversation +prompt(agent, "Create a Python project.") + +# User wants to redirect +steer(agent, UserMessage("user", [TextContent("Actually, let's use Node.js instead")], timestamp)) + +# Wait for redirection +wait_for_idle(agent) + +# Agent would normally stop, but user has more +prompt(agent, "Wait, there's one more thing...") +followUp(agent, UserMessage("user", [TextContent("Can you add tests?")], timestamp)) + +# Continue until completion +while hasQueuedMessages(agent) + wait_for_idle(agent) +end +``` + +### Example 4: Branching Conversations + +```julia +# Initial conversation +prompt(agent, "I want to build a web app.") + +# User decides to explore a different path +session.moveTo(msg_3_id) # Go back to message 3 + +# Create branch +appendBranchSummary( + session, + "User decided to explore mobile app instead", + msg_3_id, + Dict("focus" => "mobile"), +) + +# Continue on new branch +prompt(agent, "Let's build a mobile app instead.") + +# Check branches +branch = getBranch(session) +println("Current branch has $(length(branch)) entries") +``` + +## Advanced Patterns + +### Pattern 1: Long-Running Agent with Compaction + +```julia +# Configure compaction settings +MAX_TOKENS = 120000 # Stay under 128K limit +COMPACTION_THRESHOLD = 100000 + +# Agent loop with compaction +function runAgentWithCompaction(agent, session) + while true + # Get current token count + stats = getSessionStats(session) + + if stats.total_tokens > COMPACTION_THRESHOLD + # Compact session + compactSession(session) + end + + # Check if agent is idle + if !hasQueuedMessages(agent) && !isnothing(agent.active_run) + break + end + end +end + +function compactSession(session) + # Get current branch + branch = getBranch(session) + + # Calculate tokens to compact + total_tokens = 0 + for entry in branch + if entry isa MessageEntry + total_tokens += estimateTokens(entry.message) + end + end + + if total_tokens < COMPACTION_THRESHOLD + return + end + + # Identify messages to compact + messages_to_compact = [] + tokens_to_keep = 50000 # Keep recent 50K tokens + + for entry in branch + if entry isa MessageEntry + msg_tokens = estimateTokens(entry.message) + if tokens_to_keep > 0 + tokens_to_keep -= msg_tokens + else + push!(messages_to_compact, entry) + end + end + end + + # Generate summary + summary = generateSummary(messages_to_compact) + + # Create compaction entry + appendCompaction( + session, + summary, + messages_to_compact[end].id, + total_tokens, + ) + + println("Compacted $(length(messages_to_compact)) messages") +end + +function estimateTokens(message::AgentMessage)::Int64 + # Simple estimation: ~4 chars per token + content = if message isa UserMessage + join([c.text for c in message.content if c isa TextContent]) + elseif message isa AssistantMessage + join([c.text for c in message.content if c isa TextContent]) + elseif message isa ToolResultMessage + join([c.text for c in message.content if c isa TextContent]) + else + "" + end + + return ceil(Int, length(content) / 4) +end + +function generateSummary(messages::Vector{MessageEntry})::String + # Use LLM to generate summary + summary = "Conversation summary:" + for msg in messages + summary *= "\n- $(msg.message)" + end + return summary +end +``` + +### Pattern 2: Custom Tool with Context + +```julia +# Define context type +struct DatabaseContext + connection::Any + user::String +end + +# Create tool with context +function createDatabaseTool() + return AgentTool( + "database", + "database", + "Execute SQL queries", + Dict{String, Any}(), + (tool_call_id, params, signal, on_update, context) -> begin + if !isa(context, DatabaseContext) + return AgentToolResult( + [TextContent("Error: Database context not provided")], + nothing, + nothing, + nothing, + true, # terminate + ) + end + + # Execute query + query = params["query"] + result = executeQuery(context.connection, query) + + return AgentToolResult( + [TextContent(formatResult(result))], + Dict("user" => context.user), + nothing, + nothing, + nothing, + ) + end, + nothing, + EXECUTION_SEQUENTIAL, + ) +end + +# Use tool with context +db_context = DatabaseContext(connection, "alice") + +harness = AgentHarness(Dict( + :tools => [createDatabaseTool()], + :tool_context => AgentHarnessToolContextSource(db_context), +)) +``` + +### Pattern 3: Dynamic Model Selection + +```julia +# Hook to change model based on task +function dynamicModelSelection(context, signal) + # Check message content + last_message = context.message + + # If complex task, use more capable model + if contains(join(last_message.content), "analyze") + return AgentLoopTurnUpdate( + context = context.context, + model = Model("gpt-4", "GPT-4", "openai", ...), + thinking_level = THINKING_HIGH, + ) + end + + # Otherwise use cheaper model + return AgentLoopTurnUpdate( + context = context.context, + model = Model("gpt-3.5", "GPT-3.5", "openai", ...), + thinking_level = THINKING_MEDIUM, + ) +end + +# Configure agent +agent = Agent(Dict( + :prepareNextTurn => dynamicModelSelection, +)) +``` + +### Pattern 4: Rate Limiting + +```julia +# Rate limiter +struct RateLimiter + calls_per_minute::Int + last_calls::Vector{DateTime} +end + +function RateLimiter(calls_per_minute::Int) + return RateLimiter(calls_per_minute, DateTime[]) +end + +function rateLimit(limiter::RateLimiter) + now = Dates.now() + + # Remove old calls + limiter.last_calls = filter( + c -> Dates.value(now - c) / 1000 < 60, + limiter.last_calls, + ) + + # Check limit + if length(limiter.last_calls) >= limiter.calls_per_minute + return false + end + + # Record call + push!(limiter.last_calls, now) + return true +end + +# Use in hook +limiter = RateLimiter(60) # 60 calls per minute + +function rateLimitHook(event, signal) + if !rateLimit(limiter) + return BeforeProviderPayloadResult(event.payload) # Still send, but track + end + + return BeforeProviderPayloadResult(event.payload) +end + +# Configure +agent = Agent(Dict( + :beforeProviderPayload => rateLimitHook, +)) +``` + +### Pattern 5: Multi-Step Tool Execution + +```julia +# Tool that requires multiple steps +function createMultiStepTool() + return AgentTool( + "multistep", + "multistep", + "Multi-step task", + Dict{String, Any}(), + (tool_call_id, params, signal, on_update, context) -> begin + # Step 1: Prepare + on_update("Preparing...") + prepare_result = prepareStep(params) + + # Step 2: Execute + on_update("Executing...") + execute_result = executeStep(prepare_result, params) + + # Step 3: Finalize + on_update("Finalizing...") + finalize_result = finalizeStep(execute_result) + + return AgentToolResult( + [TextContent(finalize_result)], + Dict("steps" => 3), + nothing, + nothing, + nothing, + ) + end, + nothing, + EXECUTION_SEQUENTIAL, + ) +end +``` + +### Pattern 6: Image Processing + +```julia +# Create read tool with image support +image_processor = ReadImageProcessor( + (path, context) -> begin + # Load image + image_data = readImage(path) + + # Process with vision model + result = processImageWithVision(image_data) + + return ReadImageProcessorResult( + [TextContent(result.description)], + result.usage, + ) + end, + context, +) + +read_tool = createReadTool(Dict( + "image_processor" => image_processor, +)) +``` + +### Pattern 7: Session Navigation + +```julia +# Navigate to specific point +session.moveTo(entry_id) + +# Get branch from specific point +branch = getBranch(session, entry_id) + +# Create label for easy navigation +appendLabel(session, entry_id, "important-decision") + +# Find labeled entry +label = getLabel(session, "important-decision") + +# Build context from branch +context = buildSessionContext(session) + +# Get specific messages +messages = sessionEntryToContextMessages(entry, index, entries) +``` + +### Pattern 8: Batch Processing + +```julia +# Process multiple prompts in batch +prompts = [ + "What is Julia?", + "What is JavaScript?", + "What is Python?", +] + +results = [] + +for prompt_text in prompts + # Create fresh agent for each prompt + agent = Agent(Dict( + :systemPrompt => "You are a helpful assistant.", + :model => model, + :tools => [bash_tool], + )) + + # Run prompt + prompt(agent, prompt_text) + wait_for_idle(agent) + + # Get result + state = get_state(agent) + last_message = state.messages[end] + + push!(results, last_message) + + # Clean up + reset!(agent) +end + +# Process results +for result in results + println("Result: $(result)") +end +``` + +### Pattern 9: Custom Event Handling + +```julia +# Custom event types +struct CustomEvent <: AgentEvent + data::Any +end + +# Custom event handler +function customEventHandler(event, signal) + if event isa CustomEvent + println("Custom event: $(event.data)") + end +end + +# Subscribe to custom events +subscribe(agent) do event, signal + customEventHandler(event, signal) +end + +# Emit custom event +emit(CustomEvent("custom data")) +``` + +### Pattern 10: Error Handling + +```julia +# Hook for error handling +function errorHook(context, signal) + if context isa PrepareNextTurnContext + last_message = context.message + + if last_message.stop_reason == "error" + println("Error in conversation: $(last_message.error_message)") + + return AgentLoopTurnUpdate( + context = context.context, + model = context.context.model, + thinking_level = THINKING_HIGH, # Use more capable model + ) + end + end + + return nothing +end + +# Use in agent +agent = Agent(Dict( + :prepareNextTurn => errorHook, +)) +``` + +## Testing Patterns + +### Unit Testing + +```julia +# Test tool execution +@testset "Bash tool" begin + tool = createBashTool() + + # Test successful execution + result = tool.execute("tc1", Dict("command" => "echo hello"), nothing, nothing, nothing) + @test result.content[1].text == "hello\n" + @test result.details === nothing + + # Test error handling + result = tool.execute("tc2", Dict("command" => "exit 1"), nothing, nothing, nothing) + @test result.terminate === true +end + +# Test agent with mock LLM +@testset "Agent with mock" begin + # Mock stream function + function mockStreamFn(model, context, options) + # Return mock response + return MockResponse([TextContent("Hello!")]) + end + + agent = Agent(Dict( + :stream_fn => mockStreamFn, + :systemPrompt => "You are a helpful assistant.", + :model => model, + )) + + # Test prompt + prompt(agent, "Hello") + wait_for_idle(agent) + + # Verify result + state = get_state(agent) + @test length(state.messages) == 2 # User + Assistant +end +``` + +### Integration Testing + +```julia +# Test full conversation flow +@testset "Full conversation" begin + # Create session storage + storage = InMemorySessionStorage(...) + session = Session(storage) + + # Create agent + agent = Agent(Dict( + :systemPrompt => "You are a helpful assistant.", + :model => model, + :tools => [bash_tool], + :sessionId => session.getMetadata().id, + )) + + # Run conversation + prompt(agent, "What's in the directory?") + wait_for_idle(agent) + + # Verify session + context = buildSessionContext(session) + @test length(context.messages) == 2 + + # Continue conversation + prompt(agent, "What's the weather?") + wait_for_idle(agent) + + # Verify growth + context = buildSessionContext(session) + @test length(context.messages) == 4 +end +``` + +## Performance Patterns + +### Pattern 1: Caching + +```julia +# Simple caching for LLM calls +struct LLMCache + cache::Dict{String, AssistantMessage} +end + +function LLMCache() + return LLMCache(Dict{String, AssistantMessage}()) +end + +function getCached(cache::LLMCache, key::String) + return get(cache.cache, key, nothing) +end + +function setCached(cache::LLMCache, key::String, value::AssistantMessage) + cache.cache[key] = value +end + +# Use in stream function +function cachedStreamFn(model, context, options) + key = generateCacheKey(context) + + cached = getCached(cache, key) + if !isnothing(cached) + return MockResponse(cached) + end + + result = actualStreamFn(model, context, options) + setCached(cache, key, result) + return result +end +``` + +### Pattern 2: Batch LLM Calls + +```julia +# Batch multiple LLM calls +function batchLLMCalls(calls::Vector{Dict}) + results = [] + + for call in calls + result = streamFunction( + call[:model], + call[:context], + call[:options], + ) + push!(results, result) + end + + return results +end + +# Use with parallel execution +tool.execute = (id, params, signal, on_update, context) -> begin + # Batch multiple LLM calls + llm_calls = [ + Dict(:model => model, :context => context1, :options => options1), + Dict(:model => model, :context => context2, :options => options2), + ] + + results = batchLLMCalls(llm_calls) + + return AgentToolResult( + [TextContent(join([r.text for r in results], "\n"))], + nothing, + nothing, + nothing, + nothing, + ) +end +``` + +### Pattern 3: Lazy Loading + +```julia +# Lazy load skills +struct LazySkills + dir::String + skills::Union{Vector{Skill}, Nothing} +end + +function LazySkills(dir) + return LazySkills(dir, nothing) +end + +function getSkills(lazy::LazySkills) + if isnothing(lazy.skills) + lazy.skills, _ = loadSkills(lazy.dir) + end + return lazy.skills +end + +# Use in harness +harness = AgentHarness(Dict( + :resources => AgentHarnessResources( + templates, + LazySkills("/path/to/skills"), + ), +)) +``` + +## Production Patterns + +### Pattern 1: Observability + +```julia +# Logging hook +function loggingHook(event, signal) + if event isa BeforeProviderRequestEvent + println("[Request] $(event.model.id)") + elseif event isa AfterProviderResponseEvent + println("[Response] Status: $(event.status)") + elseif event isa ToolExecutionEndEvent + println("[Tool] $(event.tool_name): $(event.is_error ? "error" : "success")") + end + return nothing +end + +# Metrics hook +function metricsHook(event, signal) + if event isa AgentStartEvent + metrics.start_time = Dates.now() + elseif event isa AgentEndEvent + duration = Dates.value(Dates.now() - metrics.start_time) / 1000 + println("[Metrics] Duration: $(duration)s") + end + return nothing +end +``` + +### Pattern 2: Retry Logic + +```julia +# Retry hook +function retryHook(event, signal) + if event isa AfterProviderResponseEvent && event.status >= 500 + # Server error, retry + return BeforeProviderRequestResult(Dict( + "retry" => true, + "max_retries" => 3, + )) + end + return nothing +end + +# Use in stream options +harness = AgentHarness(Dict( + :stream_options => AgentHarnessStreamOptions( + max_retries = 3, + max_retry_delay_ms = 5000, + ), + :retry => retryHook, +)) +``` + +### Pattern 3: Security + +```julia +# Security hook +function securityHook(event, signal) + if event isa ToolCallEvent + # Validate tool call + if event.tool_name == "bash" + command = event.input["command"] + + # Block dangerous commands + dangerous_patterns = ["rm -rf /", "sudo", "curl | sh"] + for pattern in dangerous_patterns + if contains(command, pattern) + return ToolCallResult(true, "Blocked dangerous command") + end + end + end + end + + return nothing +end +``` + +## Debugging Patterns + +### Pattern 1: Conversation Trace + +```julia +# Trace conversation +trace = [] + +subscribe(agent) do event, signal + if event isa MessageEndEvent + push!(trace, Dict( + "role" => event.message.role, + "content" => event.message.content, + )) + end +end + +# Run conversation +prompt(agent, "Hello") +wait_for_idle(agent) + +# Print trace +for entry in trace + println("$(entry["role"]): $(entry["content"])") +end +``` + +### Pattern 2: Tool Call Trace + +```julia +tool_trace = [] + +subscribe(agent) do event, signal + if event isa ToolExecutionStartEvent + push!(tool_trace, Dict( + "type" => "start", + "tool" => event.tool_name, + "args" => event.args, + )) + elseif event isa ToolExecutionEndEvent + push!(tool_trace, Dict( + "type" => "end", + "tool" => event.tool_name, + "error" => event.is_error, + )) + end +end +``` + +### Pattern 3: State Dump + +```julia +function dumpState(agent) + state = get_state(agent) + + println("=== Agent State ===") + println("System prompt: $(state.system_prompt)") + println("Model: $(state.model.name)") + println("Thinking level: $(state.thinking_level)") + println("Messages: $(length(state.messages))") + println("Tools: $(length(state.tools))") + println("==================") +end + +# Use after conversation +prompt(agent, "Hello") +wait_for_idle(agent) +dumpState(agent) +``` + +## Best Practices Summary + +1. **Start simple**, add complexity gradually +2. **Use hooks for customization**, not core logic +3. **Test with mock LLM** first +4. **Monitor token usage** for long conversations +5. **Use branches** for exploration +6. **Compact periodically** to stay within limits +7. **Handle errors gracefully** +8. **Log important events** +9. **Test edge cases** +10. **Profile performance** diff --git a/learning/README.md b/learning/README.md new file mode 100644 index 0000000..e27f844 --- /dev/null +++ b/learning/README.md @@ -0,0 +1,386 @@ +# AgentCore.jl - Learning Guide + +## How to Use This Documentation + +### Top-Down Learning Approach + +This documentation is organized in a **top-down** order, starting from high-level concepts and drilling down into implementation details. Follow this sequence: + +1. **Architecture Overview** - Understand the big picture +2. **Agent Component** - Learn about state management and event streaming +3. **AgentLoop Component** - Understand the core LLM interaction loop +4. **Types & Messages** - Learn the data structures +5. **Session Management** - Understand conversation history +6. **Tools** - Learn about tool execution + +### Learning Style + +- **Visual learners**: Study the ASCII diagrams +- **Hands-on learners**: Code examples provided for each section +- **Conceptual learners**: Read summaries and overviews first + +## Quick Start + +### Minimal Example + +```julia +using AgentCore + +# Create agent +agent = Agent(Dict( + :systemPrompt => "You are a helpful assistant.", + :model => Model(...), + :tools => [bash_tool], +)) + +# Run conversation +prompt(agent, "Hello!") + +# Wait for completion +wait_for_idle(agent) +``` + +### Understanding the Flow + +``` +User Code + │ + ├─► Create Agent + │ ├─ Initialize state + │ ├─ Set up queues + │ └─ Register hooks + │ + ├─► prompt("Hello") + │ ├─ Validate input + │ └─ Start AgentLoop + │ + ├─► AgentLoop (runs in thread) + │ ├─ Stream LLM response + │ ├─ Execute tools + │ └─ Emit events + │ + └─► Event handlers receive events + ├─ MessageEndEvent + ├─ ToolExecutionEndEvent + └─ AgentEndEvent +``` + +## Core Concepts + +### Agent + +**What it is**: High-level interface for LLM interactions + +**What it does**: +- Manages conversation state +- Handles event streaming +- Queues steering/follow-up messages +- Provides hooks for customization + +**Key methods**: +- `prompt()` - Start new conversation +- `continue!()` - Continue existing conversation +- `steer()` - Queue message for next turn +- `followUp()` - Queue message after stop +- `subscribe()` - Listen to events + +### AgentLoop + +**What it is**: Core LLM interaction loop + +**What it does**: +- Calls LLM API with streaming +- Executes tool calls (parallel or sequential) +- Emits lifecycle events +- Handles steering/follow-up messages + +**Key functions**: +- `agentLoop()` - Start new conversation +- `agentLoopContinue()` - Continue conversation +- `runAgentLoop()` - Internal loop execution +- `streamAssistantResponse()` - LLM API call +- `executeToolCalls()` - Tool execution + +### Session + +**What it is**: Conversation history management + +**What it does**: +- Persists messages to storage +- Supports branching +- Implements compaction +- Manages conversation tree + +**Key methods**: +- `appendMessage()` - Add message +- `appendCompaction()` - Compress history +- `moveTo()` - Navigate branches +- `buildSessionContext()` - Build context for LLM + +### Tools + +**What it is**: Functions agents can call + +**What they do**: +- Execute external operations +- Return results to agent +- Support streaming updates +- Implement hooks + +**Built-in tools**: +- `bash` - Execute shell commands +- `read` - Read files +- `write` - Write files +- `edit` - Edit files + +## Event System + +### Event Types + +``` +AgentEvent +├─ AgentStartEvent / AgentEndEvent +├─ TurnStartEvent / TurnEndEvent +├─ MessageStartEvent / MessageEndEvent +├─ MessageUpdateEvent +├─ ToolExecutionStartEvent / ToolExecutionEndEvent +└─ ToolExecutionUpdateEvent +``` + +### Event Flow + +``` +AgentStartEvent + │ + ├─ TurnStartEvent + │ ├─ MessageStartEvent (user) + │ ├─ MessageEndEvent (user) + │ ├─ MessageStartEvent (assistant) + │ ├─ MessageUpdateEvent (streaming) + │ ├─ MessageEndEvent (assistant) + │ ├─ ToolExecutionStartEvent + │ ├─ ToolExecutionEndEvent + │ └─ TurnEndEvent + │ + └─ AgentEndEvent +``` + +## Data Flow + +### Message Transformation + +``` +AgentMessage[] (internal) + │ + ├─ transform_context() (optional) + ▼ +AgentMessage[] (transformed) + │ + ├─ convert_to_llm() + ▼ +Message[] (LLM API) +``` + +### Tool Execution Flow + +``` +ToolCall (in assistant message) + │ + ├─ before_tool_call hook + ├─ prepareToolCall() + ├─ execute() + ├─ after_tool_call hook + └─ createToolResultMessage() +``` + +## Best Practices + +### 1. Use Hooks for Customization + +```julia +# Before tool call +before_hook = (context, signal) -> begin + println("Executing: $(context.tool_call.name)") + return nothing +end + +# After tool call +after_hook = (context, signal) -> begin + if context.is_error + println("Tool failed: $(context.tool_call.name)") + end + return nothing +end +``` + +### 2. Monitor Events + +```julia +subscribe(agent) do event, signal + if event isa MessageEndEvent + println("Message: $(event.message)") + elseif event isa ToolExecutionEndEvent + println("Tool completed: $(event.tool_name)") + end +end +``` + +### 3. Use Steering for Redirection + +```julia +# Agent is going wrong direction +steer(agent, UserMessage("Actually, let's do X instead")) +``` + +### 4. Use Follow-Up for Continuation + +```julia +# Agent thinks it's done, but user wants more +followUp(agent, UserMessage("Wait, there's one more thing")) +``` + +## Common Patterns + +### Pattern 1: Conversation with Memory + +```julia +# Use Session to persist conversation +storage = JsonlSessionStorage(...) +session = Session(storage) + +# Add messages to session +appendMessage(session, user_message) +appendMessage(session, assistant_message) + +# Build context from session +context = buildSessionContext(session) +``` + +### Pattern 2: Long Conversations + +```julia +# Compact periodically to stay within context limits +if token_count > MAX_TOKENS * 0.8 + compact_id = appendCompaction( + session, + summary, + first_kept_id, + token_count, + ) +end +``` + +### Pattern 3: Branching Conversations + +```julia +# User wants to explore alternative +session.moveTo(branch_point_id) + +# Create new branch +appendBranchSummary(session, "Exploring alternative approach") +appendMessage(session, new_user_message) +``` + +### Pattern 4: Custom Tools + +```julia +# Create custom tool +custom_tool = AgentTool( + "custom", + "custom", + "Does custom thing", + ..., + execute_function, + nothing, + EXECUTION_PARALLEL, +) + +# Add to agent +agent = Agent(Dict(:tools => [custom_tool])) +``` + +## Debugging + +### Check Active Run + +```julia +if !isnothing(agent.active_run) + println("Agent is busy") +else + println("Agent is idle") +end +``` + +### Clear Queues + +```julia +clearAllQueues(agent) +``` + +### Reset State + +```julia +reset!(agent) +``` + +## Performance Tips + +1. **Use parallel execution** for independent tools +2. **Compact periodically** for long conversations +3. **Use thinking_level wisely** (higher = slower but better) +4. **Batch tool calls** when possible +5. **Cache LLM responses** when appropriate + +## Troubleshooting + +### Agent stuck in loop + +```julia +# Check if agent is still processing +if hasQueuedMessages(agent) + # Clear queues + clearAllQueues(agent) +end +``` + +### Too many tokens + +```julia +# Compact session +compact_id = appendCompaction( + session, + summary, + first_kept_id, + token_count, +) +``` + +### Tool execution failed + +```julia +# Check tool result +if result.is_error + println("Tool failed: $(result.error)") +end +``` + +## Next Steps + +1. Read **Architecture Overview** for deep understanding +2. Explore **Agent Component** for state management +3. Study **AgentLoop** for core logic +4. Learn **Types & Messages** for data structures +5. Master **Session Management** for persistence +6. Build **Tools** for custom functionality + +## Resources + +- Original TypeScript implementation: `@earendil-works/pi-agent-core` +- AgentCore.jl source code: `src/` +- Examples: `examples/` + +## Community + +For questions and discussions: +- GitHub Issues: `/issues` +- Documentation: `docs/`