# AgentCore.jl - Types and Messages Deep Dive ## Core Type Hierarchy ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ Type Hierarchy │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ ThinkingLevel (Enum) │ │ - THINKING_OFF │ │ - THINKING_MINIMAL │ │ - THINKING_LOW │ │ - THINKING_MEDIUM │ │ - THINKING_HIGH │ │ - THINKING_XHIGH │ │ - THINKING_MAX │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ ToolExecutionMode (Enum) │ │ - EXECUTION_SEQUENTIAL (Tools run one at a time) │ │ - EXECUTION_PARALLEL (Tools run concurrently) │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ QueueMode (Enum) │ │ - QUEUE_ALL (Drain all messages at once) │ │ - QUEUE_ONE_AT_A_TIME (Process one message at a time) │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ MessageContent (Abstract Type) │ │ ├── TextContent (String) │ │ └── ImageContent (data::String, mime_type::String) │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ Message (Abstract Type) │ │ ├── UserMessage │ │ │ └─ role: "user", content: Message[], timestamp: Int64 │ │ ├── AssistantMessage │ │ │ └─ role: "assistant", content: Message[], api, provider, model, │ │ │ usage: Usage, stop_reason, error_message, timestamp │ │ └── ToolResultMessage │ │ └─ role: "toolResult", tool_call_id, tool_name, content, details, │ │ usage, added_tool_names, is_error, timestamp │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ AgentMessage (Abstract Type) │ │ └─ Union of all message types above + custom types │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ AgentTool │ │ - name: String │ │ - label: String │ │ - description: String │ │ - parameters: Any │ │ - execute: Function │ │ - prepare_arguments: Union{Function, Nothing} │ │ - execution_mode: Union{ToolExecutionMode, Nothing} │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ AgentContext │ │ - system_prompt: String │ │ - messages: Vector{AgentMessage} │ │ - tools: Union{Vector{AgentTool}, Nothing} │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ AgentEvent (Abstract Type) │ │ ├── AgentStartEvent / AgentEndEvent │ │ ├── TurnStartEvent / TurnEndEvent │ │ ├── MessageStartEvent / MessageEndEvent │ │ ├── MessageUpdateEvent │ │ ├── ToolExecutionStartEvent / ToolExecutionEndEvent │ │ └── ToolExecutionUpdateEvent │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ Usage & ModelCost │ │ Usage: input, output, cache_read, cache_write, total_tokens, cost │ │ ModelCost: input, output, cache_read, cache_write (all Float64) │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ Model │ │ - id, name, api, provider, base_url, reasoning: Bool │ │ - input: Vector{String} │ │ - cost: ModelCost │ │ - context_window, max_tokens: Int64 │ └─────────────────────────────────────────────────────────────────────────────┘ ``` ## Message Types ### Type Hierarchy ``` Message (for LLM API) ├── UserMessage (role: "user") │ ├── content::Vector{MessageContent} │ │ ├── TextContent (text::String) │ │ └── ImageContent (data::String, mime_type::String) │ └── timestamp::Timestamp (Int64) ├── AssistantMessage (role: "assistant") │ ├── content::Vector{MessageContent} │ │ ├── TextContent │ │ └── ToolCall (type, id, name, arguments::Dict{String, Any}) │ ├── api::String │ ├── provider::String │ ├── model::String │ ├── usage::Usage │ │ ├── input, output, cache_read, cache_write, total_tokens::Int64 │ │ └── cost::UsageCost (input, output, cache_read, cache_write, total::Float64) │ ├── stop_reason::String │ ├── error_message::Union{String, Nothing} │ └── timestamp::Timestamp └── ToolResultMessage (role: "toolResult") ├── tool_call_id::String ├── tool_name::String ├── content::Vector{MessageContent} ├── details::Any ├── usage::Union{Usage, Nothing} ├── added_tool_names::Union{Vector{String}, Nothing} ├── is_error::Bool └── timestamp::Timestamp AgentMessage (internal, extends Message) ├── UserMessage (same as above) ├── AssistantMessage (same as above) ├── ToolResultMessage (same as above) ├── BashExecutionMessage (custom, converted to UserMessage) │ ├── role, command, output, exit_code │ ├── cancelled, truncated, full_output_path │ └── exclude_from_context::Bool ├── CompactionSummaryMessage (custom, converted to UserMessage) │ ├── summary, tokens_before, timestamp └── BranchSummaryMessage (custom, converted to UserMessage) ├── summary, from_id, timestamp ``` ### UserMessage ```julia struct UserMessage <: Message role::String # "user" content::Vector{MessageContent} timestamp::Timestamp # Int64 (Unix timestamp) end ``` **Usage**: ```julia # Simple text message UserMessage( "user", [TextContent("Hello, how are you?")], Int64(Dates.now(Dates.UTC).datetime) ) ``` **Usage**: ```julia # Simple text message UserMessage( "user", [TextContent("Hello, how are you?")], Int64(Dates.now(Dates.UTC).datetime) ) # With multiple content types UserMessage( "user", [ TextContent("Analyze this image"), ImageContent(data_base64, "image/png") ], timestamp ) ``` ### AssistantMessage ```julia struct AssistantMessage <: Message role::String # "assistant" content::Vector{MessageContent} api::String # API identifier provider::String # Provider name model::String # Model ID usage::Usage stop_reason::String # "done", "error", "aborted", "length", etc. error_message::Union{String, Nothing} timestamp::Timestamp end ``` **Content can include**: - TextContent - ToolCall ```julia AssistantMessage( "assistant", [ TextContent("I'll check the directory for you."), ToolCall( "tool", "tc_123", "bash", Dict("command" => "ls -la"), nothing ), ToolCall( "tool", "tc_456", "read", Dict("path" => "README.md"), nothing ) ], "openai", "openai", "gpt-4", Usage(100, 50, 0, 0, 150, UsageCost(0.001, 0.002, 0.0, 0.0, 0.003)), "done", nothing, timestamp ) ``` ### ToolResultMessage ```julia struct ToolResultMessage <: Message role::String # "toolResult" tool_call_id::String # Reference to original ToolCall tool_name::String # Name of tool that executed content::Vector{MessageContent} details::Any # Additional tool-specific details usage::Union{Usage, Nothing} added_tool_names::Union{Vector{String}, Nothing} is_error::Bool # True if tool execution failed timestamp::Timestamp end # Note: AgentToolResult{T} (types.jl) - generic result type with type param T # AgentToolResultMutable (agent_loop.jl) - mutable variant used internally ``` **Usage**: ```julia ToolResultMessage( "toolResult", "tc_123", "bash", [TextContent("file1.md\nfile2.md\n")], BashToolDetails(...), nothing, nothing, false, timestamp ) ``` ## AgentTool Structure ```julia struct AgentTool{TParameters, TDetails} name::String label::String description::String parameters::TParameters execute::Function prepare_arguments::Union{Function, Nothing} execution_mode::Union{ToolExecutionMode, Nothing} end ``` **Parameters**: - `name`: Unique identifier for the tool - `label`: Display name - `description`: What the tool does - `parameters`: JSON schema for tool arguments - `execute`: Main execution function - `prepare_arguments`: Optional preprocessing - `execution_mode`: Sequential or parallel **Note:** `AgentHarnessTool` (`harness_types.jl:91`) is a harness-specific variant with the same structure but uses camelCase field names (`prepareArguments`, `executionMode`) and includes additional type parameters `{TContext, TParameters, TDetails}`. ### Tool Execution Function Signature ```julia execute::Function( tool_call_id::String, params::Dict{String, Any}, signal::Union{Any, Nothing}, # Abort signal on_update::Function, # Callback for streaming updates context::Any, # Tool context )::AgentToolResult{T} ``` **Returns** (`AgentToolResult{T}` from `types.jl`): ```julia AgentToolResult{T}( content::Vector{MessageContent}, # Result content details::T, # Tool-specific details usage::Union{Usage, Nothing}, # Usage statistics added_tool_names::Union{Vector{String}, Nothing}, terminate::Union{Bool, Nothing}, # If true, stop agent after this ) ``` **Note:** `AgentToolResultMutable` (in `agent_loop.jl`) is a mutable variant used internally for intermediate results. **Note:** External types used throughout the codebase: `Context`, `AbortSignal`, `EventStream`, `Promise` are defined in external modules (not in the source files covered by this document). ## AgentContext ```julia struct AgentContext system_prompt::String messages::Vector{AgentMessage} tools::Union{Vector{AgentTool}, Nothing} end ``` **Purpose**: Read-only snapshot of agent state for LLM calls **Usage in AgentLoop**: ```julia function streamAssistantResponse( context::AgentContext, # Contains messages, tools, system prompt config::AgentLoopConfig, ... )::AssistantMessage # Convert to LLM format llm_messages = config.convert_to_llm(context.messages) # Create context for API llm_context = Context( context.system_prompt, llm_messages, context.tools, ) # Call LLM return stream_function(context.model, llm_context, config) end ``` ## Event Types ### Agent Lifecycle Events ```julia struct AgentStartEvent <: AgentEvent end struct AgentEndEvent <: AgentEvent messages::Vector{AgentMessage} end ``` ### Turn Events ```julia struct TurnStartEvent <: AgentEvent end struct TurnEndEvent <: AgentEvent message::AgentMessage tool_results::Vector{ToolResultMessage} end ``` ### Message Events ```julia struct MessageStartEvent <: AgentEvent message::AgentMessage end struct MessageUpdateEvent <: AgentEvent message::AgentMessage assistant_message_event::Any # Partial message event end struct MessageEndEvent <: AgentEvent message::AgentMessage end ``` ### Tool Execution Events ```julia struct ToolExecutionStartEvent <: AgentEvent tool_call_id::String tool_name::String args::Any end struct ToolExecutionUpdateEvent <: AgentEvent tool_call_id::String tool_name::String args::Any partial_result::Any end struct ToolExecutionEndEvent <: AgentEvent tool_call_id::String tool_name::String result::Any is_error::Bool end ``` ## Usage Statistics ```julia struct Usage input::Int64 # Input tokens output::Int64 # Output tokens cache_read::Int64 # Cache read tokens cache_write::Int64 # Cache write tokens total_tokens::Int64 # Total tokens cost::UsageCost end struct UsageCost input::Float64 output::Float64 cache_read::Float64 cache_write::Float64 total::Float64 end ``` **Example**: ```julia Usage( 1000, # input tokens 200, # output tokens 500, # cache read tokens 0, # cache write tokens 1700, # total tokens UsageCost( 0.0005, # input cost ($0.50 per 1M tokens) 0.0015, # output cost ($1.50 per 1M tokens) 0.00025, # cache read cost 0.0, # cache write cost 0.0035 # total cost ) ) ``` ## Model Type ```julia struct Model{Api} id::String # Model identifier (e.g., "gpt-4") name::String # Model name (e.g., "GPT-4") api::Api # API type (String, Symbol, or custom type) provider::String # Provider name (e.g., "openai") base_url::String # API base URL reasoning::Bool # Whether model supports reasoning input::Vector{String} # Input modes (e.g., ["text", "image"]) cost::ModelCost context_window::Int64 # Max context window (e.g., 128000) max_tokens::Int64 # Max output tokens end struct ModelCost input::Float64 output::Float64 cache_read::Float64 cache_write::Float64 end ``` ## ToolCall Type ```julia struct ToolCall type::String # "tool" id::String # Unique ID for this tool call name::String # Tool name to call arguments::Dict{String, Any} # Tool arguments as JSON-like Dict partial_json::Union{String, Nothing} # Partial JSON string end ``` **Example**: ```julia ToolCall( "tool", "call_abc123", "bash", Dict( "command" => "ls -la", "timeout" => 30 ), nothing ) ``` ## Custom Message Types ### BashExecutionMessage ```julia mutable struct BashExecutionMessage role::String # "custom" command::String output::String exit_code::Union{Int64, Nothing} cancelled::Bool truncated::Bool full_output_path::Union{String, Nothing} timestamp::Timestamp exclude_from_context::Bool end ``` ### CompactionSummaryMessage ```julia mutable struct CompactionSummaryMessage role::String # "compactionSummary" summary::String # Summary of compacted history tokens_before::Int64 # Context size before compaction timestamp::Timestamp end ``` ### BranchSummaryMessage ```julia mutable struct BranchSummaryMessage role::String # "branchSummary" summary::String # Summary of branch history from_id::String # Branch point ID timestamp::Timestamp end ``` ### CustomMessage **Note:** There are two `CustomMessage` types in the codebase: 1. **Types.CustomMessage** (`types.jl:155`) - A simple wrapper that holds another `AgentMessage` with a custom type label: ```julia struct CustomMessage <: AgentMessage message::AgentMessage custom_type::String end ``` 2. **Messages.CustomMessage{T}** (`messages.jl:42`) - A standalone mutable message with content, display flag, and details: ```julia mutable struct CustomMessage{T} role::String custom_type::String content::Union{String, Vector{MessageContent}} display::Bool details::Union{T, Nothing} timestamp::Timestamp end ``` Only `Messages.CustomMessage{T}` is converted by `convertToLlmMessage()` to a `UserMessage`. ## AgentState ```julia mutable struct AgentState system_prompt::String model::Model thinking_level::ThinkingLevel tools::Vector{AgentTool} messages::Vector{AgentMessage} is_streaming::Bool streaming_message::Union{AgentMessage, Nothing} pending_tool_calls::Set{String} error_message::Union{String, Nothing} end ``` **Purpose**: Runtime state of the Agent **Note**: AgentState is mutable and used internally by Agent ## Message Transformation Pipeline ### convertToLlm() - AgentMessage[] → Message[] ```julia function convertToLlm(messages::Vector{AgentMessage})::Vector{Message} result::Vector{Message} = Message[] for m in messages converted = convertToLlmMessage(m) if !isnothing(converted) push!(result, converted) end end return result end ``` **Data Flow**: ``` Vector{AgentMessage} (internal conversation history) │ │ Type dispatch on convertToLlmMessage(): │ │ • UserMessage → UserMessage (pass-through) │ • AssistantMessage → AssistantMessage (pass-through) │ • ToolResultMessage → ToolResultMessage (pass-through) │ │ Custom messages converted to UserMessage: │ • BashExecutionMessage → UserMessage │ (via bashExecutionToText() for display) │ • CompactionSummaryMessage → UserMessage │ (wrapped with COMPACTION_SUMMARY_PREFIX/SUFFIX) │ • BranchSummaryMessage → UserMessage │ (wrapped with BRANCH_SUMMARY_PREFIX/SUFFIX) │ • CustomMessage → UserMessage │ (content field used directly, string→TextContent) │ ▼ Vector{Message} (for LLM API) - Excludes: BashExecutionMessage (if exclude_from_context) - Includes: All standard messages + converted custom messages ``` **Example**: ```julia # Input: Vector{AgentMessage} [ UserMessage("user", [TextContent("Hello")], 1234567890), AssistantMessage("assistant", [ TextContent("Hi there!"), ToolCall("bash", "call_123", "bash", Dict("command" => "ls"), nothing) ], "openai", "openai", "gpt-4", Usage(...), "done", nothing, 1234567891), BashExecutionMessage("custom", "ls -la", "file1.md\nfile2.md\n", 0, false, false, nothing, 1234567892, false), CompactionSummaryMessage("compactionSummary", "Previous conversation compacted", 1000, 1234567893), CustomMessage("custom", "someCustomType", "Some custom content", true, nothing, 1234567894), ] # Output: Vector{Message} [ UserMessage("user", [TextContent("Hello")], 1234567890), AssistantMessage("assistant", [ TextContent("Hi there!"), ToolCall(...) ], "openai", "openai", "gpt-4", Usage(...), "done", nothing, 1234567891), UserMessage("user", [TextContent("Ran `ls -la`\n```\nfile1.md\nfile2.md\n```\n")], 1234567892), UserMessage("user", [TextContent("Previous conversation compacted")], 1234567893), UserMessage("user", [TextContent("Some custom content")], 1234567894), ] ``` ### Default convertToLlmMessage Implementations ```julia function convertToLlmMessage(m::BashExecutionMessage) if m.exclude_from_context return nothing end return UserMessage("user", [TextContent(bashExecutionToText(m))], m.timestamp) end function convertToLlmMessage(m::CompactionSummaryMessage) text = COMPACTION_SUMMARY_PREFIX * m.summary * COMPACTION_SUMMARY_SUFFIX return UserMessage("user", [TextContent(text)], m.timestamp) end function convertToLlmMessage(m::CustomMessage)::Union{UserMessage, Nothing} content = if m.content isa String [TextContent(m.content)] else m.content end return UserMessage("user", content, m.timestamp) end function convertToLlmMessage(m::BranchSummaryMessage) text = BRANCH_SUMMARY_PREFIX * m.summary * BRANCH_SUMMARY_SUFFIX return UserMessage("user", [TextContent(text)], m.timestamp) end function convertToLlmMessage(m::UserMessage) return m # Pass through end function convertToLlmMessage(m::AssistantMessage) return m # Pass through end function convertToLlmMessage(m::ToolResultMessage) return m # Pass through end ``` ## Complete Data Flow Examples ### Example 1: User Prompt → Assistant Response ``` User Input: "Hello, what's in the current directory?" ↓ prompt(agent, "Hello, what's in the current directory?") │ └─► normalizePromptInput(String) Input: "Hello, what's in the current directory?" Output: [UserMessage("user", [TextContent("Hello, what's in the current directory?")], timestamp)] ↓ AgentLoop execution: │ ├─► transform_context() (optional) │ Input: [UserMessage(...)] │ Output: [UserMessage(...)] │ ├─► convert_to_llm() │ Input: [UserMessage(...)] │ Output: [UserMessage(...)] │ ├─► stream_fn() - LLM API │ Input: model, Context(...), config │ Output: AssistantMessage with ToolCall[] │ role: "assistant" │ content: [ │ TextContent("I'll check the directory for you."), │ ToolCall("tool", "tc_123", "bash", Dict("command" => "ls -la"), nothing) │ ] │ usage: Usage(input=100, output=20, ...) │ stop_reason: "done" │ ├─► executeToolCalls() │ Input: AssistantMessage with ToolCall[] │ Output: ToolResultMessage[] │ role: "toolResult" │ tool_call_id: "tc_123" │ tool_name: "bash" │ content: [TextContent("file1.md\nfile2.md\n")] │ is_error: false │ └─► Append to context.messages ↓ Final Conversation History: [ UserMessage("user", [TextContent("Hello, what's in the current directory?")], ...), AssistantMessage("assistant", [ TextContent("I'll check the directory for you."), ToolCall("tool", "tc_123", "bash", Dict("command" => "ls -la"), nothing) ], "openai", "openai", "gpt-4", Usage(...), "done", nothing, ...), ToolResultMessage("toolResult", "tc_123", "bash", [TextContent("file1.md\nfile2.md\n")], ..., false, ...), ] ``` ### Example 2: Tool Call Execution → Tool Result ``` ToolCall from AssistantMessage │ ├─ type: "tool" ├─ id: "tc_123" ├─ name: "bash" ├─ arguments: Dict("command" => "ls -la") └─ partial_json: nothing ↓ prepareToolCall(tool_call) ↓ Finds tool by name "bash" ↓ before_tool_call hook (optional) Input: BeforeToolCallContext(...) Output: BeforeToolCallResult(block=false) or nothing ↓ validateToolArguments(tool_call) Input: Dict("command" => "ls -la") Output: Dict("command" => "ls -la") ↓ Return: PreparedToolCall("prepared", tool_call, bash_tool, validated_args) ↓ executePreparedToolCall(prepared) ↓ tool.execute("tc_123", Dict("command" => "ls -la"), signal, on_update) ↓ Bash tool executes "ls -la" command Returns: AgentToolResultMutable( content: [TextContent("file1.md\nfile2.md\n")], details: BashToolDetails(...), usage: nothing, added_tool_names: nothing, terminate: nothing ) ↓ finalizeExecutedToolCall(executed) ↓ after_tool_call hook (optional) Input: AfterToolCallContext(...) Output: AfterToolCallResult(...) or nothing ↓ Return: FinalizedToolCallOutcome( tool_call: ToolCall(...), result: AgentToolResultMutable(...), is_error: false ) ↓ createToolResultMessage(finalized) ↓ Return: ToolResultMessage( role: "toolResult", tool_call_id: "tc_123", tool_name: "bash", content: [TextContent("file1.md\nfile2.md\n")], details: BashToolDetails(...), usage: nothing, added_tool_names: nothing, is_error: false, timestamp: Int64(...) ) ``` ### Example 3: Custom Message Conversion ``` BashExecutionMessage (custom, for logging) │ role: "custom" command: "ls -la" output: "file1.md\nfile2.md\n" exit_code: 0 cancelled: false truncated: false full_output_path: nothing timestamp: 1234567890 exclude_from_context: false ↓ convertToLlmMessage(BashExecutionMessage) ↓ bashExecutionToText(msg) Output: "Ran `ls -la`\n```\nfile1.md\nfile2.md\n```\n" ↓ Return: UserMessage( "user", [TextContent("Ran `ls -la`\n```\nfile1.md\nfile2.md\n```\n")], 1234567890 ) ↓ (Excluded if exclude_from_context = true) ──────────────────────────────────────────────────────────────────── CompactionSummaryMessage (custom, for history compression) │ role: "compactionSummary" summary: "Previous 100 turns about Python programming" tokens_before: 15000 timestamp: 1234567890 ↓ convertToLlmMessage(CompactionSummaryMessage) ↓ Text = COMPACTION_SUMMARY_PREFIX + summary + COMPACTION_SUMMARY_SUFFIX Result: "\nPrevious 100 turns about Python programming\n" ↓ Return: UserMessage( "user", [TextContent("...\nPrevious 100 turns...\n")], 1234567890 ) ## Summary The type system in AgentCore.jl provides: 1. **Strong typing** for different message types 2. **Extensibility** through abstract types and multiple dispatch 3. **Clear separation** between internal (AgentMessage) and external (Message) formats 4. **Rich metadata** in Usage and Model types for cost tracking 5. **Event-driven architecture** through Event types 6. **Tool execution flexibility** through Tool types with hooks All types are designed for: - **Interoperability** with LLM APIs - **Extensibility** for custom message types - **Performance** with immutable structs where possible - **Debuggability** through rich event system