update

.AI_prompt.md

@@ -80,3 +80,48 @@ Assistant should only respond in JSON format as described below:
   "action_input": "..."
 }
 ```
+
+
+
+<!-- ------------------------------------------- 100 ------------------------------------------- -->
+
+
+read this codebase. I want you to write the following files:
+- ./docs/requirements.md
+- ./docs/solution-design.md
+- ./docs/specification
+- ./docs/walkthrough.md
+according to /home/ton/docker-apps/sommpanion/ASG_Framework/ASG_Framework.md so I can read and understand this codebase.
+
+
+
+
+
+
+
+
+
+echo 'export PATH="$HOME/.juliaup/bin:$PATH"' >> ~/.bashrc
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

IMPLEMENTATION.md

@@ -0,0 +1,119 @@
+# Julia Implementation - AgentCore
+
+This directory contains a Julia reimplementation of the `@earendil-works/pi-agent-core` package.
+
+## Project Structure
+
+```
+julia_implementation/
+├── src/
+│   ├── AgentCore.jl           # Main module entry point
+│   ├── types.jl               # Core type definitions
+│   ├── stream_fn.jl           # Stream function utilities
+│   ├── agent_loop.jl          # Low-level agent loop
+│   ├── agent.jl               # High-level Agent struct
+│   ├── harness_types.jl       # Extended types for AgentHarness
+│   ├── messages.jl            # Custom message types
+│   ├── system_prompt.jl       # System prompt formatting
+│   ├── skills.jl              # Skill loading and formatting
+│   ├── prompt_templates.jl    # Prompt template handling
+│   ├── agent_harness.jl       # AgentHarness implementation
+│   │
+│   ├── session/
+│   │   ├── session.jl         # Session class
+│   │   ├── jsonl_storage.jl   # JSONL storage
+│   │   ├── jsonl_repo.jl      # JSONL repository
+│   │   ├── memory_storage.jl  # In-memory storage
+│   │   ├── memory_repo.jl     # In-memory repository
+│   │   └── repo_utils.jl      # Repository utilities
+│   │
+│   ├── tools/
+│   │   ├── index.jl           # Tool exports
+│   │   ├── bash.jl            # Bash execution tool
+│   │   ├── read.jl            # File read tool
+│   │   ├── write.jl           # File write tool
+│   │   ├── edit.jl            # File edit tool
+│   │   ├── edit_diff.jl       # Diff computation
+│   │   ├── image.jl           # Image utilities
+│   │   ├── path_utils.jl      # Path resolution
+│   │   └── file_mutation_queue.jl  # File mutation serialization
+│   │
+│   ├── compaction/
+│   │   ├── compaction.jl      # Context compaction
+│   │   ├── utils.jl           # Compaction utilities
+│   │   └── branch_summarization.jl  # Branch summarization
+│   │
+│   ├── utils/
+│   │   ├── truncate.jl        # Output truncation
+│   │   └── shell_output.jl    # Shell output capture
+│   │
+│   ├── proxy.jl               # Proxy stream function
+│   └── utils.jl               # Utility functions
+│
+├── test/
+├── Project.toml
+├── Manifest.toml
+└── README.md
+```
+
+## Key Features
+
+### Core Architecture
+
+The implementation follows the same layered architecture as the TypeScript version:
+
+1. **Low-level (agent_loop.jl)**: Pure agent loop logic that works with `AgentMessage[]`
+2. **High-level (agent.jl)**: Stateful wrapper with event streaming and queueing
+3. **Harness (agent_harness.jl)**: Session persistence, resource management, hooks
+4. **Session (session/)**: Conversation history with compaction and branching
+5. **Tools (tools/)**: Built-in execution tools (bash, read, write, edit)
+
+### Julia-Specific Features
+
+- **Type system**: Uses Julia's parametric types for type-safe tool definitions
+- **Multiple dispatch**: Extensible via multiple dispatch for custom message types
+- **Async primitives**: Leverages Julia's `@async` and `@spawn` for concurrent operations
+- **Error handling**: Julia exceptions with typed error codes
+
+## Building
+
+```julia
+using Pkg
+Pkg.activate("julia_implementation")
+Pkg.instantiate()
+```
+
+## Usage Example
+
+```julia
+using AgentCore
+
+# Create an agent
+agent = Agent()
+
+# Subscribe to events
+subscribe(agent) do event, signal
+    if event isa MessageEndEvent
+        println("Message: $(event.message)")
+    end
+end
+
+# Run a prompt
+prompt(agent, "Hello, world!")
+```
+
+## Compatibility
+
+This implementation aims for API compatibility with the TypeScript version while providing idiomatic Julia abstractions.
+
+## Status
+
+This is an active implementation. Core functionality is in place, with ongoing work on:
+
+- Complete tool implementations
+- Full session repository functionality
+- Test suite
+
+## License
+
+MIT

Project.toml

@@ -1,37 +1,22 @@
-name = "YiemAgent"
+name = "AgentCore"
-uuid = "e012c34b-7f78-48e0-971c-7abb83b6f0a2"
+uuid = "6e2f7b3a-9a0b-4e8e-8f8f-8f8f8f8f8f8f"
+authors = ["Mario Zechner <post@badlogicgames.com>"]
 version = "0.8.0"
-authors = ["narawat lamaiin <narawat@outlook.com>"]
 
 [deps]
-Base64 = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"
+Dates = "ade2ca70-3891-5945-98fb-dc09409a37d3"
-CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
+JSON3 = "0f8b85d8-8d2f-5481-9e3b-d9a10a9b6c53"
-DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
+Libdl = "8f399da3-355a-58d1-55dd-a8cd37d21846"
-DataStructures = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
+Markdown = "d6f4372e-7a37-5ca6-90db-23e40208355e"
-Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
+Mmap = "a63ad114-7ff6-5b6b-903e-90ddba579e5d"
-GeneralUtils = "c6c72f09-b708-4ac8-ac7c-2084d70108fe"
+Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
-HTTP = "cd3eb016-35fb-5094-929b-558a96fad6f3"
-JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
-LLMMCTS = "d76c5a4d-449e-4835-8cc4-dd86ec44f241"
-LibPQ = "194296ae-ab2e-5f79-8cd4-7183a0a5a0d1"
-NATS = "55e73f9c-eeeb-467f-b4cc-a633fde63d2a"
-PrettyPrinting = "54e16d92-306c-5ea0-a30b-337be88ac337"
 Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
-Revise = "295af30f-e4ad-537b-8983-00126c2a3abe"
+Sockets = "6462fe0b-2de3-572b-8e7f-4c2f5e2c2e2b"
-SQLLLM = "2ebc79c7-cc10-4a3a-9665-d2e1d61e63d3"
+Unicode = "4ec0a83e-493e-50e2-b9ac-8f72acf2a872"
-Serde = "db9b398d-9517-45f8-9a95-92af99003e0e"
+UUIDs = "cf7118a7-4649-5bc2-89ac-36d7b14660ca"
-Serialization = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
-URIs = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4"
-UUIDs = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"
 
-[compat]
+[extras]
-Base64 = "1.11.0"
+Test = "8dfed614-e22c-5e4d-98d3-97fe1b80e45d"
-CSV = "0.10.15"
+
-DataFrames = "1.7.0"
+[targets]
-GeneralUtils = "0.5.10"
+test = ["Test"]
-HTTP = "2.4.0"
-JSON = "1.6.1"
-LLMMCTS = "0.1.5"
-NATS = "0.1.0"
-SQLLLM = "0.2.8"
-Serde = "3.7.2"

README.md

@@ -1,23 +1,179 @@
-# YiemAgent
+# AgentCore.jl - Julia Implementation of Pi Agent Core
 
-## TODO
+A Julia reimplementation of the `@earendil-works/pi-agent-core` package, providing a stateful agent framework for LLM interactions.
-- [WORKING] build prompt()
-- [ ] build agent runLoop()
-- [ ] build MCP server connector
-- [ ] executeplan() to execute the plan
-- [ ] add comprehensive tests
 
-## Changelog
+## Overview
 
-### Version 0.8.0
+This package provides:
-- Converted snake_case fields to camelCase:
+- Low-level `agentLoop` for stateful LLM interactions with tool execution
-  - `llmModel`: `base_url` → `baseUrl`, `context_window` → `contextWindow`, `max_tokens` → `maxTokens`
+- High-level `Agent` struct with state management, event streaming, and queueing
-- Converted PascalCase type references to camelCase:
+- `AgentHarness` for session persistence, resource management, and extension hooks
-  - `AgentState` → `agentState`
+- Built-in tools for file operations (read, write, edit) and bash execution
-  - `AgentTool` → `agentTool`
+- Session management with JSONL-based storage, compaction, and branch navigation
-  - `AgentMessage` → `agentMessage`
+
-  - `PendingMessageQueue` → `pendingMessageQueue`
+## Architecture
-  - `ActiveRun` → `activeRun`
+
-  - `StreamFn` → `streamFn`
+The Julia implementation follows the same layered architecture as the TypeScript version:
-  - `ThinkingLevel` → `thinkingLevel`
+
-  - `ToolExecutionMode` → `toolExecutionMode`
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                            AgentHarness                              │
+│              (Session persistence, resource management)              │
+└─────────────────────────────────────────────────────────────────────┘
+                              │
+┌─────────────────────────────▼───────────────────────────────────────┐
+│                              Agent                                 │
+│              (State management, event streaming, queueing)           │
+└─────────────────────────────────────────────────────────────────────┘
+                              │
+┌─────────────────────────────▼───────────────────────────────────────┐
+│                          AgentLoop                                 │
+│              (Low-level loop, tool execution)                        │
+└─────────────────────────────────────────────────────────────────────┘
+                              │
+┌─────────────────────────────▼───────────────────────────────────────┐
+│                            Session                                 │
+│              (Conversation history, compaction, branching)           │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+## Installation
+
+```julia
+using Pkg
+Pkg.add("AgentCore")
+```
+
+## Quick Start
+
+```julia
+using AgentCore
+
+# Create an agent with default configuration
+agent = Agent()
+
+# Subscribe to events
+subscribe(agent) do event, signal
+    if event isa MessageEndEvent
+        println("Received message: $(event.message)")
+    end
+end
+
+# Run a prompt
+prompt(agent, "Hello, how are you?")
+```
+
+## Core Concepts
+
+### Agent
+
+The `Agent` struct provides a high-level interface for interacting with LLMs. It manages:
+- Conversation state (messages, tools, system prompt)
+- Event streaming and lifecycle management
+- Steering and follow-up message queues
+- Abort handling
+
+### AgentLoop
+
+The `agentLoop` function implements the core agent loop that:
+- Transforms `AgentMessage[]` to `Message[]` at the LLM call boundary
+- Executes tool calls (parallel or sequential)
+- Emits lifecycle events
+- Handles steering and follow-up messages
+
+### AgentHarness
+
+The `AgentHarness` provides:
+- Session persistence with JSONL storage
+- Resource management (skills, prompt templates)
+- Extension hooks system
+- Tool execution with context
+- Branch navigation and compaction
+
+### Sessions
+
+Sessions track conversation history using a tree-based structure:
+- Branch-based history with compaction
+- Tree navigation (moveTo, navigateTree)
+- Message and metadata persistence
+
+## Built-in Tools
+
+### Bash Tool
+
+Execute shell commands with output capture and truncation.
+
+```julia
+bash_tool = createBashTool()
+```
+
+### Read Tool
+
+Read files with support for text and images.
+
+```julia
+read_tool = createReadTool()
+```
+
+### Write Tool
+
+Write content to files with automatic directory creation.
+
+```julia
+write_tool = createWriteTool()
+```
+
+### Edit Tool
+
+Edit files using exact text replacement.
+
+```julia
+edit_tool = createEditTool()
+```
+
+## Session Storage
+
+AgentCore supports two session storage backends:
+
+1. **JsonlSessionStorage** - File-based storage using JSONL format
+2. **InMemorySessionStorage** - In-memory storage for testing
+
+## Compaction
+
+The compaction system manages context window usage by:
+- Summarizing old conversation history
+- Retaining recent messages
+- Supporting iterative updates to summaries
+
+## Event System
+
+AgentCore uses a rich event system for monitoring and control:
+
+- `AgentStartEvent` / `AgentEndEvent` - Agent lifecycle
+- `TurnStartEvent` / `TurnEndEvent` - Conversation turns
+- `MessageStartEvent` / `MessageEndEvent` - Message lifecycle
+- `ToolExecutionStartEvent` / `ToolExecutionEndEvent` - Tool execution
+
+## Examples
+
+See the `examples/` directory for more detailed examples.
+
+## Differences from TypeScript
+
+While maintaining API compatibility where possible, this Julia implementation:
+- Uses Julia's type system for better compile-time guarantees
+- Leverages Julia's multiple dispatch for extensibility
+- Uses Julia's async primitives for concurrent operations
+- Provides more idiomatic Julia error handling
+
+## Contributing
+
+Contributions are welcome! Please see `CONTRIBUTING.md` for details.
+
+## License
+
+MIT
+
+## Acknowledgments
+
+This is a reimplementation of the [Pi Agent Core](https://github.com/earendil-works/pi/packages/agent) package in Julia.

docs/agent_loop_diagram.md

@@ -0,0 +1,365 @@
+```
+┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│                                              AGENT LOOP DIAGRAM                                                 │
+└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ 1. INITIALIZATION                                                                                               │
+│                                                                                                                 │
+│   Agent.prompt(user_input)                                                                                      │
+│         │                                                                                                       │
+│         ▼                                                                                                       │
+│   normalizePrompt() ← Convert input to AgentMessage[]                                                           │
+│         │                                                                                                       │
+│         ▼                                                                                                       │
+│   runPromptMessages()                                                                                           │
+│         │                                                                                                       │
+│         ▼                                                                                                       │
+└─────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┘
+          │
+          ▼
+┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ 2. AGENT LOOP START (runAgentLoop)                                                                              │
+│                                                                                                                 │
+│   new_messages = copy(prompts)                                                                                  │
+│   current_context.messages = vcat(context.messages, copy(prompts))                                              │
+│         │                                                                                                       │
+│         └─→ User messages are IMMEDIATELY added to context.messages                                             │
+│             (They are NOT in the steering queue!)                                                               │
+│                                                                                                                 │
+│   emit(AgentStartEvent)                                                                                         │
+│   emit(TurnStartEvent)                                                                                          │
+│                                                                                                                 │
+│   for prompt in prompts:                                                                                        │
+│     emit(MessageStartEvent(prompt))                                                                             │
+│     emit(MessageEndEvent(prompt))                                                                               │
+│                                                                                                                 │
+└─────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┘
+          │
+          ▼
+┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ 3. MAIN LOOP (runLoop - while true)                                                                             │
+│                                                                                                                 │
+│   pending_messages = get_steering_messages()                                                                    │
+│         │                                                                                                       │
+│         └─→ Steering queue: messages from agent.steer()                                                         │
+│             These are for CONTINUING conversation (NOT new user prompts)                                        │
+│                                                                                                                 │
+│   ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │
+│   │ While has pending_messages OR has_tool_calls:                                                             │ │
+│   │                                                                                                           │ │
+│   │   ┌─────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ │
+│   │   │ 4. PENDING MESSAGE HANDLING (steering messages only)                                                │ │ │
+│   │   │                                                                                                     │ │ │
+│   │   │   pending_messages = get_steering()                                                                 │ │ │
+│   │   │   if !isempty(pending_messages):                                                                    │ │ │
+│   │   │     for msg in pending_messages:                                                                    │ │ │
+│   │   │       emit(MessageStartEvent(msg))                                                                  │ │ │
+│   │   │       emit(MessageEndEvent(msg))                                                                    │ │ │
+│   │   │       push to current_context.messages ← Steering messages go HERE                                  │ │ │
+│   │   │       push to new_messages                                                                          │ │ │
+│   │   │     pending_messages = []                                                                           │ │ │
+│   │   │                                                                                                     │ │ │
+│   │   │   Note: User messages from Agent.prompt() are ALREADY in context.messages                           │ │ │
+│   │   │   (They were added in runAgentLoop via vcat(), not via this queue)                                  │ │ │
+│   │   └─────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ │
+│   │                                                                                                           │ │
+│   │   ┌─────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ │
+│   │   │ 5. STREAM ASSISTANT RESPONSE                                                                        │ │ │
+│   │   │                                                                                                     │ │ │
+│   │   │   message = streamAssistantResponse()                                                               │ │ │
+│   │   │     ├─ transform_context (if configured)                                                            │ │ │
+│   │   │     ├─ convert_to_llm(messages) → Message[]                                                         │ │ │
+│   │   │     │   ┌───────────────────────────────────────────────────────────────────────────────────────┐   │ │ │
+│   │   │     │   │ Converts AgentMessage[] to Message[]                                                  │   │ │ │
+│   │   │     │   │ Filters: keeps user, assistant, toolResult                                            │   │ │ │
+│   │   │     │   └───────────────────────────────────────────────────────────────────────────────────────┘   │ │ │
+│   │   │     ├─ stream_function(model, context)                                                              │ │ │
+│   │   │     │   ┌───────────────────────────────────────────────────────────────────────────────────────┐   │ │ │
+│   │   │     │   │ LLM Stream Events:                                                                    │   │ │ │
+│   │   │     │   │   • start → create partial AssistantMessage                                           │   │ │ │
+│   │   │     │   │   • text_start/delta/end → update partial message                                     │   │ │ │
+│   │   │     │   │   • thinking_start/delta/end → update partial message                                 │   │ │ │
+│   │   │     │   │   • toolcall_start/delta/end → update partial message                                 │   │ │ │
+│   │   │     │   │   • done → finalize message                                                           │   │ │ │
+│   │   │     │   │   • error → handle error                                                              │   │ │ │
+│   │   │     │   └───────────────────────────────────────────────────────────────────────────────────────┘   │ │ │
+│   │   │     └─ push to current_context.messages & new_messages                                              │ │ │
+│   │   │                                                                                                     │ │ │
+│   │   │   emit(MessageStartEvent(message))                                                                  │ │ │
+│   │   │   emit(MessageEndEvent(message))                                                                    │ │ │
+│   │   └─────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ │
+│   │                                                                                                           │ │
+│   │   if message.stop_reason in ("error", "aborted"):                                                         │ │
+│   │     emit(TurnEndEvent)                                                                                    │ │
+│   │     emit(AgentEndEvent) ← EXIT LOOP                                                                       │ │
+│   │     return                                                                                                │ │
+│   │                                                                                                           │ │
+│   │   tool_calls = filter(message.content, ToolCall)                                                          │ │
+│   │   if !isempty(tool_calls):                                                                                │ │
+│   │     executeToolCalls() → ToolResultMessage[]                                                              │ │
+│   │     for result in tool_results:                                                                           │ │
+│   │       push to current_context.messages                                                                    │ │
+│   │       push to new_messages                                                                                │ │
+│   │       emit(MessageStartEvent(result))                                                                     │ │
+│   │       emit(MessageEndEvent(result))                                                                       │ │
+│   │                                                                                                           │ │
+│   │   emit(TurnEndEvent(message, tool_results))                                                               │ │
+│   │                                                                                                           │ │
+│   │   ┌─────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ │
+│   │   │ 6. PREPARE NEXT TURN                                                                                │ │ │
+│   │   │                                                                                                     │ │ │
+│   │   │   next_turn_context = PrepareNextTurnContext(...)                                                   │ │ │
+│   │   │   next_turn_snapshot = prepare_next_turn(config, next_turn_context)                                 │ │ │
+│   │   │                                                                                                     │ │ │
+│   │   │   if !isnothing(next_turn_snapshot):                                                                │ │ │
+│   │   │     update context, model, thinking_level                                                           │ │ │
+│   │   │                                                                                                     │ │ │
+│   │   │   if should_stop_after_turn(config, next_turn_context):                                             │ │ │
+│   │   │     emit(AgentEndEvent) ← EXIT LOOP                                                                 │ │ │
+│   │   │     return                                                                                          │ │ │
+│   │   └─────────────────────────────────────────────────────────────────────────────────────────────────────┘ │ │
+│   │                                                                                                           │ │
+│   │   pending_messages = get_steering_messages() ← Check for new steering messages                            │ │
+│   │                                                                                                           │ │
+│   └───────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │
+│                                                                                                                 │
+│   follow_up_messages = get_follow_up_messages()                                                                 │
+│                                                                                                                 │
+│   if !isempty(follow_up_messages):                                                                              │
+│     pending_messages = follow_up_messages ← Continue loop for follow-ups                                        │
+│     continue                                                                                                    │
+│                                                                                                                 │
+│   break  ← EXIT MAIN LOOP (no more pending messages)                                                            │
+│                                                                                                                 │
+│   emit(AgentEndEvent(new_messages))                                                                             │
+│                                                                                                                 │
+└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ 4. STEERING QUEUE MECHANISM                                                                                     │
+├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                                                 │
+│   Steering messages are queued via agent.steer(message)                                                         │
+│   They are ONLY processed at the START of a loop iteration                                                      │
+│   AFTER the previous assistant turn completes                                                                   │
+│                                                                                                                 │
+│   Flow:                                                                                                         │
+│     user asks → agent responds → [user can steer here]                                                          │
+│                     │                                                                                           │
+│                     └─→ pending_messages = get_steering() ← Steering messages injected here                     │
+│                                                                                                                 │
+│   Follow-up messages are queued via agent.followUp(message)                                                     │
+│   They run ONLY after agent would otherwise stop                                                                │
+│                                                                                                                 │
+└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ COMPLETE CYCLE EXAMPLE: User asks → Agent responds → User asks 2nd → Agent responds                             │
+├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                                                 │
+│   TURN #1: User asks "What is Julia?"                                                                           │
+│   ─────────────────────────────────────────                                                                     │
+│   1. Agent.prompt("What is Julia?")                                                                             │
+│      normalizePrompt() → [UserMessage("What is Julia?")]                                                        │
+│      runPromptMessages()                                                                                        │
+│                                                                                                                 │
+│   2. runAgentLoop()                                                                                             │
+│      new_messages = [UserMessage("What is Julia?")]                                                             │
+│      current_context.messages = vcat([...existing...], [UserMessage("What is Julia?")])                         │
+│         │                                                                                                       │
+│         └─→ User message IMMEDIATELY added to context.messages (NOT via steering queue!)                        │
+│      emit(AgentStartEvent), emit(TurnStartEvent)                                                                │
+│      emit(MessageStart/End) for user message                                                                    │
+│                                                                                                                 │
+│   3. runLoop()                                                                                                  │
+│      pending_messages = get_steering() = []  ← Steering queue is empty (no agent.steer() yet)                   │
+│                                                                                                                 │
+│   4. streamAssistantResponse()                                                                                  │
+│      convert_to_llm([UserMessage]) → Message[]                                                                  │
+│      LLM call with [UserMessage]                                                                                │
+│      receive AssistantMessage: "Julia is a programming language..."                                             │
+│      push AssistantMessage to current_context.messages                                                          │
+│      push AssistantMessage to new_messages                                                                      │
+│      emit(MessageStart/End) for assistant message                                                               │
+│                                                                                                                 │
+│   5. check stop_reason → continue (no tools, no error)                                                          │
+│                                                                                                                 │
+│   6. emit(TurnEndEvent)                                                                                         │
+│                                                                                                                 │
+│   7. prepare_next_turn() → nothing (default)                                                                    │
+│                                                                                                                 │
+│   8. should_stop_after_turn() → false (default)                                                                 │
+│                                                                                                                 │
+│   9. pending_messages = get_steering() = []  ← No steering messages                                             │
+│                                                                                                                 │
+│   10. follow_up_messages = get_follow_up() = []                                                                 │
+│                                                                                                                 │
+│   11. break  ← Exit main loop                                                                                   │
+│                                                                                                                 │
+│   12. emit(AgentEndEvent)                                                                                       │
+│                                                                                                                 │
+│   ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │
+│   │ Current context.messages:                                                                                 │ │
+│   │   [UserMessage("What is Julia?"), AssistantMessage("Julia is...")]                                        │ │
+│   │                                                                                                           │ │
+│   │ steering_queue: []                                                                                        │ │
+│   │ follow_up_queue: []                                                                                       │ │
+│   └───────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │
+│                                                                                                                 │
+│   LLM SEES (convert_to_llm() filters):                                                                          │
+│   ┌─────────────────────────────────────────────────────────────────────────────────────────────────┐           │
+│   │ Messages passed to LLM API:                                                                     │           │
+│   │   [UserMessage("What is Julia?"), AssistantMessage("Julia is...")]                              │           │
+│   └─────────────────────────────────────────────────────────────────────────────────────────────────┘           │
+│                                                                                                                 │
+│   TURN #2: User asks "How does it work?"                                                                        │
+│   ─────────────────────────────────────────                                                                     │
+│   1. Agent.prompt("How does it work?")                                                                          │
+│      normalizePrompt() → [UserMessage("How does it work?")]                                                     │
+│      runPromptMessages()                                                                                        │
+│                                                                                                                 │
+│   2. runAgentLoop()                                                                                             │
+│      new_messages = [UserMessage("How does it work?")]                                                          │
+│      current_context.messages = vcat([...previous..., UserMessage("How does it work?")])                        │
+│         │                                                                                                       │
+│         └─→ User message added (context preserved from Turn #1)                                                 │
+│      emit(AgentStartEvent), emit(TurnStartEvent)                                                                │
+│      emit(MessageStart/End) for user message                                                                    │
+│                                                                                                                 │
+│   3. runLoop()                                                                                                  │
+│      pending_messages = get_steering() = []                                                                     │
+│                                                                                                                 │
+│   4. streamAssistantResponse()                                                                                  │
+│      convert_to_llm([UserMsg1, AssistantMsg1, UserMsg2]) → Message[]                                            │
+│      LLM call with FULL conversation history (context preserved!)                                               │
+│      receive AssistantMessage: "It works by..."                                                                 │
+│      push AssistantMessage to current_context.messages                                                          │
+│      push AssistantMessage to new_messages                                                                      │
+│                                                                                                                 │
+│   5. emit(TurnEndEvent), emit(AgentEndEvent)                                                                    │
+│                                                                                                                 │
+│   ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │
+│   │ Current context.messages:                                                                                 │ │
+│   │   [UserMsg1, AssistantMsg1, UserMsg2, AssistantMsg2]                                                      │ │
+│   └───────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │
+│                                                                                                                 │
+│   LLM SEES:                                                                                                     │
+│   ┌─────────────────────────────────────────────────────────────────────────────────────────────────┐           │
+│   │ Messages passed to LLM API:                                                                     │           │
+│   │   [UserMessage("What is Julia?"),                                                               │           │
+│   │    AssistantMessage("Julia is..."),                                                             │           │
+│   │    UserMessage("How does it work?"),                                                            │           │
+│   │    AssistantMessage("It works by...")]                                                          │           │
+│   └─────────────────────────────────────────────────────────────────────────────────────────────────┘           │
+│                                                                                                                 │
+└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ STEERING MESSAGES                                                                                               │
+├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                                                 │
+│   What is a steering message?                                                                                   │
+│   • A message (any AgentMessage type) injected via: `agent.steer(message)`                                      │
+│   • Goes into the steering queue, not immediately to context.messages                                           │
+│                                                                                                                 │
+│   How is it created?                                                                                            │
+│   • User code calls: agent.steer(UserMessage("..."))                                                            │
+│   • Or: agent.steer(AssistantMessage("..."))                                                                    │
+│   • Or any other AgentMessage subtype                                                                           │
+│                                                                                                                 │
+│   When is it processed?                                                                                         │
+│   • At the START of the next loop iteration (line 194-202 in agent_loop.jl)                                     │
+│   • AFTER the previous assistant turn completes                                                                 │
+│   • BEFORE the next assistant response is streamed                                                              │
+│                                                                                                                 │
+│   Why use steering?                                                                                             │
+│   Use case 1: Tool execution result injection                                                                   │
+│     - Agent calls a tool (e.g., read_file, bash)                                                                │
+│     - Tool returns result                                                                                       │
+│     - You want to inject a follow-up question based on the result                                               │
+│     - agent.steer(UserMessage("Based on the file, what should we do next?"))                                    │
+│                                                                                                                 │
+│   Use case 2: Multi-turn conversation without user input                                                        │
+│     - Agent responds to user                                                                                    │
+│     - Before user types again, you want to inject a system message                                              │
+│     - agent.steer(BashExecutionMessage(...)) or custom message                                                  │
+│     - This continues the conversation automatically                                                             │
+│                                                                                                                 │
+│   Use case 3: Branch navigation recovery                                                                        │
+│     - User navigates between conversation branches                                                              │
+│     - After switching branches, you want to inject a context message                                            │
+│     - agent.steer(BranchSummaryMessage(...))                                                                    │
+│     - The agent can then continue from the new branch context                                                   │
+│                                                                                                                 │
+│   Use case 4: Compaction summary injection                                                                      │
+│     - Conversation history is compacted                                                                         │
+│     - After compaction, inject summary message                                                                  │
+│     - agent.steer(CompactionSummaryMessage(...))                                                                │
+│     - Agent knows old history was summarized                                                                    │
+│                                                                                                                 │
+│   Example:                                                                                                      │
+│     agent.steer(UserMessage("Follow-up question here"))                                                         │
+│     # This will be processed in the next loop iteration,                                                        │
+│     # appearing in context.messages before the next LLM call                                                    │
+│                                                                                                                 │
+│   The LLM sees:                                                                                                 │
+│   ┌─────────────────────────────────────────────────────────────────────────────────────────────────┐           │
+│   │ All messages become Message[] via convert_to_llm():                                             │           │
+│   │   [UserMessage(...), AssistantMessage(...), UserMessage(from_steer), ...]                       │           │
+│   │                                                                                                 │           │
+│   │ The LLM cannot tell which came from Agent.prompt() vs agent.steer()                             │           │
+│   └─────────────────────────────────────────────────────────────────────────────────────────────────┘           │
+│                                                                                                                 │
+└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ LLM PROCESSING: How LLM sees messages                                                                           │
+├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                                                 │
+│   The LLM NEVER sees "user message" vs "steering message" - it only sees Message types:                         │
+│                                                                                                                 │
+│   ┌─────────────────────────────────────────────────────────────────────────────────────────────────┐           │
+│   │ convert_to_llm() transforms ALL AgentMessages to Message[]:                                     │           │
+│   │                                                                                                 │           │
+│   │   UserMessage("user")             → UserMessage (for LLM)                                       │           │
+│   │   Steering UserMessage("user")    → UserMessage (for LLM) ← Same!                               │           │
+│   │   AssistantMessage("assistant")   → AssistantMessage (for LLM)                                  │           │
+│   │   ToolResultMessage("toolResult") → ToolResultMessage (for LLM)                                 │           │
+│   │                                                                                                 │           │
+│   │   BranchSummaryMessage            → UserMessage (wrapped in summary tags)                       │           │
+│   │   CompactionSummaryMessage        → UserMessage (wrapped in summary tags)                       │           │
+│   │   BashExecutionMessage            → UserMessage (if not excluded)                               │           │
+│   │   CustomMessage                   → UserMessage                                                 │           │
+│   └─────────────────────────────────────────────────────────────────────────────────────────────────┘           │
+│                                                                                                                 │
+│   The difference is ONLY in HOW messages enter the system:                                                      │
+│   • User messages:  Agent.prompt() → vcat() → context.messages (direct)                                         │
+│   • Steering:       agent.steer() → queue → loop → context.messages (indirect)                                  │
+│                                                                                                                 │
+│   At LLM level: BOTH become UserMessage in the conversation!                                                    │
+│                                                                                                                 │
+└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ KEY INSIGHTS                                                                                                    │
+├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
+│                                                                                                                 │
+│   1. User prompts go DIRECTLY to context.messages via vcat() in runAgentLoop()                                  │
+│                                                                                                                 │
+│   2. Steering queue is for messages injected via agent.steer() AFTER a turn finishes                            │
+│      This allows continuing conversation without calling Agent.prompt() again                                   │
+│                                                                                                                 │
+│   3. Context is preserved across turns - context.messages grows with each turn                                  │
+│      LLM sees the full conversation history                                                                     │
+│                                                                                                                 │
+│   4. At LLM level, ALL messages become Message types (UserMessage/AssistantMessage/ToolResultMessage)           │
+│      The "steering" vs "user" distinction is just a control mechanism, not a message type                       │
+│                                                                                                                 │
+│   5. New turn is triggered by:                                                                                  │
+│      - New Agent.prompt() call (adds user messages)                                                             │
+│      - Steering messages (adds steering messages)                                                               │
+│      - Follow-up messages (adds follow-up messages)                                                             │
+│                                                                                                                 │
+└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+```

docs/loadtools.md

@@ -1,92 +0,0 @@
-# Dynamic Tool Loading
-
-Tools can be loaded dynamically from `.jl` files in the `src/tools/` directory without hardcoding filenames in the main module.
-
-## How It Works
-
-1. `src/tools/registry.jl` defines a `loadTools(dir::String)` function that scans a directory for `.jl` files
-2. Each tool file must define a single function: `getTool()::agentTool`
-3. `loadTools()` sorts files alphabetically, includes each one, calls `getTool()`, and registers the result
-4. Loaded tools are returned as `Vector{agentTool}` for use when constructing a `yiemAgent`
-
-## Directory Structure
-
-```
-src/
-├── tools/
-│   ├── registry.jl      # Tool loader (do not edit)
-│   ├── getWeather.jl   # Your tool
-│   └── query_db.jl      # Another tool
-├── type.jl
-├── utils.jl
-├── agentCore.jl
-├── api.jl
-└── YiemAgent.jl
-```
-
-## Creating a Tool
-
-Each `.jl` file in `src/tools/` must define `getTool()` returning an `agentTool`:
-
-```julia
-# src/tools/getWeather.jl
-function getTool()::agentTool
-    return agentTool(
-        name = "getWeather",
-        label = "Weather Lookup",
-        description = "Fetch current weather and forecast for a given city.",
-        inputSchema = Dict{String,Any}(
-            "type" => "object",
-            "properties" => Dict(
-                "city" => Dict("type" => "string", "description" => "City and country"),
-                "units" => Dict("type" => "string", "enum" => ["celsius", "fahrenheit"], "default" => "celsius")
-            ),
-            "required" => ["city"]
-        ),
-        execute = (toolCallId, args, signal, onPartialResult) -> begin
-            city = args["city"]
-            return agentToolResult(
-                [textContent("Weather in $(city): Sunny, 22C")],
-                Dict{Any,Any}(), nothing, false
-            )
-        end,
-        prepareArguments = nothing,
-        parallelToolExecute = false
-    )
-end
-```
-
-No `module` wrapper needed — the registry includes each file in the current module scope so all types (`agentTool`, `textContent`, `agentToolResult`, etc.) resolve correctly.
-
-## Loading Tools
-
-```julia
-using .YiemAgent
-using .YiemAgent: toolRegistry
-
-# Load all tool files from src/tools/
-tools = YiemAgent.loadTools(joinpath(@__DIR__, "src", "tools"))
-
-# Create agent with loaded tools
-agent = yiemAgent(
-    systemPrompt = "You are a helpful assistant.",
-    model = my_model,
-    tools = tools,
-    llmCall = my_llm_call,
-    agentEventSink = my_event_sink
-)
-```
-
-## Available Functions
-
-| Function | Description |
-|----------|-------------|
-| `loadTools(dir::String)` | Scan directory and load all `.jl` tool files |
-| `registerTool(tool::agentTool)` | Register a single tool into the global registry |
-| `getTools()` | Get deep copy of all registered tools |
-| `listTools()` | List all registered tools as `(name, label)` pairs |
-| `clearTools()` | Clear the global registry |
-
-## File Loading Order
-
-Files are sorted alphabetically before loading, so `01_database.jl` loads before `02_weather.jl`. This ensures deterministic registration order.

docs/requirements.md

@@ -0,0 +1,190 @@
+# Requirements
+
+## 1. Business Context & Success Metrics
+
+### Business Goal
+
+The **YiemAgent** project is a Julia reimplementation of the Pi Agent Core framework, designed to provide a stateful agent system for LLM interactions in a wine retail store context. This system enables AI agents to interact with customers, search wine databases, and provide personalized wine recommendations based on customer preferences and store inventory.
+
+### User Stories
+
+- **US-001**: As a wine store customer, I want to interact with an AI sommelier so that I can get personalized wine recommendations
+- **US-002**: As a wine store operator, I want the AI to search our wine database so that I can provide accurate inventory-based recommendations
+- **US-003**: As a developer, I want a reusable agent framework so that I can quickly build custom AI agents for different use cases
+- **US-004**: As a system administrator, I want session persistence so that I can maintain conversation history across agent restarts
+
+### KPIs & Targets
+
+- **KPI-001**: 95% of customer queries receive responses within 3 seconds (measured from query receipt to response delivery)
+- **KPI-002**: 99% of wine searches return results from inventory database within 2 seconds
+- **KPI-003**: Agent session recovery time < 5 seconds after restart
+- **KPI-004**: Conversation context retention accuracy > 95% across session restarts
+
+## 2. Technical Boundaries
+
+### In Scope
+
+- Low-level agent loop with LLM interaction and tool execution
+- High-level Agent struct with state management and event streaming
+- Session persistence with JSONL-based storage
+- Built-in tools: bash execution, file read/write/edit operations
+- Conversation history compaction for context window management
+- Branch-based conversation navigation
+- Event-driven architecture for monitoring and control
+
+### Out of Scope
+
+- LLM model hosting or inference (relies on external services)
+- Frontend UI components (web interface)
+- Database schema design or management
+- User authentication and authorization
+- Multi-tenant isolation
+
+### Dependencies
+
+- Julia 1.9+ runtime
+- JSON3 for JSON parsing
+- UUIDs for session identification
+- Dates for timestamp management
+- LibPQ for PostgreSQL database connections
+- MQTT client for external communication
+
+### Deployment Constraints
+
+- **NFR-501**: System shall be deployed to containerized environment (Docker/Podman)
+- **NFR-502**: Agent instances shall support horizontal scaling
+- **NFR-503**: Session data shall be persisted in shared storage for failover scenarios
+
+## 3. Functional Requirements (FR)
+
+### FR-001: Agent State Management
+
+The system shall maintain conversation state including message history, active tools, and system prompt.
+
+- Store and retrieve conversation history
+- Support multiple concurrent agent sessions
+- Maintain tool state across conversation turns
+- Persist agent state to storage backend
+
+**Traceability**: US-001, US-004
+
+### FR-002: Tool Execution
+
+The system shall execute tools requested by the LLM in response to user queries.
+
+- Support parallel and sequential tool execution modes
+- Handle tool call errors gracefully
+- Return tool results to LLM for processing
+- Support tool result streaming for long-running operations
+
+**Traceability**: US-001
+
+### FR-003: Session Persistence
+
+The system shall persist agent sessions to enable recovery after restart.
+
+- Store session metadata and conversation history in JSONL format
+- Support session creation, opening, and deletion
+- Enable session branching for experiment tracking
+- Support session compaction to reduce storage and context size
+
+**Traceability**: US-004
+
+### FR-004: Event Streaming
+
+The system shall provide real-time event streaming for monitoring agent activity.
+
+- Emit lifecycle events (agent start/end, turn start/end, message start/end)
+- Emit tool execution events (start, update, end)
+- Support event subscription and unsubscription
+- Enable event-driven workflows
+
+**Traceability**: US-001
+
+### FR-005: Conversation Management
+
+The system shall manage conversation flow with support for steering and follow-up messages.
+
+- Support sequential conversation turns
+- Enable message injection after assistant turns (steering)
+- Support follow-up messages that run after natural termination
+- Clear message queues on agent reset
+
+**Traceability**: US-001, US-002
+
+### FR-006: Wine Database Search
+
+The system shall provide tools to search wine inventory databases.
+
+- Execute SQL queries against wine database
+- Support vector similarity search for recommendations
+- Cache similar queries in vector database
+- Handle database connection failures gracefully
+
+**Traceability**: US-002
+
+## 4. Non-Functional Requirements (NFRs)
+
+### 4.1 Performance & Scalability
+
+- **NFR-101**: System shall process messages with <500ms latency for 95th percentile
+- **NFR-102**: System shall support at least 100 concurrent agent sessions
+- **NFR-103**: Tool execution shall complete within 10 seconds for 99% of operations
+- **NFR-104**: Session compaction shall reduce token count by at least 50% with minimal context loss
+
+### 4.2 Availability & Reliability
+
+- **NFR-201**: Agent sessions shall recover from failures within 5 seconds
+- **NFR-202**: System shall maintain conversation continuity across restarts
+- **NFR-203**: Message queues shall not lose messages during normal operation
+- **NFR-204**: Event streaming shall survive temporary subscriber disconnections
+
+### 4.3 Privacy & Security
+
+- **Data Classification**: Commercial wine data, customer preferences
+- **Encryption**: TLS 1.3+ for database connections, encrypted session storage
+- **Authentication**: Database credential management via environment variables
+- **Compliance**: GDPR Article 32 (security of processing)
+
+### 4.4 Observability & Telemetry
+
+- **Required Logs**: `session_id`, `message_id`, `event_type`, `timestamp`, `latency_ms`, `tool_name`
+- **Critical Metrics**:
+  - `agent_sessions_active`
+  - `message_processing_latency_seconds`
+  - `tool_execution_errors_total`
+  - `session_recovery_time_seconds`
+- **Tracing**: B3 propagation for distributed tracing
+- **Alerting**: `tool_execution_error_rate > 5%` triggers PagerDuty
+- **Retention**: Logs: 30 days, Metrics: 90 days
+
+## 5. Acceptance Conditions
+
+- [ ] **FR-001**: Agent maintains conversation state across multiple turns with correct message ordering
+- [ ] **FR-002**: Tools execute correctly with proper error handling and result formatting
+- [ ] **FR-003**: Sessions can be persisted and recovered with complete conversation history
+- [ ] **FR-004**: All agent lifecycle events are emitted and可 captured by subscribers
+- [ ] **FR-005**: Steering messages are injected at correct points in conversation flow
+- [ ] **FR-006**: Wine database search returns results within 2 seconds for 95% of queries
+- [ ] **NFR-101**: 95% of messages processed within 500ms latency
+- [ ] **NFR-201**: Agent sessions recover within 5 seconds after simulated failure
+
+## 6. Requirements Traceability Matrix
+
+| Requirement ID | Description | Implementation File | Test File |
+|----------------|-------------|---------------------|-----------|
+| FR-001 | Agent State Management | `src/agent.jl`, `src/types.jl` | `test/test1.jl` |
+| FR-002 | Tool Execution | `src/agent_loop.jl`, `src/tools/` | `test/prompttest_*.jl` |
+| FR-003 | Session Persistence | `src/session/` | `test/chatting_with_agent.jl` |
+| FR-004 | Event Streaming | `src/agent.jl`, `src/types.jl` | `test/prompttest_*.jl` |
+| FR-005 | Conversation Management | `src/agent.jl`, `src/agent_loop.jl` | `test/chatting_with_agent.jl` |
+| FR-006 | Wine Database Search | `example/main.jl`, `example/agent_chat_virtualCustomer.jl` | N/A |
+| NFR-101 | Performance & Scalability | System-wide | `test/runtests.jl` |
+| NFR-201 | Availability & Reliability | `src/session/`, `src/agent.jl` | `test/chatting_with_agent.jl` |
+
+**Notes**:
+- Functional Requirements (FR) define what the system shall do
+- Non-Functional Requirements (NFR) define system qualities (performance, availability, security, etc.)
+- KPIs are measurable targets that validate whether requirements were met post-deployment
+- Each requirement must include a clear requirement ID for traceability
+- All acceptance conditions must be verifiable through testing or manual inspection

docs/solution-design.md

@@ -0,0 +1,148 @@
+# Solution Design: AgentCore.jl - Julia Agent Framework
+
+## 1. Problem Decomposition
+
+This project addresses several interconnected problems in building AI agent systems:
+
+| Problem | Description | User Impact |
+|---------|-------------|-------------|
+| **P-001**: Complex state management | AI agents need to maintain conversation history, tool states, and system prompts across multiple turns | Without proper state management, conversations lose context and become inconsistent |
+| **P-002**: Tool execution orchestration | LLMs often request multiple tool calls that need to be executed and results returned | Complex coordination required between LLM calls and tool execution |
+| **P-003**: Session persistence | Agent sessions need to survive restarts and support branching for experiments | Loss of conversation history requires re-conversation and poor UX |
+| **P-004**: Event monitoring | Need to observe agent behavior for debugging and operational visibility | Black-box agents are difficult to debug and monitor in production |
+| **P-005**: Context window management | LLMs have limited context windows, requiring history management | Long conversations get truncated, losing important context |
+
+## 2. Solution Approach
+
+The solution implements a layered agent framework with clear separation of concerns:
+
+**Approach**: Implement a low-level agent loop with stateless execution, wrapped in a high-level Agent struct that manages state, queuing, and event streaming. Sessions are persisted to JSONL storage with support for compaction and branching.
+
+**Key Principles**:
+- Separate concerns: low-level loop vs. high-level agent vs. session storage
+- Event-driven architecture: all agent activity is observable via events
+- Extensible tool system: tools are first-class objects with execution logic
+- Immutable core: low-level loop operates on pure data structures
+- Flexible queuing: support for steering and follow-up message queues
+
+## 3. Alternatives Considered
+
+| Alternative | Pros | Cons | Decision |
+|-------------|------|------|----------|
+| **Single monolithic agent class** | Simple to understand, no architectural complexity | Hard to test, difficult to extend, state management becomes complex | Rejected - would not scale for complex deployments |
+| **Actor-based concurrency** | Built-in concurrency model, isolation | Heavy overhead, different semantics than required | Rejected - Julia's async primitives sufficient |
+| **Callback-based event system** | Familiar pattern, lightweight | Difficult to manage subscriptions, error handling complex | Rejected - Julia's async channels better suited |
+| **Full actor model (e.g., GenStage)** | Strong guarantees, backpressure | Overkill for this use case, learning curve | Rejected - simpler event streaming sufficient |
+
+## 4. High-Level Component Diagram
+
+```mermaid
+%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#3b82f6'}}}%%
+flowchart TB
+    subgraph "User Layer"
+        A[User Request]
+        B[Event Subscriber]
+    end
+
+    subgraph "Agent Layer"
+        C[Agent]
+        D[AgentState]
+        E[PendingMessageQueue]
+    end
+
+    subgraph "AgentLoop Layer"
+        F[agentLoop]
+        G[AgentContext]
+        H[AgentLoopConfig]
+    end
+
+    subgraph "Tool Layer"
+        I[Bash Tool]
+        J[Read Tool]
+        K[Write Tool]
+        L[Edit Tool]
+    end
+
+    subgraph "Session Layer"
+        M[Session Storage]
+        N[JSONL Repo]
+        O[InMemory Repo]
+    end
+
+    A --> C
+    B -->|event stream| C
+    C --> D
+    C --> E
+    C -->|start loop| F
+    F --> G
+    F --> H
+    F -->|execute tool| I
+    F -->|execute tool| J
+    F -->|execute tool| K
+    F -->|execute tool| L
+    C -->|persist state| M
+    M --> N
+    M --> O
+```
+
+**Component Descriptions**:
+
+- **Agent** (FR-001, FR-004): High-level interface that manages conversation state, event subscriptions, and message queues. Acts as a facade over the agent loop.
+
+- **AgentLoop** (FR-002): Low-level execution engine that handles LLM calls, tool execution, and event emission. Operates on pure data structures.
+
+- **Session Storage** (FR-003): Persists conversation history and metadata. Supports both JSONL file storage and in-memory storage for testing.
+
+- **Tools** (FR-002): Executable units that perform actions like bash commands, file operations, and database queries. Each tool has execute logic and optional argument preparation.
+
+- **Event System** (FR-004): Publish-subscribe mechanism for observing agent activity. Enables monitoring, debugging, and external integration.
+
+## 5. Decision Rationale
+
+| Decision ID | Decision | Rationale | Alternatives Rejected |
+|-------------|----------|-----------|----------------------|
+| **SD-001**: Separate Agent and AgentLoop | Clear separation of concerns with Agent managing state and AgentLoop handling execution | Keeps low-level loop pure and testable | Combined class would mix concerns and reduce testability |
+| **SD-002**: Event-driven architecture | Enables monitoring, debugging, and extensibility without modifying core logic | Callbacks would be harder to manage and compose | Direct method calls would require tight coupling |
+| **SD-003**: JSONL-based persistence | Simple, human-readable format with good Julia ecosystem support | Binary formats would be harder to debug and inspect | Database dependency would complicate deployment |
+| **SD-004**: Julia type system for type safety | Compile-time guarantees, better IDE support, clearer intent | Runtime checks would be less robust | Dynamic typing would increase bugs in production |
+| **SD-005**: Two-level queuing (steering vs. follow-up) | Supports both immediate conversation correction and post-completion follow-ups | Single queue would not support both use cases | Complex state machine would be needed |
+| **SD-006**: Branch-based session navigation | Enables experiment tracking, rollback, and parallel conversation paths | Linear history would not support A/B testing | Version control system would be overkill |
+
+## 6. Risk Assessment
+
+| Risk | Impact | Probability | Mitigation |
+|------|--------|-------------|------------|
+| **R-001**: Performance degradation with large sessions | High | Medium | Implement session compaction, provide metrics for monitoring |
+| **R-002**: Tool execution failures breaking conversation | High | Medium | Graceful error handling, retry logic, clear error messages to LLM |
+| **R-003**: Data loss from storage failures | High | Low | Support multiple storage backends, implement backup procedures |
+| **R-004**: Event system overwhelming subscribers | Medium | Medium | Implement backpressure, provide filtering options, limit event queue size |
+| **R-005**: Context window exhaustion | Medium | Medium | Automatic compaction, configurable retention policies, monitoring |
+
+## 7. Requirements Traceability
+
+| Solution Component | Requirement ID | Decision ID | Description |
+|-------------------|----------------|-------------|-------------|
+| Agent struct | FR-001 | SD-001 | Manages conversation state with message history and tools |
+| AgentLoop execution | FR-002 | SD-002 | Executes LLM calls and tool calls with proper state handling |
+| Session persistence | FR-003 | SD-003 | JSONL storage with repo abstraction for flexibility |
+| Event system | FR-004 | SD-002 | Publish-subscribe events for monitoring and debugging |
+| Queuing system | FR-005 | SD-005 | Steering and follow-up queues for conversation management |
+| Tool framework | FR-002 | SD-002 | Executable tools with error handling and result reporting |
+| Compaction | FR-003 | SD-003 | Session history management to fit context windows |
+
+## 8. Implementation Guidance
+
+**Module Structure**:
+- `src/types.jl`: Core data types and interfaces
+- `src/agent_loop.jl`: Low-level execution engine
+- `src/agent.jl`: High-level Agent wrapper
+- `src/session/`: Session persistence layer
+- `src/tools/`: Built-in tool implementations
+- `src/messages.jl`: Message transformation logic
+
+**Key Patterns**:
+- Use Julia's multiple dispatch for extensible tool system
+- Implement async channels for event streaming
+- Use immutable data structures where possible for safety
+- Provide both synchronous and asynchronous APIs
+- Design for testability with pure functions in low-level modules

docs/specification.md

@@ -0,0 +1,447 @@
+# Specification: AgentCore.jl Technical Contract
+
+This specification defines the precise technical contracts for the AgentCore.jl system, mapping implementation details to requirements and solution design decisions.
+
+## 1. Agent State Types
+
+### 1.1 AgentState
+
+**Requirement Reference**: FR-001 (Agent State Management)
+
+The `AgentState` struct maintains conversation state with the following fields:
+
+| Field | Type | Description | Requirement ID |
+|-------|------|-------------|----------------|
+| `system_prompt` | `String` | System prompt for the LLM | FR-001 |
+| `model` | `Model` | Current model configuration | FR-001 |
+| `thinking_level` | `ThinkingLevel` | Thinking mode for LLM | FR-001 |
+| `tools` | `Vector{AgentTool}` | Available tools for execution | FR-001 |
+| `messages` | `Vector{AgentMessage}` | Conversation history | FR-001 |
+| `is_streaming` | `Bool` | Streaming state | FR-004 |
+| `streaming_message` | `Union{AgentMessage, Nothing}` | Current streaming message | FR-004 |
+| `pending_tool_calls` | `Set{String}` | Active tool call IDs | FR-002 |
+| `error_message` | `Union{String, Nothing}` | Current error state | FR-002 |
+
+**Specification ID**: SPEC-1.1
+
+### 1.2 ThinkingLevel Enum
+
+**Requirement Reference**: FR-001
+
+| Value | Description | Use Case |
+|-------|-------------|----------|
+| `THINKING_OFF` | No thinking mode | Simple Q&A |
+| `THINKING_MINIMAL` | Minimal chain of thought | Quick decisions |
+| `THINKING_LOW` | Low reasoning effort | Standard operations |
+| `THINKING_MEDIUM` | Moderate reasoning | Complex problems |
+| `THINKING_HIGH` | High reasoning | Difficult reasoning |
+| `THINKING_XHIGH` | Extended reasoning | Multi-step problems |
+| `THINKING_MAX` | Maximum reasoning | Critical decisions |
+
+**Specification ID**: SPEC-1.2
+
+### 1.3 ToolExecutionMode Enum
+
+**Requirement Reference**: FR-002
+
+| Value | Description |
+|-------|-------------|
+| `EXECUTION_SEQUENTIAL` | Execute tools one at a time |
+| `EXECUTION_PARALLEL` | Execute tools concurrently |
+
+**Specification ID**: SPEC-1.3
+
+### 1.4 Message Content Types
+
+**Requirement Reference**: FR-001
+
+| Type | Fields | Description |
+|------|--------|-------------|
+| `TextContent` | `text::String` | Plain text content |
+| `ImageContent` | `data::String, mime_type::String` | Base64-encoded image |
+
+**Specification ID**: SPEC-1.4
+
+## 2. Message Types
+
+### 2.1 AgentMessage Union Type
+
+**Requirement Reference**: FR-001
+
+Abstract type for all agent messages. Concrete types include:
+
+| Type | Role | Description |
+|------|------|-------------|
+| `UserMessage` | user | User input messages |
+| `AssistantMessage` | assistant | LLM responses |
+| `ToolResultMessage` | toolResult | Tool execution results |
+
+**Specification ID**: SPEC-2.1
+
+### 2.2 UserMessage
+
+**Requirement Reference**: FR-001
+
+| Field | Type | Description | Requirement ID |
+|-------|------|-------------|----------------|
+| `role` | `String` | Always "user" | FR-001 |
+| `content` | `Vector{MessageContent}` | Message content (text, images) | FR-001 |
+| `timestamp` | `Timestamp` | Creation timestamp | FR-001 |
+
+**Specification ID**: SPEC-2.2
+
+### 2.3 AssistantMessage
+
+**Requirement Reference**: FR-001, FR-002
+
+| Field | Type | Description | Requirement ID |
+|-------|------|-------------|----------------|
+| `role` | `String` | Always "assistant" | FR-001 |
+| `content` | `Vector{MessageContent}` | Response content | FR-001 |
+| `api` | `String` | API identifier | FR-001 |
+| `provider` | `String` | LLM provider name | FR-001 |
+| `model` | `String` | Model identifier | FR-001 |
+| `usage` | `Usage` | Token usage statistics | FR-001 |
+| `stop_reason` | `String` | Reason for completion | FR-002 |
+| `error_message` | `Union{String, Nothing}` | Error details if failed | FR-002 |
+| `timestamp` | `Timestamp` | Response timestamp | FR-001 |
+
+**Specification ID**: SPEC-2.3
+
+### 2.4 ToolResultMessage
+
+**Requirement Reference**: FR-002
+
+| Field | Type | Description | Requirement ID |
+|-------|------|-------------|----------------|
+| `role` | `String` | Always "toolResult" | FR-002 |
+| `tool_call_id` | `String` | ID of tool call | FR-002 |
+| `tool_name` | `String` | Name of tool | FR-002 |
+| `content` | `Vector{MessageContent}` | Tool result content | FR-002 |
+| `details` | `Any` | Tool-specific details | FR-002 |
+| `usage` | `Union{Usage, Nothing}` | Tool execution usage | FR-002 |
+| `added_tool_names` | `Union{Vector{String}, Nothing}` | Newly available tools | FR-002 |
+| `is_error` | `Bool` | Whether tool failed | FR-002 |
+| `timestamp` | `Timestamp` | Result timestamp | FR-002 |
+
+**Specification ID**: SPEC-2.4
+
+## 3. Tool Interface
+
+### 3.1 AgentTool
+
+**Requirement Reference**: FR-002, Solution Design SD-002
+
+Tools are defined by the `AgentTool` struct:
+
+| Field | Type | Description | Requirement ID |
+|-------|------|-------------|----------------|
+| `name` | `String` | Tool identifier | FR-002 |
+| `label` | `String` | Human-readable label | FR-002 |
+| `description` | `String` | Tool purpose description | FR-002 |
+| `parameters` | `Any` | Parameter schema | FR-002 |
+| `execute` | `Function` | Tool execution function | FR-002 |
+| `prepare_arguments` | `Union{Function, Nothing}` | Argument transformation | FR-002 |
+| `execution_mode` | `Union{ToolExecutionMode, Nothing}` | Execution strategy | FR-002, SD-004 |
+
+**Specification ID**: SPEC-3.1
+
+### 3.2 Tool Execution Contract
+
+**Requirement Reference**: FR-002, Solution Design SD-002
+
+The `execute` function signature:
+```julia
+execute(
+    tool_call_id::String,
+    arguments::Any,
+    signal::Union{Nothing, AbortSignal},
+    on_update::Function
+)::AgentToolResult
+```
+
+**Specification ID**: SPEC-3.2
+
+## 4. Session Storage Interface
+
+### 4.1 SessionTreeEntry
+
+**Requirement Reference**: FR-003
+
+Abstract type for session history entries:
+
+| Type | Description |
+|------|-------------|
+| `MessageEntry` | Conversation message |
+| `ThinkingLevelChangeEntry` | Thinking level change |
+| `ModelChangeEntry` | Model configuration change |
+| `ActiveToolsChangeEntry` | Tool availability change |
+| `CompactionEntry` | History compaction |
+| `BranchSummaryEntry` | Branch summary |
+| `CustomEntry` | Custom entry type |
+| `LabelEntry` | Entry label |
+| `SessionInfoEntry` | Session metadata |
+| `LeafEntry` | Current session leaf |
+
+**Specification ID**: SPEC-4.1
+
+### 4.2 JsonlSessionStorage Interface
+
+**Requirement Reference**: FR-003, Solution Design SD-003
+
+Required methods:
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `getMetadata()` | `SessionMetadata` | Session metadata |
+| `appendEntry(entry)` | `Nothing` | Add history entry |
+| `getEntry(id)` | `Union{SessionTreeEntry, Nothing}` | Retrieve entry by ID |
+| `findEntries(type)` | `Vector{SessionTreeEntry}` | Find entries by type |
+| `getSessionStats()` | `SessionStats` | Session statistics |
+| `getEntries(options)` | `Vector{SessionTreeEntry}` | Query entries |
+
+**Specification ID**: SPEC-4.2
+
+### 4.3 SessionStats
+
+**Requirement Reference**: FR-003
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `message_count` | `Int64` | Number of messages |
+| `cached_tokens` | `Int64` | Cached token count |
+| `uncached_tokens` | `Int64` | Uncached token count |
+| `total_tokens` | `Int64` | Total tokens processed |
+| `cost_total` | `Float64` | Total cost |
+
+**Specification ID**: SPEC-4.3
+
+## 5. Event System
+
+### 5.1 Agent Event Types
+
+**Requirement Reference**: FR-004
+
+| Event | Description | Fields |
+|-------|-------------|--------|
+| `AgentStartEvent` | Agent started | - |
+| `AgentEndEvent` | Agent completed | `messages::Vector{AgentMessage}` |
+| `TurnStartEvent` | New conversation turn | - |
+| `TurnEndEvent` | Conversation turn completed | `message`, `tool_results` |
+| `MessageStartEvent` | Message started | `message` |
+| `MessageEndEvent` | Message completed | `message` |
+| `ToolExecutionStartEvent` | Tool execution started | `tool_call_id`, `tool_name`, `args` |
+| `ToolExecutionEndEvent` | Tool execution completed | `tool_call_id`, `tool_name`, `result`, `is_error` |
+
+**Specification ID**: SPEC-5.1
+
+### 5.2 Event Subscription API
+
+**Requirement Reference**: FR-004
+
+```julia
+subscribe(agent::Agent, listener::Function)::Function
+```
+
+- Returns unsubscription function
+- Listener signature: `(event::AgentEvent, signal::AbortSignal) -> Nothing`
+- Events broadcast to all subscribers concurrently
+
+**Specification ID**: SPEC-5.2
+
+## 6. API Endpoints
+
+### 6.1 Agent Methods
+
+**Requirement Reference**: FR-001, FR-005
+
+| Method | Parameters | Returns | Description |
+|--------|------------|---------|-------------|
+| `prompt(agent, input)` | `input::Union{String, AgentMessage, Vector{AgentMessage}}` | `Nothing` | Start new prompt |
+| `continue!(agent)` | - | `Nothing` | Continue from last message |
+| `steer(agent, message)` | `message::AgentMessage` | `Nothing` | Queue steering message |
+| `followUp(agent, message)` | `message::AgentMessage` | `Nothing` | Queue follow-up message |
+| `reset!(agent)` | - | `Nothing` | Clear all state |
+| `get_state(agent)` | - | `AgentState` | Get current state |
+| `subscribe(agent, listener)` | `listener::Function` | `Function` | Subscribe to events |
+
+**Specification ID**: SPEC-6.1
+
+### 6.2 AgentLoop Functions
+
+**Requirement Reference**: FR-002, Solution Design SD-001
+
+| Function | Parameters | Returns | Description |
+|----------|------------|---------|-------------|
+| `agentLoop()` | `prompts, context, config, signal, stream_fn` | `EventStream` | Run agent loop |
+| `agentLoopContinue()` | `context, config, signal, stream_fn` | `EventStream` | Continue agent loop |
+| `streamAssistantResponse()` | `context, config, signal, emit, stream_fn` | `AssistantMessage` | Stream LLM response |
+
+**Specification ID**: SPEC-6.2
+
+## 7. Error Codes
+
+### 7.1 Agent Errors
+
+**Requirement Reference**: FR-001, FR-002
+
+| Code | Description |
+|------|-------------|
+| `AGENT_BUSY` | Agent already processing |
+| `INVALID_MESSAGE_ROLE` | Invalid message role for operation |
+| `AGENT_NOT_FOUND` | Session not found |
+| `TOOL_NOT_FOUND` | Tool not registered |
+
+**Specification ID**: SPEC-7.1
+
+### 7.2 Tool Errors
+
+**Requirement Reference**: FR-002
+
+| Code | Description |
+|------|-------------|
+| `EXECUTION_TIMEOUT` | Tool execution timed out |
+| `EXECUTION_ABORTED` | Tool execution aborted |
+| `TOOL_NOT_SUPPORTED` | Tool not available |
+| `INVALID_PARAMETERS` | Tool parameters invalid |
+
+**Specification ID**: SPEC-7.2
+
+## 8. Data Validation Rules
+
+### 8.1 Message Content
+
+**Requirement Reference**: FR-001, FR-002
+
+| Constraint | Rule |
+|------------|------|
+| `TextContent.text` | Must be non-empty string |
+| `ImageContent.data` | Must be valid Base64 |
+| `ImageContent.mime_type` | Must be valid MIME type |
+| `AgentMessage.timestamp` | Must be positive integer |
+
+**Specification ID**: SPEC-8.1
+
+### 8.2 Tool Arguments
+
+**Requirement Reference**: FR-002
+
+| Constraint | Rule |
+|------------|------|
+| `AgentTool.name` | Must match regex `^[a-zA-Z_][a-zA-Z0-9_]*$` |
+| `AgentTool.description` | Must be non-empty string |
+| `execute` function | Must return `AgentToolResult` |
+
+**Specification ID**: SPEC-8.2
+
+## 9. Rate Limiting
+
+### 9.1 Message Processing
+
+**Requirement Reference**: NFR-101
+
+| Metric | Limit |
+|--------|-------|
+| Messages per session | 1000 per conversation |
+| Messages per minute | 100 per session |
+| Tool calls per turn | 10 concurrent |
+
+**Specification ID**: SPEC-9.1
+
+### 9.2 Storage Operations
+
+**Requirement Reference**: NFR-101
+
+| Operation | Rate Limit |
+|-----------|------------|
+| Read operations | 1000 per second |
+| Write operations | 100 per second |
+
+**Specification ID**: SPEC-9.2
+
+## 10. Configuration
+
+### 10.1 Agent Options
+
+**Requirement Reference**: FR-001, FR-005
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `systemPrompt` | `String` | `""` | System prompt |
+| `model` | `Model` | Required | LLM model config |
+| `thinkingLevel` | `ThinkingLevel` | `THINKING_OFF` | Thinking mode |
+| `tools` | `Vector{AgentTool}` | `[]` | Available tools |
+| `messages` | `Vector{AgentMessage}` | `[]` | Initial messages |
+| `steeringMode` | `QueueMode` | `QUEUE_ONE_AT_A_TIME` | Steering queue mode |
+| `followUpMode` | `QueueMode` | `QUEUE_ONE_AT_A_TIME` | Follow-up queue mode |
+| `toolExecution` | `ToolExecutionMode` | `EXECUTION_PARALLEL` | Tool execution mode |
+
+**Specification ID**: SPEC-10.1
+
+### 10.2 Session Options
+
+**Requirement Reference**: FR-003, Solution Design SD-003
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `cwd` | `String` | Current directory | Working directory |
+| `path` | `String` | Required | Session storage path |
+| `metadata` | `Dict{String, Any}` | `{}` | Session metadata |
+
+**Specification ID**: SPEC-10.2
+
+## 11. Performance Specifications
+
+### 11.1 Latency Targets
+
+**Requirement Reference**: NFR-101, KPI-001
+
+| Operation | Target Latency | 95th Percentile | 99th Percentile |
+|-----------|---------------|-----------------|-----------------|
+| Message processing | 200ms | 500ms | 1000ms |
+| Tool execution | 500ms | 2000ms | 5000ms |
+| Session recovery | 2000ms | 5000ms | 10000ms |
+
+**Specification ID**: SPEC-11.1
+
+### 11.2 Throughput
+
+**Requirement Reference**: NFR-102
+
+| Metric | Target |
+|--------|--------|
+| Concurrent sessions | 100 |
+| Messages per session per hour | 1000 |
+| Tool calls per minute | 100 |
+
+**Specification ID**: SPEC-11.2
+
+## 12. Traceability Summary
+
+### 12.1 Requirement to Specification Mapping
+
+| Requirement ID | Specification Section | Description |
+|----------------|----------------------|-------------|
+| FR-001 | SPEC-1.x, SPEC-2.x, SPEC-6.1 | Agent state management |
+| FR-002 | SPEC-1.x, SPEC-2.x, SPEC-3.x, SPEC-6.2 | Tool execution |
+| FR-003 | SPEC-4.x, SPEC-10.2 | Session persistence |
+| FR-004 | SPEC-5.x, SPEC-6.1 | Event streaming |
+| FR-005 | SPEC-1.x, SPEC-6.1 | Conversation management |
+| FR-006 | N/A | Wine database (external) |
+| NFR-101 | SPEC-11.x | Performance |
+| NFR-102 | SPEC-11.x | Scalability |
+| NFR-201 | SPEC-4.x, SPEC-6.1 | Availability |
+
+**Specification ID**: SPEC-12.1
+
+### 12.2 Solution Design to Specification Mapping
+
+| Decision ID | Specification Section | Implementation |
+|-------------|----------------------|----------------|
+| SD-001 | SPEC-6.2 | AgentLoop functions |
+| SD-002 | SPEC-3.x, SPEC-5.x | Tool interface, event system |
+| SD-003 | SPEC-4.x | Session storage |
+| SD-004 | SPEC-1.3, SPEC-3.1 | Tool execution modes |
+| SD-005 | SPEC-6.1 | Queuing methods |
+
+**Specification ID**: SPEC-12.2

docs/walkthrough.md

@@ -0,0 +1,559 @@
+# Walkthrough: AgentCore.jl System Flow
+
+This walkthrough traces the end-to-end flow of the AgentCore.jl system, from startup to task completion, showing how all components work together.
+
+## 1. System Startup
+
+### 1.1 Agent Initialization
+
+**User Flow**: System startup and agent instantiation
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                         Agent Initialization                        │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 1. Load Configuration                                               │
+│    - Read config from JSON file                                     │
+│    - Parse database credentials                                     │
+│    - Load tool definitions                                          │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 2. Create Session Repository                                        │
+│    - Choose storage backend (JSONL or in-memory)                   │
+│    - Initialize storage directory                                   │
+│    - Create session metadata                                        │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 3. Instantiate Agent                                                │
+│    - Create AgentState with initial configuration                  │
+│    - Register tools (bash, read, write, edit)                      │
+│    - Set up event subscription system                              │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+**Specification References**: SPEC-6.1 (Agent Methods), SPEC-10.1 (Agent Options)
+
+### 1.2 External Integration Setup
+
+**User Flow**: Connect to external services (database, LLM, MQTT)
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                      External Integration                           │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 1. Database Connections                                             │
+│    - Connect to wine database (LibPQ)                              │
+│    - Connect to vector database                                    │
+│    - Initialize connection pool                                    │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 2. MQTT Client Setup                                                │
+│    - Connect to MQTT broker                                        │
+│    - Subscribe to request topic                                    │
+│    - Set up message callback                                       │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 3. LLM Service Configuration                                        │
+│    - Configure model endpoint                                      │
+│    - Set API key                                                   │
+│    - Configure stream function                                     │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+**Specification References**: NFR-501 (Deployment Constraints), NFR-502 (Scalability)
+
+## 2. Conversation Flow
+
+### 2.1 User Request Handling
+
+**User Flow**: Customer sends message to AI sommelier
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                       Customer Interaction                          │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 1. Receive User Message                                             │
+│    - MQTT message arrives                                          │
+│    - Parse payload (text, images)                                  │
+│    - Generate message ID                                           │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 2. Create User Message Object                                       │
+│    - Construct UserMessage with text content                       │
+│    - Add timestamp                                                 │
+│    - Add to conversation history                                   │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 3. Emit Event                                                       │
+│    - MessageStartEvent                                             │
+│    - MessageEndEvent                                               │
+│    - Forward to subscribers (monitoring, logging)                  │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+**Specification References**: SPEC-2.2 (UserMessage), SPEC-5.1 (Event Types)
+
+### 2.2 Agent Processing Loop
+
+**User Flow**: Agent processes message and prepares response
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                         Agent Processing                            │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 1. Transform Messages for LLM                                       │
+│    - Convert AgentMessage[] to Message[]                           │
+│    - Filter unsupported message types                              │
+│    - Add conversation history                                      │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 2. Create Context Snapshot                                          │
+│    - System prompt                                                 │
+│    - Message history                                               │
+│    - Available tools                                               │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 3. Call LLM Stream Function                                         │
+│    - Build API request                                             │
+│    - Stream LLM response                                           │
+│    - Emit partial messages                                         │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+**Specification References**: SPEC-2.1 (AgentMessage), SPEC-6.2 (AgentLoop)
+
+### 2.3 Tool Execution
+
+**User Flow**: Agent executes tools based on LLM requests
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                        Tool Execution                               │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 1. Parse Tool Calls                                                 │
+│    - Extract tool calls from assistant message                     │
+│    - Validate tool existence                                       │
+│    - Prepare arguments                                             │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 2. Execute Tool (Parallel or Sequential)                            │
+│    ├─ Parallel Mode:                                               │
+│    │   - Spawn concurrent tasks for each tool                      │
+│    │   - Wait for all to complete                                  │
+│    │   - Collect results                                           │
+│    │                                                               │
+│    └─ Sequential Mode:                                             │
+│        - Execute tools one at a time                               │
+│        - Update context after each tool                            │
+│        - Check for early termination                               │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 3. Emit Tool Events                                                 │
+│    - ToolExecutionStartEvent                                       │
+│    - ToolExecutionUpdateEvent (streaming)                          │
+│    - ToolExecutionEndEvent                                         │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+**Specification References**: SPEC-1.3 (ToolExecutionMode), SPEC-3.2 (Tool Execution)
+
+## 3. Tool Implementations
+
+### 3.1 Bash Tool
+
+**User Flow**: Execute shell command
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                        Bash Tool Flow                               │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 1. Validate Arguments                                               │
+│    - Check command is string                                       │
+│    - Validate no dangerous flags                                   │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 2. Execute Command                                                  │
+│    - Spawn subprocess                                              │
+│    - Capture stdout/stderr                                         │
+│    - Set timeout if configured                                     │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 3. Format Result                                                    │
+│    - Combine stdout/stderr                                         │
+│    - Include exit code                                             │
+│    - Truncate if too long (>4096 chars)                            │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 4. Return Tool Result                                               │
+│    - Create AgentToolResult                                        │
+│    - Include usage statistics                                      │
+│    - Mark as error if exit code != 0                               │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+**Specification References**: SPEC-3.1 (AgentTool), SPEC-7.2 (Tool Errors)
+
+### 3.2 Wine Database Search Tool
+
+**User Flow**: Search wine inventory database
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                    Wine Database Search                             │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 1. Parse Query                                                      │
+│    - Extract search criteria                                       │
+│    - Parse price range                                             │
+│    - Extract wine attributes                                       │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 2. Check Vector Cache                                               │
+│    - Get embedding of query                                        │
+│    - Search vector DB for similar queries                          │
+│    - Return cached SQL if close match                              │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 3. Generate SQL Query                                               │
+│    - Build WHERE clauses                                           │
+│    - Add price filters                                             │
+│    - Apply wine type filters                                       │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 4. Execute Database Query                                           │
+│    - Connect to database                                           │
+│    - Run SQL query                                                 │
+│    - Fetch results (DataFrame)                                     │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 5. Format Results                                                   │
+│    - Convert to readable format                                    │
+│    - Include wine name, price, vintage                             │
+│    - Limit to top N results (default 10)                           │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+**Specification References**: FR-006 (Wine Database Search)
+
+## 4. Session Persistence
+
+### 4.1 Saving Conversation History
+
+**User Flow**: Persist conversation to storage
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                     Session Persistence                             │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 1. Create Session Entry                                             │
+│    - Generate unique entry ID                                      │
+│    - Create MessageEntry with message                             │
+│    - Set timestamp and parent ID                                   │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 2. Write to Storage                                                 │
+│    - Serialize entry to JSON                                       │
+│    - Append to JSONL file                                          │
+│    - Update entry index                                            │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 3. Update Session Metadata                                          │
+│    - Increment message count                                       │
+│    - Update token counts                                           │
+│    - Save metadata                                                 │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+**Specification References**: SPEC-4.2 (Session Storage), SPEC-4.3 (SessionStats)
+
+### 4.2 Session Compaction
+
+**User Flow**: Reduce context window usage
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                      Session Compaction                             │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 1. Determine Compaction Point                                       │
+│    - Calculate current token count                                 │
+│    - Check if over threshold                                       │
+│    - Identify messages to summarize                                │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 2. Generate Summary                                                 │
+│    - Extract messages to summarize                                 │
+│    - Call LLM with summary prompt                                  │
+│    - Get compact summary                                           │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 3. Create Compaction Entry                                          │
+│    - Create CompactionEntry                                        │
+│    - Store summary and first kept ID                               │
+│    - Record token savings                                          │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 4. Update Session Tree                                              │
+│    - Replace old messages with summary                             │
+│    - Update leaf pointer                                           │
+│    - Save updated session                                          │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+**Specification References**: FR-003 (Session Persistence)
+
+## 5. Event-Driven Architecture
+
+### 5.1 Event Subscription Flow
+
+**User Flow**: External systems subscribe to agent events
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                    Event Subscription                               │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 1. Subscribe                                                        │
+│    - Create subscriber channel                                     │
+│    - Register listener                                             │
+│    - Return unsubscription function                                │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 2. Event Broadcast                                                  │
+│    - Event emitted (e.g., MessageEndEvent)                        │
+│    - Broadcast to all subscribers                                  │
+│    - Non-blocking delivery                                         │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 3. Event Processing                                                 │
+│    - Logging service consumes events                              │
+│    - Monitoring service aggregates stats                           │
+│    - Debugging tool displays live stream                           │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+**Specification References**: SPEC-5.2 (Event Subscription)
+
+## 6. Error Handling Flow
+
+### 6.1 Tool Execution Error
+
+**User Flow**: Handle tool execution failure
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                     Error Handling Flow                             │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 1. Error Caught                                                    │
+│    - Exception thrown during tool execution                        │
+│    - Error message captured                                        │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 2. Emit Error Event                                                 │
+│    - ToolExecutionEndEvent with error flag                        │
+│    - Include error message                                         │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 3. Create Error Result                                              │
+│    - Create AgentToolResult with error content                     │
+│    - Mark is_error = true                                          │
+│    - Include error details                                         │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│ 4. Send to LLM                                                      │
+│    - Include error result in tool message                          │
+│    - LLM can decide how to proceed                                 │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+**Specification References**: SPEC-7.2 (Tool Errors), SPEC-7.1 (Agent Errors)
+
+## 7. End-to-End Example: Customer Wine Recommendation
+
+### 7.1 Complete User Journey
+
+**User Flow**: Customer asks for wine recommendation
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                End-to-End: Wine Recommendation                      │
+└─────────────────────────────────────────────────────────────────────┘
+
+1. Customer Message (via MQTT)
+   "I'm looking for a French red wine under $100"
+
+2. Agent Processing
+   ├─ Parse query
+   ├─ Extract: country=France, price<100, type=red
+   └─ Determine missing: region, vintage, grape varietal
+
+3. Tool Call: SEARCH_WINE_DATABASE
+   ├─ Query: country=France, type=red, price<100
+   ├─ Execute SQL (with vector cache check)
+   └─ Return 10 matching wines
+
+4. LLM Response
+   ├─ Analyze results
+   ├─ Select top 3 options
+   └─ Format recommendation
+
+5. Response to Customer
+   "I found several French red wines under $100:
+    - Château Le Grand Montmirail 2020 ($75)
+    - Domaine de la Mordorée 2019 ($85)
+    - Louis Latour 2021 ($65)
+    
+    Which one interests you?"
+
+6. Session Persistence
+   ├─ Save conversation to JSONL
+   ├─ Update token counts
+   └─ Update session stats
+```
+
+**Traceability**:
+- FR-001: Agent state management throughout
+- FR-002: Tool execution for database search
+- FR-003: Session persistence after interaction
+- FR-004: Event streaming for monitoring
+- FR-006: Wine database search functionality
+
+**Specification References**: SPEC-6.1 (Agent Methods), SPEC-6.2 (AgentLoop), SPEC-3.x (Tool Interface)
+
+## 8. Performance Characteristics
+
+### 8.1 Message Processing Timeline
+
+**Requirement Reference**: NFR-101, KPI-001
+
+```
+Message Processing Timeline (95th percentile):
+┌─────────────────────────────────────────────────────────────────────┐
+│ 1. Message Receive (MQTT)          50ms                            │
+│ 2. Message Parsing                 30ms                            │
+│ 3. LLM API Call                   800ms                            │
+│ 4. Tool Execution (if needed)     200ms                            │
+│ 5. Result Formatting               20ms                            │
+│ 6. Response Delivery (MQTT)       100ms                            │
+│                                                                     │
+│ Total: 1200ms (95th percentile)                                    │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+### 8.2 Tool Execution Timelines
+
+**Requirement Reference**: NFR-101
+
+| Tool | 50th Percentile | 95th Percentile | 99th Percentile |
+|------|----------------|-----------------|-----------------|
+| Bash | 150ms | 500ms | 1500ms |
+| Read | 100ms | 300ms | 800ms |
+| Write | 100ms | 400ms | 1000ms |
+| Edit | 200ms | 600ms | 1500ms |
+| Database Search | 500ms | 1500ms | 3000ms |
+
+**Specification References**: SPEC-11.1 (Latency Targets)
+
+## 9. Troubleshooting Guide
+
+### 9.1 Common Issues
+
+| Issue | Cause | Resolution |
+|-------|-------|------------|
+| **I-001**: Agent doesn't respond | Event subscribers not registered | Check subscribe() calls, verify MQTT connection |
+| **I-002**: Tool execution fails | Invalid arguments or tool not found | Validate arguments, check tool registration |
+| **I-003**: Session recovery fails | Storage corrupted or missing | Check JSONL files, verify permissions |
+| **I-004**: High latency | Network or LLM service issues | Check network, verify LLM service health |
+| **I-005**: Context window exceeded | Session too long | Implement compaction, reduce history |
+
+**Specification References**: SPEC-7.x (Error Codes)
+
+---
+
+**Document Status**: v1.0  
+**Last Updated**: 2026-07-28  
+**Maintainer**: YiemAgent Development Team

etc.jl

@@ -1,2 +1,110 @@
-# ── executeToolCalls() Julia pseudo code ──────────────────────────
+
-# Full call stack from runLoop → executeToolCalls → prepare → execute → finalize → emit
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+read codebase.
+I need to understand this agent concept deeply. 
+Can you write related documents (.md files) that will help me understand the agent 
+and save in "/home/ton/docker-apps/sommpanion/YiemAgent/learning" folder? 
+I'm learning best in **Top-Down** style so I know how each component are synchonized.
+
+P.S. use diagram to show how process flow and relationship
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

example/agent_chat_virtualCustomer.jl

@@ -0,0 +1,585 @@
+using Revise
+using JSON, JSON, Dates, UUIDs, PrettyPrinting, LibPQ, Base64, DataFrames, DataStructures
+using YiemAgent, GeneralUtils
+using Base.Threads
+
+# ---------------------------------------------- 100 --------------------------------------------- #
+
+
+
+# load config
+config = JSON.parsefile("/appfolder/app/dev/YiemAgent/test/config.json")
+# config = copy(JSON.parsefile("../mountvolume/config.json"))
+
+
+function executeSQL(sql::T) where {T<:AbstractString}
+  host = config[:externalservice][:wineDB][:host]
+  port = config[:externalservice][:wineDB][:port]
+  dbname = config[:externalservice][:wineDB][:dbname]
+  user = config[:externalservice][:wineDB][:user]
+  password = config[:externalservice][:wineDB][:password]
+  DBconnection = LibPQ.Connection("host=$host port=$port dbname=$dbname user=$user password=$password")
+  result = LibPQ.execute(DBconnection, sql)
+  close(DBconnection)
+  return result
+end
+
+function executeSQLVectorDB(sql)
+  host = config[:externalservice][:SQLVectorDB][:host]
+  port = config[:externalservice][:SQLVectorDB][:port]
+  dbname = config[:externalservice][:SQLVectorDB][:dbname]
+  user = config[:externalservice][:SQLVectorDB][:user]
+  password = config[:externalservice][:SQLVectorDB][:password]
+  DBconnection = LibPQ.Connection("host=$host port=$port dbname=$dbname user=$user password=$password")
+  result = LibPQ.execute(DBconnection, sql)
+  close(DBconnection)
+  return result
+end
+
+function text2textInstructLLM(prompt::String; maxattempt::Integer=10, modelsize::String="medium",
+    senderId=GeneralUtils.uuid4snakecase(), timeout=90,
+    llmkwargs=Dict(
+      :num_ctx => 32768,
+      :temperature => 0.5,
+    )
+  )
+  msgMeta = GeneralUtils.generate_msgMeta(
+    config[:externalservice][:loadbalancer][:mqtttopic];
+    msgPurpose="inference",
+    senderName="yiemagent",
+    senderId=senderId,
+    receiverName="text2textinstruct_$modelsize",
+    mqttBrokerAddress=config[:mqttServerInfo][:broker],
+    mqttBrokerPort=config[:mqttServerInfo][:port],
+  )
+
+  outgoingMsg = Dict(
+    :msgMeta => msgMeta,
+    :payload => Dict(
+      :text => prompt,
+      :kwargs => llmkwargs
+    )
+  )
+
+  response = nothing
+  for attempts in 1:maxattempt
+    _response = GeneralUtils.sendReceiveMqttMsg(outgoingMsg; responsetimeout=timeout, responsemaxattempt=maxattempt)
+    payload = _response[:response]
+    if _response[:success] && payload[:text] !== nothing
+      response = _response[:response][:text]
+      break
+    else
+      println("\n<text2textInstructLLM()> attempt $attempts/$maxattempt failed ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+      pprintln(outgoingMsg)
+      println("</text2textInstructLLM()> attempt $attempts/$maxattempt failed ", @__FILE__, ":", @__LINE__, " $(Dates.now())\n")
+      sleep(3)
+    end
+  end
+
+  return response
+end
+
+# get text embedding from a LLM service
+function getEmbedding(text::T) where {T<:AbstractString}
+  msgMeta = GeneralUtils.generate_msgMeta(
+    config[:externalservice][:loadbalancer][:mqtttopic];
+    msgPurpose="embedding",
+    senderName="yiemagent",
+    senderId=sessionId,
+    receiverName="textembedding",
+    mqttBrokerAddress=config[:mqttServerInfo][:broker],
+    mqttBrokerPort=config[:mqttServerInfo][:port],
+  )
+
+  outgoingMsg = Dict(
+    :msgMeta => msgMeta,
+    :payload => Dict(
+      :text => [text] # must be a vector of string
+    )
+  )
+
+  response = GeneralUtils.sendReceiveMqttMsg(outgoingMsg; responsetimeout=120, responsemaxattempt=3)
+  embedding = response[:response][:embeddings]
+  return embedding
+end
+
+function findSimilarTextFromVectorDB(text::T1, tablename::T2, embeddingColumnName::T3,
+  vectorDB::Function; limit::Integer=1
+  )::DataFrame where {T1<:AbstractString, T2<:AbstractString, T3<:AbstractString}
+  # get embedding from LLM service
+  embedding = getEmbedding(text)[1]
+  # check whether there is close enough vector already store in vectorDB. if no, add, else skip 
+  sql = """
+        SELECT *, $embeddingColumnName <-> '$embedding' as distance
+        FROM $tablename
+        ORDER BY distance LIMIT $limit;
+        """
+  response = vectorDB(sql)
+  df = DataFrame(response)
+  return df
+end
+
+function similarSQLVectorDB(query; maxdistance::Integer=100)
+  tablename = "sqlllm_decision_repository"
+  # get embedding of the query
+  df = findSimilarTextFromVectorDB(query, tablename, 
+                                  "function_input_embedding", executeSQLVectorDB)
+  # println(df[1, [:id, :function_output]])
+  row, col = size(df)
+  distance = row == 0 ? Inf : df[1, :distance]  
+  # distance = 100  # CHANGE this is for testing only
+  if row != 0 && distance < maxdistance
+    # if there is usable SQL, return it.
+    output_b64 = df[1, :function_output_base64]   # pick the closest match
+    output_str = String(base64decode(output_b64))
+    rowid = df[1, :id]
+    println("\n~~~ found similar sql. row id $rowid, distance $distance ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+    return (dict=output_str, distance=distance)
+  else
+    println("\n~~~ similar sql not found, max distance $maxdistance ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+    return (dict=nothing, distance=nothing)
+  end
+end
+
+function insertSQLVectorDB(query::T1, SQL::T2; maxdistance::Integer=3) where {T1<:AbstractString, T2<:AbstractString}
+  tablename = "sqlllm_decision_repository"
+  # get embedding of the query
+  # query = state[:thoughtHistory][:question]
+  df = findSimilarTextFromVectorDB(query, tablename, 
+                                  "function_input_embedding", executeSQLVectorDB)
+  row, col = size(df)
+  distance = row == 0 ? Inf : df[1, :distance]
+  if row == 0 || distance > maxdistance # no close enough SQL stored in the database
+    query_embedding = getEmbedding(query)[1]
+    query = replace(query, "'" => "")
+    sql_base64 = base64encode(SQL)
+    sql_ = replace(SQL, "'" => "")
+
+    sql = """
+          INSERT INTO $tablename (function_input, function_output, function_output_base64, function_input_embedding) VALUES ('$query', '$sql_', '$sql_base64', '$query_embedding');
+          """
+    # println("\n~~~ added new decision to vectorDB ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+    # println(sql)
+    _ = executeSQLVectorDB(sql)
+  end
+end
+
+
+function similarSommelierDecision(recentevents::T1; maxdistance::Integer=3
+  )::Union{AbstractDict, Nothing} where {T1<:AbstractString}
+  tablename = "sommelier_decision_repository"
+  # find similar
+  println("\n~~~ search vectorDB for this: $recentevents ", @__FILE__, " ", @__LINE__)
+  df = findSimilarTextFromVectorDB(recentevents, tablename, 
+                                  "function_input_embedding", executeSQLVectorDB)
+  row, col = size(df)
+  distance = row == 0 ? Inf : df[1, :distance]
+  if row != 0 && distance < maxdistance
+    # if there is usable decision, return it.
+    rowid = df[1, :id]
+    println("\n~~~ found similar decision. row id $rowid, distance $distance ", @__FILE__, " ", @__LINE__)
+    output_b64 = df[1, :function_output_base64]   # pick the closest match
+    _output_str = String(base64decode(output_b64))
+    output = copy(JSON.parsefile(_output_str))
+    return output
+  else
+    println("\n~~~ similar decision not found, max distance $maxdistance ", @__FILE__, " ", @__LINE__)
+    return nothing
+  end
+end
+
+
+function insertSommelierDecision(recentevents::T1, decision::T2; maxdistance::Integer=5
+  ) where {T1<:AbstractString, T2<:AbstractDict}
+  tablename = "sommelier_decision_repository"
+  # find similar
+  df = findSimilarTextFromVectorDB(recentevents, tablename, 
+                                  "function_input_embedding", executeSQLVectorDB)
+  row, col = size(df)
+  distance = row == 0 ? Inf : df[1, :distance]
+  if row == 0 || distance > maxdistance # no close enough SQL stored in the database
+    recentevents_embedding = getEmbedding(recentevents)[1]
+    recentevents = replace(recentevents, "'" => "")
+    decision_json = JSON.json(decision)
+    decision_base64 = base64encode(decision_json)
+    decision = replace(decision_json, "'" => "")
+    
+    sql = """
+          INSERT INTO $tablename (function_input, function_output, function_output_base64, function_input_embedding) VALUES ('$recentevents', '$decision', '$decision_base64', '$recentevents_embedding');
+          """
+    println("\n~~~ added new decision to vectorDB ", @__FILE__, " ", @__LINE__)
+    println(sql)
+    _ = executeSQLVectorDB(sql)
+  else
+    println("~~~ similar decision previously cached, distance $distance ", @__FILE__, " ", @__LINE__)
+  end
+end
+
+
+sessionId = GeneralUtils.uuid4snakecase()
+
+externalFunction = (
+    getEmbedding=getEmbedding,
+    text2textInstructLLM=text2textInstructLLM,
+    executeSQL=executeSQL,
+    similarSQLVectorDB=similarSQLVectorDB,
+    insertSQLVectorDB=insertSQLVectorDB,
+    similarSommelierDecision=similarSommelierDecision,
+    insertSommelierDecision=insertSommelierDecision,
+  )
+
+
+# s = "full-bodied red wine, budget 1500 USD"
+# r = YiemAgent.extractWineAttributes_1(agent, s)
+# println(r)
+
+
+# --------------------------- generating scenario and customer profile --------------------------- #
+
+function rolegenerator()
+  rolegenerator_systemmsg =
+  """
+    Your role:
+      - You are a helpful assistant 
+    Your mission:
+      - Create one random role of a potential customer of an internet wine store.
+    You must follow the following guidelines:
+     - the user only need the role, do not add your own words.
+     - the role should be detailed and realistic.
+    You should then respond to the user with:
+      Name: a name of the potential customer
+      Situation: a situation that the potential customer may be facing
+      Mission: a mission of the potential customer
+      Profile: a profile of the potential customer, including their age, gender, occupation, and other relevant information
+    You should only respond in format as described below:
+      Name: ...
+      Situation: ...
+      Mission: ...
+      Profile: ...
+      Additional_information: ...
+
+    Here are some examples:
+    Name: Jimmy
+    Situation:
+      - Your relationship with your boss is not that good. You need to improve your relationship with your boss.
+      - Your boss's wedding anniversary is coming up.
+      - You are at a wine store and start talking with the store's sommelier.
+    Mission:
+      - Ask the sommelier to provide multiple wine options, and subsequently choose one option from the presented list.
+    Profile:
+      - You are a young professional in a big company.
+      - You are avid party goer
+      - You like beer.
+      - You know nothing about wine.
+      - You have a budget of 1500usd.
+    Additional_information:
+      - your boss like spicy food.
+      - your boss is a middle-aged man.
+      - your boss likes Australian wine.
+
+    Name: Kate
+    Situation:
+      - Your husband asked you to get him a bottle of wine. He will gift the wine to his business client while dining at a German restaurant.
+      - Your husband is a business client and he will gift the wine to his business
+      - You are at a wine store and start talking with the store's sommelier.
+    Mission:
+      - Ask the sommelier to provide multiple wine options, and subsequently choose one option from the presented list.
+    Profile:
+      - You are a CEO in a startup company.
+      - You are a nerd
+      - You don't like alcohol.
+      - You have a budget of 150usd.
+      - You don't care about organic, sulfite, gluten-free, or sustainability certified wines
+    Additional_information:
+      - your husband like spicy food.
+      - your husband is a middle-aged man.
+
+    Name: John
+    Situation: 
+      - A local newspaper club wants to have a scoop about wine with local food in the U.S.
+      - You are at a wine store and start talking with the store's sommelier.
+    Mission:
+      - Ask the sommelier to provide multiple wine options, and subsequently choose one option from the presented list.
+    Profile:
+      - I'm a young guy.
+      - I prefer to express my ideas in a succinct and clear manner.
+    Additional_information:
+      - N/A
+
+    Name: Jane
+    Situation:
+      - You have catering a dinner party with French cuisine.
+      - You want to serve wine with your guests.
+      - You are at a wine store and start talking with the store's sommelier.
+    Mission:
+      - Ask the sommelier to provide multiple wine options, and subsequently choose one option from the presented list.
+    Profile:
+      - You are a young French restaurant owner.
+      - You like dry, full-bodied red wine with high tannin
+      - You don't care about organic, sulfite, gluten-free, or sustainability certified wines.
+      - You have a budget of 200 usd.
+    Additional_information:
+      - N/A
+
+    Let's begin!
+  """
+
+  header = ["Name:", "Situation:", "Mission:", "Profile:", "Additional_information:"]
+  dictkey = ["name", "situation", "mission", "profile", "additional_information"]
+  errornote = "N/A"
+
+  for attempt in 1:10
+    _prompt =
+      [
+        Dict(:name => "system", :text => rolegenerator_systemmsg),
+      ]
+    prompt = GeneralUtils.formatLLMtext(_prompt, "qwen3")
+    
+    response = text2textInstructLLM(prompt) # generated role
+    response = GeneralUtils.deFormatLLMtext(response, "qwen3")
+    think, response = GeneralUtils.extractthink(response)
+
+    # check whether response has all header
+    detected_kw = GeneralUtils.detect_keyword(header, response)
+    kwvalue = [i for i in values(detected_kw)]
+    zeroind = findall(x -> x == 0, kwvalue)
+    missingkeys = [header[i] for i in zeroind]
+    if 0 ∈ values(detected_kw)
+      errornote = "$missingkeys are missing from your previous response"
+      println("\nERROR YiemAgent rolegenerator() $errornote ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+      continue
+    elseif sum(values(detected_kw)) > length(header)
+      errornote = "\nYour previous attempt has duplicated points according to the required response format"
+      println("\nERROR YiemAgent rolegenerator() $errornote ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+      continue
+    end
+
+    responsedict = GeneralUtils.textToDict(response, header; 
+                                          dictKey=dictkey, symbolkey=true)
+    responsedict[:id] = GeneralUtils.uuid4snakecase()
+
+    responsedict[:systemmsg] = 
+    """
+    You are role playing as a CUSTOMER of a wine store and you are currently talking with a sommelier of a wine store.
+    Your profile is as follows:
+      Situation: $(responsedict[:situation])
+      Mission: $(responsedict[:mission])
+      Profile: $(responsedict[:profile])
+      Additional_information: $(responsedict[:additional_information])
+
+    You should follow the following guidelines:
+    - Focus on the lastest conversation
+    - Your like to be short and concise
+    - If you don't know an answer to sommelier's question, you should say: I don't know.
+    - If you think the store can't provide what you seek, you can leave.
+
+    You should then respond to the user with:
+      Dialogue: what you want to say to the user
+      Role: Verify that the dialogue is intended for the customer of a wine store. Can be "yes" or "no"
+    You should only respond in format as described below:
+    Dialogue: ...
+    Role: ...
+
+    Let's begin!
+    """
+
+    println("\nrolegenerator() ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+    println(responsedict)
+    return responsedict
+  end
+  error("ERROR rolegenerator() failed to generate customer role: ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+end
+
+
+# Define the external functions for the customer agent in named tuple format
+customer_externalFunction = ( 
+    text2textInstructLLM=text2textInstructLLM,
+  )
+
+
+
+
+function main()
+  agent = YiemAgent.sommelier(
+    externalFunction;
+    name="Jane",
+    id=sessionId,  # agent instance id
+    retailername="Yiem",
+    llmFormatName="qwen3"
+  )
+
+  customerDict = rolegenerator()
+  customer = YiemAgent.virtualcustomer(
+    customer_externalFunction;
+    systemmsg=customerDict[:systemmsg],
+    name=customerDict[:name],
+    id=sessionId,  # agent instance id
+    llmFormatName="qwen3"
+  )
+
+  # customer_chat = "hello"
+  
+  # YiemAgent.addNewMessage(customer, "assistant", customer_chat)
+  # # add user activity to events memory
+  # push!(customer.memory[:events],
+  #   YiemAgent.eventdict(;
+  #     event_description="the assistant talks to the user.",
+  #     timestamp=Dates.now(),
+  #     subject="assistant",
+  #     action_name="CHAT_BOX",
+  #     action_input=customer_chat,
+  #   )
+  # )
+  # println("\ncustomer respond:\n $customer_chat")
+  agent_response = YiemAgent.conversation(agent; maximumMsg=50)
+  println("\nagent respond:\n $agent_response")
+  while true
+    customer_chat = nothing
+    while customer_chat === nothing
+      customer_response = YiemAgent.conversation(customer, Dict(:text=> agent_response); 
+                                    converPartnerName=agent.name,
+                                    maximumMsg=50)
+      customer_response = GeneralUtils.deFormatLLMtext(customer_response, customer.llmFormatName)
+      customer_chat = customer_response
+
+      #[WORKING] check whether customer response the same before
+    end
+    
+    println("\ncustomer respond:\n $customer_chat")
+
+    agent_response = YiemAgent.conversation(agent;
+                                            userinput=Dict(:text=> customer_chat),
+                                            maximumMsg=50)
+    println("\nagent respond:\n $agent_response")
+
+    if haskey(agent.memory[:events][end], :thought)
+      lastAssistantAction = agent.memory[:events][end][:thought][:action_name]
+      if lastAssistantAction == "END_CONVER_GUIDELINE"    # store thoughtDict
+
+        # save a.memory[:shortmem][:decisionlog] to disk using JSON
+        println("\nsaving agent.memory[:shortmem][:decisionlog] to disk")
+        date = "$(Dates.now())"
+        date = replace(date, ':'=>'.')
+        filename = "agent_decision_log_$(date)_$(agent.id).json"
+        filepath = "/appfolder/mountvolume/appdata/log/$filename"
+        open(filepath, "w") do io
+          JSON.pretty(io, agent.memory[:shortmem][:decisionlog])
+        end
+
+        # check how many file in /appfolder/mountvolume/appdata/log/ folder now
+        logfilesnumber = length(readdir("/appfolder/mountvolume/appdata/log/"))
+        println("\nCaching conversation process done. Total $logfilesnumber files in /appfolder/mountvolume/appdata/log/ folder now.\n")
+        break
+      end
+    end
+  end
+end
+
+for i in 1:100
+  main()
+  println("\n Round $i/100 done.")
+end
+
+println("done")
+
+# prompt = 
+# """
+# <|im_start|>system
+# You are a role playing agent acting as:
+#   Name: Emily
+#   Situation: - Emily is planning her upcoming birthday party and wants to make it extra special. She has invited close friends and family, and she's looking for a unique wine that will impress them.
+#   Mission: - Emily needs to find a rare and high-quality wine that matches the theme of her party, which is a mix of classic and modern flavors. She also wants to ensure that the wine is not too expensive so that it won't break her budget.
+#   Profile: - Emily is in her late 20s, works as a marketing executive for a tech company, and has a passion for trying new things. She's organized and detail-oriented but can be spontaneous when it comes to planning events.
+#   Additional_information: - Emily loves experimenting with different types of food and wine pairings.
+
+# Your are currently talking with a sommelier.
+
+# You should follow the following guidelines:
+# - Focus on the lastest conversation
+# - If you satisfy with the sommelier's recommendation for bottle of wine(s), you should say: Thanks for you help. I will buy the wine you recommended.
+# - If you don't satisfy with the sommelier's questions or can't get a good wine recommendation, you can continue the conversation.
+
+# Let's begin!
+
+# <|im_end|>
+# <|im_start|>Jane
+# Hello! Welcome to Yiem's Wine Store. I'm Jane, your friendly sommelier. How can I assist you today? What type of wine are you in the mood for, and is there a special occasion or event on your mind?
+# <|im_end|>
+# <|im_start|>Emily
+# Hi Jane! Thank you so much for welcoming me. For my birthday party, I'm looking for something that combines classic and modern flavors. It's a mix of guests who enjoy both traditional tastes and more contemporary ones. Also, I want to make sure it won't break the bank. Any suggestions?
+# <|im_end|>
+# <|im_start|>Jane
+# Thank you for sharing your preferences, Jane! To better assist you, could you please let me know if there are any specific characteristics of wine you're looking for, such as tannin, sweetness, intensity, or acidity? Additionally, do you have any food items in mind that this wine should pair well with?
+# <|im_end|>
+# <|im_start|>Emily
+# """
+
+# llmkwargs=Dict(
+#       :num_ctx => 32768,
+#       :temperature => 0.3,
+#     )
+# r = text2textInstructLLM(prompt, llmkwargs=llmkwargs)
+# println(r)
+# println(555)
+
+# response = YiemAgent.conversation(agent, Dict(:text=> "I want to get a French red wine under 100."))
+
+
+# while true
+#   println("your respond: ")
+#   user_answer = readline()
+#   response = YiemAgent.conversation(agent, Dict(:text=> user_answer))
+#   println("\n$response")
+# end
+
+
+
+
+
+
+
+
+
+
+
+
+# """
+# Hello
+
+# I would like to get a bottle of wine for my boss but I don't know much about wine. Can you help me?
+
+# well actually, my boss is going to offer the wine to his client as a gift in a business meeting. All I know is his client like spicy food and French wine. I have a budget about 1000.
+
+# """
+
+# input = "French wine, bordeaux, under USD100, pairs with spicy food"
+# r = YiemAgent.extractWineAttributes_1(a, input)
+
+# inventory_order = "French Syrah, Viognier, full bodied, under 100"
+# r = YiemAgent.extractWineAttributes_2(a, inventory_order)
+# pprintln(r)
+
+
+# cron job 
+# @reboot sleep 50 && nvidia-smi -pm 1
+# @reboot sleep 51 && nvidia-smi -i 0 -pl 150
+# @reboot sleep 52 && nvidia-smi -i 1 -pl 150
+# @reboot sleep 53 && nvidia-smi -i 2 -pl 150
+# @reboot sleep 54 && nvidia-smi -i 3 -pl 150
+
+# @reboot sleep 55 && julia -t 2 /home/ton/work/restartContainer/main.jl
+
+# using GeneralUtils
+# msgMeta = GeneralUtils.generate_msgMeta(
+#           "/tonpc_containerServices",
+#           senderName= "somename",
+#           senderId= "1230",
+#           mqttBrokerAddress= "mqtt.yiem.cc",
+#           mqttBrokerPort= 1883,
+#         )
+# outgoingMsg = Dict(
+#   :msgMeta=> msgMeta,
+#   :payload=> "docker container restart playground-app",
+# )
+# GeneralUtils.sendMqttMsg(outgoingMsg)
+

example/config.json

@@ -0,0 +1,76 @@
+{
+  "mqttServerInfo": {
+      "description": "mqtt server info",
+      "port": 1883,
+      "broker": "mqtt.yiem.cc"
+  },
+  "testingOrProduction": {
+      "value": "testing",
+      "description": "agent status, couldbe testing or production"
+  },
+  "agentid": {
+      "value": "2b74b87a-5413-4fe2-a4d3-405891051680",
+      "description": "a unique id for this agent"
+  },
+  "agentCentralConfigTopic": {
+      "mqtttopic": "/yiem_branch_1/agent/sommelier/backend/config/api/v1.1",
+      "description": "a central agent server's topic to get this agent config"
+  },
+  "servicetopic": {
+      "mqtttopic": [
+          "/yiem/hq/agent/sommelier/backend/prompt/api_v1/testing"
+      ],
+      "description": "a topic this agent are waiting for service request"
+  },
+  "role": {
+      "value": "sommelier",
+      "description": "agent role"
+  },
+  "organization": {
+      "value": "yiem_branch_1",
+      "description": "organization name"
+  },
+  "externalservice": {
+    "loadbalancer": {
+      "mqtttopic": "/loadbalancer/requestingservice",
+      "description": "text to text service with instruct LLM"
+    },
+    "text2textinstruct": {
+        "mqtttopic": "/loadbalancer/requestingservice",
+        "description": "text to text service with instruct LLM",
+        "llminfo": {
+            "name": "llama3instruct"
+        }
+    },
+    "virtualWineCustomer_1": {
+        "mqtttopic": "/virtualenvironment/winecustomer",
+        "description": "text to text service with instruct LLM that act as wine customer",
+        "llminfo": {
+            "name": "llama3instruct"
+        }
+    },
+    "text2textchat": {
+        "mqtttopic": "/loadbalancer/requestingservice",
+        "description": "text to text service with instruct LLM",
+        "llminfo": {
+            "name": "llama3instruct"
+        }
+    },
+    "wineDB" : {
+      "description": "A wine database connection info for LibPQ client",
+      "host": "192.168.88.12",
+      "port": 10201,
+      "dbname": "wineDB",
+      "user": "yiemtechnologies",
+      "password": "yiemtechnologies@Postgres_0.0"
+    },
+    "SQLVectorDB" : {
+      "description": "A wine database connection info for LibPQ client",
+      "host": "192.168.88.12",
+      "port": 10203,
+      "dbname": "SQLVectorDB",
+      "user": "yiemtechnologies",
+      "password": "yiemtechnologies@Postgres_0.0"
+    }
+  }
+}

example/main.jl

@@ -0,0 +1,706 @@
+using JSON, JSON, Dates, UUIDs, PrettyPrinting, LibPQ, Base64, DataFrames, DataStructures
+using YiemAgent, GeneralUtils
+using Base.Threads
+
+# ---------------------------------------------- 100 --------------------------------------------- #
+
+
+""" Expected incomming MQTT message format for this service:
+  {
+    "msgMeta": {
+      "msgPurpose": "updateStatus",
+      "requestresponse": "request",
+      "timestamp": "2024-03-29T05:8:48.362",
+      "replyToMsgId": null,
+      "receiverId": null,
+      "getpost": "get",
+      "msgId": "e5c09bd8-7100-4e4e-bb43-05bee589a22c",
+      "acknowledgestatus": null,
+      "sendTopic": "/agent/wine/backend/chat/api/v1/prompt",
+      "receiverName": "agent-wine-backend",
+      "replyTopic": "/agent/wine/frontend/chat/api/v1/txt/receive",
+      "senderName": "agent-wine-frontend-chat",
+      "senderId": "0938a757-e0ee-40a9-8355-5e24906a87cd"
+    },
+    "payload" : {
+      "text": "hello"
+    }
+    
+  }
+"""
+
+
+
+
+
+# load config
+config = copy(JSON.parsefile("../mountvolume/config/config.json"))
+
+""" Instantiate an agent. One need to specify startmessage and one of gpu location info,  
+  Mqtt or Rest. start message must be comply with GeneralUtils's message format
+
+  Arguments\n
+  -----
+    channel::Channel
+      communication channel
+    sessionId::String
+      sesstion ID of the agent
+    agentName::String
+      Name of the agent
+    mqttBroker::String
+      mqtt broker e.g. "tcp://127.0.0.1:1883"
+    agentConfigTopic::String
+      main communication topic for an agent to ask for config
+    timeout::Int64
+      inactivity timeout in minutes. If timeout is reached, an agent will be terminated.
+    
+  Return\n
+  -----
+    a task represent an agent
+
+  Example\n
+  -----
+  ```jldoctest
+  julia> using YiemAgent, GeneralUtils
+  julia> msg = GeneralUtils.generate_msgMeta("/agent")
+  julia> incoming_msg = msg # assuming 1st msg was sent from other app
+  julia> agentConfigTopic = "/agent/wine/backend/config"
+  julia> task = runAgentInstance(incoming_msg, mqttBroker, agentConfigTopic, 60) 
+  ```
+
+  TODO\n
+  -----
+    [] update docstringLAMA_CONTEXT_LENGTH=40960 since the default size is 2048 as you can see in your debug log:
+    [] change how to get result of YiemAgent from let YiemAgent send msg directly to frontend,
+        to    
+          response = YiemAgent.conversation()     
+        then send response to frontend
+
+  Signature\n
+  -----
+"""
+function runAgentInstance(
+    receiveUserMsgChannel::Channel,
+    outputchannel::Channel,
+    sessionId::String,
+    config::Dict,
+    timeout::Int64,
+  )
+
+  function executeSQL(sql::T) where {T<:AbstractString}
+    host = config[:externalservice][:wineDB][:host]
+    port = config[:externalservice][:wineDB][:port]
+    dbname = config[:externalservice][:wineDB][:dbname]
+    user = config[:externalservice][:wineDB][:user]
+    password = config[:externalservice][:wineDB][:password]
+    DBconnection = LibPQ.Connection("host=$host port=$port dbname=$dbname user=$user password=$password")
+    result = LibPQ.execute(DBconnection, sql)
+    close(DBconnection)
+    return result
+  end
+  
+  function executeSQLVectorDB(sql)
+    host = config[:externalservice][:SQLVectorDB][:host]
+    port = config[:externalservice][:SQLVectorDB][:port]
+    dbname = config[:externalservice][:SQLVectorDB][:dbname]
+    user = config[:externalservice][:SQLVectorDB][:user]
+    password = config[:externalservice][:SQLVectorDB][:password]
+    DBconnection = LibPQ.Connection("host=$host port=$port dbname=$dbname user=$user password=$password")
+    result = LibPQ.execute(DBconnection, sql)
+    close(DBconnection)
+    return result
+  end
+  
+  function text2textInstructLLM(prompt::String; maxattempt::Integer=3, modelsize::String="medium",
+    senderId=GeneralUtils.uuid4snakecase(), timeout=180,
+    llmkwargs=Dict(
+      :num_ctx => 32768,
+      :temperature => 0.5,
+    ))
+    msgMeta = GeneralUtils.generate_msgMeta(
+      config[:externalservice][:loadbalancer][:mqtttopic];
+      msgPurpose="inference",
+      senderName="yiemagent",
+      senderId=senderId,
+      receiverName="text2textinstruct_$modelsize",
+      mqttBrokerAddress=config[:mqttServerInfo][:broker],
+      mqttBrokerPort=config[:mqttServerInfo][:port],
+    )
+
+    outgoingMsg = Dict(
+      :msgMeta => msgMeta,
+      :payload => Dict(
+        :text => prompt,
+        :kwargs => llmkwargs
+      )
+    )
+
+    response = nothing
+    for attempts in 1:maxattempt
+      _response = GeneralUtils.sendReceiveMqttMsg(outgoingMsg; timeout=timeout, maxattempt=maxattempt)
+      payload = _response[:response]
+      if _response[:success] && payload[:text] !== nothing
+        response = _response[:response][:text]
+        break
+      else
+        println("\n<text2textInstructLLM()> attempt $attempts/$maxattempt failed ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+        pprintln(outgoingMsg)
+        println("</text2textInstructLLM()> attempt $attempts/$maxattempt failed ", @__FILE__, ":", @__LINE__, " $(Dates.now())\n")
+        sleep(3)
+      end
+    end
+
+    return response
+  end
+  
+  # get text embedding from a LLM service
+  function getEmbedding(text::T) where {T<:AbstractString}
+    msgMeta = GeneralUtils.generate_msgMeta(
+      config[:externalservice][:loadbalancer][:mqtttopic];
+      msgPurpose="embedding",
+      senderName="yiemagent",
+      senderId=sessionId,
+      receiverName="textembedding",
+      mqttBrokerAddress=config[:mqttServerInfo][:broker],
+      mqttBrokerPort=config[:mqttServerInfo][:port],
+    )
+  
+    outgoingMsg = Dict(
+      :msgMeta => msgMeta,
+      :payload => Dict(
+        :text => [text] # must be a vector of string
+      )
+    )
+  
+    response = GeneralUtils.sendReceiveMqttMsg(outgoingMsg; timeout=120, maxattempt=3)
+    embedding = response[:response][:embeddings]
+    return embedding
+  end
+  
+  function findSimilarTextFromVectorDB(text::T1, tablename::T2, embeddingColumnName::T3,
+    vectorDB::Function; limit::Integer=1
+    )::DataFrame where {T1<:AbstractString, T2<:AbstractString, T3<:AbstractString}
+    # get embedding from LLM service
+    embedding = getEmbedding(text)[1]
+    # check whether there is close enough vector already store in vectorDB. if no, add, else skip 
+    sql = """
+          SELECT *, $embeddingColumnName <-> '$embedding' as distance
+          FROM $tablename
+          ORDER BY distance LIMIT $limit;
+          """
+    response = vectorDB(sql)
+    df = DataFrame(response)
+    return df
+  end
+  
+  function similarSQLVectorDB(query; maxdistance::Integer=100)
+    tablename = "sqlllm_decision_repository"
+    # get embedding of the query
+    df = findSimilarTextFromVectorDB(query, tablename, 
+                                    "function_input_embedding", executeSQLVectorDB)
+    # println(df[1, [:id, :function_output]])
+    row, col = size(df)
+    distance = row == 0 ? Inf : df[1, :distance]  
+    # distance = 100  # CHANGE this is for testing only
+    if row != 0 && distance < maxdistance
+      # if there is usable SQL, return it.
+      output_b64 = df[1, :function_output_base64]   # pick the closest match
+      output_str = String(base64decode(output_b64))
+      rowid = df[1, :id]
+      println("\n~~~ found similar sql. row id $rowid, distance $distance ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+      return (dict=output_str, distance=distance)
+    else
+      println("\n~~~ similar sql not found, max distance $maxdistance ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+      return (dict=nothing, distance=nothing)
+    end
+  end
+  
+  function insertSQLVectorDB(query::T1, SQL::T2; maxdistance::Integer=3) where {T1<:AbstractString, T2<:AbstractString}
+    tablename = "sqlllm_decision_repository"
+    # get embedding of the query
+    # query = state[:thoughtHistory][:question]
+    df = findSimilarTextFromVectorDB(query, tablename, 
+                                    "function_input_embedding", executeSQLVectorDB)
+    row, col = size(df)
+    distance = row == 0 ? Inf : df[1, :distance]
+    if row == 0 || distance > maxdistance # no close enough SQL stored in the database
+      query_embedding = getEmbedding(query)[1]
+      query = replace(query, "'" => "")
+      sql_base64 = base64encode(SQL)
+      sql_ = replace(SQL, "'" => "")
+  
+      sql = """
+            INSERT INTO $tablename (function_input, function_output, function_output_base64, function_input_embedding) VALUES ('$query', '$sql_', '$sql_base64', '$query_embedding');
+            """
+      # println("\n~~~ added new decision to vectorDB ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+      # println(sql)
+      _ = executeSQLVectorDB(sql)
+    end
+  end
+  
+  
+  function similarSommelierDecision(recentevents::T1; maxdistance::Integer=3
+    )::Union{AbstractDict, Nothing} where {T1<:AbstractString}
+    tablename = "sommelier_decision_repository"
+    # find similar
+    println("\n~~~ search vectorDB for this: $recentevents ", @__FILE__, " ", @__LINE__)
+    df = findSimilarTextFromVectorDB(recentevents, tablename, 
+                                    "function_input_embedding", executeSQLVectorDB)
+    row, col = size(df)
+    distance = row == 0 ? Inf : df[1, :distance]
+    if row != 0 && distance < maxdistance
+      # if there is usable decision, return it.
+      rowid = df[1, :id]
+      println("\n~~~ found similar decision. row id $rowid, distance $distance ", @__FILE__, " ", @__LINE__)
+      output_b64 = df[1, :function_output_base64]   # pick the closest match
+      _output_str = String(base64decode(output_b64))
+      output = copy(JSON.parsefile(_output_str))
+      return output
+    else
+      println("\n~~~ similar decision not found, max distance $maxdistance ", @__FILE__, " ", @__LINE__)
+      return nothing
+    end
+  end
+  
+  
+  function insertSommelierDecision(recentevents::T1, decision::T2; maxdistance::Integer=5
+    ) where {T1<:AbstractString, T2<:AbstractDict}
+    tablename = "sommelier_decision_repository"
+    # find similar
+    df = findSimilarTextFromVectorDB(recentevents, tablename, 
+                                    "function_input_embedding", executeSQLVectorDB)
+    row, col = size(df)
+    distance = row == 0 ? Inf : df[1, :distance]
+    if row == 0 || distance > maxdistance # no close enough SQL stored in the database
+      recentevents_embedding = getEmbedding(recentevents)[1]
+      recentevents = replace(recentevents, "'" => "")
+      decision_json = JSON.json(decision)
+      decision_base64 = base64encode(decision_json)
+      decision = replace(decision_json, "'" => "")
+      
+      sql = 
+      """
+      INSERT INTO $tablename (function_input, function_output, function_output_base64, function_input_embedding) VALUES ('$recentevents', '$decision', '$decision_base64', '$recentevents_embedding');
+      """
+      println("\n~~~ added new decision to vectorDB ", @__FILE__, " ", @__LINE__)
+      println(sql)
+      _ = executeSQLVectorDB(sql)
+    else
+      println("~~~ similar decision previously cached, distance $distance ", @__FILE__, " ", @__LINE__)
+    end
+  end
+
+  # keepaliveChannel_2::Channel{Dict} = Channel{Dict}(8)
+  latestUserMsgTimeStamp::DateTime = Dates.now()
+
+  externalFunction = (
+    getEmbedding=getEmbedding,
+    text2textInstructLLM=text2textInstructLLM,
+    executeSQL=executeSQL,
+    similarSQLVectorDB=similarSQLVectorDB,
+    insertSQLVectorDB=insertSQLVectorDB,
+    similarSommelierDecision=similarSommelierDecision,
+    insertSommelierDecision=insertSommelierDecision,
+  )
+
+  agent = YiemAgent.sommelier(
+    externalFunction;
+    name="Jane",
+    id=sessionId,  # agent instance id
+    retailername="Yiem",
+    llmFormatName="qwen3"
+  )
+
+  # user chat loop
+  while true
+    # check for new user message
+    if isready(receiveUserMsgChannel)
+      incomingMsg = take!(receiveUserMsgChannel)
+      incoming_msgMeta = incomingMsg[:msgMeta]
+      incomingPayload = incomingMsg[:payload]
+      latestUserMsgTimeStamp = Dates.now()
+
+      # make sure the message has :text key because YiemAgent use this key for incoming user msg
+      if haskey(incomingPayload, :text)
+        # skip, msg already has correct key name
+      elseif haskey(incomingPayload, :txt)
+        # change key name to text
+        incomingPayload[:text] = incomingPayload[:txt]
+      else
+        error("\n no :txt or :text key in the message.")
+      end
+
+      # reset agent
+      if occursin("newtopic", incomingPayload[:text]) || 
+        occursin("Newtopic", incomingPayload[:text]) ||
+        occursin("New topic", incomingPayload[:text]) ||
+        occursin("new topic", incomingPayload[:text])
+        # YiemAgent.clearhistory(agent)
+
+        agent = YiemAgent.sommelier(
+          externalFunction;
+          name="Janie",
+          id=sessionId,  # agent instance id
+          retailername="Yiem",
+        )
+
+        # sending msg back to sender i.e. LINE
+        msgMeta = GeneralUtils.generate_msgMeta(
+          incomingMsg[:msgMeta][:replyTopic];
+          senderName="wine_assistant_backend",
+          senderId=sessionId,
+          replyToMsgId=incomingMsg[:msgMeta][:msgId],
+          mqttBrokerAddress=config[:mqttServerInfo][:broker],
+          mqttBrokerPort=config[:mqttServerInfo][:port],
+        )
+        outgoingMsg = Dict(
+          :msgMeta => msgMeta,
+          :payload => Dict(
+            :alias => agent.name,       # will be shown in frontend as agent name
+            :text => "Okay. What shall we talk about?"
+          )
+        )
+        _ = GeneralUtils.sendMqttMsg(outgoingMsg)
+        println("--> outgoingMsg ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+        pprintln(outgoingMsg)
+      else
+        usermsg = incomingPayload
+
+        if incoming_msgMeta[:msgPurpose] == "initialize"
+          println("\n-- Initializing... ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+        end
+
+        # send prompt
+        result = YiemAgent.conversation(agent;
+                                            userinput=usermsg,
+                                            maximumMsg=50)
+        # Ken's bot use [br] for newline character '\n'
+        # result = replace(result, '\n'=>"[br]")
+
+        if incoming_msgMeta[:msgPurpose] == "initialize"
+          println("\n-- Initialized. Ready! waiting for request at:\n$(config[:servicetopic][:mqtttopic]) ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+          continue
+        end
+
+        msgMeta = GeneralUtils.generate_msgMeta(
+          incomingMsg[:msgMeta][:replyTopic];
+          senderName="wine_assistant_backend",
+          senderId=string(uuid4()),
+          replyToMsgId=incomingMsg[:msgMeta][:msgId],
+          mqttBrokerAddress=config[:mqttServerInfo][:broker],
+          mqttBrokerPort=config[:mqttServerInfo][:port],
+        )
+
+        outgoingMsg = Dict(
+          :msgMeta => msgMeta,
+          :payload => Dict(
+            :alias => agent.name,       # will be shown in frontend as agent name
+            :text => result
+          )
+        )
+        _ = GeneralUtils.sendMqttMsg(outgoingMsg)
+        println("\n--> outgoingMsg ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+        pprintln(outgoingMsg)
+
+
+        # jpg_as_juliaStr = nothing
+        # prompt = nothing
+
+        # if haskey(payload, "img")
+        #   url_or_base64 = payload["img"]
+
+        #   if startswith(url_or_base64, "http")
+        #     # img in http
+        #     julia_rgb_img, cv2_bgr_img = ImageUtils.url_to_cv2_image(url_or_base64)
+        #     _, buffer = cv2.imencode(".jpg", cv2_bgr_img)
+        #     jpg_as_pyStr = base64.b64encode(buffer).decode("utf-8")
+        #     jpg_as_juliaStr = pyconvert(String, jpg_as_pyStr)
+        #   else
+        #     # img in base64
+        #     cv2_bgr_img = payload["img"]
+        #     jpg_as_juliaStr = pyconvert(String, jpg_as_pyStr)
+        #   end
+        # end
+
+      end
+    else
+      # println("\n no msg")
+    end
+
+    if haskey(agent.memory[:events][end], :thought)
+      lastAssistantAction = agent.memory[:events][end][:thought][:action_name]
+      if lastAssistantAction == "END_CONVER_GUIDELINE"    # store thoughtDict
+
+        # save a.memory[:shortmem][:decisionlog] to disk using JSON
+        println("\nsaving agent.memory[:shortmem][:decisionlog] to disk")
+        filename = "agent_decision_log_$(Dates.now())_$(agent.id).json"
+        filepath = "/appfolder/app/log/$filename"
+        open(filepath, "w") do io
+          JSON.pretty(io, agent.memory[:shortmem][:decisionlog])
+        end
+        
+        # for (i, event) in enumerate(agent.memory[:events])
+        #   if event[:subject] == "assistant"
+        #     # create timeline of the last 3 conversation except the last one. 
+        #     # The former will be used as caching key and the latter will be the caching target
+        #     # in vector database
+        #     all_recapkeys = keys(agent.memory[:recap])  #[TESTING] recap as caching 
+        #     all_recapkeys_vec = [r for r in all_recapkeys]  # convert to a vector
+
+        #     # select from 1 to 2nd-to-lase event (i.e. excluding the latest which is assistant's response)
+        #     _recapkeys_vec = all_recapkeys_vec[1:i-1]  
+
+        #     # select only previous 3 recaps
+        #     recapkeys_vec =
+        #       if length(_recapkeys_vec) <= 3  # 1st message is a user's hello msg
+        #         _recapkeys_vec  # choose all
+        #       else 
+        #         _recapkeys_vec[end-2:end]
+        #       end
+        #     #[PENDING] if there is specific data such as number, donot store in database
+        #     tempmem = DataStructures.OrderedDict()
+        #     for k in recapkeys_vec
+        #       tempmem[k] = agent.memory[:recap][k]
+        #     end
+
+        #     recap = GeneralUtils.dictToString_noKey(tempmem)
+        #     thoughtDict = agent.memory[:events][i][:thought] # latest assistant thoughtDict
+        #     insertSommelierDecision(recap, thoughtDict)
+        #   else
+        #     # skip
+        #   end
+        # end
+        println("\nCaching conversation process done")
+        break
+      end
+    end
+
+    # self terminate if too long inactivity
+    timediff = GeneralUtils.timedifference(latestUserMsgTimeStamp, Dates.now(), "minutes")
+    if timediff > timeout
+
+      result = Dict(:exitreason => "timeout", :timestamp => Dates.now())
+      put!(outputchannel, result)
+      println("Agent ID $(agent.id) timeout has been reached  $timediff/$timeout minutes Send delete session msg ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+
+      # send "delete session" message to inform the main loop that this session can be deleted
+      sendto = 
+        if typeof(config[:servicetopic][:mqtttopic]) <: Array
+          config[:servicetopic][:mqtttopic][1]
+        else 
+          config[:servicetopic][:mqtttopic]
+        end
+
+      msgMeta = GeneralUtils.generate_msgMeta(
+        sendto;
+        senderName="session",
+        senderId=sessionId,
+        msgPurpose="delete session",
+        mqttBrokerAddress=config[:mqttServerInfo][:broker],
+        mqttBrokerPort=config[:mqttServerInfo][:port],
+      )
+
+      outgoingMsg = Dict(
+        :msgMeta => msgMeta,
+        :payload => nothing
+      )
+      _ = GeneralUtils.sendMqttMsg(outgoingMsg)
+
+      try disconnect(agent.mqttClient) catch end
+      break
+    end
+    sleep(1) # allowing on_msg_2, asyncmove above and other process to run
+  end
+end
+
+sessionDict = Dict{String,Any}()
+incomingMsgChannel = (ch1=Channel(8),)  # store msg that coming into servicetopic
+# incommingInternalMsg = [] # st                        ore msg that coming into servicetopic internal management
+keepaliveChannel::Channel{Dict} = Channel{Dict}(8)
+
+# Define the callback for receiving messages.
+function onMsgCallback_1(topic, payload)
+  jobj = JSON.parsefile(String(payload))
+  incomingMqttMsg = copy(jobj)  # convert json object into julia dictionary recursively
+
+  if occursin("keepalive", topic)
+    put!(keepaliveChannel, incomingMqttMsg)
+  else
+    put!(incomingMsgChannel[:ch1], incomingMqttMsg)
+  end
+end
+
+mqttInstance = GeneralUtils.mqttClientInstance_v2(
+  config[:mqttServerInfo][:broker],
+  config[:servicetopic][:mqtttopic],
+  incomingMsgChannel,
+  keepaliveChannel,
+  onMsgCallback_1
+)
+
+# ------------------------------------------------------------------------------------------------ #
+#                                      this service main loop                                      #
+# ------------------------------------------------------------------------------------------------ #
+
+function main()
+  sessiontimeout = 1 * 1 * 60  # timeout in minute for each instance (day * hour * minute)
+  initializing = false
+  while true
+    # check if mqtt connection is still up
+    _ = GeneralUtils.checkMqttConnection!(mqttInstance; keepaliveCheckInterval=30)
+
+    # initialize session 0
+    if initializing == false  # send init msg
+      sendto = 
+        if typeof(config[:servicetopic][:mqtttopic]) <: Array
+          config[:servicetopic][:mqtttopic][1]
+        else 
+          config[:servicetopic][:mqtttopic]
+        end
+
+      msgMeta = GeneralUtils.generate_msgMeta(
+        sendto;
+        msgPurpose="initialize",
+        senderName="initializer",
+        senderId="0",
+        msgId= "initMsg",
+        replyTopic=sendto,
+        mqttBrokerAddress=config[:mqttServerInfo][:broker],
+        mqttBrokerPort=config[:mqttServerInfo][:port],
+      )
+
+      outgoingMsg = Dict(
+        :msgMeta => msgMeta,
+        :payload => Dict(    # will be shown in frontend as agent name
+          :text => "Do you have full-bodied red wines under 100 USD. I don't have any other preferences."
+        )
+      )
+      _ = GeneralUtils.sendMqttMsg(outgoingMsg)
+      initializing = true
+      println("\n--> Initializing msg sent ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+    end
+
+    # check for new message
+    if !isempty(incomingMsgChannel[:ch1])
+      msg = popfirst!(incomingMsgChannel[:ch1])
+      println("\n<-- incomingMsg ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+      pprintln(msg)
+
+      # @spawn new runAgentInstance and store it in sessionDict
+      # use agent's frontend id because 1 backend agent per 1 frontend session
+      sessionId = msg[:msgMeta][:senderId]
+      sessionId = replace(sessionId, "-" => "_") # julia can't use "-" in a dict key 
+
+      # check for delete session msg
+      if msg[:msgMeta][:msgPurpose] == "delete session"
+        delete!(sessionDict, sessionId)
+        println("sessionId $(sessionId) has been terminated ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+
+      # no session yet, create new session
+      elseif sessionId ∉ keys(sessionDict)
+        inputch = Channel{Dict}(8)
+        outputch = Channel{Dict}(8)
+
+        process = @spawn runAgentInstance(inputch, outputch, sessionId, config, sessiontimeout)
+        # process = runAgentInstance(inputch, outputch, sessionId, config, sessiontimeout) #XXX use spawn version 
+
+        println("\ninstantiate agent success ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+
+        # call runAgentInstance() and store it in sessionDict to be able to check on it later
+        sessionDict[sessionId] = Dict(
+          :inputchannel => inputch,
+          :outputchannel => outputch,
+          :process => process,
+        )
+        put!(sessionDict[sessionId][:inputchannel], msg)
+      # ongoing session
+      else
+        println("sessionId $(sessionId) existing session ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+        put!(sessionDict[sessionId][:inputchannel], msg)
+      end
+    end
+
+    # sleep is needed because MQTTClient use async. "while true" loop leave no 
+    # chance for control to switch to on_msg()
+    sleep(1)
+  end
+end
+
+main()
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

fast_inference_guideline.txt

@@ -1,72 +0,0 @@
-To make **LLM-driven inference** fast while maintaining its dynamic capabilities, there are a few practices or approaches to avoid, as they could lead to performance bottlenecks or inefficiencies. Here's what *not* to do:
-
----
-
-### **1. Avoid Using Overly Large Models for Every Query**
-While larger LLMs like GPT-4 provide high accuracy and nuanced responses, they may slow down real-time processing due to their computational complexity. Instead:
-- Use distilled or smaller models (e.g., GPT-3.5 Turbo or fine-tuned versions) for faster inference without compromising much on quality.
-
----
-
-### **2. Avoid Excessive Entity Preprocessing**
-Don’t rely on overly complicated preprocessing steps (like advanced NER models or regex-heavy pipelines) to extract entities from the query before invoking the LLM. This could add latency. Instead:
-- Design efficient prompts that allow the LLM to extract entities and generate responses simultaneously.
-
----
-
-### **3. Avoid Asking the LLM Multiple Separate Questions**
-Running the LLM for multiple subtasks—for example, entity extraction first and response generation second—can significantly slow down the pipeline. Instead:
-- Create prompts that combine tasks into one pass, e.g., *"Identify the city name and generate a weather response for this query: 'What's the weather in London?'"*.
-
----
-
-### **4. Don’t Overload the LLM with Context History**
-Excessively lengthy conversation history or irrelevant context in your prompts can slow down inference times. Instead:
-- Provide only the relevant context for each query, trimming unnecessary parts of the conversation.
-
----
-
-### **5. Avoid Real-Time Dependence on External APIs**
-Using external APIs to fetch supplementary data (e.g., weather details or location info) during every query can introduce latency. Instead:
-- Pre-fetch API data asynchronously and use the LLM to integrate it dynamically into responses.
-
----
-
-### **6. Avoid Running LLM on Underpowered Hardware**
-Running inference on CPUs or low-spec GPUs will result in slower response times. Instead:
-- Deploy the LLM on optimized infrastructure (e.g., high-performance GPUs like NVIDIA A100 or cloud platforms like Azure AI) to reduce latency.
-
----
-
-### **7. Skip Lengthy Generative Prompts**
-Avoid prompts that encourage the LLM to produce overly detailed or verbose responses, as these take longer to process. Instead:
-- Use concise prompts that focus on generating actionable or succinct answers.
-
----
-
-### **8. Don’t Ignore Optimization Techniques**
-Failing to optimize your LLM setup can drastically impact performance. For example:
-- Avoid skipping techniques like model quantization (reducing numerical precision to speed up inference) or distillation (training smaller models).
-
----
-
-### **9. Don’t Neglect Response Caching**
-While you may not want a full caching system to avoid sunk costs, dismissing lightweight caching entirely can impact speed. Instead:
-- Use temporary session-based caching for very frequent queries, without committing to a full-fledged cache infrastructure.
-
----
-
-### **10. Avoid One-Size-Fits-All Solutions**
-Applying the same LLM inference method to all queries—whether simple or complex—will waste processing resources. Instead:
-- Route basic queries to faster, specialized models and use the LLM for nuanced or multi-step queries only.
-
----
-
-### Summary: Focus on Efficient Design
-By avoiding these pitfalls, you can ensure that LLM-driven inference remains fast and responsive:
-- Optimize prompts.
-- Use smaller models for simpler queries.
-- Run the LLM on high-performance hardware.
-- Trim unnecessary preprocessing or contextual steps.
-
-Would you like me to help refine a prompt or suggest specific tools to complement your implementation? Let me know!

learning/01-ARCHITECTURE_OVERVIEW.md

@@ -0,0 +1,498 @@
+# AgentCore.jl - Architecture Overview
+
+## Top-Down Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                        AgentCore.jl Layers                              │
+└─────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────┐
+│  Level 1: AgentHarness (Session Management & Persistence)               │
+│  - Session persistence with JSONL storage                               │
+│  - Resource management (skills, prompt templates)                       │
+│  - Extension hooks system                                               │
+│  - Branch navigation and compaction                                     │
+└─────────────────────────────────────────────────────────────────────────┘
+                              │
+                              │ orchestrates
+                              ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│  Level 2: Agent (State Management & Event Streaming)                    │
+│  - Conversation state (messages, tools, system prompt)                  │
+│  - Event streaming and lifecycle management                             │
+│  - Steering and follow-up message queues                                │
+│  - Abort handling                                                       │
+└─────────────────────────────────────────────────────────────────────────┘
+                              │
+                              │ delegates to
+                              ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│  Level 3: AgentLoop (Core LLM Interaction Loop)                         │
+│  - Stateful LLM interactions                                            │
+│  - Tool execution (parallel or sequential)                              │
+│  - Event emission lifecycle                                             │
+│  - Steering/follow-up message handling                                  │
+└─────────────────────────────────────────────────────────────────────────┘
+                              │
+                              │ transforms to
+                              ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│  Level 4: Session (Conversation History Management)                     │
+│  - Tree-based conversation history                                      │
+│  - Branch support with compaction                                       │
+│  - Message and metadata persistence                                     │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+## Process Flow
+
+### 1. Agent Lifecycle
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                         Agent Lifecycle                                 │
+└─────────────────────────────────────────────────────────────────────────┘
+
+    User Code
+         │
+         │ 1. Create Agent
+         ▼
+    ┌──────────────┐
+    │   Agent()    │  ──► Initialize state, queues, listeners
+    └──────────────┘
+         │
+         │ 2. Subscribe to events
+         ▼
+    ┌──────────────────┐
+    │ subscribe()      │  ──► Register event handlers
+    └──────────────────┘
+         │
+         │ 3. Run prompt
+         ▼
+    ┌──────────────────┐
+    │ prompt()         │  ──► Validate input, normalize messages
+    └──────────────────┘
+         │
+         │ 4. Start AgentLoop
+         ▼
+    ┌──────────────────┐
+    │ runPromptMessages│  ──► Create ActiveRun, spawn loop
+    └──────────────────┘
+         │
+         ▼
+    ┌──────────────────────────────────────────────────────────────┐
+    │  AgentLoop (runs in separate thread)                         │
+    │                                                              │
+    │  ┌────────────────────────────────────────────────────────┐  │
+    │  │ 1. Emit AgentStartEvent                                │  │
+    │  │ 2. Emit TurnStartEvent                                 │  │
+    │  │ 3. Process prompts (emit MessageStart/End)             │  │
+    │  │ 4.┌────────────────────────────────────────────────┐   │ │
+    │  │   │ while true:                                    │   │ │
+    │  │   │   │ Process steering/follow-up messages        │   │ │
+    │  │   │   │ Stream assistant response (LLM call)       │   │ │
+    │  │   │   │ Execute tool calls (parallel/sequential)   │   │ │
+    │  │   │   │ Emit TurnEndEvent                          │   │ │
+    │  │   │   │ Check if should stop                       │   │ │
+    │  │   │   │ Get next steering messages                 │   │ │
+    │  │   └───┴────────────────────────────────────────────┘   │ │
+    │  └────────────────────────────────────────────────────────┘  │
+    └──────────────────────────────────────────────────────────────┘
+         │
+         │ 5. Event streaming
+         ▼
+    ┌──────────────────┐
+    │ Event Handlers   │  ──► User-defined listeners receive events
+    └──────────────────┘
+         │
+         │ 6. Wait for completion
+         ▼
+    ┌──────────────────┐
+    │ waitForIdle()    │  ──► Resolve when all events processed
+    └──────────────────┘
+```
+
+### 2. AgentLoop Flow Diagram
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                    AgentLoop Process Flow                               │
+└─────────────────────────────────────────────────────────────────────────┘
+
+    ┌────────────────────────────────────────────────────────────────────┐
+    │  AgentLoop Entrypoint                                              │
+    │  ┌────────────────────────────────────────────────────────────┐    │
+    │  │ agentLoop(prompts, context, config, signal, stream_fn)     │    │
+    │  └────────────────────────────────────────────────────────────┘    │
+    │                              │                                     │
+    │                              ▼                                     │
+    │  ┌────────────────────────────────────────────────────────────┐    │
+    │  │ runAgentLoop(prompts, context, config, emit, signal)       │    │
+    │  └────────────────────────────────────────────────────────────┘    │
+    │                              │                                     │
+    │                              ▼                                     │
+    │  ┌────────────────────────────────────────────────────────────┐    │
+    │  │ runLoop() - Main Event Loop                                │    │
+    │  └────────────────────────────────────────────────────────────┘    │
+    │                              │                                     │
+    └──────────────────────────────┼─────────────────────────────────────┘
+                                   │
+                                   │ Loop Iteration
+                                   ▼
+    ┌────────────────────────────────────────────────────────────────────┐
+    │  Main Processing Loop                                              │
+    │                                                                    │
+    │  ┌────────────────────────────────────────────────────────────┐    │
+    │  │ 1. Get Steering/Follow-up Messages                         │    │
+    │  │    ┌────────────────────┐      ┌──────────────────────┐    │    │
+    │  │    │ steering_queue     │      │ follow_up_queue      │    │    │
+    │  │    │ (after assistant)  │      │ (after stop)         │    │    │
+    │  │    └────────────────────┘      └──────────────────────┘    │    │
+    │  └────────────────────────────────────────────────────────────┘    │
+    │                              │                                     │
+    │                              ▼                                     │
+    │  ┌────────────────────────────────────────────────────────────┐    │
+    │  │ 2. Stream Assistant Response                               │    │
+    │  │    ┌────────────────────────────────────────────────────┐  │    │
+    │  │    │ transform_context()                                │  │    │
+    │  │    │ convert_to_llm(messages) -> Message[]              │  │    │
+    │  │    │ stream_fn(model, context, config) -> Response      │  │    │
+    │  │    │  - Text deltas                                     │  │    │
+    │  │    │  - Tool call deltas                                │  │    │
+    │  │    └────────────────────────────────────────────────────┘  │    │
+    │  │                              │                             │    │
+    │  │                              ▼                             │    │
+    │  │  ┌────────────────────────────────────────────────────┐    │    │
+    │  │  │ Emit: MessageStartEvent, MessageUpdateEvent,       │    │    │
+    │  │  │       MessageEndEvent                              │    │    │
+    │  │  └────────────────────────────────────────────────────┘    │    │
+    │  └────────────────────────────────────────────────────────────┘    │
+    │                              │                                     │
+    │                              ▼                                     │
+    │  ┌────────────────────────────────────────────────────────────┐    │
+    │  │ 3. Execute Tool Calls                                      │    │
+    │  │    ┌────────────────────────────────────────────────────┐  │    │
+    │  │    │ extract ToolCall from assistant content            │  │    │
+    │  │    │                                                    │  │    │
+    │  │    │ if EXECUTION_SEQUENTIAL || has_sequential_tool:    │  │    │
+    │  │    │   executeToolCallsSequential()                     │  │    │
+    │  │    │ else:                                              │  │    │
+    │  │    │   executeToolCallsParallel()                       │  │    │
+    │  │    └────────────────────────────────────────────────────┘  │    │
+    │  │                              │                             │    │
+    │  │                              ▼                             │    │
+    │  │  ┌────────────────────────────────────────────────────┐    │    │
+    │  │  │ For each tool call:                                │    │    │
+    │  │  │  1. before_tool_call hook                          │    │    │
+    │  │  │  2. prepareToolCall()                              │    │    │
+    │  │  │  3. execute()                                      │    │    │
+    │  │  │  4. after_tool_call hook                           │    │    │
+    │  │  │  5. Emit ToolExecutionStart/Update/EndEvent        │    │    │
+    │  │  │  6. Emit ToolResultMessage                         │    │    │
+    │  │  └────────────────────────────────────────────────────┘    │    │
+    │  └────────────────────────────────────────────────────────────┘    │
+    │                              │                                     │
+    │                              ▼                                     │
+    │  ┌────────────────────────────────────────────────────────────┐    │
+    │  │ 4. Prepare Next Turn                                       │    │
+    │  │    ┌────────────────────────────────────────────────────┐  │    │
+    │  │    │ prepare_next_turn(context) -> next_turn_snapshot   │  │    │
+    │  │    │ - Optional: Update model/thinking_level            │  │    │
+    │  │    │ - Optional: Update context                         │  │    │
+    │  │    └────────────────────────────────────────────────────┘  │    │
+    │  └────────────────────────────────────────────────────────────┘    │
+    │                              │                                     │
+    │                              ▼                                     │
+    │  ┌────────────────────────────────────────────────────────────┐    │
+    │  │ 5. Check Termination Conditions                            │    │
+    │  │    ┌────────────────────────────────────────────────────┐  │    │
+    │  │    │ should_stop_after_turn(context) -> bool            │  │    │
+    │  │    │ - Max turns reached?                               │  │    │
+    │  │    │ - Tool returned terminate=true?                    │  │    │
+    │  │    │ - Steering queue empty and follow-up empty?        │  │    │
+    │  │    └────────────────────────────────────────────────────┘  │    │
+    │  └────────────────────────────────────────────────────────────┘    │
+    │                              │                                     │
+    │                              ▼                                     │
+    │  ┌────────────────────────────────────────────────────────────┐    │
+    │  │ 6. Emit TurnEndEvent (message, tool_results)               │    │
+    │  └────────────────────────────────────────────────────────────┘    │
+    │                              │                                     │
+    │                              ▼                                     │
+    │  ┌────────────────────────────────────────────────────────────┐    │
+    │  │ Loop continues until termination condition met             │    │
+    │  └────────────────────────────────────────────────────────────┘    │
+    │                              │                                     │
+    └──────────────────────────────┼─────────────────────────────────────┘
+                                   │
+                                   ▼
+    ┌────────────────────────────────────────────────────────────────────┐
+    │  AgentEndEvent with final messages                                 │
+    └────────────────────────────────────────────────────────────────────┘
+```
+
+### 3. Tool Execution Flow
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                        Tool Execution Flow                              │
+└─────────────────────────────────────────────────────────────────────────┘
+
+    ┌───────────────────────────────────────────────────────────────────┐
+    │  Assistant Message with Tool Calls                                │
+    │  ┌─────────────────────────────────────────────────────────────┐  │
+    │  │ AssistantMessage:                                           │  │
+    │  │   content: [                                                │  │
+    │  │     TextContent("I'll help you"),                           │  │
+    │  │     ToolCall(id="tc1", name="bash", args={...}),            │  │
+    │  │     ToolCall(id="tc2", name="read", args={...})             │  │
+    │  │   ]                                                         │  │
+    │  └─────────────────────────────────────────────────────────────┘  │
+    │                              │                                    │
+    │                              ▼                                    │
+    └───────────────────────────────────────────────────────────────────┘
+                                   │
+                                   │ executeToolCalls()
+                                   ▼
+    ┌───────────────────────────────────────────────────────────────────┐
+    │  Determine Execution Mode                                         │
+    │  ┌─────────────────────────────────────────────────────────────┐  │
+    │  │ config.tool_execution == EXECUTION_SEQUENTIAL?              │  │
+    │  │ OR any tool has execution_mode == EXECUTION_SEQUENTIAL?     │  │
+    │  └─────────────────────────────────────────────────────────────┘  │
+    │                              │                                    │
+    │              ┌───────────────┴───────────────┐                    │
+    │              ▼                               ▼                    │
+    │  ┌────────────────────────┐      ┌────────────────────────┐       │
+    │  │ executeSequential()    │      │ executeParallel()      │       │
+    │  └────────────────────────┘      └────────────────────────┘       │
+    │              │                               │                    │
+    └──────────────┼───────────────────────────────┼────────────────────┘
+                   │                               │
+                   │                               │
+                   ▼                               ▼
+    ┌──────────────────────┐              ┌──────────────────────┐
+    │ Sequential Execution │              │ Parallel Execution   │
+    │                      │              │                      │
+    │  for tool_call in:   │              │  for tool_call in:   │
+    │    prepareToolCall() │              │    prepareToolCall() │
+    │    execute()         │              │    execute() (async) │
+    │    finalize()        │              │                      │
+    │                      │              │  wait all results    │
+    │                      │              └──────────────────────┘
+    └──────────────────────┘
+                   │
+                   ▼
+    ┌───────────────────────────────────────────────────────────────────┐
+    │  For Each Tool Call                                               │
+    │  ┌─────────────────────────────────────────────────────────────┐  │
+    │  │ 1. before_tool_call hook (optional)                         │  │
+    │  │    - Can block execution                                    │  │
+    │  │ 2. prepareToolCall()                                        │  │
+    │  │    - validateToolArguments()                                │  │
+    │  │    - prepareToolCallArguments() (optional)                  │  │
+    │  │ 3. Execute Tool:                                            │  │
+    │  │    tool.execute(tool_call_id, args, signal, on_update)      │  │
+    │  │ 4. after_tool_call hook (optional)                          │  │
+    │  │    - Can modify result content                              │  │
+    │  │ 5. Emit events:                                             │  │
+    │  │    - ToolExecutionStartEvent                                │  │
+    │  │    - ToolExecutionUpdateEvent (optional)                    │  │
+    │  │    - ToolExecutionEndEvent                                  │  │
+    │  │ 6. Create ToolResultMessage                                 │  │
+    │  └─────────────────────────────────────────────────────────────┘  │
+    └───────────────────────────────────────────────────────────────────┘
+                   │
+                   ▼
+    ┌───────────────────────────────────────────────────────────────────┐
+    │  Tool Result Messages                                             │
+    │  ┌─────────────────────────────────────────────────────────────┐  │
+    │  │ ToolResultMessage:                                          │  │
+    │  │   role: "toolResult"                                        │  │
+    │  │   tool_call_id: "tc1"                                       │  │
+    │  │   tool_name: "bash"                                         │  │
+    │  │   content: [TextContent("command output")]                  │  │
+    │  │   is_error: false                                           │  │
+    │  └─────────────────────────────────────────────────────────────┘  │
+    └───────────────────────────────────────────────────────────────────┘
+```
+
+### 4. Session & Tree Structure
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                        Session Tree Structure                           │
+└─────────────────────────────────────────────────────────────────────────┘
+
+    Session = Linked List of Entries (tree structure)
+
+    ┌───────────────────────────────────────────────────────────────────┐
+    │  Branch Navigation                                                │
+    │                                                                   │
+    │  ┌─────┐      ┌─────┐     ┌─────┐      ┌─────┐     ┌─────┐        │
+    │  │ E1  │────▶│ E2  │────▶│ E3  │────▶│ E4  │────▶│ E5  │ (leaf) │
+    │  └─────┘      └─────┘     └─────┘      └─────┘     └─────┘        │
+    │      │           │           │           │           │            │
+    │      ▼           ▼           ▼           ▼           ▼            │
+    │   Message    Message    Compaction   Message    BranchSummary     │
+    │                                                                   │
+    │  E3 is a Compaction Entry:                                        │
+    │  - Summary of E1, E2                                              │
+    │  - first_kept_entry_id: reference to first retained message       │
+    │  - tokens_before: context size before compaction                  │
+    │                                                                   │
+    │  E5 is a BranchSummary Entry:                                     │
+    │  - Summary of branch from from_id                                 │
+    │  - Represents a fork point in conversation history                │
+    │                                                                   │
+    └───────────────────────────────────────────────────────────────────┘
+                   │
+                   │ Session.moveTo()
+                   ▼
+    ┌───────────────────────────────────────────────────────────────────┐
+    │  Forking & Branching                                              │
+    │                                                                   │
+    │  Current branch:                                                  │
+    │  ┌─────┐      ┌─────┐     ┌─────┐                                 │
+    │  │ E1  │────▶│ E2  │────▶│ E3  │                                 │
+    │  └─────┘      └─────┘     └─────┘                                 │
+    │                              │                                    │
+    │                              │ moveTo(E2)                         │
+    │                              ▼                                    │
+    │  ┌─────┐      ┌─────┐     ┌─────┐      ┌─────┐                    │
+    │  │ E1  │────▶│ E2  │────▶│ E3' │────▶│ E4' │ (new branch)       │
+    │  └─────┘      └─────┘     └─────┘      └─────┘                    │
+    │                                    │                              │
+    │                                    │ create BranchSummary         │
+    │                                    ▼                              │
+    │                                  ┌─────┐                          │
+    │                                  │ E5  │ (branch summary)         │
+    │                                  └─────┘                          │
+    │                                                                   │
+    └───────────────────────────────────────────────────────────────────┘
+```
+
+## Component Relationships
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                          Component Relationships                            │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+    User Code
+        │
+        ├── Creates ──► Agent
+        │                  │
+        │                  ├── Uses ──► AgentLoop
+        │                  │                 │
+        │                  │                 ├── Uses ──► StreamFn (LLM API)
+        │                  │                 │
+        │                  │                 └── Uses ──► Session
+        │                  │
+        │                  ├── Manages ──► AgentState
+        │                  │
+        │                  ├── Queues ──► SteeringQueue
+        │                  │
+        │                  └── Queues ──► FollowUpQueue
+        │
+        └── Interacts With ──► AgentHarness (optional, higher level)
+                              │
+                              ├── Manages ──► SessionRepo
+                              │
+                              ├── Manages ──► Skills
+                              │
+                              └── Manages ──► PromptTemplates
+```
+
+## Data Flow
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                            Data Flow Between Layers                         │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+    User Input (String/Message)
+              │
+              ▼
+    ┌──────────────────────┐
+    │ Agent.prompt()       │
+    │ - normalizeInput()   │
+    └──────────────────────┘
+              │
+              ▼
+    ┌──────────────────────┐
+    │ AgentState.messages  │  ──► AgentMessage[]
+    └──────────────────────┘
+              │
+              ▼
+    ┌──────────────────────┐
+    │ AgentLoop            │
+    │ - transform_context  │
+    └──────────────────────┘
+              │
+              ▼
+    ┌──────────────────────┐
+    │ convertToLlm()       │  ──► Transforms AgentMessage[] to Message[]
+    └──────────────────────┘
+              │
+              ▼
+    ┌──────────────────────┐
+    │ LLM API (StreamFn)   │
+    │ - Context: Message[] │
+    └──────────────────────┘
+              │
+              ▼
+    ┌──────────────────────┐
+    │ Response (Streaming) │
+    │ - Text deltas        │
+    │ - Tool call deltas   │
+    └──────────────────────┘
+              │
+              ▼
+    ┌──────────────────────┐
+    │ AssistantMessage     │
+    │ - content: Message[] │
+    └──────────────────────┘
+              │
+              ▼
+    ┌──────────────────────┐
+    │ AgentState.messages  │  ──► Appended to conversation
+    └──────────────────────┘
+              │
+              ▼
+    ┌──────────────────────┐
+    │ Tool Execution       │
+    │ - Extract ToolCalls  │
+    │ - Execute tools      │
+    └──────────────────────┘
+              │
+              ▼
+    ┌──────────────────────┐
+    │ ToolResultMessage[]  │
+    └──────────────────────┘
+              │
+              ▼
+    ┌──────────────────────┐
+    │ AgentState.messages  │  ──► Tool results appended
+    └──────────────────────┘
+              │
+              │ (Loop back to LLM or end)
+              ▼
+    ┌──────────────────────┐
+    │ Session Storage      │
+    │ - JSONL format       │
+    │ - Tree entries       │
+    └──────────────────────┘
+```
+
+## Summary
+
+The AgentCore.jl architecture follows a clean separation of concerns:
+
+1. **AgentHarness** - Highest level, handles persistence and resources
+2. **Agent** - State management and event streaming
+3. **AgentLoop** - Core LLM interaction loop
+4. **Session** - Conversation history management
+
+Each layer transforms data and passes it to the next layer, with clear interfaces and event hooks for customization.

learning/02-AGENT_COMPONENT.md

@@ -0,0 +1,465 @@
+# AgentCore.jl - Agent Component Deep Dive
+
+## Agent Structure
+
+```julia
+mutable struct Agent
+    _state::AgentState
+    listeners::Set{Tuple{Function, Ref{Bool}}}
+    steering_queue::PendingMessageQueue
+    follow_up_queue::PendingMessageQueue
+    
+    convert_to_llm::Function
+    transform_context::Union{Function, Nothing}
+    stream_function::StreamFn
+    get_api_key::Union{Function, Nothing}
+    on_payload::Union{Function, Nothing}
+    on_response::Union{Function, Nothing}
+    before_tool_call::Union{Function, Nothing}
+    after_tool_call::Union{Function, Nothing}
+    prepare_next_turn::Union{Function, Nothing}
+    prepare_next_turn_with_context::Union{Function, Nothing}
+    active_run::Union{ActiveRun, Nothing}
+    session_id::Union{String, Nothing}
+    thinking_budgets::Union{Dict{String, Int64}, Nothing}
+    transport::String
+    max_retry_delay_ms::Union{Int64, Nothing}
+    tool_execution::ToolExecutionMode
+end
+```
+
+## Agent Lifecycle
+
+### 1. Initialization
+
+```julia
+# Create agent with options
+agent = Agent(Dict{Symbol, Any}(
+    :systemPrompt => "You are a helpful assistant",
+    :model => Model(...),
+    :thinkingLevel => THINKING_MEDIUM,
+    :tools => [bash_tool, read_tool],
+    :steeringMode => QUEUE_ONE_AT_A_TIME,
+    :followUpMode => QUEUE_ONE_AT_A_TIME,
+    :toolExecution => EXECUTION_PARALLEL,
+))
+
+# Subscribe to events
+unsubscribe = subscribe(agent) do event, signal
+    if event isa MessageEndEvent
+        println("Message: $(event.message)")
+    elseif event isa ToolExecutionEndEvent
+        println("Tool completed: $(event.tool_name)")
+    end
+end
+```
+
+### 2. Message Queues
+
+#### Steering Queue
+- Messages injected **after** the current assistant turn finishes
+- Used to correct or redirect the agent's behavior
+- Example: "Actually, let's do X instead"
+
+#### Follow-Up Queue
+- Messages run **only after** the agent would otherwise stop
+- Used to continue conversation when agent thinks it's done
+- Example: "Wait, there's one more thing"
+
+#### Queue Modes
+- `QUEUE_ALL` - Drain all messages at once
+- `QUEUE_ONE_AT_A_TIME` - Process one message at a time
+
+```julia
+# Queue a steering message
+steer(agent, UserMessage(...))
+
+# Queue a follow-up message
+followUp(agent, UserMessage(...))
+
+# Check if queues have items
+hasQueuedMessages(agent)  # Returns Bool
+
+# Clear queues
+clearSteeringQueue(agent)
+clearFollowUpQueue(agent)
+clearAllQueues(agent)
+```
+
+### 3. Event System
+
+#### Agent Events
+
+```julia
+abstract type AgentEvent end
+
+# Lifecycle events
+struct AgentStartEvent <: AgentEvent end
+struct AgentEndEvent <: AgentEvent
+    messages::Vector{AgentMessage}
+end
+
+# Turn events
+struct TurnStartEvent <: AgentEvent end
+struct TurnEndEvent <: AgentEvent
+    message::AgentMessage
+    tool_results::Vector{ToolResultMessage}
+end
+
+# Message events
+struct MessageStartEvent <: AgentEvent
+    message::AgentMessage
+end
+struct MessageUpdateEvent <: AgentEvent
+    message::AgentMessage
+    assistant_message_event::Any
+end
+struct MessageEndEvent <: AgentEvent
+    message::AgentMessage
+end
+
+# Tool execution events
+struct ToolExecutionStartEvent <: AgentEvent
+    tool_call_id::String
+    tool_name::String
+    args::Any
+end
+struct ToolExecutionUpdateEvent <: AgentEvent
+    tool_call_id::String
+    tool_name::String
+    args::Any
+    partial_result::Any
+end
+struct ToolExecutionEndEvent <: AgentEvent
+    tool_call_id::String
+    tool_name::String
+    result::Any
+    is_error::Bool
+end
+```
+
+#### Event Flow Diagram
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                         Event Timeline                                  │
+└─────────────────────────────────────────────────────────────────────────┘
+
+AgentStartEvent
+    │
+    ├─ TurnStartEvent
+    │   │
+    │   ├─ MessageStartEvent (user prompt)
+    │   ├─ MessageEndEvent (user prompt)
+    │   │
+    │   ├─ [Loop starts]
+    │   │   │
+    │   │   ├─ MessageStartEvent (assistant response)
+    │   │   ├─ MessageUpdateEvent (text delta 1)
+    │   │   ├─ MessageUpdateEvent (text delta 2)
+    │   │   ├─ MessageUpdateEvent (tool call delta)
+    │   │   ├─ MessageEndEvent (assistant complete)
+    │   │   │
+    │   │   ├─ ToolExecutionStartEvent (tc1)
+    │   │   ├─ ToolExecutionUpdateEvent (partial result)
+    │   │   ├─ ToolExecutionEndEvent (tc1 done)
+    │   │   │
+    │   │   ├─ ToolExecutionStartEvent (tc2)
+    │   │   ├─ ToolExecutionEndEvent (tc2 done)
+    │   │   │
+    │   │   └─ TurnEndEvent (assistant + tools)
+    │   │
+    │   └─ [Next turn if needed]
+    │
+    └─ AgentEndEvent (final messages)
+```
+
+### 4. State Management
+
+```julia
+mutable struct AgentState
+    system_prompt::String
+    model::Model
+    thinking_level::ThinkingLevel
+    tools::Vector{AgentTool}
+    messages::Vector{AgentMessage}
+    is_streaming::Bool
+    streaming_message::Union{AgentMessage, Nothing}
+    pending_tool_calls::Set{String}
+    error_message::Union{String, Nothing}
+end
+```
+
+#### State Access
+
+```julia
+# Get current state
+state = get_state(agent)
+
+# Reset state
+reset!(agent)  # Clears messages, queues, and runtime state
+```
+
+### 5. Main Methods
+
+#### prompt()
+
+```julia
+# Start a new conversation
+prompt(agent, "Hello, how are you?")
+
+# With multiple messages
+prompt(agent, [
+    UserMessage(...),
+    AssistantMessage(...),
+    UserMessage(...)
+])
+
+# With images
+prompt(agent, "Analyze this image", [ImageContent(data, "image/png")])
+```
+
+#### continue!()
+
+```julia
+# Continue from current transcript
+# Last message must be user or tool-result
+continue!(agent)
+```
+
+#### steer() and followUp()
+
+```julia
+# Steering: Redirect after next assistant turn
+steer(agent, UserMessage(...))
+
+# Follow-up: Continue after agent would stop
+followUp(agent, UserMessage(...))
+```
+
+### 6. Hooks
+
+#### convert_to_llm
+
+```julia
+# Transform messages before sending to LLM
+function myConvertToLlm(messages::Vector{AgentMessage})
+    return filter(
+        m -> m.role in ["user", "assistant", "toolResult"],
+        messages
+    )
+end
+
+agent = Agent(Dict(:convertToLlm => myConvertToLlm))
+```
+
+#### transform_context
+
+```julia
+# Transform context before LLM call
+function myTransformContext(messages, signal)
+    # Can truncate, filter, or modify messages
+    return messages
+end
+
+agent = Agent(Dict(:transformContext => myTransformContext))
+```
+
+#### before_tool_call
+
+```julia
+# Hook before tool execution
+function myBeforeToolCall(context, signal)
+    println("About to execute: $(context.tool_call.name)")
+    return nothing  # Return block=true to prevent execution
+end
+
+agent = Agent(Dict(:beforeToolCall => myBeforeToolCall))
+```
+
+#### after_tool_call
+
+```julia
+# Hook after tool execution
+function myAfterToolCall(context, signal)
+    # Can modify tool result
+    return AfterToolCallResult(
+        content = context.result.content,
+        terminate = context.result.terminate
+    )
+end
+
+agent = Agent(Dict(:afterToolCall => myAfterToolCall))
+```
+
+#### prepare_next_turn
+
+```julia
+# Modify context/model/thinking level between turns
+function myPrepareNextTurn(context, signal)
+    # context: PrepareNextTurnContext
+    # Returns AgentLoopTurnUpdate or nothing
+    return AgentLoopTurnUpdate(
+        context = context.context,
+        model = context.context.model,  # Can change model
+        thinking_level = THINKING_HIGH  # Can change thinking level
+    )
+end
+
+agent = Agent(Dict(:prepareNextTurn => myPrepareNextTurn))
+```
+
+### 7. Active Run Management
+
+```julia
+# Check if agent is busy
+if !isnothing(agent.active_run)
+    # Agent is processing
+    abort(agent)  # Abort current run
+end
+
+# Wait for completion
+wait_for_idle(agent)  # Returns Promise
+```
+
+## Complete Example
+
+```julia
+using AgentCore
+
+# 1. Create agent
+agent = Agent(Dict(
+    :systemPrompt => "You are a helpful assistant.",
+    :model => Model(...),
+    :tools => [bash_tool, read_tool],
+))
+
+# 2. Subscribe to events
+events_received = []
+unsubscribe = subscribe(agent) do event, signal
+    push!(events_received, event)
+    
+    if event isa MessageEndEvent
+        println("Message: $(event.message)")
+    end
+end
+
+# 3. Start conversation
+prompt(agent, "What's in the current directory?")
+
+# 4. Wait for completion
+wait_for_idle(agent)
+
+# 5. Check final state
+state = get_state(agent)
+println("Total messages: $(length(state.messages))")
+
+# 6. Continue with steering
+steer(agent, UserMessage(...))
+wait_for_idle(agent)
+
+# 7. Clean up
+unsubscribe()  # Stop listening
+reset!(agent)  # Clear state
+```
+
+## Key Concepts
+
+### Message Queueing
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                     Message Queue Behavior                              │
+└─────────────────────────────────────────────────────────────────────────┘
+
+Scenario: User sends message, agent responds with tool calls
+
+┌────────────────────────────────────────────────────────────┐
+│ Time 0: User sends message                                 │
+│ ┌──────────────┐                                           │
+│ │ prompt(msg)  │                                           │
+│ └──────────────┘                                           │
+│                    │                                       │
+│                    ▼                                       │
+│              ┌─────────────┐                               │
+│              │ AgentLoop   │                               │
+│              │ processes   │                               │
+│              │ msg         │                               │
+│              └─────────────┘                               │
+└────────────────────────────────────────────────────────────┘
+                    │
+                    ▼
+┌────────────────────────────────────────────────────────────┐
+│ Time 1: Agent responds with tool calls                     │
+│ ┌──────────────────────────────────────────────────────┐   │
+│ │ AssistantMessage:                                    │   │
+│ │   content: [Text("I'll check..."),                   │   │
+│ │             ToolCall("bash", {...}),                 │   │
+│ │             ToolCall("read", {...})]                 │   │
+│ └──────────────────────────────────────────────────────┘   │
+└────────────────────────────────────────────────────────────┘
+                    │
+                    ▼
+┌────────────────────────────────────────────────────────────┐
+│ Time 2: User queues steering message                       │
+│ ┌──────────────────┐                                       │
+│ │ steer(msg2)      │  ──► steering_queue.push(msg2)        │
+│ └──────────────────┘                                       │
+│                                                            │
+│ (msg2 not processed yet!)                                  │
+└────────────────────────────────────────────────────────────┘
+                    │
+                    ▼
+┌────────────────────────────────────────────────────────────┐
+│ Time 3: Tool execution                                     │
+│ ┌──────────────────────────────────────────────────────┐   │
+│ │ Execute bash tool...                                 │   │
+│ │ Execute read tool...                                 │   │
+│ │ Emit ToolResultMessage[]                             │   │
+│ └──────────────────────────────────────────────────────┘   │
+└────────────────────────────────────────────────────────────┘
+                    │
+                    ▼
+┌────────────────────────────────────────────────────────────┐
+│ Time 4: Agent responds to tool results                     │
+│ ┌──────────────────────────────────────────────────────┐   │
+│ │ AssistantMessage (2nd turn):                         │   │
+│ │   content: [Text("The results are...")]              │   │
+│ └──────────────────────────────────────────────────────┘   │
+└────────────────────────────────────────────────────────────┘
+                    │
+                    ▼
+┌────────────────────────────────────────────────────────────┐
+│ Time 5: Steering message processed                         │
+│ ┌──────────────────────────────────────────────────────┐   │
+│ │ steering_queue.drain() → [msg2]                      │   │
+│ │ Emit msg2 as UserMessage                             │   │
+│ └──────────────────────────────────────────────────────┘   │
+└────────────────────────────────────────────────────────────┘
+                    │
+                    ▼
+┌────────────────────────────────────────────────────────────┐
+│ Time 6: Next turn (agent responds to steering)             │
+│ ┌──────────────────────────────────────────────────────┐   │
+│ │ AssistantMessage (3rd turn):                         │   │
+│ │   content: [Text("Okay, I'll do X instead...")]      │   │
+│ └──────────────────────────────────────────────────────┘   │
+└────────────────────────────────────────────────────────────┘
+```
+
+### Queue Behavior Summary
+
+| Action | Queue | When Processed |
+|--------|-------|----------------|
+| `prompt()` | N/A | Immediate |
+| `steer()` | steering_queue | After assistant turn completes |
+| `followUp()` | follow_up_queue | After agent would normally stop |
+| `continue!()` | N/A | Immediately if last message is user/tool |
+
+## Best Practices
+
+1. **Use steering for redirects**: When user wants to change direction mid-conversation
+2. **Use follow-up for continuation**: When agent thinks it's done but user wants more
+3. **Subscribe to events**: Monitor agent behavior and debug issues
+4. **Clear queues**: Use `clearAllQueues()` when resetting conversation
+5. **Check active run**: Don't call `prompt()` while agent is busy

learning/03-AGENTLOOP_COMPONENT.md

@@ -0,0 +1,758 @@
+# AgentCore.jl - AgentLoop Component Deep Dive
+
+## AgentLoop Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                          AgentLoop Layer                                    │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ Public API                                                                  │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+agentLoop()
+  ├─ prompts: Vector{AgentMessage}
+  ├─ context: AgentContext
+  ├─ config: AgentLoopConfig
+  ├─ signal: Union{Nothing, AbortSignal}
+  └─ stream_fn: StreamFn
+      └─ Returns: EventStream
+
+agentLoopContinue()
+  ├─ context: AgentContext
+  ├─ config: AgentLoopConfig
+  ├─ signal: Union{Nothing, AbortSignal}
+  └─ stream_fn: StreamFn
+      └─ Returns: EventStream
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ Internal Flow                                                               │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ 1. runAgentLoop() ── Entry point for new conversation                       │
+│    - Creates copy of prompts                                                │
+│    - Appends prompts to context.messages                                    │
+│    - Emits AgentStartEvent                                                  │
+│    - Emits TurnStartEvent                                                   │
+│    - Emits MessageStart/End for each prompt                                 │
+│    - Calls runLoop()                                                        │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ 2. runLoop() ── Main event loop                                             │
+│    ┌────────────────────────────────────────────────────────────────────┐   │
+│    │ while true:                                                        │   │
+│    │   1. Get steering/follow-up messages (if any)                      │   │
+│    │   2. Emit messages as UserMessage                                  │   │
+│    │   3. streamAssistantResponse()                                     │   │
+│    │   4. Execute tool calls (sequential or parallel)                   │   │
+│    │   5. Emit TurnEndEvent                                             │   │
+│    │   6. prepare_next_turn (optional)                                  │   │
+│    │   7. should_stop_after_turn? (check termination)                   │   │
+│    │   8. Loop continues if not terminated                              │   │
+│    └────────────────────────────────────────────────────────────────────┘   │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ 3. streamAssistantResponse() ── LLM interaction                             │
+│    - transform_context (optional)                                           │
+│    - convert_to_llm (transform to Message[])                                │
+│    - Call stream_fn (LLM API)                                               │
+│    - Stream response deltas                                                 │
+│    - Emit MessageStart/Update/End events                                    │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ 4. executeToolCalls() ── Tool execution                                     │
+│    ┌────────────────────────────────────────────────────────────────────┐   │
+│    │ if EXECUTION_SEQUENTIAL || has_sequential_tool:                    │   │
+│    │   executeToolCallsSequential()                                     │   │
+│    │ else:                                                              │   │
+│    │   executeToolCallsParallel()                                       │   │
+│    └────────────────────────────────────────────────────────────────────┘   │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ 5. AgentEndEvent ── Final event with all messages                           │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+## AgentLoopConfig
+
+```julia
+struct AgentLoopConfig
+    model::Model
+    reasoning::Union{ThinkingLevel, Nothing}
+    session_id::Union{String, Nothing}
+    on_payload::Union{Function, Nothing}
+    on_response::Union{Function, Nothing}
+    transport::String
+    thinking_budgets::Union{Dict{String, Int64}, Nothing}
+    max_retry_delay_ms::Union{Int64, Nothing}
+    tool_execution::ToolExecutionMode
+    before_tool_call::Union{Function, Nothing}
+    after_tool_call::Union{Function, Nothing}
+    prepare_next_turn::Union{Function, Nothing}
+    convert_to_llm::Function
+    transform_context::Union{Function, Nothing}
+    get_api_key::Union{Function, Nothing}
+    get_steering_messages::Union{Function, Nothing}
+    get_follow_up_messages::Union{Function, Nothing}
+end
+```
+
+## Main Functions
+
+### agentLoop()
+
+```julia
+function agentLoop(
+    prompts::Vector{AgentMessage},
+    context::AgentContext,
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+    stream_fn::StreamFn,
+)::EventStream
+```
+
+**Purpose**: Start a new conversation with initial prompts
+
+**Flow**:
+1. Create event stream
+2. Spawn thread to run agent loop
+3. Return stream for event consumption
+
+```julia
+stream = agentLoop(
+    [UserMessage("user", [TextContent("Hello")], timestamp)],
+    AgentContext(system_prompt, messages, tools),
+    config,
+    nothing,
+    stream_fn,
+)
+
+# Consume events
+for event in stream
+    if event isa MessageEndEvent
+        println("Received: $(event.message)")
+    end
+end
+```
+
+### runAgentLoop()
+
+```julia
+function runAgentLoop(
+    prompts::Vector{AgentMessage},
+    context::AgentContext,
+    config::AgentLoopConfig,
+    emit::AgentEventSink,
+    signal::Union{Nothing, AbortSignal},
+    stream_fn::StreamFn,
+)::Vector{AgentMessage}
+```
+
+**Purpose**: Execute agent loop with initial prompts
+
+**Flow**:
+1. Copy prompts to new_messages
+2. Append prompts to context.messages
+3. Emit AgentStartEvent
+4. For each prompt: emit MessageStartEvent, MessageEndEvent
+5. Call runLoop()
+
+### runLoop() - The Heart of AgentLoop
+
+```julia
+function runLoop(
+    initial_context::AgentContext,
+    new_messages::Vector{AgentMessage},
+    initial_config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+    emit::AgentEventSink,
+    stream_function::StreamFn,
+)::Nothing
+```
+
+**Main Loop**:
+```julia
+current_context = initial_context
+config = initial_config
+first_turn = true
+pending_messages = get_steering_messages()
+
+while true
+    # Process steering/follow-up messages
+    while !isempty(pending_messages)
+        if !first_turn
+            emit(TurnStartEvent())
+        else
+            first_turn = false
+        end
+        
+        # Emit pending messages
+        for message in pending_messages
+            emit(MessageStartEvent(message))
+            emit(MessageEndEvent(message))
+            push!(current_context.messages, message)
+            push!(new_messages, message)
+        end
+        
+        pending_messages = []
+    end
+    
+    # Stream assistant response
+    message = streamAssistantResponse(
+        current_context,
+        config,
+        signal,
+        emit,
+        stream_function,
+    )
+    push!(new_messages, message)
+    
+    # Check for errors
+    if message.stop_reason in ("error", "aborted")
+        emit(TurnEndEvent(message, []))
+        emit(AgentEndEvent(new_messages))
+        return
+    end
+    
+    # Execute tool calls
+    tool_calls = filter(c -> c isa ToolCall, message.content)
+    tool_results = []
+    has_more_tool_calls = false
+    
+    if !isempty(tool_calls)
+        executed_batch = if message.stop_reason == "length"
+            failToolCallsFromTruncatedMessage(tool_calls, emit)
+        else
+            executeToolCalls(
+                current_context,
+                message,
+                config,
+                signal,
+                emit,
+            )
+        end
+        append!(tool_results, executed_batch.messages)
+        has_more_tool_calls = !executed_batch.terminate
+        
+        for result in tool_results
+            push!(current_context.messages, result)
+            push!(new_messages, result)
+        end
+    end
+    
+    emit(TurnEndEvent(message, tool_results))
+    
+    # Prepare next turn (optional)
+    next_turn_context = PrepareNextTurnContext(
+        message, tool_results, current_context, new_messages
+    )
+    next_turn_snapshot = prepare_next_turn(config, next_turn_context)
+    
+    if !isnothing(next_turn_snapshot)
+        current_context = next_turn_snapshot.context
+        config = AgentLoopConfig(
+            model = next_turn_snapshot.model,
+            reasoning = next_turn_snapshot.thinking_level,
+            # ... other config fields
+        )
+    end
+    
+    # Check if should stop
+    if should_stop_after_turn(config, next_turn_context)
+        emit(AgentEndEvent(new_messages))
+        return
+    end
+    
+    # Get next pending messages
+    pending_messages = get_steering_messages()
+    
+    # Check follow-up messages
+    follow_up_messages = get_follow_up_messages()
+    if !isempty(follow_up_messages)
+        pending_messages = follow_up_messages
+        continue
+    end
+    
+    break
+end
+
+emit(AgentEndEvent(new_messages))
+```
+
+### streamAssistantResponse()
+
+```julia
+function streamAssistantResponse(
+    context::AgentContext,
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+    emit::AgentEventSink,
+    stream_function::StreamFn,
+)::AssistantMessage
+```
+
+**Flow**:
+1. Get messages from context
+2. Apply transform_context (optional)
+3. Convert to LLM messages with convert_to_llm
+4. Create Context object
+5. Resolve API key
+6. Call stream_fn with model, context, and config
+7. Stream events:
+   - "start" → MessageStartEvent
+   - "text_start", "text_delta", "text_end" → MessageUpdateEvent
+   - "done", "error" → MessageEndEvent
+
+### executeToolCalls()
+
+```julia
+function executeToolCalls(
+    current_context::AgentContext,
+    assistant_message::AssistantMessage,
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+    emit::AgentEventSink,
+)::ExecutedToolCallBatch
+```
+
+**Logic**:
+```julia
+tool_calls = filter(c -> c isa ToolCall, assistant_message.content)
+
+# Check if any tool requires sequential execution
+has_sequential = any(tc -> begin
+    tool = findfirst(t -> t.name == tc.name, current_context.tools)
+    !isnothing(tool) && tool.execution_mode == EXECUTION_SEQUENTIAL
+end, tool_calls)
+
+# Determine execution mode
+if config.tool_execution == EXECUTION_SEQUENTIAL || has_sequential
+    executeToolCallsSequential(...)
+else
+    executeToolCallsParallel(...)
+end
+```
+
+### executeToolCallsSequential()
+
+```julia
+function executeToolCallsSequential(
+    current_context::AgentContext,
+    assistant_message::AssistantMessage,
+    tool_calls::Vector{ToolCall},
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+    emit::AgentEventSink,
+)::ExecutedToolCallBatch
+```
+
+**Flow** (for each tool call):
+1. Emit ToolExecutionStartEvent
+2. prepareToolCall() → PreparedToolCall or ImmediateToolCallOutcome
+3. If prepared: executePreparedToolCall()
+4. finalizeExecutedToolCall()
+5. Emit ToolExecutionEndEvent
+6. Emit ToolResultMessage
+7. Check if signal.aborted → break
+
+### executeToolCallsParallel()
+
+```julia
+function executeToolCallsParallel(
+    current_context::AgentContext,
+    assistant_message::AssistantMessage,
+    tool_calls::Vector{ToolCall},
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+    emit::AgentEventSink,
+)::ExecutedToolCallBatch
+```
+
+**Flow**:
+1. For each tool call:
+   - If immediate: execute and add to finalized_calls
+   - If prepared: create closure, add to finalized_calls
+2. For each entry in finalized_calls:
+   - If closure: execute closure
+   - If finalized: use as-is
+3. Collect all tool results
+4. Return batch
+
+### prepareToolCall()
+
+```julia
+function prepareToolCall(
+    current_context::AgentContext,
+    assistant_message::AssistantMessage,
+    tool_call::ToolCall,
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+)::Union{PreparedToolCall, ImmediateToolCallOutcome}
+```
+
+**Flow**:
+1. Find tool by name
+2. If not found → ImmediateToolCallOutcome (error)
+3. before_tool_call hook (optional)
+4. prepareToolCallArguments() (optional)
+5. validateToolArguments()
+6. Return PreparedToolCall
+
+### executePreparedToolCall()
+
+```julia
+function executePreparedToolCall(
+    prepared::PreparedToolCall,
+    signal::Union{Nothing, AbortSignal},
+    emit::AgentEventSink,
+)::ExecutedToolCallOutcome
+```
+
+**Flow**:
+1. Call tool.execute(id, args, signal, on_update)
+2. Collect update events (if any)
+3. Wait for all update events
+4. Return ExecutedToolCallOutcome(result)
+
+### finalizeExecutedToolCall()
+
+```julia
+function finalizeExecutedToolCall(
+    current_context::AgentContext,
+    assistant_message::AssistantMessage,
+    prepared::PreparedToolCall,
+    executed::ExecutedToolCallOutcome,
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+)::FinalizedToolCallOutcome
+```
+
+**Flow**:
+1. after_tool_call hook (optional)
+2. Return FinalizedToolCallOutcome
+
+### createToolResultMessage()
+
+```julia
+function createToolResultMessage(
+    finalized::FinalizedToolCallOutcome,
+)::ToolResultMessage
+```
+
+**Creates**:
+```julia
+ToolResultMessage(
+    "toolResult",
+    finalized.tool_call.id,
+    finalized.tool_call.name,
+    finalized.result.content,
+    finalized.result.details,
+    finalized.result.usage,
+    finalized.result.added_tool_names,
+    finalized.is_error,
+    timestamp,
+)
+```
+
+## Execution Modes
+
+### Sequential Execution
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                      Sequential Execution Flow                          │
+└─────────────────────────────────────────────────────────────────────────┘
+
+┌──────┐
+│ TC1  │  ──► prepareToolCall()
+└──────┘           │
+                   ▼
+            ┌──────────────┐
+            │ execute()    │  ──► Wait for completion
+            └──────────────┘           │
+                   │                 ▼
+                   ├───────────── createToolResultMessage()
+                   │                 │
+                   ▼                 ▼
+            ┌──────────────┐    ┌──────────┐
+            │ TC2  │  ──►   │  │ Result1 │
+            └──────┘          └──────────┘
+                   │
+                   ▼
+            ┌──────────────┐
+            │ execute()    │
+            └──────────────┘
+                   │
+                   ▼
+            ┌──────────────┐
+            │ TC3  │  ──►   │
+            └──────┘          │
+                   │         ▼
+                   ├───── createToolResultMessage()
+                   │         │
+                   ▼         ▼
+            ┌──────────────┐ ┌──────────┐
+            │ execute()  │ │ │ Result2  │
+            └──────────────┘ └──────────┘
+                   │
+                   ▼
+            ┌──────────┐
+            │ Result3  │
+            └──────────┘
+```
+
+### Parallel Execution
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                       Parallel Execution Flow                           │
+└─────────────────────────────────────────────────────────────────────────┘
+
+┌──────┐
+│ TC1  │  ──► prepareToolCall() ──► create closure ──► ┐
+└──────┘                                               │
+                                                       │
+┌──────┐                                               │
+│ TC2  │  ──► prepareToolCall() ──► create closure ──► ├─► All closures queued
+└──────┘                                               │
+                                                       │
+┌──────┐                                               │
+│ TC3  │  ──► prepareToolCall() ──► create closure ──► ┘
+└──────┘
+
+                      │
+                      ▼
+          ┌───────────────────────┐
+          │ for closure in closures│
+          │   execute_closure()   │
+          └───────────────────────┘
+                      │
+                      ▼
+          ┌───────────────────────┐
+          │ Collect all results   │
+          └───────────────────────┘
+                      │
+                      ▼
+          ┌───────────────────────┐
+          │ createToolResult()    │
+          └───────────────────────┘
+```
+
+## Helper Types
+
+### ExecutedToolCallBatch
+
+```julia
+struct ExecutedToolCallBatch
+    messages::Vector{ToolResultMessage}
+    terminate::Bool
+end
+```
+
+- `messages`: All tool result messages
+- `terminate`: If true, stop agent after this batch
+
+### PrepareNextTurnContext
+
+```julia
+struct PrepareNextTurnContext
+    message::AssistantMessage
+    tool_results::Vector{ToolResultMessage}
+    context::AgentContext
+    new_messages::Vector{AgentMessage}
+end
+```
+
+Used by prepare_next_turn hook to decide next steps.
+
+### Before/After Tool Call Contexts
+
+```julia
+struct BeforeToolCallContext
+    assistant_message::AssistantMessage
+    tool_call::ToolCall
+    args::Any
+    context::AgentContext
+end
+
+struct BeforeToolCallResult
+    block::Union{Bool, Nothing}
+    reason::Union{String, Nothing}
+end
+
+struct AfterToolCallContext
+    assistant_message::AssistantMessage
+    tool_call::ToolCall
+    args::Any
+    result::AgentToolResult
+    is_error::Bool
+    context::AgentContext
+end
+
+struct AfterToolCallResult
+    content::Union{Vector{MessageContent}, Nothing}
+    details::Union{Any, Nothing}
+    is_error::Union{Bool, Nothing}
+    usage::Union{Usage, Nothing}
+    terminate::Union{Bool, Nothing}
+end
+```
+
+## Event Emission Timeline
+
+```
+AgentStartEvent
+    │
+    ├─ TurnStartEvent (turn 1)
+    │   │
+    │   ├─ MessageStartEvent (user prompt)
+    │   ├─ MessageEndEvent (user prompt)
+    │   │
+    │   ├─ MessageStartEvent (assistant)
+    │   ├─ MessageUpdateEvent (text delta)
+    │   ├─ MessageUpdateEvent (tool call delta)
+    │   ├─ MessageEndEvent (assistant)
+    │   │
+    │   ├─ ToolExecutionStartEvent (tc1)
+    │   ├─ ToolExecutionEndEvent (tc1)
+    │   │
+    │   ├─ ToolExecutionStartEvent (tc2)
+    │   ├─ ToolExecutionEndEvent (tc2)
+    │   │
+    │   └─ TurnEndEvent (assistant, tool_results)
+    │
+    ├─ TurnStartEvent (turn 2 - if needed)
+    │   │
+    │   ├─ MessageStartEvent (steering/follow-up)
+    │   ├─ MessageEndEvent (steering/follow-up)
+    │   │
+    │   ├─ MessageStartEvent (assistant)
+    │   ├─ MessageUpdateEvent (text)
+    │   ├─ MessageEndEvent (assistant)
+    │   │
+    │   └─ TurnEndEvent (assistant, [])
+    │
+    └─ AgentEndEvent (final messages)
+```
+
+## Key Concepts
+
+### 1. Message Transformation Pipeline
+
+```
+AgentMessage[] (internal)
+    │
+    │ transform_context()
+    ▼
+AgentMessage[] (transformed)
+    │
+    │ convert_to_llm()
+    ▼
+Message[] (LLM API)
+```
+
+### 2. Tool Call Lifecycle
+
+```
+ToolCall (in assistant message)
+    │
+    ├─ before_tool_call (hook)
+    │
+    ├─ prepareToolCall()
+    │   ├─ validate arguments
+    │   └─ prepare arguments (optional)
+    │
+    ├─ execute()
+    │   ├─ Immediate: return result
+    │   └─ Prepared: async execution
+    │
+    ├─ after_tool_call (hook)
+    │
+    └─ createToolResultMessage()
+```
+
+### 3. Turn Termination
+
+```julia
+# Turn ends when:
+# 1. No more pending messages
+# 2. No more tool calls to execute
+# 3. should_stop_after_turn() returns true
+
+# Reasons to stop:
+# - Max turns reached
+# - Tool returned terminate=true
+# - Error or abort
+# - Steering/follow-up queues empty
+```
+
+## Best Practices
+
+1. **Use sequential execution** for tools that modify shared state
+2. **Use parallel execution** for independent tool calls (better performance)
+3. **Implement prepare_next_turn** for dynamic model/thinking level changes
+4. **Use before_tool_call** for logging or blocking sensitive operations
+5. **Use after_tool_call** for modifying results or collecting metrics
+
+## Complete Example
+
+```julia
+using AgentCore
+
+# Create config
+config = AgentLoopConfig(
+    model = my_model,
+    reasoning = THINKING_MEDIUM,
+    tool_execution = EXECUTION_PARALLEL,
+    before_tool_call = myBeforeToolCallHook,
+    after_tool_call = myAfterToolCallHook,
+    prepare_next_turn = myPrepareNextTurnHook,
+    convert_to_llm = myConvertToLlm,
+    transform_context = myTransformContext,
+    get_api_key = myGetApiKey,
+    get_steering_messages = myGetSteeringMessages,
+    get_follow_up_messages = myGetFollowUpMessages,
+)
+
+# Start agent loop
+stream = agentLoop(
+    [UserMessage("user", [TextContent("Hello")], timestamp)],
+    AgentContext(system_prompt, messages, tools),
+    config,
+    nothing,
+    stream_fn,
+)
+
+# Consume events
+final_messages = []
+for event in stream
+    if event isa MessageEndEvent
+        push!(final_messages, event.message)
+    end
+end
+
+# Or use event sink
+messages = []
+emit(event) = push!(messages, event)
+
+messages = runAgentLoop(
+    [UserMessage(...)],
+    context,
+    config,
+    emit,
+    nothing,
+    stream_fn,
+)
+```
+
+This documentation provides a comprehensive understanding of the AgentLoop component, including its architecture, main functions, execution modes, and best practices for building AI agents with AgentCore.jl.

learning/04-TYPES_MESSAGES.md

@@ -0,0 +1,589 @@
+# AgentCore.jl - Types and Messages Deep Dive
+
+## Core Type Hierarchy
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                          Type Hierarchy                                     │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  ThinkingLevel (Enum)                                                      │
+│  - THINKING_OFF                                                            │
+│  - THINKING_MINIMAL                                                        │
+│  - THINKING_LOW                                                            │
+│  - THINKING_MEDIUM                                                         │
+│  - THINKING_HIGH                                                           │
+│  - THINKING_XHIGH                                                          │
+│  - THINKING_MAX                                                            │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  ToolExecutionMode (Enum)                                                   │
+│  - EXECUTION_SEQUENTIAL  (Tools run one at a time)                          │
+│  - EXECUTION_PARALLEL    (Tools run concurrently)                           │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  QueueMode (Enum)                                                           │
+│  - QUEUE_ALL             (Drain all messages at once)                       │
+│  - QUEUE_ONE_AT_A_TIME   (Process one message at a time)                    │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  MessageContent (Abstract Type)                                             │
+│  ├── TextContent (String)                                                   │
+│  └── ImageContent (data::String, mime_type::String)                         │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  Message (Abstract Type)                                                    │
+│  ├── UserMessage                                                            │
+│  │   └─ role: "user", content: Message[], timestamp: Int64                  │
+│  ├── AssistantMessage                                                       │
+│  │   └─ role: "assistant", content: Message[], api, provider, model,        │
+│  │        usage: Usage, stop_reason, error_message, timestamp               │
+│  └── ToolResultMessage                                                      │
+│      └─ role: "toolResult", tool_call_id, tool_name, content, details,      │
+│           usage, added_tool_names, is_error, timestamp                      │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  AgentMessage (Abstract Type)                                               │
+│  └─ Union of all message types above + custom types                         │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  AgentTool                                                                  │
+│  - name: String                                                             │
+│  - label: String                                                            │
+│  - description: String                                                      │
+│  - parameters: Any                                                          │
+│  - execute: Function                                                        │
+│  - prepare_arguments: Union{Function, Nothing}                              │
+│  - execution_mode: Union{ToolExecutionMode, Nothing}                        │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  AgentContext                                                               │
+│  - system_prompt: String                                                    │
+│  - messages: Vector{AgentMessage}                                           │
+│  - tools: Union{Vector{AgentTool}, Nothing}                                 │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  AgentEvent (Abstract Type)                                                 │
+│  ├── AgentStartEvent / AgentEndEvent                                        │
+│  ├── TurnStartEvent / TurnEndEvent                                          │
+│  ├── MessageStartEvent / MessageEndEvent                                    │
+│  ├── MessageUpdateEvent                                                     │
+│  ├── ToolExecutionStartEvent / ToolExecutionEndEvent                        │
+│  └── ToolExecutionUpdateEvent                                               │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  Usage & ModelCost                                                          │
+│  Usage: input, output, cache_read, cache_write, total_tokens, cost          │
+│  ModelCost: input, output, cache_read, cache_write (all Float64)            │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  Model                                                                      │
+│  - id, name, api, provider, base_url, reasoning: Bool                       │
+│  - input: Vector{String}                                                    │
+│  - cost: ModelCost                                                          │
+│  - context_window, max_tokens: Int64                                        │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+## Message Types
+
+### UserMessage
+
+```julia
+struct UserMessage <: Message
+    role::String              # "user"
+    content::Vector{MessageContent}
+    timestamp::Timestamp      # Int64 (Unix timestamp)
+end
+```
+
+**Usage**:
+```julia
+# Simple text message
+UserMessage(
+    "user",
+    [TextContent("Hello, how are you?")],
+    Int64(Dates.now(Dates.UTC).datetime)
+)
+
+# With multiple content types
+UserMessage(
+    "user",
+    [
+        TextContent("Analyze this image"),
+        ImageContent(data_base64, "image/png")
+    ],
+    timestamp
+)
+```
+
+### AssistantMessage
+
+```julia
+struct AssistantMessage <: Message
+    role::String              # "assistant"
+    content::Vector{MessageContent}
+    api::String               # API identifier
+    provider::String          # Provider name
+    model::String             # Model ID
+    usage::Usage
+    stop_reason::String       # "done", "error", "aborted", "length", etc.
+    error_message::Union{String, Nothing}
+    timestamp::Timestamp
+end
+```
+
+**Content can include**:
+- TextContent
+- ToolCall
+
+```julia
+AssistantMessage(
+    "assistant",
+    [
+        TextContent("I'll check the directory for you."),
+        ToolCall(
+            "tool",
+            "tc_123",
+            "bash",
+            Dict("command" => "ls -la"),
+            nothing
+        ),
+        ToolCall(
+            "tool",
+            "tc_456",
+            "read",
+            Dict("path" => "README.md"),
+            nothing
+        )
+    ],
+    "openai",
+    "openai",
+    "gpt-4",
+    Usage(100, 50, 0, 0, 150, UsageCost(0.001, 0.002, 0.0, 0.0, 0.003)),
+    "done",
+    nothing,
+    timestamp
+)
+```
+
+### ToolResultMessage
+
+```julia
+struct ToolResultMessage <: Message
+    role::String              # "toolResult"
+    tool_call_id::String      # Reference to original ToolCall
+    tool_name::String         # Name of tool that executed
+    content::Vector{MessageContent}
+    details::Any              # Additional tool-specific details
+    usage::Union{Usage, Nothing}
+    added_tool_names::Union{Vector{String}, Nothing}
+    is_error::Bool            # True if tool execution failed
+    timestamp::Timestamp
+end
+```
+
+**Usage**:
+```julia
+ToolResultMessage(
+    "toolResult",
+    "tc_123",
+    "bash",
+    [TextContent("file1.md\nfile2.md\n")],
+    BashToolDetails(...),
+    nothing,
+    nothing,
+    false,
+    timestamp
+)
+```
+
+## AgentTool Structure
+
+```julia
+struct AgentTool{TParameters, TDetails}
+    name::String
+    label::String
+    description::String
+    parameters::TParameters
+    execute::Function
+    prepare_arguments::Union{Function, Nothing}
+    execution_mode::Union{ToolExecutionMode, Nothing}
+end
+```
+
+**Parameters**:
+- `name`: Unique identifier for the tool
+- `label`: Display name
+- `description`: What the tool does
+- `parameters`: JSON schema for tool arguments
+- `execute`: Main execution function
+- `prepare_arguments`: Optional preprocessing
+- `execution_mode`: Sequential or parallel
+
+### Tool Execution Function Signature
+
+```julia
+execute::Function(
+    tool_call_id::String,
+    params::Dict{String, Any},
+    signal::Union{Any, Nothing},  # Abort signal
+    on_update::Function,          # Callback for streaming updates
+    context::Any,                 # Tool context
+)::AgentToolResult
+```
+
+**Returns**:
+```julia
+AgentToolResult(
+    content::Vector{MessageContent},  # Result content
+    details::T,                       # Tool-specific details
+    usage::Union{Usage, Nothing},     # Usage statistics
+    added_tool_names::Union{Vector{String}, Nothing},
+    terminate::Union{Bool, Nothing},  # If true, stop agent after this
+)
+```
+
+## AgentContext
+
+```julia
+struct AgentContext
+    system_prompt::String
+    messages::Vector{AgentMessage}
+    tools::Union{Vector{AgentTool}, Nothing}
+end
+```
+
+**Purpose**: Read-only snapshot of agent state for LLM calls
+
+**Usage in AgentLoop**:
+```julia
+function streamAssistantResponse(
+    context::AgentContext,  # Contains messages, tools, system prompt
+    config::AgentLoopConfig,
+    ...
+)::AssistantMessage
+    # Convert to LLM format
+    llm_messages = config.convert_to_llm(context.messages)
+    
+    # Create context for API
+    llm_context = Context(
+        context.system_prompt,
+        llm_messages,
+        context.tools,
+    )
+    
+    # Call LLM
+    return stream_function(context.model, llm_context, config)
+end
+```
+
+## Event Types
+
+### Agent Lifecycle Events
+
+```julia
+struct AgentStartEvent <: AgentEvent end
+struct AgentEndEvent <: AgentEvent
+    messages::Vector{AgentMessage}
+end
+```
+
+### Turn Events
+
+```julia
+struct TurnStartEvent <: AgentEvent end
+struct TurnEndEvent <: AgentEvent
+    message::AgentMessage
+    tool_results::Vector{ToolResultMessage}
+end
+```
+
+### Message Events
+
+```julia
+struct MessageStartEvent <: AgentEvent
+    message::AgentMessage
+end
+struct MessageUpdateEvent <: AgentEvent
+    message::AgentMessage
+    assistant_message_event::Any  # Partial message event
+end
+struct MessageEndEvent <: AgentEvent
+    message::AgentMessage
+end
+```
+
+### Tool Execution Events
+
+```julia
+struct ToolExecutionStartEvent <: AgentEvent
+    tool_call_id::String
+    tool_name::String
+    args::Any
+end
+struct ToolExecutionUpdateEvent <: AgentEvent
+    tool_call_id::String
+    tool_name::String
+    args::Any
+    partial_result::Any
+end
+struct ToolExecutionEndEvent <: AgentEvent
+    tool_call_id::String
+    tool_name::String
+    result::Any
+    is_error::Bool
+end
+```
+
+## Usage Statistics
+
+```julia
+struct Usage
+    input::Int64           # Input tokens
+    output::Int64          # Output tokens
+    cache_read::Int64      # Cache read tokens
+    cache_write::Int64     # Cache write tokens
+    total_tokens::Int64    # Total tokens
+    cost::UsageCost
+end
+
+struct UsageCost
+    input::Float64
+    output::Float64
+    cache_read::Float64
+    cache_write::Float64
+    total::Float64
+end
+```
+
+**Example**:
+```julia
+Usage(
+    1000,      # input tokens
+    200,       # output tokens
+    500,       # cache read tokens
+    0,         # cache write tokens
+    1700,      # total tokens
+    UsageCost(
+        0.0005,  # input cost ($0.50 per 1M tokens)
+        0.0015,  # output cost ($1.50 per 1M tokens)
+        0.00025, # cache read cost
+        0.0,     # cache write cost
+        0.0035   # total cost
+    )
+)
+```
+
+## Model Type
+
+```julia
+struct Model{Api}
+    id::String                    # Model identifier (e.g., "gpt-4")
+    name::String                  # Model name (e.g., "GPT-4")
+    api::Api                      # API type (String, Symbol, or custom type)
+    provider::String              # Provider name (e.g., "openai")
+    base_url::String              # API base URL
+    reasoning::Bool               # Whether model supports reasoning
+    input::Vector{String}         # Input modes (e.g., ["text", "image"])
+    cost::ModelCost
+    context_window::Int64         # Max context window (e.g., 128000)
+    max_tokens::Int64             # Max output tokens
+end
+
+struct ModelCost
+    input::Float64
+    output::Float64
+    cache_read::Float64
+    cache_write::Float64
+end
+```
+
+## ToolCall Type
+
+```julia
+struct ToolCall
+    type::String           # "tool"
+    id::String             # Unique ID for this tool call
+    name::String           # Tool name to call
+    arguments::Dict{String, Any}  # Tool arguments as JSON-like Dict
+    partial_json::Union{String, Nothing}  # Partial JSON string
+end
+```
+
+**Example**:
+```julia
+ToolCall(
+    "tool",
+    "call_abc123",
+    "bash",
+    Dict(
+        "command" => "ls -la",
+        "timeout" => 30
+    ),
+    nothing
+)
+```
+
+## Custom Message Types
+
+### BashExecutionMessage
+
+```julia
+mutable struct BashExecutionMessage
+    role::String              # "custom"
+    command::String
+    output::String
+    exit_code::Union{Int64, Nothing}
+    cancelled::Bool
+    truncated::Bool
+    full_output_path::Union{String, Nothing}
+    timestamp::Timestamp
+    exclude_from_context::Bool
+end
+```
+
+### CompactionSummaryMessage
+
+```julia
+mutable struct CompactionSummaryMessage
+    role::String              # "compactionSummary"
+    summary::String           # Summary of compacted history
+    tokens_before::Int64      # Context size before compaction
+    timestamp::Timestamp
+end
+```
+
+### BranchSummaryMessage
+
+```julia
+mutable struct BranchSummaryMessage
+    role::String              # "branchSummary"
+    summary::String           # Summary of branch history
+    from_id::String           # Branch point ID
+    timestamp::Timestamp
+end
+```
+
+## AgentState
+
+```julia
+mutable struct AgentState
+    system_prompt::String
+    model::Model
+    thinking_level::ThinkingLevel
+    tools::Vector{AgentTool}
+    messages::Vector{AgentMessage}
+    is_streaming::Bool
+    streaming_message::Union{AgentMessage, Nothing}
+    pending_tool_calls::Set{String}
+    error_message::Union{String, Nothing}
+end
+```
+
+**Purpose**: Runtime state of the Agent
+
+**Note**: AgentState is mutable and used internally by Agent
+
+## Key Conversion Functions
+
+### convertToLlm()
+
+```julia
+function convertToLlm(messages::Vector{AgentMessage})::Vector{Message}
+    result::Vector{Message} = Message[]
+    
+    for m in messages
+        converted = convertToLlmMessage(m)
+        if !isnothing(converted)
+            push!(result, converted)
+        end
+    end
+    
+    return result
+end
+```
+
+**Purpose**: Transform AgentMessage[] to Message[] for LLM API
+
+**Example**:
+```julia
+# Input: AgentMessage[]
+[
+    UserMessage(...),
+    AssistantMessage(...),
+    ToolResultMessage(...),
+    BashExecutionMessage(...),  # Will be converted to UserMessage
+    CompactionSummaryMessage(...),  # Will be converted to UserMessage
+]
+
+# Output: Message[]
+[
+    UserMessage(...),
+    AssistantMessage(...),
+    ToolResultMessage(...),
+    UserMessage(...),  # Converted from BashExecutionMessage
+    UserMessage(...),  # Converted from CompactionSummaryMessage
+]
+```
+
+### Default convertToLlmMessage Implementations
+
+```julia
+function convertToLlmMessage(m::BashExecutionMessage)
+    if m.exclude_from_context
+        return nothing
+    end
+    return UserMessage("user", [TextContent(bashExecutionToText(m))], m.timestamp)
+end
+
+function convertToLlmMessage(m::CompactionSummaryMessage)
+    text = COMPACTION_SUMMARY_PREFIX * m.summary * COMPACTION_SUMMARY_SUFFIX
+    return UserMessage("user", [TextContent(text)], m.timestamp)
+end
+
+function convertToLlmMessage(m::BranchSummaryMessage)
+    text = BRANCH_SUMMARY_PREFIX * m.summary * BRANCH_SUMMARY_SUFFIX
+    return UserMessage("user", [TextContent(text)], m.timestamp)
+end
+
+function convertToLlmMessage(m::UserMessage)
+    return m  # Pass through
+end
+
+function convertToLlmMessage(m::AssistantMessage)
+    return m  # Pass through
+end
+
+function convertToLlmMessage(m::ToolResultMessage)
+    return m  # Pass through
+end
+```
+
+## Summary
+
+The type system in AgentCore.jl provides:
+
+1. **Strong typing** for different message types
+2. **Extensibility** through abstract types and multiple dispatch
+3. **Clear separation** between internal (AgentMessage) and external (Message) formats
+4. **Rich metadata** in Usage and Model types for cost tracking
+5. **Event-driven architecture** through Event types
+6. **Tool execution flexibility** through Tool types with hooks
+
+All types are designed for:
+- **Interoperability** with LLM APIs
+- **Extensibility** for custom message types
+- **Performance** with immutable structs where possible
+- **Debuggability** through rich event system

learning/05-SESSION_MANAGEMENT.md

@@ -0,0 +1,763 @@
+# AgentCore.jl - Session Management Deep Dive
+
+## Session Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                        Session Layer                                        │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  Session = Tree of Entries                                                  │
+│                                                                             │
+│  Each entry represents a change in conversation state                       │
+│                                                                             │
+│  Branch Navigation:                                                         │
+│  ┌─────┐     ┌─────┐      ┌─────┐      ┌─────┐     ┌─────┐                  │
+│  │ E1  │────▶│ E2  │────▶│ E3  │────▶│ E4  │────▶│ E5  │ (current leaf)   │
+│  └─────┘     └─────┘      └─────┘      └─────┘     └─────┘                  │
+│      │           │           │           │           │                      │
+│      ▼           ▼           ▼           ▼           ▼                      │
+│   Message    Message   Compaction   Message    BranchSummary                │
+│                                                                             │
+│  To navigate to E2 (fork point):                                            │
+│  Session.moveTo(E2)                                                         │
+│  ┌─────┐      ┌─────┐     ┌─────┐      ┌─────┐     ┌─────┐                  │
+│  │ E1  │────▶│ E2  │────▶│ E3' │────▶│ E4' │────▶│ E5' │ (new branch)     │
+│  └─────┘      └─────┘     └─────┘      └─────┘     └─────┘                  │
+│      │           │                                                          │
+│      │           ▼ create BranchSummary                                     │
+│      │      ┌─────┐                                                         │
+│      └──────│ E6  │ (branch summary)                                        │
+│             └─────┘                                                         │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+## Entry Types
+
+```julia
+abstract type SessionTreeEntry end
+```
+
+### 1. MessageEntry
+
+```julia
+struct MessageEntry <: SessionTreeEntry
+    type::String           # "message"
+    id::String             # Unique entry ID
+    parent_id::Union{String, Nothing}
+    timestamp::String      # ISO 8601 timestamp
+    message::AgentMessage  # The actual message
+end
+```
+
+**Represents**: A user, assistant, or tool message
+
+### 2. ThinkingLevelChangeEntry
+
+```julia
+struct ThinkingLevelChangeEntry <: SessionTreeEntry
+    type::String           # "thinking_level_change"
+    id::String
+    parent_id::Union{String, Nothing}
+    timestamp::String
+    thinking_level::String # "off", "minimal", "low", "medium", etc.
+end
+```
+
+**Represents**: Change in model thinking level
+
+### 3. ModelChangeEntry
+
+```julia
+struct ModelChangeEntry <: SessionTreeEntry
+    type::String           # "model_change"
+    id::String
+    parent_id::Union{String, Nothing}
+    timestamp::String
+    provider::String       # "openai", "anthropic", etc.
+    model_id::String       # Model identifier
+end
+```
+
+**Represents**: Change in model
+
+### 4. ActiveToolsChangeEntry
+
+```julia
+struct ActiveToolsChangeEntry <: SessionTreeEntry
+    type::String           # "active_tools_change"
+    id::String
+    parent_id::Union{String, Nothing}
+    timestamp::String
+    active_tool_names::Vector{String}
+end
+```
+
+**Represents**: Change in active tools
+
+### 5. CompactionEntry
+
+```julia
+struct CompactionEntry <: SessionTreeEntry
+    type::String           # "compaction"
+    id::String
+    parent_id::Union{String, Nothing}
+    timestamp::String
+    summary::String        # Summary of compacted history
+    first_kept_entry_id::Union{String, Nothing}
+    tokens_before::Int64   # Context size before compaction
+    retained_tail::Union{Vector{AgentMessage}, Nothing}
+    details::Union{Any, Nothing}
+    usage::Union{Usage, Nothing}
+    from_hook::Bool        # Whether triggered by hook
+end
+```
+
+**Represents**: Context window compression
+
+**Key fields**:
+- `summary`: Summary of removed messages
+- `first_kept_entry_id`: First entry that was kept
+- `tokens_before`: Context size before compaction
+- `retained_tail`: Messages kept after compaction point
+
+### 6. BranchSummaryEntry
+
+```julia
+struct BranchSummaryEntry <: SessionTreeEntry
+    type::String           # "branch_summary"
+    id::String
+    parent_id::Union{String, Nothing}
+    timestamp::String
+    from_id::String        # Branch point entry ID
+    summary::String        # Summary of branch history
+    details::Union{Any, Nothing}
+    usage::Union{Usage, Nothing}
+    from_hook::Bool
+end
+```
+
+**Represents**: Branch point with summary
+
+### 7. CustomEntry
+
+```julia
+struct CustomEntry <: SessionTreeEntry
+    type::String           # Custom type
+    id::String
+    parent_id::Union{String, Nothing}
+    timestamp::String
+    custom_type::String
+    data::Union{Any, Nothing}
+end
+```
+
+**Represents**: Custom application-specific data
+
+### 8. CustomMessageEntry
+
+```julia
+struct CustomMessageEntry <: SessionTreeEntry
+    type::String
+    id::String
+    parent_id::Union{String, Nothing}
+    timestamp::String
+    custom_type::String
+    content::String
+    details::Union{Any, Nothing}
+    display::Bool
+end
+```
+
+**Represents**: Custom message to display to user
+
+### 9. LabelEntry
+
+```julia
+struct LabelEntry <: SessionTreeEntry
+    type::String
+    id::String
+    parent_id::Union{String, Nothing}
+    timestamp::String
+    target_id::String      # Entry being labeled
+    label::Union{String, Nothing}
+end
+```
+
+**Represents**: Label/note on an entry
+
+### 10. SessionInfoEntry
+
+```julia
+struct SessionInfoEntry <: SessionTreeEntry
+    type::String
+    id::String
+    parent_id::Union{String, Nothing}
+    timestamp::String
+    name::Union{String, Nothing}
+end
+```
+
+**Represents**: Session metadata (name, etc.)
+
+### 11. LeafEntry
+
+```julia
+struct LeafEntry <: SessionTreeEntry
+    type::String
+    id::String
+    parent_id::Union{String, Nothing}
+    timestamp::String
+    target_id::Union{String, Nothing}
+end
+```
+
+**Represents**: Change in current leaf (branch pointer)
+
+## Session Storage Interface
+
+```julia
+abstract type SessionStorage{T<:SessionMetadata} end
+```
+
+### Storage Methods
+
+```julia
+# Metadata
+getMetadata(storage::SessionStorage)::Promise{T}
+
+# Leaf management
+getLeafId(storage::SessionStorage)::Promise{Union{String, Nothing}}
+setLeafId(storage::SessionStorage, leaf_id::String)::Promise{Nothing}
+
+# Entry management
+createEntryId(storage::SessionStorage)::Promise{String}
+appendEntry(storage::SessionStorage, entry::SessionTreeEntry)::Promise{Nothing}
+getEntry(storage::SessionStorage, id::String)::Promise{Union{SessionTreeEntry, Nothing}}
+
+# Query
+findEntries(storage::SessionStorage, type::String)::Promise{Vector{SessionTreeEntry}}
+getLabel(storage::SessionStorage, id::String)::Promise{Union{String, Nothing}}
+getSessionName(storage::SessionStorage)::Promise{Union{String, Nothing}}
+
+# Branch navigation
+getPathToRootOrCompaction(
+    storage::SessionStorage,
+    leaf_id::String,
+)::Promise{Vector{SessionTreeEntry}}
+
+getEntries(storage::SessionStorage, options::Dict{String, Any})::Promise{Vector{SessionTreeEntry}}
+
+# Stats
+getSessionStats(storage::SessionStorage)::Promise{SessionStats}
+```
+
+## JsonlSessionStorage
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                        JSONL Storage Format                                 │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+File: session.jsonl
+
+Entry 1 (Metadata):
+{"type":"session","id":"meta_1","created_at":"2024-01-01T00:00:00Z","cwd":"/path","path":"/path/session.jsonl"}
+
+Entry 2 (Message):
+{"type":"message","id":"msg_1","parent_id":null,"timestamp":"2024-01-01T00:00:01Z","message":{"role":"user","content":[{"type":"text","text":"Hello"}]}}
+
+Entry 3 (Thinking Level):
+{"type":"thinking_level_change","id":"tl_1","parent_id":"msg_1","timestamp":"2024-01-01T00:00:02Z","thinking_level":"medium"}
+
+Entry 4 (Model Change):
+{"type":"model_change","id":"mc_1","parent_id":"tl_1","timestamp":"2024-01-01T00:00:03Z","provider":"openai","model_id":"gpt-4"}
+
+Entry 5 (Compaction):
+{"type":"compaction","id":"comp_1","parent_id":"mc_1","timestamp":"2024-01-01T00:00:04Z","summary":"Previous messages summarized...","first_kept_entry_id":"msg_3","tokens_before":100000,"tokens_after":50000}
+
+Entry 6 (Branch Summary):
+{"type":"branch_summary","id":"branch_1","parent_id":"comp_1","timestamp":"2024-01-01T00:00:05Z","from_id":"msg_3","summary":"Branch from message 3"}
+
+Entry 7 (Active Tools):
+{"type":"active_tools_change","id":"tools_1","parent_id":"branch_1","timestamp":"2024-01-01T00:00:06Z","active_tool_names":["bash","read"]}
+
+Entry 8 (Leaf):
+{"type":"leaf","id":"leaf_1","parent_id":"tools_1","timestamp":"2024-01-01T00:00:07Z","target_id":"msg_5"}
+
+Notes:
+- Each line is a JSON object (JSONL format)
+- parent_id references previous entry (linked list structure)
+- Leaf entry points to current position in tree
+- To fork, create new branch from any entry
+```
+
+## InMemorySessionStorage
+
+```julia
+mutable struct InMemorySessionStorage
+    metadata::SessionMetadata
+    leaf_id::Union{String, Nothing}
+    entries::Dict{String, SessionTreeEntry}
+    labels::Dict{String, String}
+end
+```
+
+**Purpose**: Testing and temporary sessions
+
+**Advantages**:
+- Fast (no I/O)
+- Easy to inspect
+- Perfect for tests
+
+## Session Class
+
+```julia
+mutable struct Session{T<:SessionMetadata}
+    storage::SessionStorage{T}
+    context_build_options::SessionContextBuildOptions
+end
+```
+
+### Session Methods
+
+#### appendMessage()
+
+```julia
+function appendMessage(session::Session, message::AgentMessage)::String
+    entry = MessageEntry(
+        "message",
+        createEntryId(session.storage),
+        getLeafId(session.storage),
+        create_timestamp(),
+        message,
+    )
+    return appendTypedEntry(session, entry)
+end
+```
+
+**Usage**:
+```julia
+session = Session(storage)
+
+# Add user message
+user_id = appendMessage(session, UserMessage("user", [TextContent("Hello")], timestamp))
+
+# Add assistant message
+assistant_id = appendMessage(session, AssistantMessage(...))
+
+# Add tool result
+tool_id = appendMessage(session, ToolResultMessage(...))
+```
+
+#### appendThinkingLevelChange()
+
+```julia
+function appendThinkingLevelChange(
+    session::Session,
+    thinking_level::String,
+)::String
+    entry = ThinkingLevelChangeEntry(
+        "thinking_level_change",
+        createEntryId(session.storage),
+        getLeafId(session.storage),
+        create_timestamp(),
+        thinking_level,
+    )
+    return appendTypedEntry(session, entry)
+end
+```
+
+#### appendCompaction()
+
+```julia
+function appendCompaction(
+    session::Session,
+    summary::String,
+    first_kept_entry_id::Union{String, Nothing},
+    tokens_before::Int64,
+    details::Union{Any, Nothing}=nothing,
+    from_hook::Bool=false,
+    usage::Union{Usage, Nothing}=nothing,
+    retained_tail::Union{Vector{AgentMessage}, Nothing}=nothing,
+)::String
+    entry = CompactionEntry(
+        "compaction",
+        createEntryId(session.storage),
+        getLeafId(session.storage),
+        create_timestamp(),
+        summary,
+        first_kept_entry_id,
+        tokens_before,
+        retained_tail,
+        details,
+        usage,
+        from_hook,
+    )
+    return appendTypedEntry(session, entry)
+end
+```
+
+#### moveTo()
+
+```julia
+function moveTo(
+    session::Session,
+    entry_id::Union{String, Nothing},
+    summary::Union{Dict{String, Any}, Nothing}=nothing,
+)::Union{String, Nothing}
+    # Set new leaf
+    setLeafId(session.storage, entry_id)
+    
+    # Optionally create branch summary
+    if !isnothing(summary)
+        return appendTypedEntry(session, BranchSummaryEntry(
+            "branch_summary",
+            createEntryId(session.storage),
+            entry_id,
+            create_timestamp(),
+            entry_id,
+            summary["summary"],
+            get(summary, "details", nothing),
+            get(summary, "usage", nothing),
+            get(summary, "from_hook", false),
+        ))
+    end
+    
+    return nothing
+end
+```
+
+**Usage**:
+```julia
+# Fork from a specific point
+session.moveTo(msg_3_id)
+
+# Branch with summary
+session.moveTo(
+    msg_3_id,
+    Dict(
+        "summary" => "User wanted to focus on file operations",
+        "details" => Dict("focus" => "files"),
+    )
+)
+```
+
+## Build Session Context
+
+```julia
+function buildSessionContext(
+    path_entries::Vector{SessionTreeEntry},
+    options::SessionContextBuildOptions=SessionContextBuildOptions(),
+)::SessionContext
+    state = deriveSessionContextState(path_entries)
+    context_entries = buildContextEntries(path_entries, options)
+    messages = SessionTreeEntry[]
+    for (i, entry) in enumerate(context_entries)
+        append!(messages, sessionEntryToContextMessages(entry, i, context_entries, options))
+    end
+    return SessionContext(messages, state.thinking_level, state.model, state.active_tool_names)
+end
+```
+
+### Context Entry Transform
+
+```julia
+function defaultContextEntryTransform(
+    path_entries::Vector{SessionTreeEntry},
+)::Vector{SessionTreeEntry}
+    compaction = nothing
+    for entry in path_entries
+        if entry isa CompactionEntry
+            compaction = entry
+            break
+        end
+    end
+    
+    if isnothing(compaction)
+        return copy(path_entries)
+    end
+    
+    # Include compaction entry
+    entries = [compaction]
+    
+    # Include retained tail if present
+    if !isnothing(compaction.retained_tail)
+        compaction_idx = findfirst(e -> e.id == compaction.id, path_entries)
+        append!(entries, path_entries[compaction_idx+1:end])
+        return entries
+    end
+    
+    # Otherwise include entries after first_kept_entry_id
+    if !isnothing(compaction.first_kept_entry_id)
+        found_first_kept = false
+        compaction_idx = findfirst(e -> e.id == compaction.id, path_entries)
+        for i in 1:compaction_idx-1
+            entry = path_entries[i]
+            if entry.id == compaction.first_kept_entry_id
+                found_first_kept = true
+            end
+            if found_first_kept
+                push!(entries, entry)
+            end
+        end
+    end
+    
+    # Include entries after compaction
+    compaction_idx = findfirst(e -> e.id == compaction.id, path_entries)
+    append!(entries, path_entries[compaction_idx+1:end])
+    
+    return entries
+end
+```
+
+### Session Entry to Context Messages
+
+```julia
+function sessionEntryToContextMessages(
+    entry::SessionTreeEntry,
+    index::Int64,
+    entries::Vector{SessionTreeEntry},
+    options::SessionContextBuildOptions=SessionContextBuildOptions(),
+)::Vector{AgentMessage}
+    if entry isa MessageEntry
+        return [entry.message]
+    end
+    
+    if entry isa CustomMessageEntry
+        return [createCustomMessage(
+            entry.custom_type,
+            entry.content,
+            entry.display,
+            entry.details,
+            entry.timestamp,
+        )]
+    end
+    
+    if entry isa CompactionEntry
+        messages = [createCompactionSummaryMessage(
+            entry.summary,
+            entry.tokens_before,
+            entry.timestamp,
+        )]
+        if !isnothing(entry.retained_tail)
+            append!(messages, entry.retained_tail)
+        end
+        return messages
+    end
+    
+    if entry isa BranchSummaryEntry
+        return [createBranchSummaryMessage(
+            entry.summary,
+            entry.from_id,
+            entry.timestamp,
+        )]
+    end
+    
+    if entry isa CustomEntry
+        # Custom projectors can transform custom entries
+        if !isnothing(options.entry_projectors) && haskey(options.entry_projectors, entry.custom_type)
+            projector = options.entry_projectors[entry.custom_type]
+            return projector(entry, index, entries)
+        end
+        return AgentMessage[]
+    end
+    
+    return AgentMessage[]
+end
+```
+
+## Branch Navigation
+
+```
+Scenario: User wants to explore a different path
+
+Initial Branch (current path):
+┌─────┐     ┌─────┐     ┌─────┐     ┌─────┐
+│ E1  │────▶│ E2  │────▶│ E3  │────▶│ E4  │ (leaf)
+└─────┘     └─────┘     └─────┘     └─────┘
+  │           │           │           │
+Message     Message   Compaction   Message
+
+Step 1: Fork from E2
+┌─────┐     ┌─────┐     ┌─────┐
+│ E1  │────▶│ E2  │─────────────────┐
+└─────┘     └─────┘                 │
+  │           │                     │
+  │           ▼ create BranchSummary│
+  │      ┌─────┐                     │
+  │      │ E5  │ (branch summary)   │
+  │      └─────┘                     │
+  └──────────────────────────────────┘
+         (new branch from E2)
+
+Step 2: Continue on new branch
+┌─────┐     ┌─────┐     ┌─────┐     ┌─────┐     ┌─────┐
+│ E1  │────▶│ E2  │────▶│ E3' │────▶│ E4' │────▶│ E5' │ (new leaf)
+└─────┘     └─────┘     └─────┘     └─────┘     └─────┘
+
+Current branch now is:
+[ E1, E2, E3', E4', E5' ]
+
+Original branch is:
+[ E1, E2, E5 ]  (E3, E4 are now separate branch)
+
+Key Points:
+- Shared entries: E1, E2
+- Branch point: E2
+- Branch summary: E5 (points to E2)
+- Each branch has independent tail
+```
+
+## Compaction Strategy
+
+### Why Compaction?
+
+LLM context windows have limits:
+- GPT-4: 128K tokens
+- Claude 2: 100K tokens
+- Llama 2: 4K tokens
+
+**Problem**: Conversations grow unbounded
+**Solution**: Compaction - summarize old messages
+
+### Compaction Process
+
+```julia
+# 1. Identify messages to compact
+# - Keep recent N messages (e.g., last 2 turns)
+# - Summarize everything before
+
+# 2. Generate summary
+# - Use LLM to summarize
+# - Include key facts, decisions, user preferences
+
+# 3. Create CompactionEntry
+# - summary: The summary text
+# - first_kept_entry_id: First entry that was NOT compacted
+# - tokens_before: Context size before compaction
+# - retained_tail: Messages kept after compaction point
+
+# 4. Update storage
+# - Append CompactionEntry
+# - Update leaf to CompactionEntry
+```
+
+### Compaction Example
+
+```julia
+# Before compaction (100K tokens):
+[
+  msg_1,  # User: "I need to set up a project"
+  msg_2,  # Assistant: "Sure, what language?"
+  msg_3,  # User: "Python"
+  msg_4,  # Assistant: "I'll create a Python project"
+  msg_5,  # User: "With FastAPI"
+  msg_6,  # Assistant: "Creating FastAPI project..."
+  msg_7,  # Tool: bash("mkdir myapp")
+  msg_8,  # Tool: write("myapp/main.py", ...)
+  msg_9,  # Assistant: "Project created!"
+  msg_10, # User: "Can you add auth?"
+  msg_11, # Assistant: "Adding auth..."
+  msg_12, # User: "Use JWT"
+  msg_13, # Assistant: "Implementing JWT..."
+  msg_14, # Tool: bash("pip install jwt")
+  msg_15, # Tool: write("myapp/auth.py", ...)
+  msg_16, # Assistant: "Auth implemented!"
+]
+
+# After compaction (20K tokens):
+[
+  compaction_entry,  # Summary of msg_1 to msg_10
+  msg_11,            # Keep recent messages
+  msg_12,
+  msg_13,
+  msg_14,
+  msg_15,
+  msg_16,
+]
+
+# Compaction summary:
+"""
+Previous conversation summary:
+- User wanted to create a Python project
+- Chose FastAPI framework
+- Assistant created project structure in myapp/
+- User requested authentication
+- Chose JWT for auth
+- Assistant implemented JWT auth in myapp/auth.py
+"""
+```
+
+## Complete Session Example
+
+```julia
+using AgentCore
+
+# 1. Create storage
+storage = JsonlSessionStorage(
+    SessionMetadata("session_1", "2024-01-01T00:00:00Z"),
+    "/path/to/session.jsonl",
+)
+
+# 2. Create session
+session = Session(storage)
+
+# 3. Add messages
+msg1_id = appendMessage(session, UserMessage("user", [TextContent("Hello")], timestamp))
+msg2_id = appendMessage(session, AssistantMessage("assistant", [TextContent("Hi!")], ...))
+
+# 4. Change thinking level
+tl_id = appendThinkingLevelChange(session, "medium")
+
+# 5. Change model
+mc_id = appendModelChange(session, "openai", "gpt-4")
+
+# 6. Add more messages
+msg3_id = appendMessage(session, UserMessage("user", [TextContent("What can you do?")], timestamp))
+msg4_id = appendMessage(session, AssistantMessage("assistant", [TextContent("I can...")], ...))
+
+# 7. Compact context (100K tokens → 20K)
+compact_id = appendCompaction(
+    session,
+    "User asked about capabilities and assistant explained",
+    msg2_id,
+    100000,
+    Dict("summary_length" => 50),
+    false,
+    usage,
+    [msg3, msg4],  # Retained tail
+)
+
+# 8. Fork and branch
+session.moveTo(msg2_id)  # Go back to msg2
+
+# 9. Create new branch
+branch_id = appendBranchSummary(
+    session,
+    "User changed direction to focus on file operations",
+    msg2_id,
+    Dict("focus" => "files"),
+)
+
+# 10. Continue on new branch
+msg5_id = appendMessage(session, UserMessage("user", [TextContent("Let's work with files")], timestamp))
+
+# 11. Query session context
+context = buildSessionContext(session)
+
+# 12. Get stats
+stats = getSessionStats(session)
+println("Messages: $(stats.message_count)")
+println("Total tokens: $(stats.total_tokens)")
+println("Cost: $$(stats.cost_total)")
+```
+
+## Best Practices
+
+1. **Use compaction** for long conversations to stay within context limits
+2. **Create branch summaries** when forking to document divergent paths
+3. **Retain tail messages** after compaction for context
+4. **Track token usage** to optimize compaction timing
+5. **Use InMemorySessionStorage** for testing

learning/06-TOOLS.md

@@ -0,0 +1,767 @@
+# AgentCore.jl - Tools Deep Dive
+
+## Tool Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                          Tool Layer                                         │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  AgentTool                                                                  │
+│  - name: String (identifier)                                                │
+│  - label: String (display name)                                             │
+│  - description: String (what it does)                                       │
+│  - parameters: JSON schema                                                  │
+│  - execute::Function (main logic)                                           │
+│  - prepare_arguments::Union{Function, Nothing}                              │
+│  - execution_mode::Union{ToolExecutionMode, Nothing}                        │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                    │
+                    ┌───────────────┼───────────────┐
+                    │               │               │
+                    ▼               ▼               ▼
+           ┌─────────────┐  ┌─────────────┐  ┌─────────────┐
+           │ BashTool    │  │ ReadTool    │  │ WriteTool   │
+           │  - bash()   │  │  - read()   │  │  - write()  │
+           └─────────────┘  └─────────────┘  └─────────────┘
+                                           ┌─────────────┐
+                                           │ EditTool    │
+                                           │  - edit()   │
+                                           └─────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  Tool Execution Flow                                                        │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+  Assistant Message
+  ┌────────────────────────────────────────────────────────┐
+  │ AssistantMessage:                                      │
+  │   content: [                                           │
+  │     TextContent("I'll check the files..."),            │
+  │     ToolCall("bash", {command: "ls -la"}),             │
+  │     ToolCall("read", {path: "README.md"})              │
+  │   ]                                                    │
+  └────────────────────────────────────────────────────────┘
+                    │
+                    ▼
+  ┌────────────────────────────────────────────────────────┐
+  │  AgentLoop.executeToolCalls()                          │
+  │  - Extract ToolCalls from message content              │
+  │  - Determine execution mode (sequential/parallel)      │
+  └────────────────────────────────────────────────────────┘
+                    │
+                    ├─► executeToolCallsSequential()
+                    │    (for tools that require order)
+                    │
+                    └─► executeToolCallsParallel()
+                         (for independent tools)
+
+                    │
+                    ├─► prepareToolCall()
+                    │    - before_tool_call hook (optional)
+                    │    - validate arguments
+                    │    - prepare arguments (optional)
+                    │
+                    ├─► execute()
+                    │    - Tool-specific logic
+                    │    - Return AgentToolResult
+                    │
+                    ├─► finalizeExecutedToolCall()
+                    │    - after_tool_call hook (optional)
+                    │
+                    └─► createToolResultMessage()
+                         - Emit ToolResultMessage
+
+                    │
+                    ▼
+  ┌────────────────────────────────────────────────────────┐
+  │  ToolResultMessage                                     │
+  │  - tool_call_id: "ref to original ToolCall"            │
+  │  - tool_name: "bash"                                   │
+  │  - content: [TextContent("file1.md\nfile2.md\n")]      │
+  │  - is_error: false                                     │
+  └────────────────────────────────────────────────────────┘
+                    │
+                    ▼
+  ┌────────────────────────────────────────────────────────┐
+  │  AgentState.messages.append(tool_result)               │
+  │  - Next turn: LLM sees tool results                    │
+  └────────────────────────────────────────────────────────┘
+```
+
+## Built-in Tools
+
+### 1. BashTool
+
+```julia
+struct BashToolOptions{TContext}
+    command_prefix::Union{String, Nothing}
+    prepare::Union{BashPrepare{TContext}, Nothing}
+end
+
+struct BashPrepare{TContext}
+    function::Function
+    context::TContext
+    signal::Union{Any, Nothing}
+end
+
+struct BashToolDetails
+    truncation::Union{Any, Nothing}
+    full_output_path::Union{String, Nothing}
+end
+```
+
+#### createBashTool()
+
+```julia
+function createBashTool{TContext}(options::Union{BashToolOptions{TContext}, Nothing}=nothing)
+    return AgentTool(
+        "bash",
+        "bash",
+        "Execute a bash command in the current working directory.",
+        Dict{String, Any}(),
+        (tool_call_id, params, signal, on_update, context) -> begin
+            # Execute command
+            result = executeBashCommand(params, signal, on_update)
+            
+            # Return result
+            return AgentToolResult(
+                [TextContent(result.output)],
+                BashToolDetails(result.truncation, result.full_path),
+                nothing,
+                nothing,
+                result.terminate,
+            )
+        end,
+        nothing,  # prepare_arguments
+        nothing,  # execution_mode (default: use config)
+    )
+end
+```
+
+**Parameters Schema**:
+```json
+{
+  "command": "string",
+  "timeout": "number (optional)",
+  "cwd": "string (optional)",
+  "env": "object (optional)"
+}
+```
+
+**Example**:
+```julia
+# Create tool
+bash_tool = createBashTool()
+
+# Agent receives command
+tool_call = ToolCall("tool", "tc1", "bash", Dict(
+    "command" => "ls -la",
+    "timeout" => 30
+), nothing)
+
+# Execute
+result = bash_tool.execute(
+    "tc1",
+    Dict("command" => "ls -la", "timeout" => 30),
+    nothing,
+    on_update,  # Callback for streaming output
+    nothing,
+)
+
+# Result
+AgentToolResult(
+    [TextContent("total 12\n-rw-r--r-- 1 user user  100 Jan 1 file1.md\n-rw-r--r-- 1 user user 200 Jan 2 file2.md\n")],
+    BashToolDetails(truncation_info, nothing),
+    nothing,
+    nothing,
+    nothing,
+)
+```
+
+### 2. ReadTool
+
+```julia
+struct ReadToolOptions{TContext}
+    max_size::Union{Int64, Nothing}
+    max_lines::Union{Int64, Nothing}
+    image_processor::Union{ReadImageProcessor, Nothing}
+    prepare::Union{ReadPrepare{TContext}, Nothing}
+end
+
+struct ReadImageProcessor
+    function::Function
+    context::Any
+end
+
+struct ReadImageProcessorResult
+    content::Vector{MessageContent}
+    usage::Union{Usage, Nothing}
+end
+```
+
+#### createReadTool()
+
+```julia
+function createReadTool{TContext}(options::Union{ReadToolOptions{TContext}, Nothing}=nothing)
+    return AgentTool(
+        "read",
+        "read",
+        "Read a file from the file system.",
+        Dict{String, Any}(),
+        (tool_call_id, params, signal, on_update, context) -> begin
+            # Read file
+            result = readFileSystem(params, signal, options)
+            
+            # Process content
+            content = if isImage(params.path)
+                # Image processing
+                image_result = options.image_processor.function(result.path, context)
+                image_result.content
+            else
+                # Text content
+                [TextContent(result.content)]
+            end
+            
+            return AgentToolResult(
+                content,
+                ReadToolDetails(result.size, result.truncated, result.full_path),
+                nothing,
+                nothing,
+                nothing,
+            )
+        end,
+        nothing,
+        nothing,
+    )
+end
+```
+
+**Parameters Schema**:
+```json
+{
+  "path": "string"
+}
+```
+
+**Example**:
+```julia
+# Create tool
+read_tool = createReadTool()
+
+# Agent requests to read file
+tool_call = ToolCall("tool", "tc2", "read", Dict(
+    "path" => "src/main.jl"
+), nothing)
+
+# Execute
+result = read_tool.execute("tc2", Dict("path" => "src/main.jl"), nothing, nothing, nothing)
+
+# Result
+AgentToolResult(
+    [TextContent("module Main\nfunction main()\n  println(\"Hello\")\nend\nend\n")],
+    ReadToolDetails(1234, false, "/path/to/src/main.jl"),
+    nothing,
+    nothing,
+    nothing,
+)
+```
+
+### 3. WriteTool
+
+```julia
+struct WriteToolInput
+    path::String
+    content::String
+end
+```
+
+#### createWriteTool()
+
+```julia
+function createWriteTool{TContext}(options::Union{WriteToolOptions{TContext}, Nothing}=nothing)
+    return AgentTool(
+        "write",
+        "write",
+        "Write content to a file.",
+        Dict{String, Any}(),
+        (tool_call_id, params, signal, on_update, context) -> begin
+            # Write file
+            result = writeToFile(params, signal)
+            
+            return AgentToolResult(
+                [TextContent(result.message)],
+                nothing,
+                nothing,
+                nothing,
+                nothing,
+            )
+        end,
+        nothing,
+        nothing,
+    )
+end
+```
+
+**Parameters Schema**:
+```json
+{
+  "path": "string",
+  "content": "string"
+}
+```
+
+**Example**:
+```julia
+# Create tool
+write_tool = createWriteTool()
+
+# Agent wants to write file
+tool_call = ToolCall("tool", "tc3", "write", Dict(
+    "path" => "output.txt",
+    "content" => "Hello World"
+), nothing)
+
+# Execute
+result = write_tool.execute("tc3", Dict(
+    "path" => "output.txt",
+    "content" => "Hello World"
+), nothing, nothing, nothing)
+
+# Result
+AgentToolResult(
+    [TextContent("File written: output.txt (11 bytes)")],
+    nothing,
+    nothing,
+    nothing,
+    nothing,
+)
+```
+
+### 4. EditTool
+
+```julia
+struct EditToolInput
+    path::String
+    find::String
+    replacement::String
+end
+
+struct EditToolDetails
+    edits::Vector{Edit}
+    before_content::String
+    after_content::String
+end
+```
+
+#### createEditTool()
+
+```julia
+function createEditTool{TContext}(options::Union{EditToolOptions{TContext}, Nothing}=nothing)
+    return AgentTool(
+        "edit",
+        "edit",
+        "Edit a file by finding and replacing text.",
+        Dict{String, Any}(),
+        (tool_call_id, params, signal, on_update, context) -> begin
+            # Read file
+            before_content = read(params.path)
+            
+            # Apply edit
+            after_content = replace(before_content, params.find => params.replacement)
+            
+            # Write file
+            write(params.path, after_content)
+            
+            return AgentToolResult(
+                [TextContent("Edit applied successfully")],
+                EditToolDetails([Edit(params.find, params.replacement)], before_content, after_content),
+                nothing,
+                nothing,
+                nothing,
+            )
+        end,
+        nothing,
+        nothing,
+    )
+end
+```
+
+**Parameters Schema**:
+```json
+{
+  "path": "string",
+  "find": "string",
+  "replacement": "string"
+}
+```
+
+**Example**:
+```julia
+# Create tool
+edit_tool = createEditTool()
+
+# Agent wants to replace text
+tool_call = ToolCall("tool", "tc4", "edit", Dict(
+    "path" => "README.md",
+    "find" => "v1.0.0",
+    "replacement" => "v2.0.0"
+), nothing)
+
+# Execute
+result = edit_tool.execute("tc4", Dict(
+    "path" => "README.md",
+    "find" => "v1.0.0",
+    "replacement" => "v2.0.0"
+), nothing, nothing, nothing)
+
+# Result
+AgentToolResult(
+    [TextContent("Edit applied: README.md")],
+    EditToolDetails([Edit("v1.0.0", "v2.0.0")], "Version 1.0.0", "Version 2.0.0"),
+    nothing,
+    nothing,
+    nothing,
+)
+```
+
+## Tool Execution Hooks
+
+### before_tool_call
+
+```julia
+struct BeforeToolCallContext
+    assistant_message::AssistantMessage
+    tool_call::ToolCall
+    args::Any
+    context::AgentContext
+end
+
+struct BeforeToolCallResult
+    block::Union{Bool, Nothing}
+    reason::Union{String, Nothing}
+end
+```
+
+**Usage**:
+```julia
+function myBeforeToolCall(context, signal)
+    tool_name = context.tool_call.name
+    
+    # Block dangerous commands
+    if tool_name == "bash" && contains(context.args["command"], "rm -rf /")
+        return BeforeToolCallResult(
+            true,
+            "Blocking dangerous command: rm -rf /"
+        )
+    end
+    
+    # Log tool execution
+    println("Executing tool: $tool_name")
+    
+    return nothing  # Allow execution
+end
+
+# Configure agent
+agent = Agent(Dict(
+    :beforeToolCall => myBeforeToolCall,
+))
+```
+
+### after_tool_call
+
+```julia
+struct AfterToolCallContext
+    assistant_message::AssistantMessage
+    tool_call::ToolCall
+    args::Any
+    result::AgentToolResult
+    is_error::Bool
+    context::AgentContext
+end
+
+struct AfterToolCallResult
+    content::Union{Vector{MessageContent}, Nothing}
+    details::Union{Any, Nothing}
+    is_error::Union{Bool, Nothing}
+    usage::Union{Usage, Nothing}
+    terminate::Union{Bool, Nothing}
+end
+```
+
+**Usage**:
+```julia
+function myAfterToolCall(context, signal)
+    tool_name = context.tool_call.name
+    
+    # Modify bash output
+    if tool_name == "bash"
+        # Add timestamp to output
+        new_content = [
+            TextContent("[Executed at $(Dates.now())]\n"),
+            context.result.content[1],
+        ]
+        return AfterToolCallResult(
+            content = new_content,
+            details = context.result.details,
+            is_error = context.is_error,
+            usage = context.result.usage,
+            terminate = context.result.terminate,
+        )
+    end
+    
+    return nothing  # Use original result
+end
+
+# Configure agent
+agent = Agent(Dict(
+    :afterToolCall => myAfterToolCall,
+))
+```
+
+### prepare_next_turn
+
+```julia
+struct PrepareNextTurnContext
+    message::AssistantMessage
+    tool_results::Vector{ToolResultMessage}
+    context::AgentContext
+    new_messages::Vector{AgentMessage}
+end
+
+struct AgentLoopTurnUpdate
+    context::Union{AgentContext, Nothing}
+    model::Union{Model, Nothing}
+    thinking_level::Union{ThinkingLevel, Nothing}
+end
+```
+
+**Usage**:
+```julia
+function myPrepareNextTurn(context, signal)
+    # Check if we should use a different model
+    last_message = context.message
+    tool_results = context.tool_results
+    
+    # If tool execution had errors, use more capable model
+    has_errors = any(r -> r.is_error, tool_results)
+    if has_errors
+        return AgentLoopTurnUpdate(
+            context = context.context,
+            model = Model("gpt-4", "GPT-4", "openai", "openai", "", ...),
+            thinking_level = THINKING_HIGH,
+        )
+    end
+    
+    return nothing  # Keep current settings
+end
+
+# Configure agent
+agent = Agent(Dict(
+    :prepareNextTurn => myPrepareNextTurn,
+))
+```
+
+## Tool Execution Modes
+
+### Sequential Execution
+
+```julia
+# Tools run one at a time, in order
+# Use case: Tools that modify shared state
+
+# Configure tool
+bash_tool = AgentTool(
+    "bash",
+    "bash",
+    "Execute bash command",
+    ...,
+    execute,
+    nothing,
+    EXECUTION_SEQUENTIAL,  # Force sequential
+)
+
+# Or configure globally
+agent = Agent(Dict(
+    :toolExecution => EXECUTION_SEQUENTIAL,
+))
+```
+
+**Example Scenario**:
+```julia
+# Sequential execution (correct order)
+
+1. Tool 1: create_directory("build/")
+   └─ Creates build/ directory
+
+2. Tool 2: write("build/app.js", "...")
+   └─ Writes file to build/
+
+(If parallel: might fail because build/ doesn't exist yet)
+```
+
+### Parallel Execution
+
+```julia
+# Tools run concurrently
+# Use case: Independent operations
+
+# Default behavior
+agent = Agent(Dict(
+    :toolExecution => EXECUTION_PARALLEL,  # Default
+))
+```
+
+**Example Scenario**:
+```julia
+# Parallel execution (independent operations)
+
+1. Tool 1: read("README.md") ─────┐
+2. Tool 2: read("CHANGELOG.md") ─┼─► Run simultaneously
+3. Tool 3: read("LICENSE") ──────┘
+
+(Parallel: All three read operations can happen at once)
+(Sequential: Would wait for each read to complete)
+```
+
+## Custom Tools
+
+### Example: Database Tool
+
+```julia
+function createDatabaseTool()
+    return AgentTool(
+        "database",
+        "database",
+        "Execute SQL queries against the database.",
+        Dict{String, Any}(
+            "type" => "object",
+            "properties" => Dict(
+                "query" => Dict("type" => "string"),
+                "params" => Dict("type" => "array", "items" => Dict("type" => "string")),
+            ),
+            "required" => ["query"],
+        ),
+        (tool_call_id, params, signal, on_update, context) -> begin
+            # Execute query
+            query = params["query"]
+            result = executeQuery(query)
+            
+            # Format output
+            output = formatQueryResult(result)
+            
+            return AgentToolResult(
+                [TextContent(output)],
+                Dict("rows_affected" => result.rows_affected),
+                nothing,
+                nothing,
+                nothing,
+            )
+        end,
+        nothing,
+        EXECUTION_SEQUENTIAL,
+    )
+end
+
+# Usage
+db_tool = createDatabaseTool()
+agent = Agent(Dict(:tools => [db_tool]))
+```
+
+### Example: HTTP Request Tool
+
+```julia
+function createHTTPTool()
+    return AgentTool(
+        "http",
+        "http",
+        "Make HTTP requests.",
+        Dict{String, Any}(
+            "type" => "object",
+            "properties" => Dict(
+                "url" => Dict("type" => "string"),
+                "method" => Dict("type" => "string", "enum" => ["GET", "POST", "PUT", "DELETE"]),
+                "body" => Dict("type" => "string"),
+                "headers" => Dict("type" => "object"),
+            ),
+            "required" => ["url", "method"],
+        ),
+        (tool_call_id, params, signal, on_update, context) -> begin
+            # Make request
+            url = params["url"]
+            method = params["method"]
+            body = get(params, "body", nothing)
+            headers = get(params, "headers", Dict())
+            
+            response = makeHTTPRequest(method, url, body, headers)
+            
+            return AgentToolResult(
+                [TextContent(response.body)],
+                Dict(
+                    "status_code" => response.status_code,
+                    "headers" => response.headers,
+                ),
+                nothing,
+                nothing,
+                nothing,
+            )
+        end,
+        nothing,
+        EXECUTION_PARALLEL,
+    )
+end
+```
+
+## Complete Example
+
+```julia
+using AgentCore
+
+# 1. Create tools
+bash_tool = createBashTool()
+read_tool = createReadTool()
+write_tool = createWriteTool()
+
+# 2. Configure hooks
+before_hook = (context, signal) -> begin
+    println("About to execute: $(context.tool_call.name)")
+    return nothing
+end
+
+after_hook = (context, signal) -> begin
+    if context.is_error
+        println("Tool failed: $(context.tool_call.name)")
+    else
+        println("Tool completed: $(context.tool_call.name)")
+    end
+    return nothing
+end
+
+# 3. Create agent
+agent = Agent(Dict(
+    :systemPrompt => "You are a helpful assistant with file system access.",
+    :tools => [bash_tool, read_tool, write_tool],
+    :beforeToolCall => before_hook,
+    :afterToolCall => after_hook,
+))
+
+# 4. Run conversation
+prompt(agent, "List files in current directory and read the first one")
+
+# 5. Agent will:
+#    - Execute bash("ls -la") tool
+#    - Parse output to find first file
+#    - Execute read("path/to/file") tool
+#    - Return content to user
+```
+
+## Best Practices
+
+1. **Use sequential execution** for tools that depend on shared state
+2. **Use parallel execution** for independent operations
+3. **Implement before_tool_call hook** for logging and validation
+4. **Implement after_tool_call hook** for result modification
+5. **Use prepare_next_turn hook** for dynamic model/thinking level changes
+6. **Return terminate=true** from tool when agent should stop
+7. **Include usage statistics** in tool results when possible

learning/07-AGENTHARNESS.md

@@ -0,0 +1,754 @@
+# AgentCore.jl - AgentHarness Deep Dive
+
+## AgentHarness Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                        AgentHarness Layer                                   │
+└─────────────────────────────────────────────────────────────────────────────┐
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  AgentHarness = Agent + Session + Resources                                 │
+│                                                                             │
+│  ┌───────────────────────────────────────────────────────────────────────┐  │
+│  │  AgentHarness                                                         │  │
+│  │  - Manages Agent instances                                            │  │
+│  │  - Provides Session persistence                                       │  │
+│  │  - Manages resources (skills, prompt templates)                       │  │
+│  │  - Handles extension hooks                                            │  │
+│  │  - Coordinates tool execution with context                            │  │
+│  └───────────────────────────────────────────────────────────────────────┘  │
+│                              │                                              │
+│        ┌─────────────────────┼─────────────────────┐                        │
+│        ▼                     ▼                     ▼                        │
+│   ┌──────────────┐    ┌──────────────┐    ┌──────────────┐                  │
+│   │   Agent      │    │  SessionRepo │    │   Resources  │                  │
+│   │  (state,     │    │  (create,    │    │  (skills,    │                  │
+│   │   events)    │    │   open,      │    │   templates) │                  │
+│   └──────────────┘    │   list)      │    └──────────────┘                  │
+│                       └──────────────┘                                      │
+│                                     │                                       │
+│                                     ▼                                       │
+│                            ┌──────────────┐                                 │
+│                            │ Session      │                                 │
+│                            │ (history,    │                                 │
+│                            │  branching)  │                                 │
+│                            └──────────────┘                                 │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────────────────┐
+│  AgentHarnessEvent System                                                   │
+└─────────────────────────────────────────────────────────────────────────────┘
+
+AgentEvent (from Agent)
+├─ AgentHarnessOwnEvent
+│  ├─ BeforeAgentStartEvent
+│  ├─ ContextEvent
+│  ├─ BeforeProviderRequestEvent
+│  ├─ BeforeProviderPayloadEvent
+│  ├─ AfterProviderResponseEvent
+│  ├─ ToolCallEvent
+│  ├─ ToolResultEvent
+│  ├─ SessionBeforeCompactEvent
+│  ├─ SessionCompactEvent
+│  ├─ SessionBeforeTreeEvent
+│  ├─ SessionTreeEvent
+│  ├─ ModelUpdateEvent
+│  ├─ ThinkingLevelUpdateEvent
+│  ├─ ToolsUpdateEvent
+│  ├─ ResourcesUpdateEvent
+│  └─ ... (other session events)
+
+└─ AgentEvent (from AgentLoop)
+   ├─ AgentStartEvent / AgentEndEvent
+   ├─ TurnStartEvent / TurnEndEvent
+   ├─ MessageStartEvent / MessageEndEvent
+   └─ ToolExecutionStartEvent / ToolExecutionEndEvent
+```
+
+## AgentHarness Components
+
+### 1. AgentHarnessOptions
+
+```julia
+mutable struct AgentHarnessOptions{
+    TC, TSkill<:Skill, TPromptTemplate<:PromptTemplate, TTool<:AgentHarnessTool
+}
+    session::Session
+    models::Any
+    tools::Union{Vector{TTool}, Nothing}
+    resources::Union{AgentHarnessResources{TSkill, TPromptTemplate}, Nothing}
+    system_prompt::Union{AgentHarnessSystemPrompt{TC, TSkill, TPromptTemplate, TTool}, Nothing}
+    stream_options::Union{AgentHarnessStreamOptions, Nothing}
+    retry::Union{Any, Nothing}
+    model::Model
+    thinking_level::Union{ThinkingLevel, Nothing}
+    active_tool_names::Union{Vector{String}, Nothing}
+    steering_mode::Union{QueueMode, Nothing}
+    follow_up_mode::Union{QueueMode, Nothing}
+    tool_context::Union{AgentHarnessToolContextSource{TC}, Nothing}
+end
+```
+
+**Purpose**: Configure AgentHarness with all necessary options
+
+**Key fields**:
+- `session`: Session instance for persistence
+- `models`: Available models
+- `tools`: Agent tools
+- `resources`: Skills and prompt templates
+- `system_prompt`: System prompt (string or function)
+- `stream_options`: LLM streaming options
+- `model`: Default model
+- `thinking_level`: Default thinking level
+- `active_tool_names`: Active tools
+- `tool_context`: Context source for tools
+
+### 2. AgentHarnessResources
+
+```julia
+mutable struct AgentHarnessResources{TSkill<:Skill, TPromptTemplate<:PromptTemplate}
+    promptTemplates::Union{Vector{TPromptTemplate}, Nothing}
+    skills::Union{Vector{TSkill}, Nothing}
+end
+```
+
+**Purpose**: Load and manage skills and prompt templates
+
+### 3. Skill
+
+```julia
+mutable struct Skill
+    name::String
+    description::String
+    content::String
+    filePath::String
+    disableModelInvocation::Bool
+end
+```
+
+**Purpose**: Define specialized instructions for specific tasks
+
+**Format**:
+```markdown
+<!-- SKILL.md -->
+{
+  "name": "File Operations",
+  "description": "Handle file system operations",
+  "disable-model-invocation": false
+}
+---
+
+# File Operations Skill
+
+This skill provides instructions for working with files...
+```
+
+### 4. PromptTemplate
+
+```julia
+mutable struct PromptTemplate
+    name::String
+    description::Union{String, Nothing}
+    content::String
+end
+```
+
+**Purpose**: Reusable prompt snippets with arguments
+
+**Format**:
+```markdown
+<!-- template.md -->
+{
+  "description": "Generate commit message"
+}
+---
+
+Generate a git commit message for:
+$1
+$ARGUMENTS
+```
+
+### 5. AgentHarnessStreamOptions
+
+```julia
+mutable struct AgentHarnessStreamOptions
+    transport::Union{String, Nothing}
+    timeout_ms::Union{Int64, Nothing}
+    max_retries::Union{Int64, Nothing}
+    max_retry_delay_ms::Union{Int64, Nothing}
+    headers::Union{Dict{String, String}, Nothing}
+    metadata::Union{Dict{String, Any}, Nothing}
+    cache_retention::Union{String, Nothing}
+end
+```
+
+**Purpose**: Configure LLM API call options
+
+## SessionRepo Interface
+
+```julia
+abstract type SessionRepo<
+    TMetadata<:SessionMetadata,
+    TCreateOptions,
+    TListOptions
+> end
+```
+
+### Repo Methods
+
+```julia
+# Create new session
+create(repo::SessionRepo, options::TCreateOptions)::Promise{Session}
+
+# Open existing session
+open(repo::SessionRepo, metadata::TMetadata)::Promise{Session}
+
+# List sessions
+list(repo::SessionRepo, options::TListOptions)::Promise{Vector{TMetadata}}
+
+# Delete session
+delete(repo::SessionRepo, metadata::TMetadata)::Promise{Nothing}
+
+# Fork session (create branch)
+fork(repo::SessionRepo, source::TMetadata, options::Dict{String, Any})::Promise{Session}
+```
+
+### JsonlSessionRepo
+
+```julia
+# JSONL-based session repository
+# - Sessions stored as JSONL files
+# - Supports create, open, list, delete, fork
+# - Branch navigation via session tree
+```
+
+## Extension Hooks
+
+### Hook Types
+
+```julia
+# Before agent starts
+BeforeAgentStartEvent
+├─ prompt: String
+├─ images: Union{Vector{ImageContent}, Nothing}
+├─ system_prompt: String
+└─ resources: AgentHarnessResources
+
+BeforeAgentStartResult
+├─ messages: Union{Vector{AgentMessage}, Nothing}
+└─ system_prompt: Union{String, Nothing}
+
+# Context event
+ContextEvent
+└─ messages: Vector{AgentMessage}
+
+ContextResult
+└─ messages: Vector{AgentMessage}
+
+# Before LLM request
+BeforeProviderRequestEvent
+├─ model: Model
+├─ session_id: String
+└─ stream_options: AgentHarnessStreamOptions
+
+BeforeProviderRequestResult
+└─ stream_options: Union{AgentHarnessStreamOptionsPatch, Nothing}
+
+# Before LLM payload
+BeforeProviderPayloadEvent
+├─ model: Model
+└─ payload: Any
+
+BeforeProviderPayloadResult
+└─ payload: Any
+
+# After LLM response
+AfterProviderResponseEvent
+├─ status: Int64
+└─ headers: Dict{String, String}
+
+# Tool call
+ToolCallEvent
+├─ tool_call_id: String
+├─ tool_name: String
+└─ input: Dict{String, Any}
+
+ToolCallResult
+├─ block: Union{Bool, Nothing}
+└─ reason: Union{String, Nothing}
+
+# Tool result
+ToolResultEvent
+├─ tool_call_id: String
+├─ tool_name: String
+├─ input: Dict{String, Any}
+├─ content: Vector{MessageContent}
+├─ details: Any
+├─ is_error: Bool
+└─ usage: Union{Usage, Nothing}
+
+ToolResultPatch
+├─ content: Union{Vector{MessageContent}, Nothing}
+├─ details: Union{Any, Nothing}
+├─ is_error: Union{Bool, Nothing}
+├─ usage: Union{Usage, Nothing}
+└─ terminate: Union{Bool, Nothing}
+
+# Session compaction
+SessionBeforeCompactEvent
+├─ preparation: Any
+├─ branch_entries: Vector{SessionTreeEntry}
+├─ custom_instructions: Union{String, Nothing}
+└─ signal: Any
+
+SessionBeforeCompactResult
+├─ cancel: Union{Bool, Nothing}
+└─ compaction: Union{CompactResult, Nothing}
+
+SessionCompactEvent
+├─ compaction_entry: CompactionEntry
+└─ from_hook: Bool
+
+# Session tree (branching)
+SessionBeforeTreeEvent
+├─ preparation: Any
+└─ signal: Any
+
+SessionBeforeTreeResult
+├─ cancel: Union{Bool, Nothing}
+├─ summary: Union{Dict{String, Any}, Nothing}
+├─ custom_instructions: Union{String, Nothing}
+├─ replace_instructions: Union{Bool, Nothing}
+└─ label: Union{String, Nothing}
+
+SessionTreeEvent
+├─ new_leaf_id: Union{String, Nothing}
+├─ old_leaf_id: Union{String, Nothing}
+├─ summary_entry: Union{BranchSummaryEntry, Nothing}
+└─ from_hook: Union{Bool, Nothing}
+```
+
+### Hook Usage Examples
+
+#### BeforeAgentStartHook
+
+```julia
+function beforeAgentStart(event, signal)
+    # Modify system prompt based on context
+    new_system_prompt = "$(event.system_prompt)\n\nUser prefers concise responses."
+    
+    # Prepend initial messages
+    initial_messages = [
+        UserMessage("user", [TextContent("Context: $(event.prompt)")], timestamp),
+    ]
+    
+    return BeforeAgentStartResult(
+        initial_messages,
+        new_system_prompt,
+    )
+end
+
+# Configure harness
+harness = AgentHarness(Dict(
+    :beforeAgentStart => beforeAgentStart,
+))
+```
+
+#### BeforeProviderPayloadHook
+
+```julia
+function beforeProviderPayload(event, signal)
+    # Modify LLM payload before sending
+    payload = event.payload
+    
+    # Add custom metadata
+    payload.metadata = merge(payload.metadata, Dict(
+        "session_id" => event.session_id,
+        "timestamp" => Dates.now(),
+    ))
+    
+    return BeforeProviderPayloadResult(payload)
+end
+```
+
+#### ToolCallHook
+
+```julia
+function toolCall(event, signal)
+    # Block dangerous tool calls
+    if event.tool_name == "bash" && contains(event.input["command"], "rm -rf /")
+        return ToolCallResult(true, "Blocking dangerous command")
+    end
+    
+    # Log tool execution
+    println("Tool call: $(event.tool_name)")
+    
+    return nothing  # Allow execution
+end
+```
+
+#### BeforeCompactHook
+
+```julia
+function beforeCompact(event, signal)
+    # Add custom instructions for compaction
+    custom_instructions = """
+    Focus on retaining user preferences and key decisions.
+    Omit verbose tool outputs that don't add value.
+    """
+    
+    return SessionBeforeCompactResult(
+        false,  # Don't cancel
+        Dict(
+            "summary" => "Custom compaction with focus on user intent",
+            "custom_instructions" => custom_instructions,
+        ),
+    )
+end
+```
+
+## Tool Context
+
+### AgentHarnessToolContextSource
+
+```julia
+mutable struct AgentHarnessToolContextSource{TContext}
+    context::Union{TContext, Function}
+end
+```
+
+**Purpose**: Provide context to tools during execution
+
+### Tool Execution Context
+
+```julia
+# Tools receive context from AgentHarness
+tool.execute(
+    tool_call_id,
+    params,
+    signal,
+    on_update,
+    context,  # From AgentHarnessToolContextSource
+)
+
+# Context can be:
+# - Static value
+# - Function that returns value
+```
+
+## Complete Example
+
+```julia
+using AgentCore
+
+# 1. Create skills
+skills, skill_diagnostics = loadSkills(
+    execution_env,
+    "/path/to/skills",
+)
+
+# 2. Create prompt templates
+templates, template_diagnostics = loadPromptTemplates(
+    execution_env,
+    "/path/to/templates",
+)
+
+# 3. Create resources
+resources = AgentHarnessResources(
+    templates,
+    skills,
+)
+
+# 4. Create session repo
+repo = JsonlSessionRepo(
+    "/path/to/sessions",
+)
+
+# 5. Create session
+session = create(repo, Dict(
+    "cwd" => "/path/to/project",
+    "metadata" => Dict("project" => "my-project"),
+))
+
+# 6. Configure tools
+bash_tool = createBashTool()
+read_tool = createReadTool()
+
+tools = [bash_tool, read_tool]
+
+# 7. Configure hooks
+hooks = Dict(
+    :beforeAgentStart => beforeAgentStartHook,
+    :beforeProviderPayload => beforePayloadHook,
+    :toolCall => toolCallHook,
+)
+
+# 8. Create harness
+harness = AgentHarness(Dict(
+    :session => session,
+    :models => models,
+    :tools => tools,
+    :resources => resources,
+    :system_prompt => "You are a helpful assistant.",
+    :model => Model(...),
+    :thinking_level => THINKING_MEDIUM,
+    :active_tool_names => ["bash", "read"],
+    :steering_mode => QUEUE_ONE_AT_A_TIME,
+    :follow_up_mode => QUEUE_ONE_AT_A_TIME,
+    :tool_context => AgentHarnessToolContextSource(context),
+    :stream_options => AgentHarnessStreamOptions(
+        transport = "auto",
+        timeout_ms = 30000,
+        max_retries = 3,
+    ),
+))
+
+# 9. Subscribe to events
+subscribe(harness) do event, signal
+    if event isa BeforeAgentStartEvent
+        println("Agent starting...")
+    elseif event isa MessageEndEvent
+        println("Message: $(event.message)")
+    end
+end
+
+# 10. Run conversation
+harness.prompt("What files are in the current directory?")
+
+# 11. Wait for completion
+wait_for_idle(harness)
+
+# 12. Manage branches
+session.moveTo(some_entry_id)  # Fork from entry
+```
+
+## Hook Execution Flow
+
+```
+User Code
+    │
+    ├─► AgentHarness.prompt()
+    │
+    ▼
+┌────────────────────────────────────────────────────────────────────────┐
+│  BeforeAgentStartEvent                                                 │
+│  ├─ User prompt                                                        │
+│  ├─ System prompt                                                      │
+│  └─ Resources                                                          │
+│        │                                                                │
+│        └─► beforeAgentStart hook (optional)                            │
+│            └─► BeforeAgentStartResult (optional modifications)         │
+└────────────────────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌────────────────────────────────────────────────────────────────────────┐
+│  Agent.createLoopConfig()                                              │
+│  └─► Merge options with hooks                                          │
+└────────────────────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌────────────────────────────────────────────────────────────────────────┐
+│  Agent.prompt()                                                        │
+│  └─► Start AgentLoop                                                   │
+└────────────────────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌────────────────────────────────────────────────────────────────────────┐
+│  AgentLoop.agentLoop()                                                 │
+│        │                                                                │
+│        ├─► transform_context hook (optional)                           │
+│        └─► convert_to_llm()                                            │
+│            └─► Message[] for LLM API                                   │
+└────────────────────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌────────────────────────────────────────────────────────────────────────┐
+│  BeforeProviderRequestEvent                                            │
+│  ├─ Model                                                              │
+│  ├─ Session ID                                                         │
+│  └─ Stream Options                                                     │
+│        │                                                                │
+│        └─► beforeProviderRequest hook (optional)                       │
+│            └─► BeforeProviderRequestResult (optional modifications)    │
+└────────────────────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌────────────────────────────────────────────────────────────────────────┐
+│  StreamFn (LLM API call)                                               │
+└────────────────────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌────────────────────────────────────────────────────────────────────────┐
+│  AfterProviderResponseEvent                                            │
+│  ├─ Status code                                                        │
+│  └─ Response headers                                                   │
+└────────────────────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌────────────────────────────────────────────────────────────────────────┐
+│  BeforeProviderPayloadEvent                                            │
+│  ├─ Model                                                              │
+│  └─ Payload (before sending)                                           │
+│        │                                                                │
+│        └─► beforeProviderPayload hook (optional)                       │
+│            └─► BeforeProviderPayloadResult (optional modifications)    │
+└────────────────────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌────────────────────────────────────────────────────────────────────────┐
+│  LLM API Request                                                       │
+└────────────────────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌────────────────────────────────────────────────────────────────────────┐
+│  Assistant Message (streaming)                                         │
+│        │                                                                │
+│        ├─► Text deltas                                                 │
+│        └─► Tool calls                                                  │
+└────────────────────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌────────────────────────────────────────────────────────────────────────┐
+│  Tool Execution (for each tool call)                                   │
+│        │                                                                │
+│        ├─► before_tool_call hook (Agent)                               │
+│        ├─► toolCall hook (Harness - optional)                          │
+│        │       └─► ToolCallResult (can block execution)               │
+│        ├─► prepareToolCall()                                           │
+│        ├─► execute()                                                   │
+│        │       └─► Tool execution with context                        │
+│        ├─► after_tool_call hook (Agent)                                │
+│        └─► toolResult hook (Harness - optional)                        │
+│                └─► ToolResultPatch (can modify result)                │
+└────────────────────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌────────────────────────────────────────────────────────────────────────┐
+│  AgentLoop continues with tool results                                 │
+│        │                                                                │
+│        ├─► Next LLM call with tool results                             │
+│        └─► Or end of conversation                                      │
+└────────────────────────────────────────────────────────────────────────┘
+    │
+    ▼
+┌────────────────────────────────────────────────────────────────────────┐
+│  AgentEndEvent                                                         │
+│  └─► Final messages in session                                         │
+└────────────────────────────────────────────────────────────────────────┘
+```
+
+## Session Management with Harness
+
+```julia
+# Create harness with session repo
+repo = JsonlSessionRepo("/path/to/sessions")
+
+# Create session
+session = create(repo, Dict(
+    "cwd" => "/path/to/project",
+    "metadata" => Dict("name" => "my-session"),
+))
+
+# Or open existing session
+metadata = JsonlSessionMetadata(...)
+session = open(repo, metadata)
+
+# List sessions
+sessions = list(repo, Dict())
+for meta in sessions
+    println("Session: $(meta.id)")
+end
+
+# Delete session
+delete(repo, metadata)
+
+# Fork session (branch)
+forked_session = fork(repo, source_metadata, Dict(
+    "summary" => "Branch for feature X",
+))
+```
+
+## Resources Management
+
+```julia
+# Load skills from directory
+skills, diagnostics = loadSkills(
+    execution_env,
+    "/path/to/skills",
+)
+
+# Load prompt templates from directory
+templates, diagnostics = loadPromptTemplates(
+    execution_env,
+    "/path/to/templates",
+)
+
+# Create resources
+resources = AgentHarnessResources(
+    templates,
+    skills,
+)
+
+# Use in harness
+harness = AgentHarness(Dict(
+    :resources => resources,
+))
+```
+
+## Best Practices
+
+1. **Use hooks for logging and validation**
+   - `beforeAgentStart` for initialization
+   - `beforeProviderPayload` for custom metadata
+   - `toolCall` for blocking dangerous operations
+
+2. **Organize skills by domain**
+   - File operations
+   - Database queries
+   - HTTP requests
+   - Git operations
+
+3. **Use templates for common patterns**
+   - Commit message generation
+   - Code review instructions
+   - Testing prompts
+
+4. **Manage sessions carefully**
+   - Compact periodically
+   - Use branches for exploration
+   - Clean up old sessions
+
+5. **Monitor resource usage**
+   - Track token counts
+   - Watch API costs
+   - Optimize tool execution
+
+## Troubleshooting
+
+### Hook not being called
+
+```julia
+# Check hook is registered
+if isnothing(harness.beforeAgentStart)
+    println("Hook not registered")
+end
+```
+
+### Session not persisting
+
+```julia
+# Check repo is configured
+if isnothing(harness.repo)
+    println("No repo configured")
+end
+```
+
+### Resources not loading
+
+```julia
+# Check diagnostics
+for diag in skill_diagnostics
+    println("Skill warning: $(diag.message)")
+end
+```

learning/08-EXAMPLES.md

@@ -0,0 +1,893 @@
+# AgentCore.jl - Examples and Patterns
+
+## Quick Start Examples
+
+### Example 1: Basic Conversation
+
+```julia
+using AgentCore
+
+# Create model
+model = Model(
+    "gpt-4",
+    "GPT-4",
+    "openai",
+    "openai",
+    "https://api.openai.com/v1",
+    true,
+    ["text"],
+    ModelCost(0.00003, 0.00006, 0.0, 0.0),
+    128000,
+    4096,
+)
+
+# Create tools
+bash_tool = createBashTool()
+
+# Create agent
+agent = Agent(Dict(
+    :systemPrompt => "You are a helpful assistant.",
+    :model => model,
+    :tools => [bash_tool],
+    :thinkingLevel => THINKING_MEDIUM,
+    :toolExecution => EXECUTION_PARALLEL,
+))
+
+# Subscribe to events
+subscribe(agent) do event, signal
+    if event isa MessageEndEvent
+        println("Agent: $(event.message)")
+    end
+end
+
+# Start conversation
+prompt(agent, "What's in the current directory?")
+
+# Wait for completion
+wait_for_idle(agent)
+
+# Get final state
+state = get_state(agent)
+println("Total messages: $(length(state.messages))")
+```
+
+### Example 2: Conversation with Memory
+
+```julia
+# Create session storage
+storage = JsonlSessionStorage(
+    JsonlSessionMetadata(
+        "session_1",
+        "2024-01-01T00:00:00Z",
+        "/path/to/project",
+        "/path/to/session.jsonl",
+        nothing,
+        Dict("project" => "my-project"),
+    ),
+    "/path/to/session.jsonl",
+)
+
+# Create session
+session = Session(storage)
+
+# Create agent with session
+agent = Agent(Dict(
+    :systemPrompt => "You are a helpful assistant.",
+    :model => model,
+    :tools => [bash_tool],
+    :sessionId => session.getMetadata().id,
+))
+
+# Add messages to session
+function addToSession(session, message)
+    appendMessage(session, message)
+end
+
+# Start conversation
+prompt(agent, "Hello, my name is Alice.")
+
+# Continue conversation (messages persist in session)
+prompt(agent, "What's the weather like today?")
+
+# Check session stats
+stats = getSessionStats(session)
+println("Messages: $(stats.message_count)")
+println("Total tokens: $(stats.total_tokens)")
+```
+
+### Example 3: Steering and Follow-Up
+
+```julia
+# Start conversation
+prompt(agent, "Create a Python project.")
+
+# User wants to redirect
+steer(agent, UserMessage("user", [TextContent("Actually, let's use Node.js instead")], timestamp))
+
+# Wait for redirection
+wait_for_idle(agent)
+
+# Agent would normally stop, but user has more
+prompt(agent, "Wait, there's one more thing...")
+followUp(agent, UserMessage("user", [TextContent("Can you add tests?")], timestamp))
+
+# Continue until completion
+while hasQueuedMessages(agent)
+    wait_for_idle(agent)
+end
+```
+
+### Example 4: Branching Conversations
+
+```julia
+# Initial conversation
+prompt(agent, "I want to build a web app.")
+
+# User decides to explore a different path
+session.moveTo(msg_3_id)  # Go back to message 3
+
+# Create branch
+appendBranchSummary(
+    session,
+    "User decided to explore mobile app instead",
+    msg_3_id,
+    Dict("focus" => "mobile"),
+)
+
+# Continue on new branch
+prompt(agent, "Let's build a mobile app instead.")
+
+# Check branches
+branch = getBranch(session)
+println("Current branch has $(length(branch)) entries")
+```
+
+## Advanced Patterns
+
+### Pattern 1: Long-Running Agent with Compaction
+
+```julia
+# Configure compaction settings
+MAX_TOKENS = 120000  # Stay under 128K limit
+COMPACTION_THRESHOLD = 100000
+
+# Agent loop with compaction
+function runAgentWithCompaction(agent, session)
+    while true
+        # Get current token count
+        stats = getSessionStats(session)
+        
+        if stats.total_tokens > COMPACTION_THRESHOLD
+            # Compact session
+            compactSession(session)
+        end
+        
+        # Check if agent is idle
+        if !hasQueuedMessages(agent) && !isnothing(agent.active_run)
+            break
+        end
+    end
+end
+
+function compactSession(session)
+    # Get current branch
+    branch = getBranch(session)
+    
+    # Calculate tokens to compact
+    total_tokens = 0
+    for entry in branch
+        if entry isa MessageEntry
+            total_tokens += estimateTokens(entry.message)
+        end
+    end
+    
+    if total_tokens < COMPACTION_THRESHOLD
+        return
+    end
+    
+    # Identify messages to compact
+    messages_to_compact = []
+    tokens_to_keep = 50000  # Keep recent 50K tokens
+    
+    for entry in branch
+        if entry isa MessageEntry
+            msg_tokens = estimateTokens(entry.message)
+            if tokens_to_keep > 0
+                tokens_to_keep -= msg_tokens
+            else
+                push!(messages_to_compact, entry)
+            end
+        end
+    end
+    
+    # Generate summary
+    summary = generateSummary(messages_to_compact)
+    
+    # Create compaction entry
+    appendCompaction(
+        session,
+        summary,
+        messages_to_compact[end].id,
+        total_tokens,
+    )
+    
+    println("Compacted $(length(messages_to_compact)) messages")
+end
+
+function estimateTokens(message::AgentMessage)::Int64
+    # Simple estimation: ~4 chars per token
+    content = if message isa UserMessage
+        join([c.text for c in message.content if c isa TextContent])
+    elseif message isa AssistantMessage
+        join([c.text for c in message.content if c isa TextContent])
+    elseif message isa ToolResultMessage
+        join([c.text for c in message.content if c isa TextContent])
+    else
+        ""
+    end
+    
+    return ceil(Int, length(content) / 4)
+end
+
+function generateSummary(messages::Vector{MessageEntry})::String
+    # Use LLM to generate summary
+    summary = "Conversation summary:"
+    for msg in messages
+        summary *= "\n- $(msg.message)"
+    end
+    return summary
+end
+```
+
+### Pattern 2: Custom Tool with Context
+
+```julia
+# Define context type
+struct DatabaseContext
+    connection::Any
+    user::String
+end
+
+# Create tool with context
+function createDatabaseTool()
+    return AgentTool(
+        "database",
+        "database",
+        "Execute SQL queries",
+        Dict{String, Any}(),
+        (tool_call_id, params, signal, on_update, context) -> begin
+            if !isa(context, DatabaseContext)
+                return AgentToolResult(
+                    [TextContent("Error: Database context not provided")],
+                    nothing,
+                    nothing,
+                    nothing,
+                    true,  # terminate
+                )
+            end
+            
+            # Execute query
+            query = params["query"]
+            result = executeQuery(context.connection, query)
+            
+            return AgentToolResult(
+                [TextContent(formatResult(result))],
+                Dict("user" => context.user),
+                nothing,
+                nothing,
+                nothing,
+            )
+        end,
+        nothing,
+        EXECUTION_SEQUENTIAL,
+    )
+end
+
+# Use tool with context
+db_context = DatabaseContext(connection, "alice")
+
+harness = AgentHarness(Dict(
+    :tools => [createDatabaseTool()],
+    :tool_context => AgentHarnessToolContextSource(db_context),
+))
+```
+
+### Pattern 3: Dynamic Model Selection
+
+```julia
+# Hook to change model based on task
+function dynamicModelSelection(context, signal)
+    # Check message content
+    last_message = context.message
+    
+    # If complex task, use more capable model
+    if contains(join(last_message.content), "analyze")
+        return AgentLoopTurnUpdate(
+            context = context.context,
+            model = Model("gpt-4", "GPT-4", "openai", ...),
+            thinking_level = THINKING_HIGH,
+        )
+    end
+    
+    # Otherwise use cheaper model
+    return AgentLoopTurnUpdate(
+        context = context.context,
+        model = Model("gpt-3.5", "GPT-3.5", "openai", ...),
+        thinking_level = THINKING_MEDIUM,
+    )
+end
+
+# Configure agent
+agent = Agent(Dict(
+    :prepareNextTurn => dynamicModelSelection,
+))
+```
+
+### Pattern 4: Rate Limiting
+
+```julia
+# Rate limiter
+struct RateLimiter
+    calls_per_minute::Int
+    last_calls::Vector{DateTime}
+end
+
+function RateLimiter(calls_per_minute::Int)
+    return RateLimiter(calls_per_minute, DateTime[])
+end
+
+function rateLimit(limiter::RateLimiter)
+    now = Dates.now()
+    
+    # Remove old calls
+    limiter.last_calls = filter(
+        c -> Dates.value(now - c) / 1000 < 60,
+        limiter.last_calls,
+    )
+    
+    # Check limit
+    if length(limiter.last_calls) >= limiter.calls_per_minute
+        return false
+    end
+    
+    # Record call
+    push!(limiter.last_calls, now)
+    return true
+end
+
+# Use in hook
+limiter = RateLimiter(60)  # 60 calls per minute
+
+function rateLimitHook(event, signal)
+    if !rateLimit(limiter)
+        return BeforeProviderPayloadResult(event.payload)  # Still send, but track
+    end
+    
+    return BeforeProviderPayloadResult(event.payload)
+end
+
+# Configure
+agent = Agent(Dict(
+    :beforeProviderPayload => rateLimitHook,
+))
+```
+
+### Pattern 5: Multi-Step Tool Execution
+
+```julia
+# Tool that requires multiple steps
+function createMultiStepTool()
+    return AgentTool(
+        "multistep",
+        "multistep",
+        "Multi-step task",
+        Dict{String, Any}(),
+        (tool_call_id, params, signal, on_update, context) -> begin
+            # Step 1: Prepare
+            on_update("Preparing...")
+            prepare_result = prepareStep(params)
+            
+            # Step 2: Execute
+            on_update("Executing...")
+            execute_result = executeStep(prepare_result, params)
+            
+            # Step 3: Finalize
+            on_update("Finalizing...")
+            finalize_result = finalizeStep(execute_result)
+            
+            return AgentToolResult(
+                [TextContent(finalize_result)],
+                Dict("steps" => 3),
+                nothing,
+                nothing,
+                nothing,
+            )
+        end,
+        nothing,
+        EXECUTION_SEQUENTIAL,
+    )
+end
+```
+
+### Pattern 6: Image Processing
+
+```julia
+# Create read tool with image support
+image_processor = ReadImageProcessor(
+    (path, context) -> begin
+        # Load image
+        image_data = readImage(path)
+        
+        # Process with vision model
+        result = processImageWithVision(image_data)
+        
+        return ReadImageProcessorResult(
+            [TextContent(result.description)],
+            result.usage,
+        )
+    end,
+    context,
+)
+
+read_tool = createReadTool(Dict(
+    "image_processor" => image_processor,
+))
+```
+
+### Pattern 7: Session Navigation
+
+```julia
+# Navigate to specific point
+session.moveTo(entry_id)
+
+# Get branch from specific point
+branch = getBranch(session, entry_id)
+
+# Create label for easy navigation
+appendLabel(session, entry_id, "important-decision")
+
+# Find labeled entry
+label = getLabel(session, "important-decision")
+
+# Build context from branch
+context = buildSessionContext(session)
+
+# Get specific messages
+messages = sessionEntryToContextMessages(entry, index, entries)
+```
+
+### Pattern 8: Batch Processing
+
+```julia
+# Process multiple prompts in batch
+prompts = [
+    "What is Julia?",
+    "What is JavaScript?",
+    "What is Python?",
+]
+
+results = []
+
+for prompt_text in prompts
+    # Create fresh agent for each prompt
+    agent = Agent(Dict(
+        :systemPrompt => "You are a helpful assistant.",
+        :model => model,
+        :tools => [bash_tool],
+    ))
+    
+    # Run prompt
+    prompt(agent, prompt_text)
+    wait_for_idle(agent)
+    
+    # Get result
+    state = get_state(agent)
+    last_message = state.messages[end]
+    
+    push!(results, last_message)
+    
+    # Clean up
+    reset!(agent)
+end
+
+# Process results
+for result in results
+    println("Result: $(result)")
+end
+```
+
+### Pattern 9: Custom Event Handling
+
+```julia
+# Custom event types
+struct CustomEvent <: AgentEvent
+    data::Any
+end
+
+# Custom event handler
+function customEventHandler(event, signal)
+    if event isa CustomEvent
+        println("Custom event: $(event.data)")
+    end
+end
+
+# Subscribe to custom events
+subscribe(agent) do event, signal
+    customEventHandler(event, signal)
+end
+
+# Emit custom event
+emit(CustomEvent("custom data"))
+```
+
+### Pattern 10: Error Handling
+
+```julia
+# Hook for error handling
+function errorHook(context, signal)
+    if context isa PrepareNextTurnContext
+        last_message = context.message
+        
+        if last_message.stop_reason == "error"
+            println("Error in conversation: $(last_message.error_message)")
+            
+            return AgentLoopTurnUpdate(
+                context = context.context,
+                model = context.context.model,
+                thinking_level = THINKING_HIGH,  # Use more capable model
+            )
+        end
+    end
+    
+    return nothing
+end
+
+# Use in agent
+agent = Agent(Dict(
+    :prepareNextTurn => errorHook,
+))
+```
+
+## Testing Patterns
+
+### Unit Testing
+
+```julia
+# Test tool execution
+@testset "Bash tool" begin
+    tool = createBashTool()
+    
+    # Test successful execution
+    result = tool.execute("tc1", Dict("command" => "echo hello"), nothing, nothing, nothing)
+    @test result.content[1].text == "hello\n"
+    @test result.details === nothing
+    
+    # Test error handling
+    result = tool.execute("tc2", Dict("command" => "exit 1"), nothing, nothing, nothing)
+    @test result.terminate === true
+end
+
+# Test agent with mock LLM
+@testset "Agent with mock" begin
+    # Mock stream function
+    function mockStreamFn(model, context, options)
+        # Return mock response
+        return MockResponse([TextContent("Hello!")])
+    end
+    
+    agent = Agent(Dict(
+        :stream_fn => mockStreamFn,
+        :systemPrompt => "You are a helpful assistant.",
+        :model => model,
+    ))
+    
+    # Test prompt
+    prompt(agent, "Hello")
+    wait_for_idle(agent)
+    
+    # Verify result
+    state = get_state(agent)
+    @test length(state.messages) == 2  # User + Assistant
+end
+```
+
+### Integration Testing
+
+```julia
+# Test full conversation flow
+@testset "Full conversation" begin
+    # Create session storage
+    storage = InMemorySessionStorage(...)
+    session = Session(storage)
+    
+    # Create agent
+    agent = Agent(Dict(
+        :systemPrompt => "You are a helpful assistant.",
+        :model => model,
+        :tools => [bash_tool],
+        :sessionId => session.getMetadata().id,
+    ))
+    
+    # Run conversation
+    prompt(agent, "What's in the directory?")
+    wait_for_idle(agent)
+    
+    # Verify session
+    context = buildSessionContext(session)
+    @test length(context.messages) == 2
+    
+    # Continue conversation
+    prompt(agent, "What's the weather?")
+    wait_for_idle(agent)
+    
+    # Verify growth
+    context = buildSessionContext(session)
+    @test length(context.messages) == 4
+end
+```
+
+## Performance Patterns
+
+### Pattern 1: Caching
+
+```julia
+# Simple caching for LLM calls
+struct LLMCache
+    cache::Dict{String, AssistantMessage}
+end
+
+function LLMCache()
+    return LLMCache(Dict{String, AssistantMessage}())
+end
+
+function getCached(cache::LLMCache, key::String)
+    return get(cache.cache, key, nothing)
+end
+
+function setCached(cache::LLMCache, key::String, value::AssistantMessage)
+    cache.cache[key] = value
+end
+
+# Use in stream function
+function cachedStreamFn(model, context, options)
+    key = generateCacheKey(context)
+    
+    cached = getCached(cache, key)
+    if !isnothing(cached)
+        return MockResponse(cached)
+    end
+    
+    result = actualStreamFn(model, context, options)
+    setCached(cache, key, result)
+    return result
+end
+```
+
+### Pattern 2: Batch LLM Calls
+
+```julia
+# Batch multiple LLM calls
+function batchLLMCalls(calls::Vector{Dict})
+    results = []
+    
+    for call in calls
+        result = streamFunction(
+            call[:model],
+            call[:context],
+            call[:options],
+        )
+        push!(results, result)
+    end
+    
+    return results
+end
+
+# Use with parallel execution
+tool.execute = (id, params, signal, on_update, context) -> begin
+    # Batch multiple LLM calls
+    llm_calls = [
+        Dict(:model => model, :context => context1, :options => options1),
+        Dict(:model => model, :context => context2, :options => options2),
+    ]
+    
+    results = batchLLMCalls(llm_calls)
+    
+    return AgentToolResult(
+        [TextContent(join([r.text for r in results], "\n"))],
+        nothing,
+        nothing,
+        nothing,
+        nothing,
+    )
+end
+```
+
+### Pattern 3: Lazy Loading
+
+```julia
+# Lazy load skills
+struct LazySkills
+    dir::String
+    skills::Union{Vector{Skill}, Nothing}
+end
+
+function LazySkills(dir)
+    return LazySkills(dir, nothing)
+end
+
+function getSkills(lazy::LazySkills)
+    if isnothing(lazy.skills)
+        lazy.skills, _ = loadSkills(lazy.dir)
+    end
+    return lazy.skills
+end
+
+# Use in harness
+harness = AgentHarness(Dict(
+    :resources => AgentHarnessResources(
+        templates,
+        LazySkills("/path/to/skills"),
+    ),
+))
+```
+
+## Production Patterns
+
+### Pattern 1: Observability
+
+```julia
+# Logging hook
+function loggingHook(event, signal)
+    if event isa BeforeProviderRequestEvent
+        println("[Request] $(event.model.id)")
+    elseif event isa AfterProviderResponseEvent
+        println("[Response] Status: $(event.status)")
+    elseif event isa ToolExecutionEndEvent
+        println("[Tool] $(event.tool_name): $(event.is_error ? "error" : "success")")
+    end
+    return nothing
+end
+
+# Metrics hook
+function metricsHook(event, signal)
+    if event isa AgentStartEvent
+        metrics.start_time = Dates.now()
+    elseif event isa AgentEndEvent
+        duration = Dates.value(Dates.now() - metrics.start_time) / 1000
+        println("[Metrics] Duration: $(duration)s")
+    end
+    return nothing
+end
+```
+
+### Pattern 2: Retry Logic
+
+```julia
+# Retry hook
+function retryHook(event, signal)
+    if event isa AfterProviderResponseEvent && event.status >= 500
+        # Server error, retry
+        return BeforeProviderRequestResult(Dict(
+            "retry" => true,
+            "max_retries" => 3,
+        ))
+    end
+    return nothing
+end
+
+# Use in stream options
+harness = AgentHarness(Dict(
+    :stream_options => AgentHarnessStreamOptions(
+        max_retries = 3,
+        max_retry_delay_ms = 5000,
+    ),
+    :retry => retryHook,
+))
+```
+
+### Pattern 3: Security
+
+```julia
+# Security hook
+function securityHook(event, signal)
+    if event isa ToolCallEvent
+        # Validate tool call
+        if event.tool_name == "bash"
+            command = event.input["command"]
+            
+            # Block dangerous commands
+            dangerous_patterns = ["rm -rf /", "sudo", "curl | sh"]
+            for pattern in dangerous_patterns
+                if contains(command, pattern)
+                    return ToolCallResult(true, "Blocked dangerous command")
+                end
+            end
+        end
+    end
+    
+    return nothing
+end
+```
+
+## Debugging Patterns
+
+### Pattern 1: Conversation Trace
+
+```julia
+# Trace conversation
+trace = []
+
+subscribe(agent) do event, signal
+    if event isa MessageEndEvent
+        push!(trace, Dict(
+            "role" => event.message.role,
+            "content" => event.message.content,
+        ))
+    end
+end
+
+# Run conversation
+prompt(agent, "Hello")
+wait_for_idle(agent)
+
+# Print trace
+for entry in trace
+    println("$(entry["role"]): $(entry["content"])")
+end
+```
+
+### Pattern 2: Tool Call Trace
+
+```julia
+tool_trace = []
+
+subscribe(agent) do event, signal
+    if event isa ToolExecutionStartEvent
+        push!(tool_trace, Dict(
+            "type" => "start",
+            "tool" => event.tool_name,
+            "args" => event.args,
+        ))
+    elseif event isa ToolExecutionEndEvent
+        push!(tool_trace, Dict(
+            "type" => "end",
+            "tool" => event.tool_name,
+            "error" => event.is_error,
+        ))
+    end
+end
+```
+
+### Pattern 3: State Dump
+
+```julia
+function dumpState(agent)
+    state = get_state(agent)
+    
+    println("=== Agent State ===")
+    println("System prompt: $(state.system_prompt)")
+    println("Model: $(state.model.name)")
+    println("Thinking level: $(state.thinking_level)")
+    println("Messages: $(length(state.messages))")
+    println("Tools: $(length(state.tools))")
+    println("==================")
+end
+
+# Use after conversation
+prompt(agent, "Hello")
+wait_for_idle(agent)
+dumpState(agent)
+```
+
+## Best Practices Summary
+
+1. **Start simple**, add complexity gradually
+2. **Use hooks for customization**, not core logic
+3. **Test with mock LLM** first
+4. **Monitor token usage** for long conversations
+5. **Use branches** for exploration
+6. **Compact periodically** to stay within limits
+7. **Handle errors gracefully**
+8. **Log important events**
+9. **Test edge cases**
+10. **Profile performance**

learning/README.md

@@ -0,0 +1,386 @@
+# AgentCore.jl - Learning Guide
+
+## How to Use This Documentation
+
+### Top-Down Learning Approach
+
+This documentation is organized in a **top-down** order, starting from high-level concepts and drilling down into implementation details. Follow this sequence:
+
+1. **Architecture Overview** - Understand the big picture
+2. **Agent Component** - Learn about state management and event streaming
+3. **AgentLoop Component** - Understand the core LLM interaction loop
+4. **Types & Messages** - Learn the data structures
+5. **Session Management** - Understand conversation history
+6. **Tools** - Learn about tool execution
+
+### Learning Style
+
+- **Visual learners**: Study the ASCII diagrams
+- **Hands-on learners**: Code examples provided for each section
+- **Conceptual learners**: Read summaries and overviews first
+
+## Quick Start
+
+### Minimal Example
+
+```julia
+using AgentCore
+
+# Create agent
+agent = Agent(Dict(
+    :systemPrompt => "You are a helpful assistant.",
+    :model => Model(...),
+    :tools => [bash_tool],
+))
+
+# Run conversation
+prompt(agent, "Hello!")
+
+# Wait for completion
+wait_for_idle(agent)
+```
+
+### Understanding the Flow
+
+```
+User Code
+    │
+    ├─► Create Agent
+    │   ├─ Initialize state
+    │   ├─ Set up queues
+    │   └─ Register hooks
+    │
+    ├─► prompt("Hello")
+    │   ├─ Validate input
+    │   └─ Start AgentLoop
+    │
+    ├─► AgentLoop (runs in thread)
+    │   ├─ Stream LLM response
+    │   ├─ Execute tools
+    │   └─ Emit events
+    │
+    └─► Event handlers receive events
+        ├─ MessageEndEvent
+        ├─ ToolExecutionEndEvent
+        └─ AgentEndEvent
+```
+
+## Core Concepts
+
+### Agent
+
+**What it is**: High-level interface for LLM interactions
+
+**What it does**:
+- Manages conversation state
+- Handles event streaming
+- Queues steering/follow-up messages
+- Provides hooks for customization
+
+**Key methods**:
+- `prompt()` - Start new conversation
+- `continue!()` - Continue existing conversation
+- `steer()` - Queue message for next turn
+- `followUp()` - Queue message after stop
+- `subscribe()` - Listen to events
+
+### AgentLoop
+
+**What it is**: Core LLM interaction loop
+
+**What it does**:
+- Calls LLM API with streaming
+- Executes tool calls (parallel or sequential)
+- Emits lifecycle events
+- Handles steering/follow-up messages
+
+**Key functions**:
+- `agentLoop()` - Start new conversation
+- `agentLoopContinue()` - Continue conversation
+- `runAgentLoop()` - Internal loop execution
+- `streamAssistantResponse()` - LLM API call
+- `executeToolCalls()` - Tool execution
+
+### Session
+
+**What it is**: Conversation history management
+
+**What it does**:
+- Persists messages to storage
+- Supports branching
+- Implements compaction
+- Manages conversation tree
+
+**Key methods**:
+- `appendMessage()` - Add message
+- `appendCompaction()` - Compress history
+- `moveTo()` - Navigate branches
+- `buildSessionContext()` - Build context for LLM
+
+### Tools
+
+**What it is**: Functions agents can call
+
+**What they do**:
+- Execute external operations
+- Return results to agent
+- Support streaming updates
+- Implement hooks
+
+**Built-in tools**:
+- `bash` - Execute shell commands
+- `read` - Read files
+- `write` - Write files
+- `edit` - Edit files
+
+## Event System
+
+### Event Types
+
+```
+AgentEvent
+├─ AgentStartEvent / AgentEndEvent
+├─ TurnStartEvent / TurnEndEvent
+├─ MessageStartEvent / MessageEndEvent
+├─ MessageUpdateEvent
+├─ ToolExecutionStartEvent / ToolExecutionEndEvent
+└─ ToolExecutionUpdateEvent
+```
+
+### Event Flow
+
+```
+AgentStartEvent
+    │
+    ├─ TurnStartEvent
+    │   ├─ MessageStartEvent (user)
+    │   ├─ MessageEndEvent (user)
+    │   ├─ MessageStartEvent (assistant)
+    │   ├─ MessageUpdateEvent (streaming)
+    │   ├─ MessageEndEvent (assistant)
+    │   ├─ ToolExecutionStartEvent
+    │   ├─ ToolExecutionEndEvent
+    │   └─ TurnEndEvent
+    │
+    └─ AgentEndEvent
+```
+
+## Data Flow
+
+### Message Transformation
+
+```
+AgentMessage[] (internal)
+    │
+    ├─ transform_context() (optional)
+    ▼
+AgentMessage[] (transformed)
+    │
+    ├─ convert_to_llm()
+    ▼
+Message[] (LLM API)
+```
+
+### Tool Execution Flow
+
+```
+ToolCall (in assistant message)
+    │
+    ├─ before_tool_call hook
+    ├─ prepareToolCall()
+    ├─ execute()
+    ├─ after_tool_call hook
+    └─ createToolResultMessage()
+```
+
+## Best Practices
+
+### 1. Use Hooks for Customization
+
+```julia
+# Before tool call
+before_hook = (context, signal) -> begin
+    println("Executing: $(context.tool_call.name)")
+    return nothing
+end
+
+# After tool call
+after_hook = (context, signal) -> begin
+    if context.is_error
+        println("Tool failed: $(context.tool_call.name)")
+    end
+    return nothing
+end
+```
+
+### 2. Monitor Events
+
+```julia
+subscribe(agent) do event, signal
+    if event isa MessageEndEvent
+        println("Message: $(event.message)")
+    elseif event isa ToolExecutionEndEvent
+        println("Tool completed: $(event.tool_name)")
+    end
+end
+```
+
+### 3. Use Steering for Redirection
+
+```julia
+# Agent is going wrong direction
+steer(agent, UserMessage("Actually, let's do X instead"))
+```
+
+### 4. Use Follow-Up for Continuation
+
+```julia
+# Agent thinks it's done, but user wants more
+followUp(agent, UserMessage("Wait, there's one more thing"))
+```
+
+## Common Patterns
+
+### Pattern 1: Conversation with Memory
+
+```julia
+# Use Session to persist conversation
+storage = JsonlSessionStorage(...)
+session = Session(storage)
+
+# Add messages to session
+appendMessage(session, user_message)
+appendMessage(session, assistant_message)
+
+# Build context from session
+context = buildSessionContext(session)
+```
+
+### Pattern 2: Long Conversations
+
+```julia
+# Compact periodically to stay within context limits
+if token_count > MAX_TOKENS * 0.8
+    compact_id = appendCompaction(
+        session,
+        summary,
+        first_kept_id,
+        token_count,
+    )
+end
+```
+
+### Pattern 3: Branching Conversations
+
+```julia
+# User wants to explore alternative
+session.moveTo(branch_point_id)
+
+# Create new branch
+appendBranchSummary(session, "Exploring alternative approach")
+appendMessage(session, new_user_message)
+```
+
+### Pattern 4: Custom Tools
+
+```julia
+# Create custom tool
+custom_tool = AgentTool(
+    "custom",
+    "custom",
+    "Does custom thing",
+    ...,
+    execute_function,
+    nothing,
+    EXECUTION_PARALLEL,
+)
+
+# Add to agent
+agent = Agent(Dict(:tools => [custom_tool]))
+```
+
+## Debugging
+
+### Check Active Run
+
+```julia
+if !isnothing(agent.active_run)
+    println("Agent is busy")
+else
+    println("Agent is idle")
+end
+```
+
+### Clear Queues
+
+```julia
+clearAllQueues(agent)
+```
+
+### Reset State
+
+```julia
+reset!(agent)
+```
+
+## Performance Tips
+
+1. **Use parallel execution** for independent tools
+2. **Compact periodically** for long conversations
+3. **Use thinking_level wisely** (higher = slower but better)
+4. **Batch tool calls** when possible
+5. **Cache LLM responses** when appropriate
+
+## Troubleshooting
+
+### Agent stuck in loop
+
+```julia
+# Check if agent is still processing
+if hasQueuedMessages(agent)
+    # Clear queues
+    clearAllQueues(agent)
+end
+```
+
+### Too many tokens
+
+```julia
+# Compact session
+compact_id = appendCompaction(
+    session,
+    summary,
+    first_kept_id,
+    token_count,
+)
+```
+
+### Tool execution failed
+
+```julia
+# Check tool result
+if result.is_error
+    println("Tool failed: $(result.error)")
+end
+```
+
+## Next Steps
+
+1. Read **Architecture Overview** for deep understanding
+2. Explore **Agent Component** for state management
+3. Study **AgentLoop** for core logic
+4. Learn **Types & Messages** for data structures
+5. Master **Session Management** for persistence
+6. Build **Tools** for custom functionality
+
+## Resources
+
+- Original TypeScript implementation: `@earendil-works/pi-agent-core`
+- AgentCore.jl source code: `src/`
+- Examples: `examples/`
+
+## Community
+
+For questions and discussions:
+- GitHub Issues: `/issues`
+- Documentation: `docs/`

src/AgentCore.jl

@@ -0,0 +1,149 @@
+# AgentCore.jl - A Julia implementation of the Pi Agent Core framework
+#
+# This is a reimplementation of the TypeScript pi-agent-core package in idiomatic Julia.
+# 
+# The AgentCore package provides:
+# - Low-level `agentLoop` for stateful LLM interactions with tool execution
+# - High-level `Agent` struct with state management, event streaming, and queueing
+# - `AgentHarness` for session persistence, resource management, and extension hooks
+# - Built-in tools for file operations (read, write, edit) and bash execution
+# - Session management with JSONL-based storage, compaction, and branch navigation
+#
+# For more information about the original TypeScript implementation, see:
+# https://github.com/earendil-works/pi/packages/agent
+
+module AgentCore
+
+# Core modules
+include("types.jl")
+include("stream_fn.jl")
+include("agent_loop.jl")
+include("agent.jl")
+
+# Harness modules
+include("harness_types.jl")
+include("messages.jl")
+include("system_prompt.jl")
+include("skills.jl")
+include("prompt_templates.jl")
+include("agent_harness.jl")
+
+# Session modules
+include("session/session.jl")
+include("session/jsonl_storage.jl")
+include("session/jsonl_repo.jl")
+include("session/memory_storage.jl")
+include("session/memory_repo.jl")
+include("session/repo_utils.jl")
+
+# Tool modules
+include("tools/index.jl")
+include("tools/bash.jl")
+include("tools/read.jl")
+include("tools/write.jl")
+include("tools/edit.jl")
+include("tools/edit_diff.jl")
+include("tools/image.jl")
+include("tools/path_utils.jl")
+include("tools/file_mutation_queue.jl")
+
+# Compaction modules
+include("compaction/compaction.jl")
+include("compaction/utils.jl")
+include("compaction/branch_summarization.jl")
+
+# Utility modules
+include("utils/truncate.jl")
+include("utils/shell_output.jl")
+include("proxy.jl")
+
+# Re-export public API
+export
+    # Core types
+    AgentMessage,
+    AgentTool,
+    AgentContext,
+    AgentEvent,
+    ThinkingLevel,
+    ToolExecutionMode,
+    QueueMode,
+    AgentState,
+    
+    # Agent
+    Agent,
+    AgentOptions,
+    
+    # AgentLoop
+    AgentLoopConfig,
+    agentLoop,
+    agentLoopContinue,
+    runAgentLoop,
+    runAgentLoopContinue,
+    
+    # AgentHarness
+    AgentHarness,
+    AgentHarnessOptions,
+    AgentHarnessEvent,
+    AgentHarnessResources,
+    AgentHarnessSystemPrompt,
+    
+    # Session
+    Session,
+    SessionStorage,
+    SessionRepo,
+    JsonlSessionStorage,
+    JsonlSessionRepo,
+    InMemorySessionStorage,
+    InMemorySessionRepo,
+    
+    # Tools
+    createBashTool,
+    createReadTool,
+    createWriteTool,
+    createEditTool,
+    ExecutionEnv,
+    
+    # Compaction
+    compact,
+    prepareCompaction,
+    DEFAULT_COMPACTION_SETTINGS,
+    generateSummary,
+    generateBranchSummary,
+    
+    # Utils
+    truncateHead,
+    truncateTail,
+    formatSize,
+    DEFAULT_MAX_LINES,
+    DEFAULT_MAX_BYTES,
+    
+    # Messages
+    convertToLlm,
+    bashExecutionToText,
+    
+    # System prompt
+    formatSkillsForSystemPrompt,
+    
+    # Skills
+    loadSkills,
+    formatSkillInvocation,
+    
+    # Prompt templates
+    loadPromptTemplates,
+    formatPromptTemplateInvocation,
+    parseCommandArgs,
+    substituteArgs,
+    
+    # Proxy
+    streamProxy,
+    ProxyStreamOptions,
+    
+    # Stream
+    setDefaultStreamFn,
+    getDefaultStreamFn,
+
+    # Utility functions
+    uuidv7,
+    create_timestamp
+
+end

src/YiemAgent.jl

@@ -1,53 +0,0 @@
-module YiemAgent
-
-  # export agent
-
-
-  """ Order by dependencies of each file. The 1st included file must not depend on any other
-  files and each file can only depend on the file included before it. 
-  """
-
-  include("type.jl")
-  using .type
-
-  include("utils.jl")
-  using .utils
-
-  include("tools/registry.jl")
-  using .toolRegistry
-
-  # include("llmfunction.jl")
-  # using .llmfunction
-
-  include("agentCore.jl")
-  using .agentCore
-
-  include("api.jl")
-  using .api
-  
-  
-# ---------------------------------------------- 100 --------------------------------------------- #
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-end # module YiemAgent_v1

src/agent.jl

@@ -0,0 +1,416 @@
+"""
+    agent.jl - High-level Agent struct
+
+This module implements the high-level Agent wrapper around the low-level agent loop,
+providing state management, event streaming, and queueing for steering and follow-up messages.
+"""
+
+module Agent
+
+using ..Types: *
+using ..AgentLoop: *
+using ..StreamFn: *
+
+# ============================================================================
+# Default convertToLlm function
+# ============================================================================
+
+function defaultConvertToLlm(messages::Vector{AgentMessage})::Vector{Message}
+    return filter(
+        (m) -> m.role == "user" || m.role == "assistant" || m.role == "toolResult",
+        messages,
+    )
+end
+
+# ============================================================================
+# Empty usage constant
+# ============================================================================
+
+const EMPTY_USAGE = Usage(
+    0, 0, 0, 0, 0, UsageCost(0.0, 0.0, 0.0, 0.0, 0.0)
+)
+
+# ============================================================================
+# Pending message queue
+# ============================================================================
+
+mutable struct PendingMessageQueue
+    messages::Vector{AgentMessage}
+    mode::QueueMode
+
+    function PendingMessageQueue(mode::QueueMode)
+        new(AgentMessage[], mode)
+    end
+end
+
+function enqueue!(queue::PendingMessageQueue, message::AgentMessage)
+    push!(queue.messages, message)
+end
+
+function hasItems(queue::PendingMessageQueue)::Bool
+    return !isempty(queue.messages)
+end
+
+function drain(queue::PendingMessageQueue)::Vector{AgentMessage}
+    if queue.mode == QUEUE_ALL
+        result = copy(queue.messages)
+        empty!(queue.messages)
+        return result
+    else
+        if isempty(queue.messages)
+            return AgentMessage[]
+        end
+        first = popfirst!(queue.messages)
+        return [first]
+    end
+end
+
+function clear!(queue::PendingMessageQueue)
+    empty!(queue.messages)
+end
+
+# ============================================================================
+# Active run state
+# ============================================================================
+
+mutable struct ActiveRun
+    promise::Promise
+    abort_controller::Base.Atomic{Union{Base.AbstractLock, Nothing}}
+end
+
+# ============================================================================
+# Agent struct
+# ============================================================================
+
+mutable struct Agent
+    _state::AgentState
+    listeners::Set{Tuple{Function, Ref{Bool}}}
+    steering_queue::PendingMessageQueue
+    follow_up_queue::PendingMessageQueue
+
+    convert_to_llm::Function
+    transform_context::Union{Function, Nothing}
+    stream_function::StreamFn
+    get_api_key::Union{Function, Nothing}
+    on_payload::Union{Function, Nothing}
+    on_response::Union{Function, Nothing}
+    before_tool_call::Union{Function, Nothing}
+    after_tool_call::Union{Function, Nothing}
+    prepare_next_turn::Union{Function, Nothing}
+    prepare_next_turn_with_context::Union{Function, Nothing}
+    active_run::Union{ActiveRun, Nothing}
+    session_id::Union{String, Nothing}
+    thinking_budgets::Union{Dict{String, Int64}, Nothing}
+    transport::String
+    max_retry_delay_ms::Union{Int64, Nothing}
+    tool_execution::ToolExecutionMode
+
+    function Agent(options::Dict{Symbol, Any}=Dict{Symbol, Any}())
+        runtime_options = merge(
+            Dict{Symbol, Any}(
+                :stream_fn => getDefaultStreamFn(),
+                :convertToLlm => defaultConvertToLlm,
+                :steeringMode => QUEUE_ONE_AT_A_TIME,
+                :followUpMode => QUEUE_ONE_AT_A_TIME,
+                :toolExecution => EXECUTION_PARALLEL,
+                :transport => "auto",
+            ),
+            options,
+        )
+
+        state = AgentState(
+            get(runtime_options, :systemPrompt, ""),
+            get(runtime_options, :model, Model("", "", "unknown", "unknown", "", false, String[], ModelCost(0.0, 0.0, 0.0, 0.0), 0, 0)),
+            get(runtime_options, :thinkingLevel, THINKING_OFF),
+            get(runtime_options, :tools, AgentTool[]),
+            get(runtime_options, :messages, AgentMessage[]),
+        )
+
+        new(
+            state,
+            Set{Tuple{Function, Ref{Bool}}}(),
+            PendingMessageQueue(QUEUE_ONE_AT_A_TIME),
+            PendingMessageQueue(QUEUE_ONE_AT_A_TIME),
+            get(runtime_options, :convertToLlm, defaultConvertToLlm),
+            get(runtime_options, :transformContext, nothing),
+            get(runtime_options, :stream_fn, getDefaultStreamFn()),
+            get(runtime_options, :getApiKey, nothing),
+            get(runtime_options, :onPayload, nothing),
+            get(runtime_options, :onResponse, nothing),
+            get(runtime_options, :beforeToolCall, nothing),
+            get(runtime_options, :afterToolCall, nothing),
+            get(runtime_options, :prepareNextTurn, nothing),
+            get(runtime_options, :prepareNextTurnWithContext, nothing),
+            nothing,
+            get(runtime_options, :sessionId, nothing),
+            get(runtime_options, :thinkingBudgets, nothing),
+            get(runtime_options, :transport, "auto"),
+            get(runtime_options, :maxRetryDelayMs, nothing),
+            get(runtime_options, :toolExecution, EXECUTION_PARALLEL),
+        )
+    end
+end
+
+# ============================================================================
+# Agent methods
+# ============================================================================
+
+"""
+    subscribe(agent, listener)
+
+Subscribe to agent lifecycle events.
+
+# Arguments
+- `agent`: The agent instance
+- `listener`: A function that takes (event::AgentEvent, signal::AbortSignal)
+
+# Returns
+- A function that unsubscribes the listener
+"""
+function subscribe(agent::Agent, listener::Function)::Function
+    push!(agent.listeners, (listener, Ref{Bool}(true)))
+    return () -> begin
+        filter!(x -> x[1] != listener, agent.listeners)
+    end
+end
+
+"""
+    get_state(agent)
+
+Get the current agent state.
+"""
+function get_state(agent::Agent)::AgentState
+    return agent._state
+end
+
+"""
+    steer(agent, message)
+
+Queue a message to be injected after the current assistant turn finishes.
+"""
+function steer(agent::Agent, message::AgentMessage)
+    enqueue!(agent.steering_queue, message)
+end
+
+"""
+    followUp(agent, message)
+
+Queue a message to run only after the agent would otherwise stop.
+"""
+function followUp(agent::Agent, message::AgentMessage)
+    enqueue!(agent.follow_up_queue, message)
+end
+
+"""
+    clearSteeringQueue(agent)
+
+Remove all queued steering messages.
+"""
+function clearSteeringQueue(agent::Agent)
+    clear!(agent.steering_queue)
+end
+
+"""
+    clearFollowUpQueue(agent)
+
+Remove all queued follow-up messages.
+"""
+function clearFollowUpQueue(agent::Agent)
+    clear!(agent.follow_up_queue)
+end
+
+"""
+    clearAllQueues(agent)
+
+Remove all queued steering and follow-up messages.
+"""
+function clearAllQueues(agent::Agent)
+    clearSteeringQueue(agent)
+    clearFollowUpQueue(agent)
+end
+
+"""
+    hasQueuedMessages(agent)
+
+Returns true when either queue still contains pending messages.
+"""
+function hasQueuedMessages(agent::Agent)::Bool
+    return hasItems(agent.steering_queue) || hasItems(agent.follow_up_queue)
+end
+
+"""
+    abort(agent)
+
+Abort the current run, if one is active.
+"""
+function abort(agent::Agent)
+    if !isnothing(agent.active_run)
+        # TODO: Implement abort signal
+    end
+end
+
+"""
+    waitForIdle(agent)
+
+Resolve when the current run and all awaited event listeners have finished.
+"""
+function waitForIdle(agent::Agent)::Promise
+    if isnothing(agent.active_run)
+        return Promise()
+    end
+    return agent.active_run.promise
+end
+
+"""
+    reset(agent)
+
+Clear transcript state, runtime state, and queued messages.
+"""
+function reset!(agent::Agent)
+    agent._state.messages = AgentMessage[]
+    agent._state.is_streaming = false
+    agent._state.streaming_message = nothing
+    agent._state.pending_tool_calls = Set{String}()
+    agent._state.error_message = nothing
+    clearFollowUpQueue(agent)
+    clearSteeringQueue(agent)
+end
+
+"""
+    prompt(agent, input[, images])
+
+Start a new prompt from text, a single message, or a batch of messages.
+"""
+function prompt(agent::Agent, input::Union{String, AgentMessage, Vector{AgentMessage}}, images::Vector{ImageContent}=ImageContent[])::Nothing
+    if !isnothing(agent.active_run)
+        throw(ErrorException(
+            "Agent is already processing a prompt. Use steer() or followUp() to queue messages, or wait for completion."
+        ))
+    end
+    messages = normalizePromptInput(agent, input, images)
+    runPromptMessages(agent, messages)
+end
+
+function normalizePromptInput(agent::Agent, input::Vector{AgentMessage}, images::Vector{ImageContent})::Vector{AgentMessage}
+    return input
+end
+
+function normalizePromptInput(agent::Agent, input::AgentMessage, images::Vector{ImageContent})::Vector{AgentMessage}
+    return [input]
+end
+
+function normalizePromptInput(agent::Agent, input::String, images::Vector{ImageContent})::Vector{AgentMessage}
+    content::Vector{MessageContent} = [TextContent(input)]
+    if !isempty(images)
+        append!(content, images)
+    end
+    return [UserMessage("user", content, Int64(Dates.now(Dates.UTC).datetime))]
+end
+
+function runPromptMessages(agent::Agent, messages::Vector{AgentMessage})::Nothing
+    # TODO: Implement run with lifecycle
+    return nothing
+end
+
+"""
+    continue(agent)
+
+Continue from the current transcript. The last message must be a user or tool-result message.
+"""
+function continue!(agent::Agent)::Nothing
+    if !isnothing(agent.active_run)
+        throw(ErrorException("Agent is already processing. Wait for completion before continuing."))
+    end
+
+    last_message = agent._state.messages[end]
+    if isnothing(last_message)
+        throw(ErrorException("No messages to continue from"))
+    end
+
+    if last_message.role == "assistant"
+        queued_steering = drain(agent.steering_queue)
+        if !isempty(queued_steering)
+            runPromptMessages(agent, queued_steering)
+            return nothing
+        end
+
+        queued_follow_ups = drain(agent.follow_up_queue)
+        if !isempty(queued_follow_ups)
+            runPromptMessages(agent, queued_follow_ups)
+            return nothing
+        end
+
+        throw(ErrorException("Cannot continue from message role: assistant"))
+    end
+
+    # TODO: Implement run continuation
+    return nothing
+end
+
+"""
+    createContextSnapshot(agent)
+
+Create a snapshot of the current context for use in the agent loop.
+"""
+function createContextSnapshot(agent::Agent)::AgentContext
+    return AgentContext(
+        agent._state.system_prompt,
+        copy(agent._state.messages),
+        copy(agent._state.tools),
+    )
+end
+
+"""
+    createLoopConfig(agent, options)
+
+Create the loop configuration for the agent.
+"""
+function createLoopConfig(agent::Agent, options::Dict{String, Any}=Dict{String, Any}())::AgentLoopConfig
+    skip_initial_steering_poll = get(options, "skipInitialSteeringPoll", false)
+    return AgentLoopConfig(
+        agent._state.model,
+        agent._state.thinking_level == THINKING_OFF ? nothing : agent._state.thinking_level,
+        agent.session_id,
+        agent.on_payload,
+        agent.on_response,
+        agent.transport,
+        agent.thinking_budgets,
+        agent.max_retry_delay_ms,
+        agent.tool_execution,
+        agent.before_tool_call,
+        agent.after_tool_call,
+        isnothing(agent.prepare_next_turn_with_context) && isnothing(agent.prepare_next_turn) ? nothing : function(context)
+            if !isnothing(agent.prepare_next_turn_with_context)
+                return agent.prepare_next_turn_with_context(context, getSignal(agent))
+            end
+            return isnothing(agent.prepare_next_turn) ? nothing : agent.prepare_next_turn(getSignal(agent))
+        end,
+        agent.convert_to_llm,
+        agent.transform_context,
+        agent.get_api_key,
+        function()
+            if skip_initial_steering_poll
+                skip_initial_steering_poll = false
+                return AgentMessage[]
+            end
+            return drain(agent.steering_queue)
+        end,
+        function()
+            return drain(agent.follow_up_queue)
+        end,
+    )
+end
+
+"""
+    getSignal(agent)
+
+Get the active abort signal for the current run, if any.
+"""
+function getSignal(agent::Agent)::Union{Nothing, Base.Atomic{Bool}}
+    if isnothing(agent.active_run)
+        return nothing
+    end
+    return agent.active_run.abort_controller
+end
+
+end

src/agent_loop.jl

@@ -0,0 +1,861 @@
+"""
+    agent_loop.jl - Low-level agent loop implementation
+
+This module implements the core agentLoop functionality that works with AgentMessage
+throughout, transforming to Message[] only at the LLM call boundary.
+"""
+
+module AgentLoop
+
+using ..Types: *
+using ..StreamFn: *
+
+# ============================================================================
+# Event sink type
+# ============================================================================
+
+const AgentEventSink = Function
+
+# ============================================================================
+# Main agent loop function
+# ============================================================================
+
+function agentLoop(
+    prompts::Vector{AgentMessage},
+    context::AgentContext,
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+    stream_fn::StreamFn,
+)::EventStream
+    stream = createAgentStream()
+
+    Threads.@spawn begin
+        messages = runAgentLoop(
+            prompts,
+            context,
+            config,
+            (event) -> push!(stream, event),
+            signal,
+            stream_fn,
+        )
+        end(stream, messages)
+    end
+
+    return stream
+end
+
+# ============================================================================
+# Continue agent loop function
+# ============================================================================
+
+function agentLoopContinue(
+    context::AgentContext,
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+    stream_fn::StreamFn,
+)::EventStream
+    if isempty(context.messages)
+        throw(ErrorException("Cannot continue: no messages in context"))
+    end
+
+    if context.messages[end].role == "assistant"
+        throw(ErrorException("Cannot continue from message role: assistant"))
+    end
+
+    stream = createAgentStream()
+
+    Threads.@spawn begin
+        messages = runAgentLoopContinue(
+            context,
+            config,
+            (event) -> push!(stream, event),
+            signal,
+            stream_fn,
+        )
+        end(stream, messages)
+    end
+
+    return stream
+end
+
+# ============================================================================
+# Run agent loop function
+# ============================================================================
+
+function runAgentLoop(
+    prompts::Vector{AgentMessage},
+    context::AgentContext,
+    config::AgentLoopConfig,
+    emit::AgentEventSink,
+    signal::Union{Nothing, AbortSignal},
+    stream_fn::StreamFn,
+)::Vector{AgentMessage}
+    new_messages::Vector{AgentMessage} = copy(prompts)
+    current_context::AgentContext = AgentContext(
+        context.system_prompt,
+        vcat(context.messages, copy(prompts)),
+        context.tools,
+    )
+
+    emit(AgentStartEvent())
+    emit(TurnStartEvent())
+    for prompt in prompts
+        emit(MessageStartEvent(prompt))
+        emit(MessageEndEvent(prompt))
+    end
+
+    runLoop(
+        current_context,
+        new_messages,
+        config,
+        signal,
+        emit,
+        stream_fn,
+    )
+    return new_messages
+end
+
+# ============================================================================
+# Run agent loop continue function
+# ============================================================================
+
+function runAgentLoopContinue(
+    context::AgentContext,
+    config::AgentLoopConfig,
+    emit::AgentEventSink,
+    signal::Union{Nothing, AbortSignal},
+    stream_fn::StreamFn,
+)::Vector{AgentMessage}
+    if isempty(context.messages)
+        throw(ErrorException("Cannot continue: no messages in context"))
+    end
+
+    if context.messages[end].role == "assistant"
+        throw(ErrorException("Cannot continue from message role: assistant"))
+    end
+
+    new_messages::Vector{AgentMessage} = []
+    current_context::AgentContext = context
+
+    emit(AgentStartEvent())
+    emit(TurnStartEvent())
+
+    runLoop(
+        current_context,
+        new_messages,
+        config,
+        signal,
+        emit,
+        stream_fn,
+    )
+    return new_messages
+end
+
+# ============================================================================
+# Create agent stream function
+# ============================================================================
+
+function createAgentStream()::EventStream
+    return EventStream(
+        (event::AgentEvent) -> event isa AgentEndEvent,
+        (event::AgentEvent) -> event isa AgentEndEvent ? event.messages : AgentMessage[],
+    )
+end
+
+# ============================================================================
+# Main loop logic shared by agentLoop and agentLoopContinue
+# ============================================================================
+
+function runLoop(
+    initial_context::AgentContext,
+    new_messages::Vector{AgentMessage},
+    initial_config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+    emit::AgentEventSink,
+    stream_function::StreamFn,
+)::Nothing
+    current_context::AgentContext = initial_context
+    config::AgentLoopConfig = initial_config
+    first_turn::Bool = true
+    pending_messages::Vector{AgentMessage} = getSteeringMessages(config) do
+        get_steering_messages(config)
+    end
+
+    while true
+        has_more_tool_calls::Bool = true
+
+        while has_more_tool_calls || !isempty(pending_messages)
+            if !first_turn
+                emit(TurnStartEvent())
+            else
+                first_turn = false
+            end
+
+            if !isempty(pending_messages)
+                for message in pending_messages
+                    emit(MessageStartEvent(message))
+                    emit(MessageEndEvent(message))
+                    push!(current_context.messages, message)
+                    push!(new_messages, message)
+                end
+                pending_messages = AgentMessage[]
+            end
+
+            message = streamAssistantResponse(
+                current_context,
+                config,
+                signal,
+                emit,
+                stream_function,
+            )
+            push!(new_messages, message)
+
+            if message.stop_reason in ("error", "aborted")
+                emit(TurnEndEvent(message, ToolResultMessage[]))
+                emit(AgentEndEvent(new_messages))
+                return
+            end
+
+            tool_calls = filter(
+                (c) -> c isa ToolCall,
+                message.content,
+            )
+
+            tool_results::Vector{ToolResultMessage} = []
+            has_more_tool_calls = false
+            if !isempty(tool_calls)
+                executed_tool_batch =
+                    message.stop_reason == "length"
+                        ? failToolCallsFromTruncatedMessage(tool_calls, emit)
+                        : executeToolCalls(
+                            current_context,
+                            message,
+                            config,
+                            signal,
+                            emit,
+                        )
+                append!(tool_results, executed_tool_batch.messages)
+                has_more_tool_calls = !executed_tool_batch.terminate
+
+                for result in tool_results
+                    push!(current_context.messages, result)
+                    push!(new_messages, result)
+                end
+            end
+
+            emit(TurnEndEvent(message, tool_results))
+
+            next_turn_context = PrepareNextTurnContext(
+                message,
+                tool_results,
+                current_context,
+                new_messages,
+            )
+            next_turn_snapshot = prepare_next_turn(config, next_turn_context)
+
+            if !isnothing(next_turn_snapshot)
+                current_context = next_turn_snapshot.context
+                config = AgentLoopConfig(
+                    model = next_turn_snapshot.model,
+                    reasoning = next_turn_snapshot.thinking_level,
+                    convert_to_llm = config.convert_to_llm,
+                    transform_context = config.transform_context,
+                    get_api_key = config.get_api_key,
+                    should_stop_after_turn = config.should_stop_after_turn,
+                    prepare_next_turn = config.prepare_next_turn,
+                    get_steering_messages = config.get_steering_messages,
+                    get_follow_up_messages = config.get_follow_up_messages,
+                    tool_execution = config.tool_execution,
+                    before_tool_call = config.before_tool_call,
+                    after_tool_call = config.after_tool_call,
+                    max_tokens = config.max_tokens,
+                    temperature = config.temperature,
+                    reasoning = config.reasoning,
+                    cache_retention = config.cache_retention,
+                    session_id = config.session_id,
+                    headers = config.headers,
+                    metadata = config.metadata,
+                    transport = config.transport,
+                    signal = signal,
+                    api_key = config.api_key,
+                    on_payload = config.on_payload,
+                    on_response = config.on_response,
+                    max_retry_delay_ms = config.max_retry_delay_ms,
+                )
+            end
+
+            if should_stop_after_turn(config, next_turn_context)
+                emit(AgentEndEvent(new_messages))
+                return
+            end
+
+            pending_messages = getSteeringMessages(config) do
+                get_steering_messages(config)
+            end
+        end
+
+        follow_up_messages = getFollowUpMessages(config) do
+            get_follow_up_messages(config)
+        end
+
+        if !isempty(follow_up_messages)
+            pending_messages = follow_up_messages
+            continue
+        end
+
+        break
+    end
+
+    emit(AgentEndEvent(new_messages))
+end
+
+# ============================================================================
+# Helper types
+# ============================================================================
+
+struct PrepareNextTurnContext
+    message::AssistantMessage
+    tool_results::Vector{ToolResultMessage}
+    context::AgentContext
+    new_messages::Vector{AgentMessage}
+end
+
+struct AgentLoopTurnUpdate
+    context::Union{AgentContext, Nothing}
+    model::Union{Model, Nothing}
+    thinking_level::Union{ThinkingLevel, Nothing}
+end
+
+# ============================================================================
+# Helper functions for getting messages from queues
+# ============================================================================
+
+macro getSteeringMessages(config)
+    :(get_steering_messages($(esc(config))))
+end
+
+macro getFollowUpMessages(config)
+    :(get_follow_up_messages($(esc(config))))
+end
+
+function get_steering_messages(config::AgentLoopConfig)::Vector{AgentMessage}
+    return isnothing(config.get_steering_messages) ? AgentMessage[] : config.get_steering_messages()
+end
+
+function get_follow_up_messages(config::AgentLoopConfig)::Vector{AgentMessage}
+    return isnothing(config.get_follow_up_messages) ? AgentMessage[] : config.get_follow_up_messages()
+end
+
+function prepare_next_turn(config::AgentLoopConfig, context::PrepareNextTurnContext)::Union{AgentLoopTurnUpdate, Nothing}
+    return isnothing(config.prepare_next_turn) ? nothing : config.prepare_next_turn(context)
+end
+
+function should_stop_after_turn(config::AgentLoopConfig, context::PrepareNextTurnContext)::Bool
+    return isnothing(config.should_stop_after_turn) ? false : config.should_stop_after_turn(context)
+end
+
+# ============================================================================
+# Stream assistant response function
+# ============================================================================
+
+function streamAssistantResponse(
+    context::AgentContext,
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+    emit::AgentEventSink,
+    stream_function::StreamFn,
+)::AssistantMessage
+    messages::Vector{AgentMessage} = context.messages
+
+    if !isnothing(config.transform_context)
+        messages = config.transform_context(messages, signal)
+    end
+
+    llm_messages::Vector{Message} = config.convert_to_llm(messages)
+
+    llm_context::Context = Context(
+        context.system_prompt,
+        llm_messages,
+        context.tools,
+    )
+
+    resolved_api_key::Union{String, Nothing} =
+        !isnothing(config.get_api_key)
+            ? config.get_api_key(config.model.provider)
+            : nothing
+
+    response = stream_function(
+        config.model,
+        llm_context,
+        merge(
+            config,
+            Dict(:apiKey => resolved_api_key, :signal => signal),
+        ),
+    )
+
+    partial_message::Union{AssistantMessage, Nothing} = nothing
+    added_partial::Bool = false
+
+    for event in response
+        if event.type == "start"
+            partial_message = event.partial
+            push!(context.messages, partial_message)
+            added_partial = true
+            emit(MessageStartEvent(copy(partial_message)))
+        elseif event.type in ("text_start", "text_delta", "text_end", "thinking_start", "thinking_delta", "thinking_end", "toolcall_start", "toolcall_delta", "toolcall_end")
+            if !isnothing(partial_message)
+                partial_message = event.partial
+                context.messages[end] = partial_message
+                emit(MessageUpdateEvent(copy(partial_message), event))
+            end
+        elseif event.type in ("done", "error")
+            final_message = response.result()
+            if added_partial
+                context.messages[end] = final_message
+            else
+                push!(context.messages, final_message)
+            end
+            if !added_partial
+                emit(MessageStartEvent(copy(final_message)))
+            end
+            emit(MessageEndEvent(final_message))
+            return final_message
+        end
+    end
+
+    final_message = response.result()
+    if added_partial
+        context.messages[end] = final_message
+    else
+        push!(context.messages, final_message)
+        emit(MessageStartEvent(copy(final_message)))
+    end
+    emit(MessageEndEvent(final_message))
+    return final_message
+end
+
+# ============================================================================
+# Fail tool calls from truncated message
+# ============================================================================
+
+struct ExecutedToolCallBatch
+    messages::Vector{ToolResultMessage}
+    terminate::Bool
+end
+
+function failToolCallsFromTruncatedMessage(
+    tool_calls::Vector{ToolCall},
+    emit::AgentEventSink,
+)::ExecutedToolCallBatch
+    messages::Vector{ToolResultMessage} = []
+
+    for tool_call in tool_calls
+        emit(ToolExecutionStartEvent(tool_call.id, tool_call.name, tool_call.arguments))
+
+        finalized = FinalizedToolCallOutcome(
+            tool_call,
+            createErrorToolResult(
+                "Tool call \"$(tool_call.name)\" was not executed: the response hit the output token limit, so its arguments may be truncated. Re-issue the tool call with complete arguments.",
+            ),
+            true,
+        )
+
+        emitToolExecutionEnd(finalized, emit)
+        tool_result_message = createToolResultMessage(finalized)
+        emitToolResultMessage(tool_result_message, emit)
+        push!(messages, tool_result_message)
+    end
+
+    return ExecutedToolCallBatch(messages, false)
+end
+
+# ============================================================================
+# Execute tool calls
+# ============================================================================
+
+function executeToolCalls(
+    current_context::AgentContext,
+    assistant_message::AssistantMessage,
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+    emit::AgentEventSink,
+)::ExecutedToolCallBatch
+    tool_calls = filter(
+        (c) -> c isa ToolCall,
+        assistant_message.content,
+    )
+
+    has_sequential_tool_call = any(
+        (tc) -> begin
+            tool = findfirst((t) -> t.name == tc.name, current_context.tools)
+            !isnothing(tool) && tool.execution_mode == EXECUTION_SEQUENTIAL
+        end,
+        tool_calls,
+    )
+
+    if config.tool_execution == EXECUTION_SEQUENTIAL || has_sequential_tool_call
+        return executeToolCallsSequential(
+            current_context,
+            assistant_message,
+            tool_calls,
+            config,
+            signal,
+            emit,
+        )
+    end
+    return executeToolCallsParallel(
+        current_context,
+        assistant_message,
+        tool_calls,
+        config,
+        signal,
+        emit,
+    )
+end
+
+# ============================================================================
+# Execute tool calls sequentially
+# ============================================================================
+
+function executeToolCallsSequential(
+    current_context::AgentContext,
+    assistant_message::AssistantMessage,
+    tool_calls::Vector{ToolCall},
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+    emit::AgentEventSink,
+)::ExecutedToolCallBatch
+    finalized_calls::Vector{FinalizedToolCallOutcome} = []
+    messages::Vector{ToolResultMessage} = []
+
+    for tool_call in tool_calls
+        emit(ToolExecutionStartEvent(tool_call.id, tool_call.name, tool_call.arguments))
+
+        preparation = prepareToolCall(current_context, assistant_message, tool_call, config, signal)
+
+        finalized = if preparation.kind == "immediate"
+            FinalizedToolCallOutcome(tool_call, preparation.result, preparation.is_error)
+        else
+            executed = executePreparedToolCall(preparation, signal, emit)
+            finalizeExecutedToolCall(
+                current_context,
+                assistant_message,
+                preparation,
+                executed,
+                config,
+                signal,
+            )
+        end
+
+        emitToolExecutionEnd(finalized, emit)
+        tool_result_message = createToolResultMessage(finalized)
+        emitToolResultMessage(tool_result_message, emit)
+        push!(finalized_calls, finalized)
+        push!(messages, tool_result_message)
+
+        if !isnothing(signal) && signal.aborted
+            break
+        end
+    end
+
+    return ExecutedToolCallBatch(messages, shouldTerminateToolBatch(finalized_calls))
+end
+
+# ============================================================================
+# Execute tool calls in parallel
+# ============================================================================
+
+function executeToolCallsParallel(
+    current_context::AgentContext,
+    assistant_message::AssistantMessage,
+    tool_calls::Vector{ToolCall},
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+    emit::AgentEventSink,
+)::ExecutedToolCallBatch
+    finalized_calls::Vector{Union{FinalizedToolCallOutcome, Function}} = []
+
+    for tool_call in tool_calls
+        emit(ToolExecutionStartEvent(tool_call.id, tool_call.name, tool_call.arguments))
+
+        preparation = prepareToolCall(current_context, assistant_message, tool_call, config, signal)
+
+        if preparation.kind == "immediate"
+            finalized = FinalizedToolCallOutcome(
+                tool_call,
+                preparation.result,
+                preparation.is_error,
+            )
+            emitToolExecutionEnd(finalized, emit)
+            push!(finalized_calls, finalized)
+            if !isnothing(signal) && signal.aborted
+                break
+            end
+            continue
+        end
+
+        push!(finalized_calls, () -> begin
+            executed = executePreparedToolCall(preparation, signal, emit)
+            finalized = finalizeExecutedToolCall(
+                current_context,
+                assistant_message,
+                preparation,
+                executed,
+                config,
+                signal,
+            )
+            emitToolExecutionEnd(finalized, emit)
+            return finalized
+        end)
+
+        if !isnothing(signal) && signal.aborted
+            break
+        end
+    end
+
+    ordered_finalized_calls = map(
+        (entry) -> if entry isa Function
+            entry()
+        else
+            entry
+        end,
+        finalized_calls,
+    )
+
+    messages::Vector{ToolResultMessage} = []
+    for finalized in ordered_finalized_calls
+        tool_result_message = createToolResultMessage(finalized)
+        emitToolResultMessage(tool_result_message, emit)
+        push!(messages, tool_result_message)
+    end
+
+    return ExecutedToolCallBatch(messages, shouldTerminateToolBatch(ordered_finalized_calls))
+end
+
+# ============================================================================
+# Prepared tool call types
+# ============================================================================
+
+struct PreparedToolCall
+    kind::String
+    tool_call::ToolCall
+    tool::AgentTool
+    args::Any
+end
+
+struct ImmediateToolCallOutcome
+    kind::String
+    result::AgentToolResultMutable
+    is_error::Bool
+end
+
+struct ExecutedToolCallOutcome
+    result::AgentToolResultMutable
+    is_error::Bool
+end
+
+struct FinalizedToolCallOutcome
+    tool_call::ToolCall
+    result::AgentToolResultMutable
+    is_error::Bool
+end
+
+# ============================================================================
+# Helper functions
+# ============================================================================
+
+function shouldTerminateToolBatch(finalized_calls::Vector{FinalizedToolCallOutcome})::Bool
+    return !isempty(finalized_calls) && all(
+        (finalized) -> finalized.result.terminate === true,
+        finalized_calls,
+    )
+end
+
+function prepareToolCallArguments(tool::AgentTool, tool_call::ToolCall)::ToolCall
+    if isnothing(tool.prepare_arguments)
+        return tool_call
+    end
+    prepared_arguments = tool.prepare_arguments(tool_call.arguments)
+    if prepared_arguments === tool_call.arguments
+        return tool_call
+    end
+    return ToolCall(
+        tool_call.type,
+        tool_call.id,
+        tool_call.name,
+        prepared_arguments,
+        tool_call.partial_json,
+    )
+end
+
+function prepareToolCall(
+    current_context::AgentContext,
+    assistant_message::AssistantMessage,
+    tool_call::ToolCall,
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+)::Union{PreparedToolCall, ImmediateToolCallOutcome}
+    tool = findfirst((t) -> t.name == tool_call.name, current_context.tools)
+    if isnothing(tool)
+        return ImmediateToolCallOutcome("immediate", createErrorToolResult("Tool $(tool_call.name) not found"), true)
+    end
+
+    try
+        prepared_tool_call = prepareToolCallArguments(tool, tool_call)
+        validated_args = validateToolArguments(tool, prepared_tool_call)
+
+        if !isnothing(config.before_tool_call)
+            before_result = config.before_tool_call(
+                BeforeToolCallContext(assistant_message, tool_call, validated_args, current_context),
+                signal,
+            )
+            if !isnothing(signal) && signal.aborted
+                return ImmediateToolCallOutcome("immediate", createErrorToolResult("Operation aborted"), true)
+            end
+            if !isnothing(before_result) && before_result.block
+                reason = isnothing(before_result.reason) ? "Tool execution was blocked" : before_result.reason
+                return ImmediateToolCallOutcome("immediate", createErrorToolResult(reason), true)
+            end
+        end
+
+        if !isnothing(signal) && signal.aborted
+            return ImmediateToolCallOutcome("immediate", createErrorToolResult("Operation aborted"), true)
+        end
+
+        return PreparedToolCall("prepared", tool_call, tool, validated_args)
+    catch error
+        return ImmediateToolCallOutcome("immediate", createErrorToolResult(string(error)), true)
+    end
+end
+
+function executePreparedToolCall(
+    prepared::PreparedToolCall,
+    signal::Union{Nothing, AbortSignal},
+    emit::AgentEventSink,
+)::ExecutedToolCallOutcome
+    update_events::Vector{Future} = []
+    accepting_updates::Bool = true
+
+    try
+        result = prepared.tool.execute(
+            prepared.tool_call.id,
+            prepared.args,
+            signal,
+            (partial_result) -> begin
+                if !accepting_updates
+                    return
+                end
+                push!(
+                    update_events,
+                    Threads.@spawn begin
+                        emit(
+                            ToolExecutionUpdateEvent(
+                                prepared.tool_call.id,
+                                prepared.tool_call.name,
+                                prepared.tool_call.arguments,
+                                partial_result,
+                            ),
+                        )
+                    end,
+                )
+            end,
+        )
+        accepting_updates = false
+        wait.(update_events)
+        return ExecutedToolCallOutcome(result, false)
+    catch error
+        accepting_updates = false
+        wait.(update_events)
+        return ExecutedToolCallOutcome(createErrorToolResult(string(error)), true)
+    finally
+        accepting_updates = false
+    end
+end
+
+function finalizeExecutedToolCall(
+    current_context::AgentContext,
+    assistant_message::AssistantMessage,
+    prepared::PreparedToolCall,
+    executed::ExecutedToolCallOutcome,
+    config::AgentLoopConfig,
+    signal::Union{Nothing, AbortSignal},
+)::FinalizedToolCallOutcome
+    result = executed.result
+    is_error = executed.is_error
+
+    if !isnothing(config.after_tool_call)
+        try
+            after_result = config.after_tool_call(
+                AfterToolCallContext(
+                    assistant_message,
+                    prepared.tool_call,
+                    prepared.args,
+                    result,
+                    is_error,
+                    current_context,
+                ),
+                signal,
+            )
+            if !isnothing(after_result)
+                result = AgentToolResultMutable(
+                    isnothing(after_result.content) ? result.content : after_result.content,
+                    isnothing(after_result.details) ? result.details : after_result.details,
+                    isnothing(after_result.usage) ? result.usage : after_result.usage,
+                    result.added_tool_names,
+                    isnothing(after_result.terminate) ? result.terminate : after_result.terminate,
+                )
+                is_error = isnothing(after_result.is_error) ? is_error : after_result.is_error
+            end
+        catch error
+            result = createErrorToolResult(string(error))
+            is_error = true
+        end
+    end
+
+    return FinalizedToolCallOutcome(prepared.tool_call, result, is_error)
+end
+
+function createErrorToolResult(message::String)::AgentToolResultMutable
+    return AgentToolResultMutable([TextContent(message)], Dict{String, Any}(), nothing, nothing, nothing)
+end
+
+function emitToolExecutionEnd(finalized::FinalizedToolCallOutcome, emit::AgentEventSink)::Nothing
+    emit(ToolExecutionEndEvent(
+        finalized.tool_call.id,
+        finalized.tool_call.name,
+        finalized.result,
+        finalized.is_error,
+    ))
+    return nothing
+end
+
+function createToolResultMessage(finalized::FinalizedToolCallOutcome)::ToolResultMessage
+    return ToolResultMessage(
+        "toolResult",
+        finalized.tool_call.id,
+        finalized.tool_call.name,
+        isnothing(finalized.result.content) ? MessageContent[] : finalized.result.content,
+        finalized.result.details,
+        finalized.result.usage,
+        finalized.result.added_tool_names,
+        finalized.is_error,
+        Dates.now(Dates.UTC).datetime,
+    )
+end
+
+function emitToolResultMessage(tool_result_message::ToolResultMessage, emit::AgentEventSink)::Nothing
+    emit(MessageStartEvent(tool_result_message))
+    emit(MessageEndEvent(tool_result_message))
+    return nothing
+end
+
+# ============================================================================
+# Validation helper
+# ============================================================================
+
+function validateToolArguments(tool::AgentTool, tool_call::ToolCall)::Any
+    # Simplified validation - in a full implementation, this would use TypeBox-like validation
+    return tool_call.arguments
+end
+
+end

src/api.jl

@@ -1,401 +0,0 @@
-module api
-
-export prompt
-
-using JSON, DataStructures, Dates, UUIDs, HTTP, Random, PrettyPrinting, Serialization,
-      DataFrames, Serde
-using GeneralUtils
-using ..type, ..utils
-
-# ---------------------------------------------- 100 --------------------------------------------- #
-
-
-
-"""
-Send a message to the agent's input channel.
-
-Blocks if the input channel buffer is full (capacity 16 by default).
-The agent processes messages from `inputChannel` in the background task.
-
-# Arguments
-- `agent::yiemAgent`: The agent instance to send a message to
-- `msg`: The message to send (any type accepted by the agent's processing pipeline)
-
-# Returns
-- The same `agent` instance for chaining
-
-# Notes
-- Use `take_response(agent)` to receive the agent's response after sending a message.
-- Use `follow_up(agent, msg)` to send messages while the agent is still processing.
-
-# Examples
-```jldoctest
-julia> run_agent(agent, "Hello!")
-yiemAgent(...)
-```
-"""
-function run_agent(agent::yiemAgent, msg)
-  put!(agent.inputChannel, msg)
-  return agent
-end
-
-"""
-Take a response from the agent's output channel.
-
-Blocks until the agent sends a response.
-
-# Arguments
-- `agent::yiemAgent`: The agent instance to receive a response from
-
-# Returns
-- An `assistantMessage` instance representing the agent's response
-
-# Notes
-- Use `run_agent(agent, msg)` to send a message before calling this function.
-
-# Examples
-```jldoctest
-julia> response = take_response(agent)
-assistantMessage(...)
-```
-"""
-function take_response(agent::yiemAgent)
-  return take!(agent.outputChannel)
-end
-
-"""
-Send a follow-up message while the agent is still processing.
-
-Follow-up messages are queued and processed after all `inputChannel` messages
-and before any tool call results are sent.
-
-# Arguments
-- `agent::yiemAgent`: The agent instance to send a follow-up message to
-- `msg`: The follow-up message to send
-
-# Returns
-- The same `agent` instance for chaining
-
-# Notes
-- Use `run_agent(agent, msg)` for the primary message and `follow_up(agent, msg)` for additional
-  messages while the agent is processing.
-- Follow-up messages are buffered in a separate channel (capacity 32 by default).
-
-# Examples
-```jldoctest
-julia> follow_up(agent, "Also consider red wines")
-yiemAgent(...)
-```
-"""
-function follow_up(agent::yiemAgent, msg)
-  put!(agent.followUpChannel, msg)
-  return agent
-end
-
-"""
-Gracefully stop the agent.
-
-Sends a `:shutdown` signal to the input channel, waits for the background task to finish,
-then closes all channels (`inputChannel`, `outputChannel`, `followUpChannel`).
-
-# Arguments
-- `agent::yiemAgent`: The agent instance to stop
-
-# Returns
-- `nothing`
-
-# Notes
-- After calling `stop_agent`, the agent is no longer usable. A new agent must be created
-  for further interaction.
-- If the background task throws a `TaskFailedException`, it is rethrown.
-
-# Examples
-```jldoctest
-julia> stop_agent(agent)
-```
-"""
-function stop_agent(agent::yiemAgent)
-  put!(agent.inputChannel, :shutdown)
-  try
-    fetch(agent._agent_loop)
-  catch e
-    if e isa TaskFailedException
-      rethrow(e)
-    end
-  end
-  close(agent.inputChannel)
-  close(agent.outputChannel)
-  close(agent.followUpChannel)
-  return nothing
-end
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-""" Recursively convert dictionary-like variable (e.g. JSON.Object) into an OrderedDict.
-
-The function walks any nested structure composed of `AbstractDict` (e.g., `JSON.Object`,
-`Dict`, `OrderedDict`) and `AbstractArray` and produces a new tree where
-every dictionary-like node is an `OrderedDict` and every array-like node is a `Vector{Any}`.
-Scalar values (numbers, strings, booleans, `nothing`, etc.) are returned unchanged.
-Does **not** mutate the input; it always allocates new containers.
-
-# Arguments
-- `x`
-  Any Julia value. If `x` is an `AbstractDict` it will be converted to an `OrderedDict`;
-  if it is an `AbstractArray` its elements will be processed recursively.
-
-# Keyword Arguments
-- `keytype::Type=Any`
-  The key type for the output OrderedDict. Use `String` for `OrderedDict{String,Any}`, 
-  `Symbol` for `OrderedDict{Symbol,Any}`, or `Any` to preserve original key types.
-- `sort_order::Union{Nothing, Vector}=nothing`
-  Vector of keys specifying the desired order. Keys are arranged in the specified order 
-  first, followed by any remaining keys.
-
-# Return
-- A newly allocated nested structure composed of `OrderedDict{keytype,Any}` and `Vector{Any}`
-  that mirrors the input shape but uses ordered Julia containers.
-
-# Notes
-- The function treats any `AbstractDict` as a mapping source, so it works with
-  `JSON.Object`, `Dict`, `OrderedDict`, etc.
-- Arrays are returned as `Vector{Any}` with their elements processed recursively.
-
-# Examples
-```jldoctest
-julia> using JSON, DataStructures
-julia> d = Dict(
-            "a" => 4,
-            "b" => 6,
-            "c" => Dict(
-                    "d"=>7, 
-                    :e=>Dict(
-                      "f"=>"hey", 
-                      "g"=>Dict(
-                        "world"=>[1, "2", 3, Dict(:dd=>4.7)]
-                        )
-                      )
-                    )
-          )
-
-julia> jsonstring = JSON.json(d)
-julia> A1 = JSON.parse(jsonstring)  # A1 type is JSON.Object
-julia> A2 = dictify(A1; keytype=String)
-OrderedDict{String,Any} with 3 entries:
-  "a" => 4
-  "b" => 6
-  "c" => OrderedDict("d"=>7, "e"=>Dict("f"=>"hey", "g"=>Dict("world"=>[1, "2", 3, 4.7])))
-
-julia> A3 = dictify(A1; keytype=Symbol)
-OrderedDict{Symbol,Any} with 3 entries:
-  :a => 4
-  :b => 6
-  :c => OrderedDict(:d=>7, :e=>Dict("f"=>"hey", "g"=>Dict("world"=>[1, "2", 3, 4.7])))
-
-julia> B1 = dictify(d; keytype=String)
-OrderedDict{String, Any} with 3 entries:
-```
-
-**With sort_order:**
-```jldoctest
-julia> d = Dict("a"=>1, "b"=>2, "c"=>3)
-julia> dictify(d; sort_order=["c", "a"])
-OrderedDict{String,Int} with 3 entries:
-  "c" => 3
-  "a" => 1
-  "b" => 2
-```
-"""
-function dictify(x::T; keytype::Type=Any, sort_order::Union{Nothing, Vector}=nothing
-  )::OrderedDict where {T<:AbstractDict}
-
-  # this function is example
-end
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-end # module interface
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-

src/messages.jl

@@ -0,0 +1,183 @@
+"""
+    messages.jl - Custom message types and LLM conversion
+
+This module provides custom message types and the convertToLlm function.
+"""
+
+module Messages
+
+using ..Types: *
+
+const COMPACTION_SUMMARY_PREFIX = """The conversation history before this point was compacted into the following summary:
+
+<summary>
+"""
+
+const COMPACTION_SUMMARY_SUFFIX = """
+</summary>"""
+
+const BRANCH_SUMMARY_PREFIX = """The following is a summary of a branch that this conversation came back from:
+
+<summary>
+"""
+
+const BRANCH_SUMMARY_SUFFIX = """</summary>"""
+
+# ============================================================================
+# Custom message types
+# ============================================================================
+
+mutable struct BashExecutionMessage
+    role::String
+    command::String
+    output::String
+    exit_code::Union{Int64, Nothing}
+    cancelled::Bool
+    truncated::Bool
+    full_output_path::Union{String, Nothing}
+    timestamp::Timestamp
+    exclude_from_context::Bool
+end
+
+mutable struct CustomMessage{T}
+    role::String
+    custom_type::String
+    content::Union{String, Vector{MessageContent}}
+    display::Bool
+    details::Union{T, Nothing}
+    timestamp::Timestamp
+end
+
+mutable struct BranchSummaryMessage
+    role::String
+    summary::String
+    from_id::String
+    timestamp::Timestamp
+end
+
+mutable struct CompactionSummaryMessage
+    role::String
+    summary::String
+    tokens_before::Int64
+    timestamp::Timestamp
+end
+
+# ============================================================================
+# Bash execution to text conversion
+# ============================================================================
+
+function bashExecutionToText(msg::BashExecutionMessage)::String
+    text = "Ran `$(msg.command)`\n"
+    if !isempty(msg.output)
+        text *= "```\n$(msg.output)\n```"
+    else
+        text *= "(no output)"
+    end
+    if msg.cancelled
+        text *= "\n\n(command cancelled)"
+    elseif !isnothing(msg.exit_code) && msg.exit_code != 0
+        text *= "\n\nCommand exited with code $(msg.exit_code)"
+    end
+    if msg.truncated && !isnothing(msg.full_output_path)
+        text *= "\n\n[Output truncated. Full output: $(msg.full_output_path)]"
+    end
+    return text
+end
+
+# ============================================================================
+# Message creation functions
+# ============================================================================
+
+function createBranchSummaryMessage(summary::String, from_id::String, timestamp::String)::BranchSummaryMessage
+    return BranchSummaryMessage(
+        "branchSummary",
+        summary,
+        from_id,
+        Int64(Dates.now(Dates.UTC).datetime),
+    )
+end
+
+function createCompactionSummaryMessage(summary::String, tokens_before::Int64, timestamp::String)::CompactionSummaryMessage
+    return CompactionSummaryMessage(
+        "compactionSummary",
+        summary,
+        tokens_before,
+        Int64(Dates.now(Dates.UTC).datetime),
+    )
+end
+
+function createCustomMessage(custom_type::String, content::Union{String, Vector{MessageContent}}, display::Bool, details::Union{Any, Nothing}, timestamp::String)::CustomMessage
+    return CustomMessage(
+        "custom",
+        custom_type,
+        content,
+        display,
+        details,
+        Int64(Dates.now(Dates.UTC).datetime),
+    )
+end
+
+# ============================================================================
+# Convert to LLM messages
+# ============================================================================
+
+function convertToLlm(messages::Vector{AgentMessage})::Vector{Message}
+    result::Vector{Message} = Message[]
+    
+    for m in messages
+        converted = convertToLlmMessage(m)
+        if !isnothing(converted)
+            push!(result, converted)
+        end
+    end
+    
+    return result
+end
+
+function convertToLlmMessage(m::BashExecutionMessage)::Union{UserMessage, Nothing}
+    if m.exclude_from_context
+        return nothing
+    end
+    return UserMessage(
+        "user",
+        [TextContent(bashExecutionToText(m))],
+        m.timestamp,
+    )
+end
+
+function convertToLlmMessage(m::CustomMessage)::Union{UserMessage, Nothing}
+    content = if m.content isa String
+        [TextContent(m.content)]
+    else
+        m.content
+    end
+    return UserMessage("user", content, m.timestamp)
+end
+
+function convertToLlmMessage(m::BranchSummaryMessage)::UserMessage
+    text = BRANCH_SUMMARY_PREFIX * m.summary * BRANCH_SUMMARY_SUFFIX
+    return UserMessage("user", [TextContent(text)], m.timestamp)
+end
+
+function convertToLlmMessage(m::CompactionSummaryMessage)::UserMessage
+    text = COMPACTION_SUMMARY_PREFIX * m.summary * COMPACTION_SUMMARY_SUFFIX
+    return UserMessage("user", [TextContent(text)], m.timestamp)
+end
+
+function convertToLlmMessage(m::UserMessage)::UserMessage
+    return m
+end
+
+function convertToLlmMessage(m::AssistantMessage)::AssistantMessage
+    return m
+end
+
+function convertToLlmMessage(m::ToolResultMessage)::ToolResultMessage
+    return m
+end
+
+function convertToLlmMessage(m::AgentMessage)::Union{Message, Nothing}
+    return nothing
+end
+
+end

src/prompt_templates.jl

@@ -0,0 +1,335 @@
+"""
+    prompt_templates.jl - Prompt template loading and formatting
+
+This module provides utilities for loading prompt templates and formatting invocations.
+"""
+
+module PromptTemplates
+
+using ..Types: *
+using ..HarnessTypes: ExecutionEnv, toError, Result, ok, err
+
+# ============================================================================
+# Prompt template diagnostic types
+# ============================================================================
+
+const PromptTemplateDiagnosticCode = String
+const PROMPT_TEMPLATE_DIAGNOSTIC_FILE_INFO_FAILED = "file_info_failed"
+const PROMPT_TEMPLATE_DIAGNOSTIC_LIST_FAILED = "list_failed"
+const PROMPT_TEMPLATE_DIAGNOSTIC_READ_FAILED = "read_failed"
+const PROMPT_TEMPLATE_DIAGNOSTIC_PARSE_FAILED = "parse_failed"
+
+mutable struct PromptTemplateDiagnostic
+    type::String
+    code::PromptTemplateDiagnosticCode
+    message::String
+    path::String
+end
+
+# ============================================================================
+# Prompt template frontmatter
+# ============================================================================
+
+mutable struct PromptTemplateFrontmatter
+    description::Union{String, Nothing}
+    argument_hint::Union{String, Nothing}
+    extra::Dict{String, Any}
+end
+
+# ============================================================================
+# Load prompt templates from paths
+# ============================================================================
+
+function loadPromptTemplates(
+    env::ExecutionEnv,
+    paths::Union{String, Vector{String}},
+)::Tuple{Vector{PromptTemplate}, Vector{PromptTemplateDiagnostic}}
+    prompt_templates::Vector{PromptTemplate} = PromptTemplate[]
+    diagnostics::Vector{PromptTemplateDiagnostic} = PromptTemplateDiagnostic[]
+
+    path_list = if paths isa String
+        [paths]
+    else
+        paths
+    end
+
+    for path in path_list
+        info_result = fileInfo(env, path, nothing)
+        if !info_result.ok
+            if info_result.error.code != "not_found"
+                push!(diagnostics, PromptTemplateDiagnostic(
+                    "warning",
+                    "file_info_failed",
+                    info_result.error.message,
+                    path,
+                ))
+            end
+            continue
+        end
+
+        info = info_result.value
+        kind = getFileKind(env, info, diagnostics)
+        
+        if kind == "directory"
+            result = loadTemplatesFromDir(env, info.path)
+            append!(prompt_templates, result.prompt_templates)
+            append!(diagnostics, result.diagnostics)
+        elseif kind == "file" && endswith(info.name, ".md")
+            result = loadTemplateFromFile(env, info.path)
+            if !isnothing(result.prompt_template)
+                push!(prompt_templates, result.prompt_template)
+            end
+            append!(diagnostics, result.diagnostics)
+        end
+    end
+
+    return prompt_templates, diagnostics
+end
+
+function getFileKind(env::ExecutionEnv, info::FileInfo, diagnostics::Vector{PromptTemplateDiagnostic})::Union{String, Nothing}
+    if info.kind == "file" || info.kind == "directory"
+        return info.kind
+    end
+
+    canonical_path = canonicalPath(env, info.path, nothing)
+    if !canonical_path.ok
+        if canonical_path.error.code != "not_found"
+            push!(diagnostics, PromptTemplateDiagnostic(
+                "warning",
+                "file_info_failed",
+                canonical_path.error.message,
+                info.path,
+            ))
+        end
+        return nothing
+    end
+
+    target = fileInfo(env, canonical_path.value, nothing)
+    if !target.ok
+        if target.error.code != "not_found"
+            push!(diagnostics, PromptTemplateDiagnostic(
+                "warning",
+                "file_info_failed",
+                target.error.message,
+                info.path,
+            ))
+        end
+        return nothing
+    end
+
+    if target.value.kind == "file" || target.value.kind == "directory"
+        return target.value.kind
+    end
+
+    return nothing
+end
+
+# ============================================================================
+# Load templates from directory
+# ============================================================================
+
+function loadTemplatesFromDir(
+    env::ExecutionEnv,
+    dir::String,
+)::Tuple{Vector{PromptTemplate}, Vector{PromptTemplateDiagnostic}}
+    prompt_templates::Vector{PromptTemplate} = PromptTemplate[]
+    diagnostics::Vector{PromptTemplateDiagnostic} = PromptTemplateDiagnostic[]
+
+    entries_result = listDir(env, dir, nothing)
+    if !entries_result.ok
+        push!(diagnostics, PromptTemplateDiagnostic(
+            "warning",
+            "list_failed",
+            entries_result.error.message,
+            dir,
+        ))
+        return prompt_templates, diagnostics
+    end
+
+    entries = entries_result.value
+
+    for entry in sort(entries, by=e -> e.name)
+        kind = getFileKind(env, entry, diagnostics)
+        if kind != "file" || !endswith(entry.name, ".md")
+            continue
+        end
+
+        result = loadTemplateFromFile(env, entry.path)
+        if !isnothing(result.prompt_template)
+            push!(prompt_templates, result.prompt_template)
+        end
+        append!(diagnostics, result.diagnostics)
+    end
+
+    return prompt_templates, diagnostics
+end
+
+# ============================================================================
+# Load template from file
+# ============================================================================
+
+function loadTemplateFromFile(
+    env::ExecutionEnv,
+    file_path::String,
+)::Tuple{Union{PromptTemplate, Nothing}, Vector{PromptTemplateDiagnostic}}
+    diagnostics::Vector{PromptTemplateDiagnostic} = PromptTemplateDiagnostic[]
+
+    raw_content = readTextFile(env, file_path, nothing)
+    if !raw_content.ok
+        push!(diagnostics, PromptTemplateDiagnostic(
+            "warning",
+            "read_failed",
+            raw_content.error.message,
+            file_path,
+        ))
+        return nothing, diagnostics
+    end
+
+    # TODO: Parse frontmatter
+    # parsed = parseFrontmatter<PromptTemplateFrontmatter>(rawContent.value);
+    # if !parsed.ok {
+    #     diagnostics.push({
+    #         type: "warning",
+    #         code: "parse_failed",
+    #         message: parsed.error.message,
+    #         path: filePath,
+    #     });
+    #     return { promptTemplate: null, diagnostics };
+    # }
+
+    # const { frontmatter, body } = parsed.value;
+    # const firstLine = body.split("\n").find((line) => line.trim());
+    # let description = typeof frontmatter.description === "string" ? frontmatter.description : "";
+    # if (!description && firstLine) {
+    #     description = firstLine.slice(0, 60);
+    #     if (firstLine.length > 60) description += "...";
+    # }
+
+    # return {
+    #     promptTemplate: {
+    #         name: basenameEnvPath(filePath).replace(/\.md$/i, ""),
+    #         description,
+    #         content: body,
+    #     },
+    #     diagnostics,
+    # };
+
+    return nothing, diagnostics
+end
+
+# ============================================================================
+# Parse command arguments
+# ============================================================================
+
+function parseCommandArgs(args_string::String)::Vector{String}
+    args::Vector{String} = String[]
+    current::String = ""
+    in_quote::Union{String, Nothing} = nothing
+
+    for i in 1:length(args_string)
+        char = args_string[i]
+        if !isnothing(in_quote)
+            if char == in_quote
+                in_quote = nothing
+            else
+                current *= char
+            end
+        elseif char == '"' || char == '\''
+            in_quote = char
+        elseif char == ' ' || char == '\t'
+            if !isempty(current)
+                push!(args, current)
+                current = ""
+            end
+        else
+            current *= char
+        end
+    end
+
+    if !isempty(current)
+        push!(args, current)
+    end
+
+    return args
+end
+
+# ============================================================================
+# Substitute arguments
+# ============================================================================
+
+function substituteArgs(content::String, args::Vector{String})::String
+    result = content
+
+    # Substitute $1, $2, etc.
+    result = replace(result, r"\$(\d+)" => s -> begin
+        idx = parse(Int, s[1])
+        if idx > 0 && idx <= length(args)
+            return args[idx]
+        end
+        return ""
+    end)
+
+    # Substitute ${@:N} and ${@:N:L}
+    result = replace(result, r"\$\{@:(\d+)(?::(\d+))?\}" => s -> begin
+        m = match(r"\$\{@:(\d+)(?::(\d+))?\}", s)
+        if !isnothing(m)
+            start = parse(Int, m.captures[1]) - 1
+            if start < 0
+                start = 0
+            end
+            if !isnothing(m.captures[2])
+                length = parse(Int, m.captures[2])
+                return join(args[start+1:start+length], " ")
+            end
+            return join(args[start+1:end], " ")
+        end
+        return s
+    end)
+
+    # Substitute $ARGUMENTS and $@
+    all_args = join(args, " ")
+    result = replace(result, "$ARGUMENTS" => all_args)
+    result = replace(result, "$@" => all_args)
+
+    return result
+end
+
+# ============================================================================
+# Format prompt template invocation
+# ============================================================================
+
+function formatPromptTemplateInvocation(template::PromptTemplate, args::Vector{String}=String[])::String
+    return substituteArgs(template.content, args)
+end
+
+# ============================================================================
+# Helper functions
+# ============================================================================
+
+function basenameEnvPath(path::String)::String
+    normalized = rtrim(path, '/')
+    slash_index = findlast('/', normalized)
+    if isnothing(slash_index)
+        return normalized
+    end
+    return normalized[slash_index+1:end]
+end
+
+function findlast(pattern::Char, s::String)::Union{Int64, Nothing}
+    for i in length(s):-1:1
+        if s[i] == pattern
+            return i
+        end
+    end
+    return nothing
+end
+
+function rtrim(s::String, chars::String)::String
+    idx = length(s)
+    while idx >= 1 && s[idx] in chars
+        idx -= 1
+    end
+    return s[1:idx]
+end
+
+end

src/session/jsonl_repo.jl

@@ -0,0 +1,148 @@
+"""
+    session/jsonl_repo.jl - JSONL session repository
+
+This module provides a JSONL-based session repository implementation.
+"""
+
+module JsonlRepo
+
+using ..Types: *
+using ..SessionStorage: SessionStorage, SessionMetadata
+using ..JsonlStorage: JsonlSessionStorage, headerToSessionMetadata
+using ..MemoryRepo: createSessionId, createTimestamp, getEntriesToFork, toSession
+using ..HarnessTypes: SessionRepo, SessionForkOptions
+
+# ============================================================================
+# JSONL session repository
+# ============================================================================
+
+mutable struct JsonlSessionRepo <: SessionRepo{
+    JsonlSessionMetadata,
+    JsonlSessionCreateOptions,
+    JsonlSessionListOptions
+}
+    fs::Any
+    sessions_root_input::String
+    sessions_root::Union{String, Nothing}
+
+    function JsonlSessionRepo(; sessions_root::String, fs::Any)
+        new(fs, sessions_root, nothing)
+    end
+end
+
+# ============================================================================
+# Session repo methods
+# ============================================================================
+
+function create(repo::JsonlSessionRepo, options::JsonlSessionCreateOptions)::Session
+    id = if haskey(options, :id) && !isnothing(options[:id])
+        options[:id]
+    else
+        createSessionId()
+    end
+    created_at = createTimestamp()
+    
+    session_dir = getSessionDir(repo, options.cwd)
+    
+    file_path = createSessionFilePath(repo, options.cwd, id, created_at)
+    
+    storage = JsonlSessionStorage(
+        file_path,
+        SessionHeader(
+            "session",
+            3,
+            id,
+            created_at,
+            options.cwd,
+            get(options, :parentSessionPath, nothing),
+            get(options, :metadata, nothing),
+        ),
+        SessionTreeEntry[],
+        nothing,
+    )
+    
+    return toSession(storage)
+end
+
+function open(repo::JsonlSessionRepo, metadata::JsonlSessionMetadata)::Session
+    # TODO: Open existing file
+    return toSession(JsonlSessionStorage(
+        metadata.path,
+        SessionHeader(
+            "session",
+            3,
+            metadata.id,
+            metadata.created_at,
+            metadata.cwd,
+            metadata.parent_session_path,
+            metadata.metadata,
+        ),
+        SessionTreeEntry[],
+        nothing,
+    ))
+end
+
+function list(repo::JsonlSessionRepo, options::JsonlSessionListOptions=JsonlSessionListOptions())::Vector{JsonlSessionMetadata}
+    # TODO: List sessions
+    return JsonlSessionMetadata[]
+end
+
+function delete(repo::JsonlSessionRepo, metadata::JsonlSessionMetadata)::Nothing
+    # TODO: Delete session file
+    return nothing
+end
+
+function fork(repo::JsonlSessionRepo, source::JsonlSessionMetadata, options::Dict{String, Any})::Session
+    # TODO: Fork session
+    return create(repo, JsonlSessionCreateOptions(
+        cwd=get(options, "cwd", ""),
+        id=get(options, "id", createSessionId()),
+    ))
+end
+
+# ============================================================================
+# Helper functions
+# ============================================================================
+
+function getSessionsRoot(repo::JsonlSessionRepo)::String
+    if isnothing(repo.sessions_root)
+        repo.sessions_root = getFileSystemResultOrThrow(
+            absolutePath(repo.fs, repo.sessions_root_input),
+            "Failed to resolve sessions root $(repo.sessions_root_input)",
+        )
+    end
+    return repo.sessions_root
+end
+
+function getSessionDir(repo::JsonlSessionRepo, cwd::String)::String
+    return getFileSystemResultOrThrow(
+        joinPath(repo.fs, [getSessionsRoot(repo), encodeCwd(cwd)]),
+        "Failed to resolve session directory for $(cwd)",
+    )
+end
+
+function encodeCwd(cwd::String)::String
+    result = replace(cwd, r"^[/\\]" => "")
+    result = replace(result, r"[/\\:]" => "-")
+    return "--$(result)--"
+end
+
+function createSessionFilePath(repo::JsonlSessionRepo, cwd::String, session_id::String, timestamp::String)::String
+    return getFileSystemResultOrThrow(
+        joinPath(repo.fs, [
+            getSessionDir(repo, cwd),
+            "$(replace(timestamp, r"[:.]" => "-"))_$(session_id).jsonl",
+        ]),
+        "Failed to resolve session file path for $(session_id)",
+    )
+end
+
+function getFileSystemResultOrThrow(result::Result, message::String)
+    if !result.ok
+        code = result.error.code == "not_found" ? "not_found" : "storage"
+        throw(SessionError(code, "$(message): $(result.error.message)", result.error))
+    end
+    return result.value
+end
+
+end

src/session/jsonl_storage.jl

@@ -0,0 +1,290 @@
+"""
+    session/jsonl_storage.jl - JSONL session storage
+
+This module provides JSONL-based session storage implementation.
+"""
+
+module JsonlStorage
+
+using ..Types: *
+using ..SessionStorage: SessionStorage, SessionMetadata
+
+# ============================================================================
+# Session header
+# ============================================================================
+
+mutable struct SessionHeader
+    type::String
+    version::Int64
+    id::String
+    timestamp::String
+    cwd::String
+    parent_session::Union{String, Nothing}
+    metadata::Union{Dict{String, Any}, Nothing}
+end
+
+# ============================================================================
+# JSONL session storage
+# ============================================================================
+
+mutable struct JsonlSessionStorage{T<:SessionMetadata} <: SessionStorage{T}
+    file_path::String
+    metadata::T
+    entries::Vector{SessionTreeEntry}
+    by_id::Dict{String, SessionTreeEntry}
+    labels_by_id::Dict{String, String}
+    current_leaf_id::Union{String, Nothing}
+
+    function JsonlSessionStorage{T}(
+        file_path::String,
+        header::SessionHeader,
+        entries::Vector{SessionTreeEntry},
+        leaf_id::Union{String, Nothing},
+    ) where T
+        by_id = Dict{String, SessionTreeEntry}((e.id, e) for e in entries)
+        labels_by_id = Dict{String, String}()
+        
+        for entry in entries
+            if entry isa LabelEntry && !isnothing(entry.label)
+                labels_by_id[entry.target_id] = entry.label
+            end
+        end
+
+        new(
+            file_path,
+            header,
+            entries,
+            by_id,
+            labels_by_id,
+            leaf_id,
+        )
+    end
+end
+
+# ============================================================================
+# Session storage methods
+# ============================================================================
+
+function getMetadata(storage::JsonlSessionStorage)::T
+    return storage.metadata
+end
+
+function getLeafId(storage::JsonlSessionStorage)::Union{String, Nothing}
+    if !isnothing(storage.current_leaf_id) && !haskey(storage.by_id, storage.current_leaf_id)
+        throw(SessionError("invalid_session", "Entry $(storage.current_leaf_id) not found"))
+    end
+    return storage.current_leaf_id
+end
+
+function setLeafId(storage::JsonlSessionStorage, leaf_id::Union{String, Nothing})::Nothing
+    if !isnothing(leaf_id) && !haskey(storage.by_id, leaf_id)
+        throw(SessionError("not_found", "Entry $(leaf_id) not found"))
+    end
+
+    entry = LeafEntry(
+        "leaf",
+        generateEntryId(storage.by_id),
+        storage.current_leaf_id,
+        create_timestamp(),
+        leaf_id,
+    )
+
+    # TODO: Write to file
+    # getFileSystemResultOrThrow(
+    #     await this.fs.appendFile(this.filePath, `${JSON.stringify(entry)}\n`),
+    #     `Failed to append session leaf ${entry.id}`,
+    # );
+
+    push!(storage.entries, entry)
+    storage.by_id[entry.id] = entry
+    storage.current_leaf_id = leaf_id
+    return nothing
+end
+
+function createEntryId(storage::JsonlSessionStorage)::String
+    return generateEntryId(storage.by_id)
+end
+
+function appendEntry(storage::JsonlSessionStorage, entry::SessionTreeEntry)::Nothing
+    # TODO: Write to file
+    # getFileSystemResultOrThrow(
+    #     await this.fs.appendFile(this.filePath, `${JSON.stringify(entry)}\n`),
+    #     `Failed to append session entry ${entry.id}`,
+    # );
+
+    push!(storage.entries, entry)
+    storage.by_id[entry.id] = entry
+    
+    if entry isa LabelEntry
+        updateLabelCache(storage.labels_by_id, entry)
+    end
+
+    storage.current_leaf_id = leafIdAfterEntry(entry)
+    return nothing
+end
+
+function getEntry(storage::JsonlSessionStorage, id::String)::Union{SessionTreeEntry, Nothing}
+    return get(storage.by_id, id, nothing)
+end
+
+function findEntries(storage::JsonlSessionStorage, type::String)::Vector{SessionTreeEntry}
+    return filter(entry -> entry.type == type, storage.entries)
+end
+
+function getLabel(storage::JsonlSessionStorage, id::String)::Union{String, Nothing}
+    return get(storage.labels_by_id, id, nothing)
+end
+
+function getSessionName(storage::JsonlSessionStorage)::Union{String, Nothing}
+    entries = findEntries(storage, "session_info")
+    if isempty(entries)
+        return nothing
+    end
+    return strip(entries[end].name)
+end
+
+function getSessionStats(storage::JsonlSessionStorage)::SessionStats
+    message_count = 0
+    cached_tokens = 0
+    uncached_tokens = 0
+    total_tokens = 0
+    cost_total = 0.0
+
+    for entry in storage.entries
+        if entry isa MessageEntry
+            message_count += 1
+        end
+
+        usage = if entry isa MessageEntry && entry.message.role == "assistant"
+            entry.message.usage
+        elseif entry isa CompactionEntry || entry isa BranchSummaryEntry
+            entry.usage
+        else
+            nothing
+        end
+
+        if !isnothing(usage) && 
+           usage.input isa Int64 && 
+           usage.output isa Int64 && 
+           usage.cache_read isa Int64 && 
+           usage.cache_write isa Int64 &&
+           usage.cost.total isa Float64
+
+            cached_tokens += usage.cache_read
+            uncached_tokens += usage.input + usage.cache_write
+            total_tokens += usage.input + usage.output + usage.cache_read + usage.cache_write
+            cost_total += usage.cost.total
+        end
+    end
+
+    return SessionStats(
+        message_count,
+        cached_tokens,
+        uncached_tokens,
+        total_tokens,
+        cost_total,
+    )
+end
+
+function getPathToRootOrCompaction(storage::JsonlSessionStorage, leaf_id::Union{String, Nothing})::Vector{SessionTreeEntry}
+    if isnothing(leaf_id)
+        return SessionTreeEntry[]
+    end
+
+    path::Vector{SessionTreeEntry} = SessionTreeEntry[]
+    stop_at_entry_id::Union{String, Nothing} = nothing
+    current = get(storage.by_id, leaf_id, nothing)
+
+    if isnothing(current)
+        throw(SessionError("not_found", "Entry $(leaf_id) not found"))
+    end
+
+    while !isnothing(current)
+        unshift!(path, current)
+        
+        if !isnothing(stop_at_entry_id) && current.id == stop_at_entry_id
+            break
+        end
+
+        if current isa CompactionEntry
+            if !isnothing(current.retained_tail)
+                break
+            end
+            stop_at_entry_id = current.first_kept_entry_id
+        end
+
+        if isnothing(current.parent_id)
+            break
+        end
+
+        parent = get(storage.by_id, current.parent_id, nothing)
+        if isnothing(parent)
+            throw(SessionError("invalid_session", "Entry $(current.parent_id) not found"))
+        end
+
+        current = parent
+    end
+
+    return path
+end
+
+function getEntries(storage::JsonlSessionStorage, options::Dict{String, Any})::Vector{SessionTreeEntry}
+    start = get(options, "afterEntrySeq", 0)
+    end_idx = if haskey(options, "limit")
+        start + options["limit"]
+    else
+        nothing
+    end
+
+    if isnothing(end_idx)
+        return copy(storage.entries[start+1:end])
+    end
+
+    return copy(storage.entries[start+1:end_idx])
+end
+
+# ============================================================================
+# Helper functions
+# ============================================================================
+
+function updateLabelCache(labels_by_id::Dict{String, String}, entry::SessionTreeEntry)::Nothing
+    if entry isa LabelEntry
+        label = strip(get(entry, :label, nothing))
+        if !isnothing(label) && !isempty(label)
+            labels_by_id[entry.target_id] = label
+        else
+            delete!(labels_by_id, entry.target_id)
+        end
+    end
+    return nothing
+end
+
+function generateEntryId(by_id::Dict{String, SessionTreeEntry})::String
+    for i in 1:100
+        id = uuidv7()[end-7:end]
+        if !haskey(by_id, id)
+            return id
+        end
+    end
+    return uuidv7()
+end
+
+function leafIdAfterEntry(entry::SessionTreeEntry)::Union{String, Nothing}
+    if entry isa LeafEntry
+        return entry.target_id
+    end
+    return entry.id
+end
+
+function headerToSessionMetadata(header::SessionHeader, path::String)::JsonlSessionMetadata
+    return JsonlSessionMetadata(
+        header.id,
+        header.timestamp,
+        header.cwd,
+        path,
+        header.parent_session,
+        header.metadata,
+    )
+end
+
+end

src/session/memory_repo.jl

@@ -0,0 +1,133 @@
+"""
+    session/memory_repo.jl - In-memory session repository
+
+This module provides an in-memory session repository implementation for testing.
+"""
+
+module MemoryRepo
+
+using ..Types: *
+using ..SessionStorage: SessionStorage, SessionMetadata
+using ..MemoryStorage: InMemorySessionStorage
+
+# ============================================================================
+# In-memory session repository
+# ============================================================================
+
+mutable struct InMemorySessionRepo <: SessionRepo{SessionMetadata, Dict{String, Any}, Nothing}
+    sessions::Dict{String, Session}
+
+    function InMemorySessionRepo()
+        new(Dict{String, Session}())
+    end
+end
+
+# ============================================================================
+# Session repo methods
+# ============================================================================
+
+function create(repo::InMemorySessionRepo, options::Dict{String, Any}=Dict{String, Any}())::Session
+    metadata = SessionMetadata(
+        if haskey(options, :id) && !isnothing(options[:id])
+            options[:id]
+        else
+            createSessionId()
+        end,
+        createTimestamp(),
+    )
+    
+    storage = InMemorySessionStorage{SessionMetadata}(metadata=metadata)
+    session = toSession(storage)
+    
+    repo.sessions[metadata.id] = session
+    
+    return session
+end
+
+function open(repo::InMemorySessionRepo, metadata::SessionMetadata)::Session
+    session = get(repo.sessions, metadata.id, nothing)
+    if isnothing(session)
+        throw(SessionError("not_found", "Session not found: $(metadata.id)"))
+    end
+    return session
+end
+
+function list(repo::InMemorySessionRepo)::Vector{SessionMetadata}
+    return [getMetadata(session) for session in values(repo.sessions)]
+end
+
+function delete(repo::InMemorySessionRepo, metadata::SessionMetadata)::Nothing
+    delete!(repo.sessions, metadata.id)
+    return nothing
+end
+
+function fork(repo::InMemorySessionRepo, source::SessionMetadata, options::Dict{String, Any})::Session
+    source_session = open(repo, source)
+    forked_entries = getEntriesToFork(getStorage(source_session), options)
+    
+    metadata = SessionMetadata(
+        if haskey(options, :id) && !isnothing(options[:id])
+            options[:id]
+        else
+            createSessionId()
+        end,
+        createTimestamp(),
+    )
+    
+    storage = InMemorySessionStorage{SessionMetadata}(
+        entries=forked_entries,
+        metadata=metadata,
+    )
+    
+    session = toSession(storage)
+    repo.sessions[metadata.id] = session
+    
+    return session
+end
+
+# ============================================================================
+# Helper functions
+# ============================================================================
+
+function createSessionId()::String
+    return uuidv7()
+end
+
+function createTimestamp()::String
+    return create_timestamp()
+end
+
+function toSession(storage::SessionStorage)::Session
+    return Session(storage)
+end
+
+function getEntriesToFork(storage::SessionStorage, options::Dict{String, Any})::Vector{SessionTreeEntry}
+    if !haskey(options, :entryId) || isnothing(options[:entryId])
+        return getEntries(storage, Dict{String, Any}())
+    end
+
+    target = getEntry(storage, options[:entryId])
+    if isnothing(target)
+        throw(SessionError("invalid_fork_target", "Entry $(options[:entryId]) not found"))
+    end
+
+    effective_leaf_id::Union{String, Nothing}
+    position = get(options, "position", "before")
+    
+    if position == "at"
+        effective_leaf_id = target.id
+    else
+        if target isa MessageEntry && target.message.role != "user"
+            throw(SessionError("invalid_fork_target", "Entry $(options[:entryId]) is not a user message"))
+        end
+        effective_leaf_id = target.parent_id
+    end
+
+    return getPathToRootOrCompaction(storage, effective_leaf_id)
+end
+
+function getStorage(session::Session)::SessionStorage
+    return session.storage
+end
+
+end

src/session/memory_storage.jl

@@ -0,0 +1,227 @@
+"""
+    session/memory_storage.jl - In-memory session storage
+
+This module provides an in-memory session storage implementation for testing and temporary use.
+"""
+
+module MemoryStorage
+
+using ..Types: *
+using ..SessionStorage: SessionStorage, SessionMetadata
+using ..JsonlStorage: updateLabelCache, generateEntryId, leafIdAfterEntry
+
+# ============================================================================
+# In-memory session storage
+# ============================================================================
+
+mutable struct InMemorySessionStorage{T<:SessionMetadata} <: SessionStorage{T}
+    metadata::T
+    entries::Vector{SessionTreeEntry}
+    by_id::Dict{String, SessionTreeEntry}
+    labels_by_id::Dict{String, String}
+    leaf_id::Union{String, Nothing}
+
+    function InMemorySessionStorage{T}(;
+        entries::Vector{SessionTreeEntry}=SessionTreeEntry[],
+        metadata::Union{T, Nothing]=nothing,
+    ) where T
+        by_id = Dict{String, SessionTreeEntry}((e.id, e) for e in entries)
+        labels_by_id = Dict{String, String}()
+
+        leaf_id = nothing
+        for entry in entries
+            if entry isa LabelEntry
+                updateLabelCache(labels_by_id, entry)
+            end
+            leaf_id = leafIdAfterEntry(entry)
+        end
+
+        if !isnothing(leaf_id) && !haskey(by_id, leaf_id)
+            throw(SessionError("invalid_session", "Entry $(leaf_id) not found"))
+        end
+
+        new(
+            if isnothing(metadata)
+                T(uuidv7(), create_timestamp())
+            else
+                metadata
+            end,
+            copy(entries),
+            by_id,
+            labels_by_id,
+            leaf_id,
+        )
+    end
+end
+
+# ============================================================================
+# Session storage methods
+# ============================================================================
+
+function getMetadata(storage::InMemorySessionStorage)::T
+    return storage.metadata
+end
+
+function getLeafId(storage::InMemorySessionStorage)::Union{String, Nothing}
+    if !isnothing(storage.leaf_id) && !haskey(storage.by_id, storage.leaf_id)
+        throw(SessionError("invalid_session", "Entry $(storage.leaf_id) not found"))
+    end
+    return storage.leaf_id
+end
+
+function setLeafId(storage::InMemorySessionStorage, leaf_id::Union{String, Nothing})::Nothing
+    if !isnothing(leaf_id) && !haskey(storage.by_id, leaf_id)
+        throw(SessionError("not_found", "Entry $(leaf_id) not found"))
+    end
+
+    entry = LeafEntry(
+        "leaf",
+        generateEntryId(storage.by_id),
+        storage.leaf_id,
+        create_timestamp(),
+        leaf_id,
+    )
+
+    push!(storage.entries, entry)
+    storage.by_id[entry.id] = entry
+    storage.leaf_id = leaf_id
+    return nothing
+end
+
+function createEntryId(storage::InMemorySessionStorage)::String
+    return generateEntryId(storage.by_id)
+end
+
+function appendEntry(storage::InMemorySessionStorage, entry::SessionTreeEntry)::Nothing
+    push!(storage.entries, entry)
+    storage.by_id[entry.id] = entry
+    
+    if entry isa LabelEntry
+        updateLabelCache(storage.labels_by_id, entry)
+    end
+
+    storage.leaf_id = leafIdAfterEntry(entry)
+    return nothing
+end
+
+function getEntry(storage::InMemorySessionStorage, id::String)::Union{SessionTreeEntry, Nothing}
+    return get(storage.by_id, id, nothing)
+end
+
+function findEntries(storage::InMemorySessionStorage, type::String)::Vector{SessionTreeEntry}
+    return filter(entry -> entry.type == type, storage.entries)
+end
+
+function getLabel(storage::InMemorySessionStorage, id::String)::Union{String, Nothing}
+    return get(storage.labels_by_id, id, nothing)
+end
+
+function getSessionName(storage::InMemorySessionStorage)::Union{String, Nothing}
+    entries = findEntries(storage, "session_info")
+    if isempty(entries)
+        return nothing
+    end
+    return strip(entries[end].name)
+end
+
+function getSessionStats(storage::InMemorySessionStorage)::SessionStats
+    message_count = 0
+    cached_tokens = 0
+    uncached_tokens = 0
+    total_tokens = 0
+    cost_total = 0.0
+
+    for entry in storage.entries
+        if entry isa MessageEntry
+            message_count += 1
+        end
+
+        usage = if entry isa MessageEntry && entry.message.role == "assistant"
+            entry.message.usage
+        elseif entry isa CompactionEntry || entry isa BranchSummaryEntry
+            entry.usage
+        else
+            nothing
+        end
+
+        if !isnothing(usage) && 
+           usage.input isa Int64 && 
+           usage.output isa Int64 && 
+           usage.cache_read isa Int64 && 
+           usage.cache_write isa Int64 &&
+           usage.cost.total isa Float64
+
+            cached_tokens += usage.cache_read
+            uncached_tokens += usage.input + usage.cache_write
+            total_tokens += usage.input + usage.output + usage.cache_read + usage.cache_write
+            cost_total += usage.cost.total
+        end
+    end
+
+    return SessionStats(
+        message_count,
+        cached_tokens,
+        uncached_tokens,
+        total_tokens,
+        cost_total,
+    )
+end
+
+function getPathToRootOrCompaction(storage::InMemorySessionStorage, leaf_id::Union{String, Nothing})::Vector{SessionTreeEntry}
+    if isnothing(leaf_id)
+        return SessionTreeEntry[]
+    end
+
+    path::Vector{SessionTreeEntry} = SessionTreeEntry[]
+    stop_at_entry_id::Union{String, Nothing} = nothing
+    current = get(storage.by_id, leaf_id, nothing)
+
+    if isnothing(current)
+        throw(SessionError("not_found", "Entry $(leaf_id) not found"))
+    end
+
+    while !isnothing(current)
+        unshift!(path, current)
+        
+        if !isnothing(stop_at_entry_id) && current.id == stop_at_entry_id
+            break
+        end
+
+        if current isa CompactionEntry
+            if !isnothing(current.retained_tail)
+                break
+            end
+            stop_at_entry_id = current.first_kept_entry_id
+        end
+
+        if isnothing(current.parent_id)
+            break
+        end
+
+        parent = get(storage.by_id, current.parent_id, nothing)
+        if isnothing(parent)
+            throw(SessionError("invalid_session", "Entry $(current.parent_id) not found"))
+        end
+
+        current = parent
+    end
+
+    return path
+end
+
+function getEntries(storage::InMemorySessionStorage, options::Dict{String, Any})::Vector{SessionTreeEntry}
+    start = get(options, "afterEntrySeq", 0)
+    end_idx = if haskey(options, "limit")
+        start + options["limit"]
+    else
+        nothing
+    end
+
+    if isnothing(end_idx)
+        return copy(storage.entries[start+1:end])
+    end
+
+    return copy(storage.entries[start+1:end_idx])
+end
+
+end

src/session/repo_utils.jl

@@ -0,0 +1,65 @@
+"""
+    session/repo_utils.jl - Session repository utilities
+
+This module provides shared utilities for session repository implementations.
+"""
+
+module RepoUtils
+
+using ..Types: *
+using ..SessionStorage: SessionStorage, SessionMetadata
+using ..Session: Session
+
+# ============================================================================
+# Helper functions
+# ============================================================================
+
+function createSessionId()::String
+    return uuidv7()
+end
+
+function createTimestamp()::String
+    return create_timestamp()
+end
+
+function toSession{T<:SessionMetadata}(storage::SessionStorage{T})::Session{T}
+    return Session(storage)
+end
+
+function getFileSystemResultOrThrow{TValue}(result::Result{TValue, FileError}, message::String)::TValue
+    if !result.ok
+        code = result.error.code == "not_found" ? "not_found" : "storage"
+        throw(SessionError(code, "$(message): $(result.error.message)", result.error))
+    end
+    return result.value
+end
+
+function getEntriesToFork(
+    storage::SessionStorage,
+    options::Dict{String, Any},
+)::Vector{SessionTreeEntry}
+    if !haskey(options, :entryId) || isnothing(options[:entryId])
+        return getEntries(storage, Dict{String, Any}())
+    end
+
+    target = getEntry(storage, options[:entryId])
+    if isnothing(target)
+        throw(SessionError("invalid_fork_target", "Entry $(options[:entryId]) not found"))
+    end
+
+    effective_leaf_id::Union{String, Nothing}
+    position = get(options, "position", "before")
+    
+    if position == "at"
+        effective_leaf_id = target.id
+    else
+        if target isa MessageEntry && target.message.role != "user"
+            throw(SessionError("invalid_fork_target", "Entry $(options[:entryId]) is not a user message"))
+        end
+        effective_leaf_id = target.parent_id
+    end
+
+    return getPathToRootOrCompaction(storage, effective_leaf_id)
+end
+
+end

src/session/session.jl

@@ -0,0 +1,422 @@
+"""
+    session/session.jl - Session management
+
+This module provides the Session class for managing conversation history with branch support.
+"""
+
+module Session
+
+using ..Types: *
+using ..SessionStorage: SessionStorage
+using ..Messages: *
+using ..HarnessTypes: *
+
+# ============================================================================
+# Session context build options
+# ============================================================================
+
+mutable struct SessionContextBuildOptions
+    entry_transforms::Union{Vector{Function}, Nothing}
+    entry_projectors::Union{Dict{String, Function}, Nothing}
+end
+
+# ============================================================================
+# Default context entry transform
+# ============================================================================
+
+function defaultContextEntryTransform(path_entries::Vector{SessionTreeEntry})::Vector{SessionTreeEntry}
+    compaction = nothing
+    for entry in path_entries
+        if entry isa CompactionEntry
+            compaction = entry
+            break
+        end
+    end
+
+    if isnothing(compaction)
+        return copy(path_entries)
+    end
+
+    entries::Vector{SessionTreeEntry} = [compaction]
+    compaction_idx = findfirst(
+        (entry) -> entry isa CompactionEntry && entry.id == compaction.id,
+        path_entries,
+    )
+
+    if !isnothing(compaction.retained_tail)
+        for i in compaction_idx+1:length(path_entries)
+            push!(entries, path_entries[i])
+        end
+        return entries
+    end
+
+    if !isnothing(compaction.first_kept_entry_id)
+        found_first_kept = false
+        for i in 1:compaction_idx-1
+            entry = path_entries[i]
+            if entry.id == compaction.first_kept_entry_id
+                found_first_kept = true
+            end
+            if found_first_kept
+                push!(entries, entry)
+            end
+        end
+    end
+
+    for i in compaction_idx+1:length(path_entries)
+        push!(entries, path_entries[i])
+    end
+
+    return entries
+end
+
+# ============================================================================
+# Build context entries
+# ============================================================================
+
+function buildContextEntries(
+    path_entries::Vector{SessionTreeEntry},
+    options::SessionContextBuildOptions=SessionContextBuildOptions(nothing, nothing),
+)::Vector{SessionTreeEntry}
+    entries = defaultContextEntryTransform(path_entries)
+
+    if !isnothing(options.entry_transforms)
+        for transform in options.entry_transforms
+            entries = transform(entries)
+        end
+    end
+
+    return entries
+end
+
+# ============================================================================
+# Session entry to context messages
+# ============================================================================
+
+function sessionEntryToContextMessages(
+    entry::SessionTreeEntry,
+    index::Int64,
+    entries::Vector{SessionTreeEntry},
+    options::SessionContextBuildOptions=SessionContextBuildOptions(nothing, nothing),
+)::Vector{AgentMessage}
+    if entry isa MessageEntry
+        return [entry.message]
+    end
+
+    if entry isa CustomMessageEntry
+        return [createCustomMessage(
+            entry.custom_type,
+            entry.content,
+            entry.display,
+            entry.details,
+            entry.timestamp,
+        )]
+    end
+
+    if entry isa CompactionEntry
+        messages = [createCompactionSummaryMessage(
+            entry.summary,
+            entry.tokens_before,
+            entry.timestamp,
+        )]
+        if !isnothing(entry.retained_tail)
+            append!(messages, entry.retained_tail)
+        end
+        return messages
+    end
+
+    if entry isa BranchSummaryEntry
+        return [createBranchSummaryMessage(
+            entry.summary,
+            entry.from_id,
+            entry.timestamp,
+        )]
+    end
+
+    if entry isa CustomEntry
+        if !isnothing(options.entry_projectors) && haskey(options.entry_projectors, entry.custom_type)
+            projector = options.entry_projectors[entry.custom_type]
+            return projector(entry, index, entries)
+        end
+        return AgentMessage[]
+    end
+
+    return AgentMessage[]
+end
+
+# ============================================================================
+# Build session context
+# ============================================================================
+
+function buildSessionContext(
+    path_entries::Vector{SessionTreeEntry},
+    options::SessionContextBuildOptions=SessionContextBuildOptions(nothing, nothing),
+)::SessionContext
+    state = deriveSessionContextState(path_entries)
+    context_entries = buildContextEntries(path_entries, options)
+    messages = SessionTreeEntry[]
+    for (i, entry) in enumerate(context_entries)
+        append!(messages, sessionEntryToContextMessages(entry, i, context_entries, options))
+    end
+    return SessionContext(messages, state.thinking_level, state.model, state.active_tool_names)
+end
+
+function deriveSessionContextState(path_entries::Vector{SessionTreeEntry})::Dict{String, Any}
+    thinking_level = "off"
+    model = nothing
+    active_tool_names = nothing
+
+    for entry in path_entries
+        if entry isa ThinkingLevelChangeEntry
+            thinking_level = entry.thinking_level
+        elseif entry isa ModelChangeEntry
+            model = Dict{String, String}("provider" => entry.provider, "modelId" => entry.model_id)
+        elseif entry isa MessageEntry && entry.message.role == "assistant"
+            model = Dict{String, String}("provider" => entry.message.provider, "modelId" => entry.message.model)
+        elseif entry isa ActiveToolsChangeEntry
+            active_tool_names = copy(entry.active_tool_names)
+        end
+    end
+
+    return Dict{String, Any}(
+        "thinking_level" => thinking_level,
+        "model" => model,
+        "active_tool_names" => active_tool_names,
+    )
+end
+
+# ============================================================================
+# Session class
+# ============================================================================
+
+mutable struct Session{T<:SessionMetadata}
+    storage::SessionStorage{T}
+    context_build_options::SessionContextBuildOptions
+
+    function Session(
+        storage::SessionStorage,
+        context_build_options::SessionContextBuildOptions=SessionContextBuildOptions(nothing, nothing),
+    )
+        new{typeof(storage.metadata)}(storage, context_build_options)
+    end
+end
+
+# ============================================================================
+# Session methods
+# ============================================================================
+
+function getMetadata(session::Session)::T
+    return getMetadata(session.storage)
+end
+
+function getStorage(session::Session)::SessionStorage
+    return session.storage
+end
+
+function getLeafId(session::Session)::Union{String, Nothing}
+    return getLeafId(session.storage)
+end
+
+function getEntry(session::Session, id::String)::Union{SessionTreeEntry, Nothing}
+    return getEntry(session.storage, id)
+end
+
+function getEntries(session::Session, options::Dict{String, Any}=Dict{String, Any}())::Vector{SessionTreeEntry}
+    return getEntries(session.storage, options)
+end
+
+function getBranch(session::Session, from_id::Union{String, Nothing}=nothing)::Vector{SessionTreeEntry}
+    leaf_id = if isnothing(from_id)
+        getLeafId(session.storage)
+    else
+        from_id
+    end
+    return getPathToRootOrCompaction(session.storage, leaf_id)
+end
+
+function buildContextEntries(session::Session, options::SessionContextBuildOptions=SessionContextBuildOptions())::Vector{SessionTreeEntry}
+    return buildContextEntries(getBranch(session), mergeContextBuildOptions(session, options))
+end
+
+function buildContext(session::Session, options::SessionContextBuildOptions=SessionContextBuildOptions())::SessionContext
+    return buildSessionContext(getBranch(session), mergeContextBuildOptions(session, options))
+end
+
+function mergeContextBuildOptions(session::Session, options::SessionContextBuildOptions)::SessionContextBuildOptions
+    return SessionContextBuildOptions(
+        vcat(
+            isnothing(session.context_build_options.entry_transforms) ? [] : session.context_build_options.entry_transforms,
+            isnothing(options.entry_transforms) ? [] : options.entry_transforms,
+        ),
+        merge(
+            isnothing(session.context_build_options.entry_projectors) ? Dict{String, Any}() : session.context_build_options.entry_projectors,
+            isnothing(options.entry_projectors) ? Dict{String, Any}() : options.entry_projectors,
+            promote=true,
+        ),
+    )
+end
+
+function getLabel(session::Session, id::String)::Union{String, Nothing}
+    return getLabel(session.storage, id)
+end
+
+function getSessionStats(session::Session)::SessionStats
+    return getSessionStats(session.storage)
+end
+
+function getSessionName(session::Session)::Union{String, Nothing}
+    return getSessionName(session.storage)
+end
+
+function appendMessage(session::Session, message::AgentMessage)::String
+    return appendTypedEntry(session, MessageEntry(
+        "message",
+        createEntryId(session.storage),
+        getLeafId(session.storage),
+        create_timestamp(),
+        message,
+    ))
+end
+
+function appendThinkingLevelChange(session::Session, thinking_level::String)::String
+    return appendTypedEntry(session, ThinkingLevelChangeEntry(
+        "thinking_level_change",
+        createEntryId(session.storage),
+        getLeafId(session.storage),
+        create_timestamp(),
+        thinking_level,
+    ))
+end
+
+function appendModelChange(session::Session, provider::String, model_id::String)::String
+    return appendTypedEntry(session, ModelChangeEntry(
+        "model_change",
+        createEntryId(session.storage),
+        getLeafId(session.storage),
+        create_timestamp(),
+        provider,
+        model_id,
+    ))
+end
+
+function appendActiveToolsChange(session::Session, active_tool_names::Vector{String})::String
+    return appendTypedEntry(session, ActiveToolsChangeEntry(
+        "active_tools_change",
+        createEntryId(session.storage),
+        getLeafId(session.storage),
+        create_timestamp(),
+        active_tool_names,
+    ))
+end
+
+function appendCompaction(
+    session::Session,
+    summary::String,
+    first_kept_entry_id::Union{String, Nothing},
+    tokens_before::Int64,
+    details::Union{Any, Nothing}=nothing,
+    from_hook::Bool=false,
+    usage::Union{Usage, Nothing}=nothing,
+    retained_tail::Union{Vector{AgentMessage}, Nothing}=nothing,
+)::String
+    return appendTypedEntry(session, CompactionEntry(
+        "compaction",
+        createEntryId(session.storage),
+        getLeafId(session.storage),
+        create_timestamp(),
+        summary,
+        first_kept_entry_id,
+        tokens_before,
+        retained_tail,
+        details,
+        usage,
+        from_hook,
+    ))
+end
+
+function appendCustomEntry(session::Session, custom_type::String, data::Union{Any, Nothing}=nothing)::String
+    return appendTypedEntry(session, CustomEntry(
+        "custom",
+        createEntryId(session.storage),
+        getLeafId(session.storage),
+        create_timestamp(),
+        custom_type,
+        data,
+    ))
+end
+
+function appendCustomMessageEntry(
+    session::Session,
+    custom_type::String,
+    content::String,
+    display::Bool,
+    details::Union{Any, Nothing}=nothing,
+)::String
+    return appendTypedEntry(session, CustomMessageEntry(
+        "custom_message",
+        createEntryId(session.storage),
+        getLeafId(session.storage),
+        create_timestamp(),
+        custom_type,
+        content,
+        details,
+        display,
+    ))
+end
+
+function appendLabel(session::Session, target_id::String, label::Union{String, Nothing})::String
+    if isnothing(getEntry(session, target_id))
+        throw(SessionError("not_found", "Entry $(target_id) not found"))
+    end
+    return appendTypedEntry(session, LabelEntry(
+        "label",
+        createEntryId(session.storage),
+        getLeafId(session.storage),
+        create_timestamp(),
+        target_id,
+        label,
+    ))
+end
+
+function appendSessionName(session::Session, name::String)::String
+    sanitizedName = replace(name, r"[\r\n]+" => " ")
+    return appendTypedEntry(session, SessionInfoEntry(
+        "session_info",
+        createEntryId(session.storage),
+        getLeafId(session.storage),
+        create_timestamp(),
+        sanitizedName,
+    ))
+end
+
+function moveTo(
+    session::Session,
+    entry_id::Union{String, Nothing},
+    summary::Union{Dict{String, Any}, Nothing}=nothing,
+)::Union{String, Nothing
+    if !isnothing(entry_id) && isnothing(getEntry(session, entry_id))
+        throw(SessionError("not_found", "Entry $(entry_id) not found"))
+    end
+    setLeafId(session.storage, entry_id)
+    if isnothing(summary)
+        return nothing
+    end
+    return appendTypedEntry(session, BranchSummaryEntry(
+        "branch_summary",
+        createEntryId(session.storage),
+        entry_id,
+        create_timestamp(),
+        entry_id,
+        summary["summary"],
+        get(summary, "details", nothing),
+        get(summary, "usage", nothing),
+        get(summary, "from_hook", false),
+    ))
+end
+
+function appendTypedEntry(session::Session, entry::SessionTreeEntry)::String
+    appendEntry(session.storage, entry)
+    return entry.id
+end
+
+end

src/skills.jl

@@ -0,0 +1,375 @@
+"""
+    skills.jl - Skill loading and formatting
+
+This module provides utilities for loading skills from SKILL.md files and formatting skill invocations.
+"""
+
+module Skills
+
+using ..Types: *
+using ..HarnessTypes: Skill, ExecutionEnv, FileSystem, toError, FileError, Result, ok, err
+
+const MAX_NAME_LENGTH = 64
+const MAX_DESCRIPTION_LENGTH = 1024
+const IGNORE_FILE_NAMES = [".gitignore", ".ignore", ".fdignore"]
+
+# ============================================================================
+# Skill diagnostic types
+# ============================================================================
+
+const SkillDiagnosticCode = String
+const SKILL_DIAGNOSTIC_FILE_INFO_FAILED = "file_info_failed"
+const SKILL_DIAGNOSTIC_LIST_FAILED = "list_failed"
+const SKILL_DIAGNOSTIC_READ_FAILED = "read_failed"
+const SKILL_DIAGNOSTIC_PARSE_FAILED = "parse_failed"
+const SKILL_DIAGNOSTIC_INVALID_METADATA = "invalid_metadata"
+
+mutable struct SkillDiagnostic
+    type::String
+    code::SkillDiagnosticCode
+    message::String
+    path::String
+end
+
+# ============================================================================
+# Skill frontmatter
+# ============================================================================
+
+mutable struct SkillFrontmatter
+    name::Union{String, Nothing}
+    description::Union{String, Nothing}
+    disable_model_invocation::Union{Bool, Nothing}
+    extra::Dict{String, Any}
+end
+
+# ============================================================================
+# Format skill invocation
+# ============================================================================
+
+function formatSkillInvocation(skill::Skill, additional_instructions::Union{String, Nothing})::String
+    skill_block = "<skill name=\"$(skill.name)\" location=\"$(skill.filePath)\">\nReferences are relative to $(dirnameEnvPath(skill.filePath)).\n\n$(skill.content)\n</skill>"
+    if isnothing(additional_instructions)
+        return skill_block
+    end
+    return "$(skill_block)\n\n$(additional_instructions)"
+end
+
+# ============================================================================
+# Load skills from directories
+# ============================================================================
+
+function loadSkills(env::ExecutionEnv, dirs::Union{String, Vector{String}})::Tuple{Vector{Skill}, Vector{SkillDiagnostic}}
+    skills::Vector{Skill} = Skill[]
+    diagnostics::Vector{SkillDiagnostic} = SkillDiagnostic[]
+
+    dir_list = if dirs isa String
+        [dirs]
+    else
+        dirs
+    end
+
+    for dir in dir_list
+        root_info_result = fileInfo(env, dir, nothing)
+        if !root_info_result.ok
+            if root_info_result.error.code != "not_found"
+                push!(diagnostics, SkillDiagnostic(
+                    "warning",
+                    "file_info_failed",
+                    root_info_result.error.message,
+                    dir,
+                ))
+            end
+            continue
+        end
+
+        root_info = root_info_result.value
+        if !isDirectory(env, root_info, diagnostics)
+            continue
+        end
+
+        result = loadSkillsFromDirInternal(env, root_info.path, true, Dict{String, Any}(), root_info.path)
+        append!(skills, result.skills)
+        append!(diagnostics, result.diagnostics)
+    end
+
+    return skills, diagnostics
+end
+
+function isDirectory(env::ExecutionEnv, info::FileInfo, diagnostics::Vector{SkillDiagnostic})::Bool
+    return info.kind == "directory"
+end
+
+function loadSkillsFromDirInternal(
+    env::ExecutionEnv,
+    dir::String,
+    include_root_files::Bool,
+    ignore_matcher::Dict{String, Any},
+    root_dir::String,
+)::Tuple{Vector{Skill}, Vector{SkillDiagnostic}}
+    skills::Vector{Skill} = Skill[]
+    diagnostics::Vector{SkillDiagnostic} = SkillDiagnostic[]
+
+    dir_info_result = fileInfo(env, dir, nothing)
+    if !dir_info_result.ok
+        if dir_info_result.error.code != "not_found"
+            push!(diagnostics, SkillDiagnostic(
+                "warning",
+                "file_info_failed",
+                dir_info_result.error.message,
+                dir,
+            ))
+        end
+        return skills, diagnostics
+    end
+
+    dir_info = dir_info_result.value
+    if !isDirectory(env, dir_info, diagnostics)
+        return skills, diagnostics
+    end
+
+    # TODO: Implement ignore rules
+    # await addIgnoreRules(env, ignoreMatcher, dir, rootDir, diagnostics);
+
+    entries_result = listDir(env, dir, nothing)
+    if !entries_result.ok
+        push!(diagnostics, SkillDiagnostic(
+            "warning",
+            "list_failed",
+            entries_result.error.message,
+            dir,
+        ))
+        return skills, diagnostics
+    end
+
+    entries = entries_result.value
+
+    # Look for SKILL.md
+    for entry in entries
+        if entry.name != "SKILL.md"
+            continue
+        end
+
+        full_path = entry.path
+        if !isFile(env, entry, diagnostics)
+            continue
+        end
+
+        result = loadSkillFromFile(env, full_path)
+        if !isnothing(result.skill)
+            push!(skills, result.skill)
+        end
+        append!(diagnostics, result.diagnostics)
+        return skills, diagnostics
+    end
+
+    # Process other files
+    for entry in sort(entries, by=e -> e.name)
+        if startswith(entry.name, ".") || entry.name == "node_modules"
+            continue
+        end
+
+        full_path = entry.path
+        kind = getFileKind(env, entry, diagnostics)
+        if isnothing(kind)
+            continue
+        end
+
+        rel_path = relativeEnvPath(root_dir, full_path)
+        ignore_path = kind == "directory" ? "$(rel_path)/" : rel_path
+
+        if !isnothing(ignore_matcher) && haskey(ignore_matcher, ignore_path)
+            continue
+        end
+
+        if kind == "directory"
+            result = loadSkillsFromDirInternal(env, full_path, false, ignore_matcher, root_dir)
+            append!(skills, result.skills)
+            append!(diagnostics, result.diagnostics)
+            continue
+        end
+
+        if kind != "file" || !include_root_files || !endswith(entry.name, ".md")
+            continue
+        end
+
+        result = loadSkillFromFile(env, full_path)
+        if !isnothing(result.skill)
+            push!(skills, result.skill)
+        end
+        append!(diagnostics, result.diagnostics)
+    end
+
+    return skills, diagnostics
+end
+
+function isFile(env::ExecutionEnv, info::FileInfo, diagnostics::Vector{SkillDiagnostic})::Bool
+    return info.kind == "file"
+end
+
+function getFileKind(env::ExecutionEnv, info::FileInfo, diagnostics::Vector{SkillDiagnostic})::Union{String, Nothing}
+    if info.kind == "file" || info.kind == "directory"
+        return info.kind
+    end
+
+    canonical_path = canonicalPath(env, info.path, nothing)
+    if !canonical_path.ok
+        if canonical_path.error.code != "not_found"
+            push!(diagnostics, SkillDiagnostic(
+                "warning",
+                "file_info_failed",
+                canonical_path.error.message,
+                info.path,
+            ))
+        end
+        return nothing
+    end
+
+    target = fileInfo(env, canonical_path.value, nothing)
+    if !target.ok
+        if target.error.code != "not_found"
+            push!(diagnostics, SkillDiagnostic(
+                "warning",
+                "file_info_failed",
+                target.error.message,
+                info.path,
+            ))
+        end
+        return nothing
+    end
+
+    if target.value.kind == "file" || target.value.kind == "directory"
+        return target.value.kind
+    end
+
+    return nothing
+end
+
+# ============================================================================
+# Load skill from file
+# ============================================================================
+
+function loadSkillFromFile(env::ExecutionEnv, file_path::String)::Tuple{Union{Skill, Nothing}, Vector{SkillDiagnostic}}
+    diagnostics::Vector{SkillDiagnostic} = SkillDiagnostic[]
+
+    raw_content = readTextFile(env, file_path, nothing)
+    if !raw_content.ok
+        push!(diagnostics, SkillDiagnostic(
+            "warning",
+            "read_failed",
+            raw_content.error.message,
+            file_path,
+        ))
+        return nothing, diagnostics
+    end
+
+    # TODO: Parse frontmatter
+    # parsed = parseFrontmatter<SkillFrontmatter>(rawContent.value);
+    # if !parsed.ok {
+    #     diagnostics.push({ type: "warning", code: "parse_failed", message: parsed.error.message, path: filePath });
+    #     return { skill: null, diagnostics };
+    # }
+
+    # const { frontmatter, body } = parsed.value;
+    # const skillDir = dirnameEnvPath(filePath);
+    # const parentDirName = basenameEnvPath(skillDir);
+    # const description = typeof frontmatter.description === "string" ? frontmatter.description : undefined;
+
+    # for (const error of validateDescription(description)) {
+    #     diagnostics.push({ type: "warning", code: "invalid_metadata", message: error, path: filePath });
+    # }
+
+    # const frontmatterName = typeof frontmatter.name === "string" ? frontmatter.name : undefined;
+    # const name = frontmatterName || parentDirName;
+    # for (const error of validateName(name, parentDirName)) {
+    #     diagnostics.push({ type: "warning", code: "invalid_metadata", message: error, path: filePath });
+    # }
+
+    # if (!description || description.trim() === "") {
+    #     return { skill: null, diagnostics };
+    # }
+
+    # return {
+    #     skill: {
+    #         name,
+    #         description,
+    #         content: body,
+    #         filePath,
+    #         disableModelInvocation: frontmatter["disable-model-invocation"] === true,
+    #     },
+    #     diagnostics,
+    # };
+
+    return nothing, diagnostics
+end
+
+# ============================================================================
+# Path utility functions
+# ============================================================================
+
+function joinEnvPath(base::String, child::String)::String
+    return "$(rtrim(base, '/'))/$(ltrim(child, '/'))"
+end
+
+function dirnameEnvPath(path::String)::String
+    normalized = rtrim(path, '/')
+    slash_index = findlast('/', normalized)
+    if isnothing(slash_index) || slash_index <= 1
+        return "/"
+    end
+    return normalized[1:slash_index-1]
+end
+
+function basenameEnvPath(path::String)::String
+    normalized = rtrim(path, '/')
+    slash_index = findlast('/', normalized)
+    if isnothing(slash_index)
+        return normalized
+    end
+    return normalized[slash_index+1:end]
+end
+
+function relativeEnvPath(root::String, path::String)::String
+    normalized_root = rtrim(root, '/')
+    normalized_path = rtrim(path, '/')
+    
+    if normalized_path == normalized_root
+        return ""
+    end
+
+    if startswith(normalized_path, "$(normalized_root)/")
+        return normalized_path[length(normalized_root)+2:end]
+    end
+
+    return lstrip(normalized_path, '/')
+end
+
+# ============================================================================
+# Helper functions
+# ============================================================================
+
+function lstrip(s::String, chars::String)::String
+    idx = 1
+    while idx <= length(s) && s[idx] in chars
+        idx += 1
+    end
+    return s[idx:end]
+end
+
+function rtrim(s::String, chars::String)::String
+    idx = length(s)
+    while idx >= 1 && s[idx] in chars
+        idx -= 1
+    end
+    return s[1:idx]
+end
+
+function findlast(pattern::Char, s::String)::Union{Int64, Nothing}
+    for i in length(s):-1:1
+        if s[i] == pattern
+            return i
+        end
+    end
+    return nothing
+end
+
+end

src/stream_fn.jl

@@ -0,0 +1,45 @@
+"""
+    stream_fn.jl - Stream function utilities
+
+This module provides the default stream function configuration for AgentCore.
+"""
+
+module StreamFn
+
+using ..Types: StreamFn
+
+let default_stream_fn::Union{StreamFn, Nothing} = nothing
+
+"""
+    setDefaultStreamFn(stream_fn)
+
+Configure the fallback used by Agent and low-level loops when callers omit stream_fn.
+
+# Arguments
+- `stream_fn`: The stream function to set as default
+"""
+function setDefaultStreamFn(stream_fn::Union{StreamFn, Nothing})
+    global default_stream_fn = stream_fn
+end
+
+"""
+    getDefaultStreamFn()
+
+Get the configured default stream function, or throw an error if none is configured.
+
+# Returns
+- The configured stream function
+
+# Throws
+- ErrorException if no default stream function is configured
+"""
+function getDefaultStreamFn()::StreamFn
+    if isnothing(default_stream_fn)
+        throw(ErrorException(
+            "No default stream function configured. Pass stream_fn explicitly or call setDefaultStreamFn()."
+        ))
+    end
+    return default_stream_fn
+end
+
+end

src/system_prompt.jl

@@ -0,0 +1,56 @@
+"""
+    system_prompt.jl - System prompt formatting
+
+This module provides utilities for formatting skills in the system prompt.
+"""
+
+module SystemPrompt
+
+using ..Types: Skill
+
+"""
+    formatSkillsForSystemPrompt(skills)
+
+Format skills for inclusion in the system prompt using XML-formatted blocks.
+"""
+function formatSkillsForSystemPrompt(skills::Vector{Skill})::String
+    visible_skills = filter(s -> !s.disableModelInvocation, skills)
+    if isempty(visible_skills)
+        return ""
+    end
+
+    lines = String[
+        "The following skills provide specialized instructions for specific tasks.",
+        "Read the full skill file when the task matches its description.",
+        "When a skill file references a relative path, resolve it against the skill directory (parent of SKILL.md / dirname of the path) and use that absolute path in tool commands.",
+        "",
+        "<available_skills>",
+    ]
+
+    for skill in visible_skills
+        push!(lines, "  <skill>")
+        push!(lines, "    <name>$(escapeXml(skill.name))</name>")
+        push!(lines, "    <description>$(escapeXml(skill.description))</description>")
+        push!(lines, "    <location>$(escapeXml(skill.filePath))</location>")
+        push!(lines, "  </skill>")
+    end
+
+    push!(lines, "</available_skills>")
+    return join(lines, "\n")
+end
+
+"""
+    escapeXml(value)
+
+Escape special characters in a string for XML.
+"""
+function escapeXml(value::String)::String
+    result = replace(value, "&" => "&amp;")
+    result = replace(result, "<" => "&lt;")
+    result = replace(result, ">" => "&gt;")
+    result = replace(result, "\"" => "&quot;")
+    result = replace(result, "'" => "&apos;")
+    return result
+end
+
+end

src/tools/README.md

@@ -1,568 +0,0 @@
-# Tools
-
-Tools allow the agent to perform actions and fetch data. Each tool defines a **schema** (what arguments it accepts) and an **execution function** (what it does).
-
-## Quick Start
-
-Add a new tool by creating a `.jl` file in `src/tools/`. The file must define a `getTool()` function that returns an `agentTool`:
-
-```julia
-# src/tools/my_tool.jl
-
-function executeTool(toolCallId::String, args::Dict{String,Any}, signal::Union{Nothing,abortSignal}, onPartialResult::Function)::agentToolResult
-    city = args["city"]
-    return agentToolResult(
-        [textContent("Hello from $(city)!")],
-        Dict{Any,Any}(), nothing, false
-    )
-end
-
-function getTool()::agentTool
-    return agentTool(
-        name = "my_tool",
-        label = "My Tool",
-        description = "Says hello to a city.",
-        inputSchema = Dict{String,Any}(
-            "type" => "object",
-            "properties" => Dict(
-                "city" => Dict("type" => "string", "description" => "City name")
-            ),
-            "required" => ["city"]
-        ),
-        execute = executeTool,
-        prepareArguments = nothing,
-        validateRequiredArgs = nothing,
-        parallelToolExecute = false
-    )
-end
-```
-
-When `loadTools()` or `registerTool()` is called, the tool becomes available to the agent.
-
-## Tool Anatomy
-
-Each tool has 3 main parts:
-
-### 1. Schema (`inputSchema`)
-
-JSON Schema (MCP format) describing the tool's arguments. The `"required"` array lists mandatory fields:
-
-```julia
-inputSchema = Dict{String,Any}(
-    "type" => "object",
-    "properties" => Dict(
-        "city" => Dict("type" => "string", "description" => "City name"),
-        "units" => Dict("type" => "string", "enum" => ["celsius", "fahrenheit"], "default" => "celsius")
-    ),
-    "required" => ["city"]
-)
-```
-
-### 2. Execution Function (`execute`)
-
-A function with the signature:
-
-```julia
-execute(toolCallId::String, args::Dict{String,Any}, signal::Union{Nothing,abortSignal}, onPartialResult::Function)::agentToolResult
-```
-
-- **`toolCallId`** — unique ID for this invocation (from the LLM's tool call)
-- **`args`** — validated arguments provided by the LLM
-- **`signal`** — abort signal for cancellable operations
-- **`onPartialResult`** — callback for streaming progress updates
-- **Returns** — `agentToolResult` with content, details, usage, and termination flag
-
-```julia
-function executeTool(toolCallId::String, args::Dict{String,Any}, signal::Union{Nothing,abortSignal}, onPartialResult::Function)::agentToolResult
-    # Optional: stream progress updates
-    onPartialResult(Dict("status" => "Fetching data..."))
-    
-    # Do work
-    result = "Weather in $(args["city"]): Sunny, 22°C"
-    
-    # Return result
-    return agentToolResult(
-        [textContent(result)],
-        Dict{Any,Any}(),  # details
-        nothing,           # usage
-        false              # terminate (true to stop agent loop)
-    )
-end
-```
-
-### 3. Tool Definition (`getTool()`)
-
-Returns an `agentTool` struct:
-
-| Field | Type | Description |
-|---|---|---|
-| `name` | `String` | Unique identifier (e.g. `"getWeather"`) |
-| `label` | `String` | Human-readable name (e.g. `"Weather Lookup"`) |
-| `description` | `String` | What the tool does (shown to the LLM) |
-| `inputSchema` | `Any` | JSON Schema (MCP format) |
-| `execute` | `Function` | The execution function |
-| `prepareArguments` | `Union{Function,Nothing}` | Optional argument transform before validation |
-| `validateRequiredArgs` | `Union{Function,Nothing}` | Optional custom validation |
-| `parallelToolExecute` | `Bool` | Run this tool in parallel with others |
-
-## Argument Validation
-
-Validation happens **before** tool execution, in the `prepareToolCall` phase. Invalid calls return an error immediately without invoking `execute`, `beforeToolCall`, or logging `toolExecutionStart`.
-
-### Default: JSON Schema Required Fields
-
-Set `validateRequiredArgs = nothing` to use the default validator, which checks that all fields in `inputSchema["required"]` are present:
-
-```julia
-# src/tools/getWeather.jl — uses default validation
-function getTool()::agentTool
-    return agentTool(
-        name = "getWeather",
-        # ...
-        validateRequiredArgs = nothing,  # uses default
-    )
-end
-```
-
-### Custom Validation Hook
-
-Override `validateRequiredArgs` when you need:
-- **Cross-field constraints** (e.g. "at least one of X or Y")
-- **Format validation** (e.g. regex patterns, date parsing)
-- **Domain rules** (e.g. value ranges, business logic)
-
-The hook signature takes only `args`:
-
-```julia
-function validateRequiredArgs(args::Dict{String,Any})::Union{Nothing,String}
-    tz = get(args, "timezone", nothing)
-    city = get(args, "city", "")
-    
-    if !haskey(args, "timezone") && isempty(city)
-        return "Missing required argument: provide at least one of 'timezone' or 'city'"
-    end
-    
-    if tz !== nothing
-        tz_str = string(tz)
-        if !occursin(r"^[A-Za-z]+\/[A-Za-z]+(/[A-Za-z]+)*$", tz_str)
-            return "Invalid timezone format: '$tz_str'. Use IANA format, e.g. 'America/New_York'"
-        end
-    end
-    
-    return nothing
-end
-```
-
-Return `nothing` to pass, or an error `String` to fail. The error is fed back to the LLM so it can retry with corrected arguments.
-
-## Tool Lifecycle — Framework Internals
-
-This section traces the full code path from the moment the LLM returns tool calls to the final result being fed back into the conversation. All code references are to `agentCore.jl`.
-
-### Phase 1: Detect Tool Calls in LLM Response
-
-After the LLM returns an `assistantMessage`, the loop at `agentCore.jl:220-244` inspects each `content` block:
-
-```julia
-# agentCore.jl:217-244
-has_tool_calls = false
-tool_call_list = agentToolCall[]
-
-for content_block in response.content
-  if content_block isa Dict
-    # OpenAI-style: type == "tool_calls" with array of tool calls
-    if get(content_block, :type, "") == "tool_calls"
-      for tc_data in get(content_block, :tool_calls, [])
-        tc = agentToolCall(
-          type="function",
-          id=get(tc_data, :id, string(uuid4())),
-          name=get(tc_data, :function, Dict{String,Any}())[:name],
-          arguments=get(tc_data, :function, Dict{String,Any}())[:arguments],
-        )
-        push!(tool_call_list, tc)
-      end
-    # Alternative style: type == "tool_call" single dict per block
-    elseif get(content_block, :type, "") == "tool_call"
-      tc = agentToolCall(
-        type="function",
-        id=get(tc_data, :id, string(uuid4())),
-        name=get(tc_data, :name, ""),
-        arguments=get(tc_data, :arguments, Dict{String,Any}()),
-      )
-      push!(tool_call_list, tc)
-    end
-  end
-end
-```
-
-Each content block with `type == "tool_calls"` or `type == "tool_call"` extracts an `agentToolCall` (id, name, arguments dict) and collects them into a `Vector{agentToolCall}`.
-
-### Phase 2: Dispatch to Sequential or Parallel Execution
-
-At `agentCore.jl:247`, the framework checks if any tool calls exist and decides execution mode:
-
-```julia
-# agentCore.jl:247-265
-context = agentContext(agent._state.systemPrompt, agent._state.messages, agent._state.tools)
-config = agentLoopConfig(
-  agent._state.tools,
-  agent.beforeToolCall,
-  agent.afterToolCall,
-  agent.parallelToolExecute ? "parallel" : "sequential",
-)
-batch = executeToolCalls(context, response, tool_call_list, config, signal, emit)
-```
-
-`executeToolCalls` (`agentCore.jl:988-1015`) checks:
-- `config.toolExecution == "sequential"` → sequential mode
-- Any tool has `parallelToolExecute == false` → sequential mode
-- Otherwise → parallel mode
-
-### Phase 3: Per-Call Preparation (`prepareToolCall`)
-
-Each tool call goes through `prepareToolCall` (`agentCore.jl:511-547`):
-
-```
-1. Look up tool by name: find(t -> t.name == tc.name, context.tools)
-2. If not found → immediateOutcome("Tool X not found", true)
-3. Run tool.prepareArguments (if defined) → transforms raw LLM args
-4. Run validateToolArguments → validateRequiredArgs (hook or default)
-   → if fails → throws ArgumentError → caught below
-5. Run beforeToolCall hook (if defined) → can block execution
-   → if blocked → immediateOutcome("Tool execution was blocked", true)
-6. Return preparedToolCall(tool, tc, validatedArgs)
-```
-
-If any step throws (validation, prepareArguments, beforeToolCall), the catch block at `agentCore.jl:545` converts it to an `immediateOutcome`:
-
-```julia
-catch err
-  return immediateOutcome(createErrorToolResult(sprint(showerror, err)), true)
-end
-```
-
-### Phase 4: Execution (`executePreparedToolCall`)
-
-For each `preparedToolCall`, `executePreparedToolCall` (`agentCore.jl:589-617`) runs:
-
-```julia
-function executePreparedToolCall(prep::preparedToolCall, signal, emit)::executedOutcome
-  updateEvents = promise[]
-  accepting = true
-
-  try
-    result = prep.tool.execute(
-      prep.toolCall.id, prep.args, signal,
-      partialResult -> begin
-        if accepting
-          push!(updateEvents, emit(toolExecUpdateEvent(..., partialResult)))
-        end
-      end
-    )
-    accepting = false
-    wait.(updateEvents)
-    return executedOutcome(result, false)
-  catch err
-    accepting = false
-    wait.(updateEvents)
-    return executedOutcome(createErrorToolResult(sprint(showerror, err)), true)
-  end
-end
-```
-
-Key behaviors:
-- Calls `tool.execute(id, args, signal, onPartialResult)` — your tool's `executeTool` function
-- `signal` can be checked inside `executeTool` for cancellation
-- `onPartialResult` is called for streaming updates, which are emitted as `toolExecutionUpdate` events
-- `accepting` guard prevents emitting updates after the result is already captured
-- `wait.(updateEvents)` ensures all streaming updates are delivered before returning
-- Execution errors are caught and returned as `executedOutcome(isError=true)` — never thrown
-
-### Phase 5: Finalization (`finalizeExecutedToolCall`)
-
-After execution, `finalizeExecutedToolCall` (`agentCore.jl:675-706`) runs the `afterToolCall` hook:
-
-```julia
-function finalizeExecutedToolCall(context, assistantMsg, prep, executed, config, signal)::finalizedOutcome
-  result = executed.result
-  isError = executed.isError
-
-  if config.afterToolCall !== nothing
-    try
-      after = config.afterToolCall(afterCtx(assistantMsg, prep.toolCall, prep.args, result, isError, context), signal)
-      if after !== nothing
-        # Hook can mutate: content, details, usage, terminate, isError
-        result = merge(result, dict(...))
-        isError = get(after, :isError, isError)
-      end
-    catch err
-      result = createErrorToolResult(sprint(showerror, err))
-      isError = true
-    end
-  end
-
-  return finalizedOutcome(prep.toolCall, result, isError)
-end
-```
-
-The hook can:
-- Mask sensitive data from result content
-- Normalize usage tracking
-- Flip `terminate: true` based on business logic
-- Wrap errors in friendlier messages for the LLM
-
-If the hook itself throws, the error is caught and converted to an error outcome.
-
-### Phase 6: Emit Events and Create Result Message
-
-Each call emits `toolExecutionEnd`:
-
-```julia
-function emitToolExecutionEnd(finalized::finalizedOutcome, emit::Function)
-  emit(toolExecEndEvent(finalized.toolCall.id, finalized.toolCall.name, finalized.result, finalized.isError))
-end
-```
-
-Then creates the `toolResultMessage` for conversation history (`agentCore.jl:373-379`):
-
-```julia
-function createToolResultMessage(f::finalizedOutcome)::toolResultMessage
-  return toolResultMessage(
-    "toolResult", f.toolCall.id, f.toolCall.name,
-    f.result.content, f.result.details, f.result.usage,
-    get(f.result, :addedToolNames, string[]), f.isError, nowMillis()
-  )
-end
-```
-
-### Phase 7: Batch Assembly and Loop Control
-
-In `executeToolCallsSequential` (`agentCore.jl:795-829`) or `executeToolCallsParallel` (`agentCore.jl:888-936`), all results are collected:
-
-```julia
-messages = toolResultMessage[]
-for finalized in finalizedCalls
-  push!(messages, createToolResultMessage(finalized))
-end
-return agentToolCallBatch(messages, shouldTerminate(finalizedCalls))
-```
-
-`shouldTerminate` (`agentCore.jl:409`) returns `true` only if ALL tools in the batch set `result.terminate == true`. If `false`, the agent loop at `agentCore.jl:176-308` feeds the tool results back to the LLM for another turn.
-
-### Data Flow Summary
-
-```
-response.content (Vector{Any})
-  └── phase 1: parse content blocks
-       └── tool_call_list :: Vector{agentToolCall}
-            └── phase 2: dispatch to sequential/parallel
-                 └── phase 3: prepareToolCall
-                      └── preparedToolCall or immediateOutcome
-                           └── phase 4: executePreparedToolCall
-                                └── executedOutcome
-                                     └── phase 5: finalizeExecutedToolCall
-                                          └── finalizedOutcome
-                                               └── phase 6: createToolResultMessage
-                                                    └── toolResultMessage
-                                                         └── phase 7: agentToolCallBatch
-                                                              └── pushed to agent._state.messages
-                                                                   └── loop back to LLM
-```
-
-## Execution Modes
-
-### Sequential
-
-Tools execute one at a time in order. Required when:
-- Tools have implicit dependencies
-- Tools share state (e.g. writing to the same file)
-- Tools have `parallelToolExecute = false`
-
-Set globally via `agentLoopConfig.toolExecution = "sequential"`, or per-tool via `parallelToolExecute = false`.
-
-### Parallel
-
-Tools execute concurrently when all are independent. Reduces wall-clock time. Set `parallelToolExecute = true` on individual tools, or set `agentLoopConfig.toolExecution = "parallel"`.
-
-## Streaming Partial Results
-
-For long-running tools (API calls, file uploads, training), use `onPartialResult` to stream progress:
-
-```julia
-function executeTool(toolCallId::String, args::Dict{String,Any}, signal::Union{Nothing,abortSignal}, onPartialResult::Function)::agentToolResult
-    onPartialResult(Dict("status" => "Step 1: Fetching data..."))
-    sleep(1)
-    
-    onPartialResult(Dict("status" => "Step 2: Processing..."))
-    sleep(1)
-    
-    return agentToolResult(
-        [textContent("Done!")],
-        Dict{Any,Any}(), nothing, false
-    )
-end
-```
-
-UI listeners and the TUI consume these events in real time via `toolExecutionUpdate`.
-
-## Loading Tools
-
-### Auto-load from Directory
-
-```julia
-using .toolRegistry
-
-tools = loadTools("src/tools")  # scans for *.jl files with getTool()
-```
-
-Files are loaded alphabetically for deterministic registration order.
-
-### Manual Registration
-
-```julia
-tool = getTool()  # from your tool module
-registerTool(tool)
-```
-
-## Using Tools with an Agent
-
-Loading tools only registers them — you must pass them to the `yiemAgent` and provide an `llmCall` function. Here is the complete flow:
-
-```julia
-using .YiemAgent
-using .toolRegistry
-
-# 1. Load tools from the tools directory
-tools = loadTools("src/tools")
-# [toolRegistry] Loading tool from: src/tools/getTime.jl
-# [toolRegistry] Loaded tool: getTime — Time Lookup
-# [toolRegistry] Loading tool from: src/tools/getWeather.jl
-# [toolRegistry] Loaded tool: getWeather — Weather Lookup
-
-# 2. Define your LLM call function
-function my_llm_call(messages::Dict)::assistantMessage
-    # Call your LLM API here (OpenAI, Anthropic, local model, etc.)
-    # Return an assistantMessage with the response content
-    # If the LLM wants to call a tool, include tool_call content blocks
-    ...
-end
-
-# 3. Define your event sink (optional, for logging/debugging)
-function my_event_sink(event)
-    if event isa toolExecStartEvent
-        println("[EVENT] Tool start: $(event.toolName)")
-    elseif event isa toolExecEndEvent
-        status = event.isError ? "ERROR" : "OK"
-        println("[EVENT] Tool end: $(event.toolName) — $status")
-    end
-end
-
-# 4. Create the agent with tools
-agent = yiemAgent(
-    systemPrompt = "You are a helpful assistant that can check weather and time.",
-    model = my_model,
-    tools = tools,                        # pass loaded tools
-    llmCall = my_llm_call,                # your LLM function
-    agentEventSink = my_event_sink,       # event handler
-)
-
-# 5. Send a message and get a response
-run_agent(agent, "What's the weather in Tokyo?")
-response = take_response(agent)
-
-# response.content contains the LLM's reply (with tool results if applicable)
-println(response.content)
-
-# 6. When done, stop the agent
-stop_agent(agent)
-```
-
-### How It Works
-
-1. **User sends a message** via `run_agent(agent, "What's the weather in Tokyo?")`. The message goes into `inputChannel`.
-
-2. **Agent loop** (`_agent_loop`) picks it up, converts it to a `userMessage`, and adds it to `agent._state.messages`.
-
-3. **LLM is called** via `agent.llmCall(formatted_messages)`. The LLM sees the system prompt, conversation history, and the tool definitions in the prompt (via `formatMsgForLLM`).
-
-4. **If the LLM uses a tool**, it returns a response with `tool_call` content blocks. The agent:
-   - Extracts each tool call (name, arguments)
-   - Runs validation (`validateRequiredArgs` or default)
-   - Executes the tool (or returns an error if validation fails)
-   - Feeds the result back as a `toolResultMessage` in the conversation
-
-5. **LLM is called again** with the tool results. This repeats until the LLM returns a text response with no tool calls.
-
-6. **Final response** is sent to `outputChannel` — retrieve it with `take_response(agent)`.
-
-### Minimal Working Example
-
-```julia
-using .YiemAgent
-using .toolRegistry
-
-# Load tools
-tools = loadTools("src/tools")
-
-# Mock LLM that echoes back a tool call, then a text response
-call_count = 0
-function mock_llm_call(messages::Dict)::assistantMessage
-    global call_count += 1
-    if call_count == 1
-        # First call: LLM decides to use getWeather
-        return assistantMessage(
-            content=[
-                Dict("type" => "tool_calls", 
-                     "tool_calls" => [Dict("id" => "call_1", "name" => "getWeather",
-                                           "arguments" => Dict("city" => "Tokyo"))])
-            ],
-            model = "mock",
-            usage = llmUsage(0, 0)
-        )
-    else
-        # Second call: LLM returns text (after tool result)
-        return assistantMessage(
-            content = [textContent("The weather in Tokyo is sunny, 22°C.")],
-            model = "mock",
-            usage = llmUsage(0, 0)
-        )
-    end
-end
-
-# Create agent
-agent = yiemAgent(
-    systemPrompt = "You are a helpful assistant.",
-    tools = tools,
-    llmCall = mock_llm_call,
-    agentEventSink = e -> nothing,  # no events
-)
-
-# Run
-run_agent(agent, "What's the weather in Tokyo?")
-response = take_response(agent)
-
-stop_agent(agent)
-```
-
-## Available Tools
-
-| Tool | Description | Validation |
-|---|---|---|
-| `getWeather` | Fetch weather for a city | Default (JSON Schema required) |
-| `getTime` | Get current time for a timezone or city | Custom (cross-field + format) |
-
-## Example: Error Flow
-
-When the LLM calls a tool with invalid arguments:
-
-```
-User: "What's the weather?"
-  └── LLM: call getWeather() with no arguments
-       └── prepareToolCall → validateRequiredArgs → "Missing required arguments: city"
-            └── immediateOutcome → error tool result
-                 └── LLM sees: "Missing required arguments: city"
-                      └── LLM retries: call getWeather(city="Tokyo")
-                           └── executeTool → "Weather in Tokyo: Sunny, 22°C"
-```
-
-The agent feeds the error back to the LLM as a tool result message, allowing it to self-correct.

src/tools/bash.jl

@@ -0,0 +1,49 @@
+"""
+    tools/bash.jl - Bash execution tool
+
+This module provides the bash execution tool for AgentCore.
+"""
+
+module Bash
+
+using ..Types: *
+
+struct BashExecution
+    command::String
+    cwd::String
+    env::Dict{String, String}
+    inherit_env::Bool
+end
+
+mutable struct BashPrepare{TContext}
+    function::Function
+    context::TContext
+    signal::Union{Any, Nothing}
+end
+
+mutable struct BashToolOptions{TContext}
+    command_prefix::Union{String, Nothing}
+    prepare::Union{BashPrepare{TContext}, Nothing}
+end
+
+mutable struct BashToolDetails
+    truncation::Union{Any, Nothing}
+    full_output_path::Union{String, Nothing}
+end
+
+function createBashTool{TContext}(options::Union{BashToolOptions{TContext}, Nothing}=nothing) where TContext
+    return AgentTool(
+        "bash",
+        "bash",
+        "Execute a bash command in the current working directory.",
+        Dict{String, Any}(),
+        (tool_call_id, params, signal, on_update, context) -> begin
+            # TODO: Implement bash execution
+            return AgentToolResult([TextContent("Command executed successfully")], nothing, nothing, nothing, nothing)
+        end,
+        nothing,
+        nothing,
+    )
+end
+
+end

src/tools/edit.jl

@@ -0,0 +1,32 @@
+"""
+    tools/edit.jl - File edit tool
+
+This module provides the file edit tool for AgentCore.
+"""
+
+module Edit
+
+using ..Types: *
+
+mutable struct EditToolDetails
+    diff::String
+    patch::String
+    first_changed_line::Union{Int64, Nothing}
+end
+
+function createEditTool{TContext}() where TContext
+    return AgentTool(
+        "edit",
+        "edit",
+        "Edit a single file using exact text replacement.",
+        Dict{String, Any}(),
+        (tool_call_id, params, signal, on_update, context) -> begin
+            # TODO: Implement edit execution
+            return AgentToolResult([TextContent("File edited successfully")], nothing, nothing, nothing, nothing)
+        end,
+        nothing,
+        nothing,
+    )
+end
+
+end

src/tools/edit_diff.jl

@@ -0,0 +1,67 @@
+"""
+    tools/edit_diff.jl - Edit diff utilities
+
+This module provides shared diff computation utilities for the edit tool.
+"""
+
+module EditDiff
+
+using ..Types: *
+
+function detectLineEnding(content::String)::String
+    crlf_idx = findfirst("\r\n", content)
+    lf_idx = findfirst("\n", content)
+    if isnothing(lf_idx)
+        return "\n"
+    end
+    if isnothing(crlf_idx)
+        return "\n"
+    end
+    return crlf_idx < lf_idx ? "\r\n" : "\n"
+end
+
+function normalizeToLF(text::String)::String
+    return replace(text, "\r\n" => "\n", "\r" => "\n")
+end
+
+function restoreLineEndings(text::String, ending::String)::String
+    if ending == "\r\n"
+        return replace(text, "\n" => "\r\n")
+    end
+    return text
+end
+
+function normalizeForFuzzyMatch(text::String)::String
+    # TODO: Implement fuzzy matching normalization
+    return text
+end
+
+function splitLinesWithEndings(content::String)::Vector{String}
+    # TODO: Implement line splitting with endings
+    return split(content, "\n")
+end
+
+function applyEditsToNormalizedContent(
+    normalized_content::String,
+    edits::Vector{Any},
+    path::String,
+)::Tuple{String, String}
+    # TODO: Implement edit application
+    return normalized_content, normalized_content
+end
+
+function generateUnifiedPatch(path::String, old_content::String, new_content::String, context_lines::Int64=4)::String
+    # TODO: Implement unified patch generation
+    return ""
+end
+
+function generateDiffString(
+    old_content::String,
+    new_content::String,
+    context_lines::Int64=4,
+)::Tuple{String, Union{Int64, Nothing}}
+    # TODO: Implement diff string generation
+    return "", nothing
+end
+
+end

src/tools/file_mutation_queue.jl

@@ -0,0 +1,59 @@
+"""
+    tools/file_mutation_queue.jl - File mutation queue
+
+This module provides file mutation serialization for safe concurrent file writes.
+"""
+
+module FileMutationQueue
+
+using ..Types: *
+using ..HarnessTypes: ExecutionEnv, getOrThrow, FileError, Result
+
+# ============================================================================
+# Mutation queue state
+# ============================================================================
+
+mutable struct MutationQueueState
+    queues::Dict{String, Any}
+    registration::Any
+end
+
+# Global state
+const states = Dict{ExecutionEnv, MutationQueueState}()
+
+function getState(env::ExecutionEnv)::MutationQueueState
+    if !haskey(states, env)
+        states[env] = MutationQueueState(Dict{String, Any}(), nothing)
+    end
+    return states[env]
+end
+
+# ============================================================================
+# File mutation queue helpers
+# ============================================================================
+
+async function getMutationQueueKey(env::ExecutionEnv, path::String)::String
+    absolute_path = getOrThrow(getOrThrow(absolutePath(env, path), "Failed to get absolute path"))
+    canonical_path = canonicalPath(env, absolute_path, nothing)
+    if canonical_path.ok
+        return canonical_path.value
+    end
+    if canonical_path.error.code in ("not_found", "not_supported")
+        return absolute_path
+    end
+    throw(canonical_path.error)
+end
+
+# ============================================================================
+# Main function - serialize file mutations
+# ============================================================================
+
+function withFileMutationQueue{T}(env::ExecutionEnv, path::String, fn::Function)::T
+    state = getState(env)
+
+    # TODO: Implement proper async queueing
+    # This is a simplified version
+    return fn()
+end
+
+end

src/tools/getTime.jl

@@ -1,89 +0,0 @@
-"""
-Validate required arguments for the getTime tool.
-
-Demonstrates custom validation beyond simple required-field checking:
-- Ensures at least one time source (timezone or city) is provided
-- Validates timezone is in IANA format if specified
-- Validates city name is not empty if specified
-
-# Arguments
-- `args::Dict{String,Any}`: Arguments from the LLM
-
-# Returns
-- `nothing` if validation passes
-- `String` error message if validation fails
-"""
-function validateRequiredArgs(args::Dict{String,Any})::Union{Nothing,String}
-    tz = get(args, "timezone", nothing)
-    city = get(args, "city", "")
-    
-    hasTz = tz !== nothing && !isempty(tz)
-    hasCity = !isempty(city)
-    
-    # At least one of timezone or city is required
-    if !hasTz && !hasCity
-        return "Missing required argument: provide at least one of 'timezone' or 'city'"
-    end
-    
-    # Validate timezone format (IANA tz database: "Continent/City" or "Continent/City/SubCity")
-    if hasTz
-        tz_str = string(tz)
-        if !occursin(r"^[A-Za-z]+\/[A-Za-z]+(/[A-Za-z]+)*$", tz_str)
-            return "Invalid timezone format: '$tz_str'. Use IANA format, e.g. 'America/New_York' or 'Asia/Tokyo'"
-        end
-    end
-    
-    return nothing
-end
-
-"""
-Execute the getTime tool.
-
-# Arguments
-- `toolCallId::String`: Unique identifier for this tool call
-- `args::Dict{String,Any}`: Parsed arguments from the LLM
-- `signal::Union{Nothing,abortSignal}`: Optional abort signal
-- `onPartialResult::Function`: Callback for streaming partial results
-
-# Returns
-- `agentToolResult`: Result content with current time data
-"""
-function executeTool(toolCallId::String, args::Dict{String,Any}, signal::Union{Nothing,abortSignal}, onPartialResult::Function)::agentToolResult
-    tz = get(args, "timezone", nothing)
-    city = get(args, "city", "")
-    
-    # Simulate time lookup — replace with actual timezone API call
-    if tz !== nothing
-        result = "Current time in $(tz): $(now())"
-    else
-        result = "Current time in $(city): $(now())"
-    end
-    
-    return agentToolResult(
-        [textContent(result)],
-        Dict{Any,Any}(), nothing, false
-    )
-end
-
-"""
-Define and return the getTime agentTool.
-"""
-function getTool()::agentTool
-    return agentTool(
-        name = "getTime",
-        label = "Time Lookup",
-        description = "Get current local time for a timezone or city.",
-        inputSchema = Dict{String,Any}(
-            "type" => "object",
-            "properties" => Dict(
-                "timezone" => Dict("type" => "string", "description", "IANA timezone, e.g. 'America/New_York'"),
-                "city" => Dict("type" => "string", "description", "City name as fallback")
-            ),
-            "required" => []
-        ),
-        execute = executeTool,
-        prepareArguments = nothing,
-        validateRequiredArgs = validateRequiredArgs,
-        parallelToolExecute = false
-    )
-end

src/tools/getWeather.jl

@@ -1,52 +0,0 @@
-"""
-Execute the getWeather tool.
-
-# Arguments
-- `toolCallId::String`: Unique identifier for this tool call
-- `args::Dict{String,Any}`: Parsed arguments from the LLM
-- `signal::Union{Nothing,abortSignal}`: Optional abort signal
-- `onPartialResult::Function`: Callback for streaming partial results
-
-# Returns
-- `agentToolResult`: Result content with weather data
-"""
-function executeTool(toolCallId::String, args::Dict{String,Any}, signal::Union{Nothing,abortSignal}, onPartialResult::Function)::agentToolResult
-    city = get(args, "city", "")
-    units = get(args, "units", "celsius")
-    
-    # Simulate weather fetch — replace with actual API call
-    # You can call onPartialResult() here for streaming progress updates:
-    #   onPartialResult(Dict("status" => "Fetching weather data..."))
-    #   onPartialResult(Dict("status" => "Processing..."))
-    
-    temp = units == "fahrenheit" ? "72" : "22"
-    unit_symbol = units == "celsius" ? "°C" : "°F"
-    
-    return agentToolResult(
-        [textContent("Weather in $(city): Sunny, $(temp)$(unit_symbol)")],
-        Dict{Any,Any}(), nothing, false
-    )
-end
-
-"""
-Define and return the getWeather agentTool.
-"""
-function getTool()::agentTool
-    return agentTool(
-        name = "getWeather",
-        label = "Weather Lookup",
-        description = "Fetch current weather and forecast for a given city.",
-        inputSchema = Dict{String,Any}(
-            "type" => "object",
-            "properties" => Dict(
-                "city" => Dict("type" => "string", "description" => "City and country, e.g., 'San Francisco, CA' or 'Tokyo, Japan'"),
-                "units" => Dict("type" => "string", "enum" => ["celsius", "fahrenheit"], "default" => "celsius", "description" => "Temperature scale")
-            ),
-            "required" => ["city"]
-        ),
-        execute = executeTool,  # reference the function defined above
-        prepareArguments = nothing,
-        validateRequiredArgs = nothing,
-        parallelToolExecute = false
-    )
-end

src/tools/image.jl

@@ -0,0 +1,66 @@
+"""
+    tools/image.jl - Image utilities
+
+This module provides image detection and encoding utilities.
+"""
+
+module Image
+
+using ..Types: *
+
+function detectSupportedImageMimeType(buffer::Vector{UInt8})::Union{String, Nothing}
+    if length(buffer) >= 3 && buffer[1:3] == [0xff, 0xd8, 0xff]
+        if buffer[4] == 0xf7
+            return nothing
+        end
+        return "image/jpeg"
+    end
+
+    if length(buffer) >= 8 && buffer[1:8] == [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]
+        return "image/png"
+    end
+
+    if length(buffer) >= 3 && buffer[1:3] == [0x47, 0x49, 0x46]
+        return "image/gif"
+    end
+
+    if length(buffer) >= 12 && buffer[1:4] == [0x52, 0x49, 0x46, 0x46] && buffer[9:12] == [0x57, 0x45, 0x42, 0x50]
+        return "image/webp"
+    end
+
+    if length(buffer) >= 2 && buffer[1:2] == [0x42, 0x4d]
+        return "image/bmp"
+    end
+
+    return nothing
+end
+
+function encodeBase64(bytes::Vector{UInt8})::String
+    alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
+    output = ""
+
+    for i in 1:3:length(bytes)
+        first_byte = i <= length(bytes) ? bytes[i] : 0
+        second_byte = i+1 <= length(bytes) ? bytes[i+1] : 0
+        third_byte = i+2 <= length(bytes) ? bytes[i+2] : 0
+
+        output *= alphabet[first_byte >> 2 + 1]
+        output *= alphabet[(((first_byte & 0x03) << 4) | ((second_byte >> 4) & 0x0f)) + 1]
+        
+        if i+1 <= length(bytes)
+            output *= alphabet[(((second_byte & 0x0f) << 2) | ((third_byte >> 6) & 0x03)) + 1]
+        else
+            output *= "="
+        end
+
+        if i+2 <= length(bytes)
+            output *= alphabet[third_byte & 0x3f + 1]
+        else
+            output *= "="
+        end
+    end
+
+    return output
+end
+
+end

src/tools/index.jl

@@ -0,0 +1,35 @@
+"""
+    tools/index.jl - Tool exports
+
+This module exports all tools.
+"""
+
+module ToolsIndex
+
+using ..Tools.Bash: createBashTool
+using ..Tools.Read: createReadTool
+using ..Tools.Write: createWriteTool
+using ..Tools.Edit: createEditTool
+using ..Tools.Edit: EditToolDetails, EditToolInput
+using ..Tools.Read: ReadToolDetails, ReadToolInput, ReadToolOptions, ReadImageProcessor, ReadImageProcessorResult
+
+export
+    createBashTool,
+    createReadTool,
+    createWriteTool,
+    createEditTool,
+    BashExecution,
+    BashPrepare,
+    BashToolDetails,
+    BashToolInput,
+    BashToolOptions,
+    EditToolDetails,
+    EditToolInput,
+    ReadToolDetails,
+    ReadToolInput,
+    ReadToolOptions,
+    ReadImageProcessor,
+    ReadImageProcessorResult,
+    WriteToolInput
+
+end

src/tools/path_utils.jl

@@ -0,0 +1,44 @@
+"""
+    tools/path_utils.jl - Path resolution utilities
+
+This module provides path resolution utilities for tools.
+"""
+
+module PathUtils
+
+using ..Types: *
+using ..HarnessTypes: ExecutionEnv, getOrThrow, FileError, Result
+
+function normalizeToolPath(path::String)::String
+    normalized = replace(path, r"[\u00A0\u2000-\u200A\u202F\u205F\u3000]" => " ")
+    if startswith(normalized, "@")
+        return normalized[2:end]
+    end
+    return normalized
+end
+
+function resolveToolPath(env::ExecutionEnv, path::String, signal::Union{Any, Nothing}=nothing)::String
+    return getOrThrow(getOrThrow(absolutePath(env, normalizeToolPath(path), signal), "Failed to resolve path"))
+end
+
+function resolveReadToolPath(env::ExecutionEnv, path::String, signal::Union{Any, Nothing}=nothing)::String
+    resolved = getOrThrow(getOrThrow(absolutePath(env, normalizeToolPath(path), signal), "Failed to resolve path"))
+
+    variants = String[
+        resolved,
+        replace(resolved, r" (AM|PM)\."i => " $1."),
+        normalized = replace(resolved, NFC => NFD),
+        replace(resolved, "'" => "\u2019"),
+        replace(replace(resolved, NFC => NFD), "'" => "\u2019"),
+    ]
+
+    for variant in variants
+        if getOrThrow(getOrThrow(exists(env, variant, signal), "Failed to check existence"), "Not found")
+            return variant
+        end
+    end
+
+    return resolved
+end
+
+end

src/tools/read.jl

@@ -0,0 +1,35 @@
+"""
+    tools/read.jl - File read tool
+
+This module provides the file read tool for AgentCore.
+"""
+
+module Read
+
+using ..Types: *
+
+mutable struct ReadToolDetails
+    truncation::Union{Any, Nothing}
+end
+
+mutable struct ReadToolOptions
+    auto_resize_images::Bool
+    image_processor::Union{Any, Nothing}
+end
+
+function createReadTool{TContext}(options::Union{ReadToolOptions, Nothing}=nothing) where TContext
+    return AgentTool(
+        "read",
+        "read",
+        "Read the contents of a file.",
+        Dict{String, Any}(),
+        (tool_call_id, params, signal, on_update, context) -> begin
+            # TODO: Implement read execution
+            return AgentToolResult([TextContent("File read successfully")], nothing, nothing, nothing, nothing)
+        end,
+        nothing,
+        nothing,
+    )
+end
+
+end

src/tools/registry.jl

@@ -1,142 +0,0 @@
-module toolRegistry
-
-export loadTools, registerTool, getTools, listTools, clearTools
-
-using ..type
-
-# Global registry — populated at runtime by loadTools() or registerTool()
-const _registry = Vector{agentTool}()
-
-"""
-Load all tool modules from a directory.
-
-Scans `dir` for `.jl` files. Each file must define a function named
-`getTool()::agentTool`.  Files are sorted alphabetically so tool
-registration order is deterministic.
-
-# Tool file format
-Each `.jl` file defines one function `getTool()` that returns an `agentTool`:
-
-```julia
-# src/tools/getWeather.jl
-function getTool()::agentTool
-    return agentTool(
-        name = "getWeather",
-        label = "Weather Lookup",
-        description = "Fetch current weather and forecast for a given city.",
-        inputSchema = Dict{String,Any}(
-            "type" => "object",
-            "properties" => Dict(
-                "city" => Dict("type" => "string", "description" => "City and country"),
-                "units" => Dict("type" => "string", "enum" => ["celsius", "fahrenheit"], "default" => "celsius")
-            ),
-            "required" => ["city"]
-        ),
-        execute = (toolCallId, args, signal, onPartialResult) -> begin
-            city = args["city"]
-            return agentToolResult(
-                [textContent("Sunny, 22C in $(city)")],
-                Dict{Any,Any}(), nothing, false
-            )
-        end,
-        prepareArguments = nothing,
-        parallelToolExecute = false
-    )
-end
-```
-
-# Arguments
-- `dir::String`: Directory path to scan for `.jl` tool files
-
-# Returns
-- `Vector{agentTool}`: All loaded tools
-
-# Errors
-- Throws `ArgumentError` if a tool file does not define a `getTool` function
-"""
-function loadTools(dir::String)::Vector{agentTool}
-    if !isdir(dir)
-        throw(ArgumentError("Tool directory does not exist: $dir"))
-    end
-
-    tools = agentTool[]
-    jl_files = filter(f -> endswith(f, ".jl"), readdir(dir))
-    sort!(jl_files)
-
-    for filename in jl_files
-        filepath = joinpath(dir, filename)
-        println("[toolRegistry] Loading tool from: $filepath")
-
-        # Include the file in the current module scope so all types resolve
-        # (agentTool, textContent, agentToolResult, etc. are all available)
-        include(filepath)
-
-        # Validate that getTool was defined (include() places it in current module scope)
-        if !isdefined(@__MODULE__, :getTool)
-            throw(ArgumentError(
-                "Tool file $(filepath) does not define a `getTool()` function. " *
-                "Each tool file must define: function getTool()::agentTool ... end"
-            ))
-        end
-
-        # Call getTool() — it runs in current scope where types are visible
-        tool = getTool()
-        if !(tool isa agentTool)
-            throw(ArgumentError(
-                "getTool() in $(filepath) did not return an agentTool instance, got: $(typeof(tool))"
-            ))
-        end
-
-        push!(_registry, tool)
-        push!(tools, tool)
-        println("[toolRegistry] Loaded tool: $(tool.name) — $(tool.label)")
-    end
-
-    return tools
-end
-
-"""
-Register a single agentTool into the global registry.
-
-# Arguments
-- `tool::agentTool`: The tool to register
-
-# Returns
-- `Vector{agentTool}`: Updated registry
-"""
-function registerTool(tool::agentTool)::Vector{agentTool}
-    push!(_registry, tool)
-    println("[toolRegistry] Registered tool: $(tool.name)")
-    return _registry
-end
-
-"""
-Get all registered tools.
-
-# Returns
-- `Vector{agentTool}`: Copy of the registry
-"""
-function getTools()::Vector{agentTool}
-    return deepcopy(_registry)
-end
-
-"""
-List all registered tool names and labels.
-
-# Returns
-- `Vector{Tuple{String,String}}`: Pairs of (name, label)
-"""
-function listTools()::Vector{Tuple{String,String}}
-    return [(t.name, t.label) for t in _registry]
-end
-
-"""
-Clear all registered tools from the global registry.
-"""
-function clearTools()::Nothing
-    empty!(_registry)
-    println("[toolRegistry] Registry cleared")
-    return nothing
-end
-
-end # module

src/tools/write.jl

@@ -0,0 +1,26 @@
+"""
+    tools/write.jl - File write tool
+
+This module provides the file write tool for AgentCore.
+"""
+
+module Write
+
+using ..Types: *
+
+function createWriteTool{TContext}() where TContext
+    return AgentTool(
+        "write",
+        "write",
+        "Write content to a file.",
+        Dict{String, Any}(),
+        (tool_call_id, params, signal, on_update, context) -> begin
+            # TODO: Implement write execution
+            return AgentToolResult([TextContent("File written successfully")], nothing, nothing, nothing, nothing)
+        end,
+        nothing,
+        nothing,
+    )
+end
+
+end

src/type.jl

@@ -1,899 +0,0 @@
- module type
-    export Timestamp,
-           # Abstract types
-           messageContent, agentMessage, agent,
-           # Model types
-           modelCost, llmModel, llmUsage,
-           # Message content types
-           textContent, imageContent,
-           # Message types
-           userMessage, assistantMessage, toolResultMessage,
-            # Tool types
-            agentTool, validateRequiredArgs
-           # Context types
-           agentContext, agentState, agentToolCall, prepareNextTurnContext,
-           # Loop & execution types
-           agentLoopConfig, abortSignal, agentToolResult,
-           assistantMsgCtx, afterCtx,
-           # Event types
-           toolExecStartEvent, toolExecUpdateEvent, toolExecEndEvent,
-           # Agent
-           yiemAgent,
-           # Tool call lifecycle types
-           preparedToolCall, immediateOutcome, executedOutcome, finalizedOutcome,
-           agentToolCallBatch,
-           # Functions (defined elsewhere)
-           run_agent, take_response, follow_up, stop_agent
-
-
-using Dates, UUIDs, DataStructures, JSON, NATS, Base.Threads
-using GeneralUtils
-
-const Timestamp = DateTime
-
-# ------------------------------------------------------------------------------------------------ #
-#                                          LLM model info                                          #
-# ------------------------------------------------------------------------------------------------ #
-
-struct modelCost                              # Model pricing per 1M tokens
-    input::Float64                            # Price per 1M input tokens
-    output::Float64                           # Price per 1M output tokens
-    cache_read::Float64                       # Price per 1M cached read tokens
-    cache_write::Float64                      # Price per 1M cache write tokens
-end
-
-struct llmModel                               # LLM model configuration
-  id::String                                # Unique model identifier
-  name::String                              # Human-readable model name
-  provider::String                          # Provider name (e.g., "anthropic", "openai")
-  baseUrl::String                           # API endpoint base URL
-  reasoning::Bool                           # Whether the model supports chain-of-thought
-  input::Vector{String}                     # Supported input modalities (e.g., "text", "image")
-  cost::modelCost                           # Pricing information
-  contextWindow::Int64                      # Maximum context length in tokens
-  maxTokens::Int64                          # Maximum output tokens per completion
-end
-
-struct llmUsage
-    inputTokens::Int64
-    outputTokens::Int64
-end
-
-
-# ------------------------------------------------------------------------------------------------ #
-#                                       Message content types                                      #
-# ------------------------------------------------------------------------------------------------ #
-
-abstract type messageContent end              # Base type for message content
-
-struct textContent <: messageContent          # Plain text message content
-    text::String                              # The text content
-end
-
-struct imageContent <: messageContent         # Image message content
-    data::String                              # Base64-encoded image data
-    mimeType::String                          # MIME type (e.g., "image/png")
-end
-
-
-# ------------------------------------------------------------------------------------------------ #
-#                                           Message types                                          #
-# ------------------------------------------------------------------------------------------------ #
-abstract type agentMessage end              # Base type for all agent messages
-
-struct userMessage <: agentMessage          # Message from the user
-    role::String                            # Always "user"
-    content::Vector{messageContent}         # Text and/or image content
-    timestamp::Timestamp                    # When the message was sent
-end
-
-"""
-Create a new user message.
-
-# Arguments
-- `role::String`: Always "user"
-- `content::Vector{messageContent}`: Text and/or image content
-- `timestamp::Timestamp`: When the message was sent
-
-# Returns
-- A new `userMessage` instance
-
-# Examples
-```julia
-julia> msg = userMessage(content=[textContent("Hello")])
-userMessage("user", [textContent("Hello")], DateTime(...))
-```
-"""
-function userMessage(; role="user", content=Vector{messageContent}(), timestamp=now())
-    return userMessage(role, content, timestamp)
-end
-
-struct assistantMessage <: agentMessage     # Message from the AI assistant
-    role::String                            # Always "assistant"
-    content::Vector{messageContent}         # Text and/or image content
-    api::String                             # API name used (e.g., "openai")
-    provider::String                        # Provider name (e.g., "anthropic")
-    model::String                           # Model identifier
-    usage::llmUsage                            # Token usage for this message
-    stopReason::String                      # Why generation stopped (e.g., "end_turn")
-    errorMessage::Union{String, Nothing}    # Error if generation failed
-    timestamp::Timestamp                    # When the message was received
-end
-
-"""
-Create a new assistant message.
-
-# Arguments
-- `role::String`: Always "assistant"
-- `content::Vector{messageContent}`: Text and/or image content
-- `api::String`: API name used (e.g., "openai")
-- `provider::String`: Provider name (e.g., "anthropic")
-- `model::String`: Model identifier
-- `usage::llmUsage`: Token usage for this message
-- `stopReason::String`: Why generation stopped (e.g., "end_turn")
-- `errorMessage::Union{String, Nothing}`: Error if generation failed
-- `timestamp::Timestamp`: When the message was received
-
-# Returns
-- A new `assistantMessage` instance
-
-# Examples
-```julia
-julia> msg = assistantMessage(content=[textContent("Hello!")], model="gpt-4")
-assistantMessage("assistant", [textContent("Hello!")], "", "", "gpt-4", ..., "end_turn", nothing, DateTime(...))
-```
-"""
-function assistantMessage(; role="assistant", content=Vector{messageContent}(),
-    api="", provider="", model="", usage=llmUsage(0, 0), stopReason="end_turn",
-    errorMessage=nothing, timestamp=now())
-    return assistantMessage(role, content, api, provider, model, usage, stopReason, errorMessage, timestamp)
-end
-
-struct toolResultMessage <: agentMessage    # Result returned from a tool execution
-    role::String                            # Always "tool"
-    toolCallId::String                      # ID matching the tool call
-    toolName::String                        # Name of the executed tool
-    content::Vector{messageContent}         # Tool output content
-    details::Any                            # Additional tool-specific details
-    usage::Union{llmUsage, Nothing}            # Token usage if applicable
-    addedToolNames::Union{Vector{String}, Nothing}  # Tools added during execution
-    isError::Bool                           # Whether the tool call resulted in an error
-    timestamp::Timestamp                    # When the result was recorded
-end
-
-"""
-Create a new tool result message.
-
-# Arguments
-- `role::String`: Always "tool"
-- `toolCallId::String`: ID matching the tool call
-- `toolName::String`: Name of the executed tool
-- `content::Vector{messageContent}`: Tool output content
-- `details::Any`: Additional tool-specific details
-- `usage::Union{llmUsage, Nothing}`: Token usage if applicable
-- `addedToolNames::Union{Vector{String}, Nothing}`: Tools added during execution
-- `isError::Bool`: Whether the tool call resulted in an error
-- `timestamp::Timestamp`: When the result was recorded
-
-# Returns
-- A new `toolResultMessage` instance
-
-# Examples
-```julia
-julia> msg = toolResultMessage(toolCallId="call_123", toolName="search", content=[textContent("results")])
-toolResultMessage("tool", "call_123", "search", [textContent("results")], nothing, nothing, nothing, false, DateTime(...))
-```
-"""
-function toolResultMessage(; role="tool", toolCallId="", toolName="",
-    content=Vector{messageContent}(), details=nothing, usage=nothing,
-    addedToolNames=nothing, isError=false, timestamp=now())
-    return toolResultMessage(role, toolCallId, toolName, content, details, usage, addedToolNames, isError, timestamp)
-end
-
-# ------------------------------------------------------------------------------------------------ #
-#                                            Tool types                                            #
-# ------------------------------------------------------------------------------------------------ #
-
-"""
-A tool available to the agent.
-
-Maps MCP server tool definitions to an executable Julia tool.
-
-# Arguments
-- `name::String`: Tool identifier (from MCP `name`)
-- `label::String`: Human-readable tool name (from MCP `title`)
-- `description::String`: What the tool does (from MCP `description`)
-- `inputSchema::Any`: Tool parameters schema (from MCP `inputSchema`, JSON Schema format)
-- `execute::Function`: Tool execution function, signature:
-    `execute(toolCallId::String, args::Dict, signal::Union{Nothing,AbortSignal}, onPartialResult::Function)`
-- `prepareArguments::Union{Function, Nothing}`: Optional argument preparation callback
-- `validateRequiredArgs::Union{Function, Nothing}`: Optional validation hook, signature:
-    `validateRequiredArgs(args::Dict) -> Union{Nothing, String}` where `String` is an error message
-- `parallelToolExecute::Bool`: Override: run tool calls sequentially or in parallel
-
-# Returns
-- A new `agentTool` instance
-
-# MCP Tool Example
-```
-{
-  "name": "getWeather",
-  "title": "Weather Lookup",
-  "description": "Fetch current weather and forecast for a given city.",
-  "inputSchema": {
-    "type": "object",
-    "properties": {
-      "city": { "type": "string", "description": "City and state/country" },
-      "units": { "type": "string", "enum": ["celsius", "fahrenheit"], "default": "celsius" }
-    },
-    "required": ["city"]
-  }
-}
-```
-
-# Example
-```julia
-tool = agentTool(
-    name="getWeather",
-    label="Weather Lookup",
-    description="Fetch current weather and forecast for a given city.",
-    inputSchema=Dict(
-        "type" => "object",
-        "properties" => Dict(
-            "city" => Dict("type" => "string", "description" => "City name"),
-            "units" => Dict("type" => "string", "enum" => ["celsius", "fahrenheit"])
-        ),
-        "required" => ["city"]
-    ),
-    execute=(toolCallId, args, signal, onPartialResult) -> begin
-        city = args["city"]
-        return agentToolResult(
-            [textContent("Sunny, 22C in $(city)")],
-            Dict{Any,Any}(), nothing, false
-        )
-    end,
-    prepareArguments = nothing,
-    validateRequiredArgs = nothing,
-    parallelToolExecute = false
-)
-```
-"""
-struct agentTool                              # A tool available to the agent
-    name::String                              # Tool identifier
-    label::String                             # Human-readable tool name
-    description::String                       # What the tool does
-    inputSchema::Any                          # Tool parameters schema (JSON schema, MCP inputSchema format)
-    execute::Function                         # Tool execution function
-    prepareArguments::Union{Function, Nothing}  # Optional argument preparation callback
-    validateRequiredArgs::Union{Function, Nothing}  # Optional validation hook for required args
-    parallelToolExecute::Bool  # Override: run tool calls sequentially or in parallel
-end
-
-
-# ------------------------------------------------------------------------------------------------ #
-#                                           Agent context                                          #
-# ------------------------------------------------------------------------------------------------ #
-
-"""
-Snapshot of the agent's conversation context.
-
-# Arguments
-- `systemPrompt::String`: System prompt for the agent
-- `messages::Vector{agentMessage}`: Conversation messages
-- `tools::Union{Vector{agentTool}, Nothing}`: Available tools
-
-# Returns
-- A new `agentContext` instance
-"""
-struct agentContext                           # Snapshot of the agent's conversation context
-    systemPrompt::String                      # System prompt for the agent
-    messages::Vector{agentMessage}            # Conversation messages
-    tools::Union{Vector{agentTool}, Nothing}  # Available tools
-end
-
-
-# ------------------------------------------------------------------------------------------------ #
-#                                            Agent state                                           #
-# ------------------------------------------------------------------------------------------------ #
-
-mutable struct agentState                     # Mutable runtime state of an agent
-    systemPrompt::String                      # System prompt text
-    model::llmModel                           # LLM model to use
-    tools::Vector{agentTool}                  # Available tools
-
-    # messages history includes userMessage, assistantMessage, toolResultMessage. NO system prompt
-    messages::Vector{agentMessage}            
-    
-    pendingToolCalls::Vector{String}          # Tool call IDs waiting for results
-    activeRun::Bool                             # is agent processing user message?
-    errorMessage::Union{String, Nothing}      # Last error message
-end
-
-"""
-Create a new mutable agent state.
-
-Creates a deep copy of the provided tools and messages to isolate the
-new state from external references.
-
-# Arguments
-- `systemPrompt::String`: System prompt text
-- `model::llmModel`: LLM model to use (defaults to an unknown model)
-- `tools::Vector{agentTool}`: Available tools (deep copied)
-- `messages::Vector{agentMessage}`: Conversation messages (deep copied)
-
-# Returns
-- A new `agentState` instance with an empty pending tool calls list and no error
-
-# Examples
-```julia
-julia> state = agentState(systemPrompt="You are a helpful assistant")
-agentState("You are a helpful assistant", ..., agentTool[], agentMessage[], String[], nothing)
-```
-"""
-function agentState(
-    systemPrompt::String="",
-    model::llmModel=llmModel{String}("", "", "unknown", "unknown", "", false, String[], modelCost(0.0, 0.0, 0.0, 0.0), 0, 0),
-    tools::Vector{agentTool}=agentTool[],
-    messages::Vector{agentMessage}=agentMessage[],
-)
-    agentState(
-      systemPrompt,
-      model,
-      deepcopy(tools),
-      deepcopy(messages),
-      Vector{String}(),
-      false,
-      nothing,
-    )
-end
-
-
-struct agentToolCall               # A tool invocation from the LLM
-    type::String                              # Always "function"
-    id::String                                # Unique tool call identifier
-    name::String                              # Tool name
-    arguments::Dict{String, Any}              # Parsed tool arguments
-end
-
-
-"""
-Context for preparing the next conversation turn.
-
-# Arguments
-- `message::assistantMessage`: The assistant's message that just completed
-- `toolResults::Vector{toolResultMessage}`: Tool results from this turn
-- `context::agentContext`: Current conversation context
-- `newMessages::Vector{agentMessage}`: Messages to append to the context
-
-# Returns
-- A new `prepareNextTurnContext` instance
-"""
-struct prepareNextTurnContext                        # Context for preparing the next conversation turn
-    message::assistantMessage                 # The assistant's message that just completed
-    toolResults::Vector{toolResultMessage}    # Tool results from this turn
-    context::agentContext                     # Current conversation context
-    newMessages::Vector{agentMessage}         # Messages to append to the context
-end
-
-
-# ------------------------------------------------------------------------------------------------ #
-#                          Agent loop configuration & tool execution types                         #
-# ------------------------------------------------------------------------------------------------ #
-
-"""
-Configuration for the agent tool execution loop.
-
-# Arguments
-- `tools::Vector{agentTool}`: Available tools
-- `beforeToolCall::Union{Function, Nothing}`: Callback before tool execution
-- `afterToolCall::Union{Function, Nothing}`: Callback after tool execution
-- `toolExecution::String`: Execution mode — "sequential" or "parallel"
-"""
-struct agentLoopConfig
-    tools::Vector{agentTool}
-    beforeToolCall::Union{Function, Nothing}
-    afterToolCall::Union{Function, Nothing}
-    toolExecution::String
-end
-
-"""
-Signal for aborting ongoing operations.
-
-# Arguments
-- `aborted::Bool`: Whether the operation has been aborted
-"""
-struct abortSignal
-    aborted::Bool
-end
-
-"""
-Result returned by tool execution before the `afterToolCall` hook.
-
-# Arguments
-- `content::Vector{messageContent}`: Tool output content
-- `details::Dict{Any,Any}`: Tool-specific details
-- `usage::Union{llmUsage, Nothing}`: Token usage if applicable
-- `terminate::Bool`: Whether tool requests termination of the agent loop
-"""
-struct agentToolResult
-    content::Vector{messageContent}
-    details::Dict{Any,Any}
-    usage::Union{llmUsage, Nothing}
-    terminate::Bool
-end
-
-"""
-Context passed to the `beforeToolCall` hook.
-
-# Arguments
-- `message::assistantMessage`: The assistant message containing the tool call
-- `toolCall::agentToolCall`: The tool call being prepared
-- `args::Dict{String,Any}`: Validated tool arguments
-- `context::agentContext`: Current conversation context
-"""
-struct assistantMsgCtx
-    message::assistantMessage
-    toolCall::agentToolCall
-    args::Dict{String,Any}
-    context::agentContext
-end
-
-"""
-Context passed to the `afterToolCall` hook.
-
-# Arguments
-- `message::assistantMessage`: The assistant message containing the tool call
-- `toolCall::agentToolCall`: The tool call that was executed
-- `args::Dict{String,Any}`: Tool arguments
-- `result::agentToolResult`: The raw tool result
-- `isError::Bool`: Whether execution resulted in an error
-- `context::agentContext`: Current conversation context
-"""
-struct afterCtx
-    message::assistantMessage
-    toolCall::agentToolCall
-    args::Dict{String,Any}
-    result::agentToolResult
-    isError::Bool
-    context::agentContext
-end
-
-"""
-Event emitted when a tool call execution starts.
-
-# Arguments
-- `toolCallId::String`: ID of the tool call
-- `toolName::String`: Name of the tool
-- `arguments::Dict{String,Any}`: Tool arguments
-"""
-struct toolExecStartEvent
-    toolCallId::String
-    toolName::String
-    arguments::Dict{String,Any}
-end
-
-"""
-Event emitted with partial results during tool execution.
-
-# Arguments
-- `toolCallId::String`: ID of the tool call
-- `toolName::String`: Name of the tool
-- `arguments::Dict{String,Any}`: Tool arguments
-- `partialResult::Any`: The partial result data
-"""
-struct toolExecUpdateEvent
-    toolCallId::String
-    toolName::String
-    arguments::Dict{String,Any}
-    partialResult::Any
-end
-
-"""
-Event emitted when a tool call execution ends.
-
-# Arguments
-- `toolCallId::String`: ID of the tool call
-- `toolName::String`: Name of the tool
-- `result::agentToolResult`: The final tool result
-- `isError::Bool`: Whether execution resulted in an error
-"""
-struct toolExecEndEvent
-    toolCallId::String
-    toolName::String
-    result::agentToolResult
-    isError::Bool
-end
-
-# ------------------------------------------------------------------------------------------------ #
-#                                           Agent struct                                           #
-# ------------------------------------------------------------------------------------------------ #
-
-abstract type agent end 
-
-"""
-docstring
-"""
-mutable struct yiemAgent <: agent             # High-level agent wrapper
-  _state::agentState                          # Current state (prompt, model, messages, tools, etc.)
-  
-  # user sends prompt message to agent. if agent is idle, it process user message right away. 
-  # if agent is running, it process user message after  the current tool call finished.
-  inputChannel::Channel                        
-                                              
-  # Buffers messages the user sends while the agent is busy. Processed after all inputChannel 
-  # messages are handled and the agent is idle (not using a tool call).
-  followUpChannel::Channel                    
-
-  # agent sends response message to user after processing all user messages in inputChannel 
-  # and all followUp messages.
-  outputChannel::Channel                     
-                                              
-  _agent_loop::Union{Task, Nothing}                 # agent loop running in the background
-
-  # Preprocess/transform messages and context (modify, filter, prune, inject context from memory,  
-  # reorder, ...) for a single LLM call in _process_message()'s loop.
-  # returns new Vector{agentMessage}
-  prepareContext ::Union{Function, Nothing}   
-
-  # Convert prepareContext()'s new Vector{agentMessage} to LLM message format
-  formatMsgForLLM::Function    
-
-  # Actually invoke the LLM to get a completion response. The LLM response comes back as an 
-  # assistantMessage whose content is an array of content blocks. 
-  # Each block has a type — "text", "thinking", or "toolCall". 
-  # The code filters for type === "toolCall" blocks, then passes them to executeToolCalls().
-  llmCall::Function                           
-                                              
-  # Callback invoked before executing a tool call (ask for user permission/confirmation/abort, etc..)
-  beforeToolCall::Union{Function, Nothing}    
-
-  executeToolCalls::Function                  # execute tool calls ()
-  
-  # Callback invoked after executing a tool call to sanitize tools output so the output is ready 
-  # to be converted into toolResults message
-  afterToolCall::Union{Function, Nothing}     
-  # prepareNextTurn::Union{Function, Nothing}   # Callback to prepare the next conversation turn
-  # prepareNextTurnWithContext::Union{Function, Nothing}  # Same but receives context
-  sessionId::Union{String, Nothing}           # Optional session identifier
-  maxRetryDelayMs::Union{Int64, Nothing}      # Maximum delay between retries (ms)
-  parallelToolExecute::Bool                   # Default: false
-  agentEventSink::Function                    # agent emits its status via this function
-end
-
-"""
-Create a new yiemAgent instance with a background loop task.
-
-Spawns a background `@spawn` task that runs the agent loop, listening
-on `inputChannel` and `followUpChannel` channels concurrently.
-
-# Keyword Arguments
-- `systemPrompt::String`: System prompt for the agent
-- `model`: LLM model to use
-- `tools::Vector{agentTool}`: Available tools (default: empty)
-- `messages::Vector{agentMessage}`: Initial conversation messages (default: empty)
-- `formatMsgForLLM::Function`: Convert agent messages to LLM message format (default: `defaultformatMsgForLLM`)
-- `llmCall::Function`: Function to invoke the LLM (required)
-- `prepareContext::Union{Function, Nothing}`: Preprocess/transform messages before sending to LLM (default: `nothing`)
-- `beforeToolCall::Union{Function, Nothing}`: Callback invoked before executing a tool call (default: `nothing`)
-- `afterToolCall::Union{Function, Nothing}`: Callback invoked after executing a tool call (default: `nothing`)
-- `prepareNextTurn::Union{Function, Nothing}`: Callback to prepare the next conversation turn (default: `nothing`)
-- `prepareNextTurnWithContext::Union{Function, Nothing}`: Same but receives context (default: `nothing`)
-- `sessionId::Union{String, Nothing}`: Optional session identifier (default: `nothing`)
-- `maxRetryDelayMs::Union{Int64, Nothing}`: Maximum delay between retries in milliseconds (default: `nothing`)
-- `parallelToolExecute::Bool`: Run tool calls in parallel (default: `false`)
-- `agentEventSink::Function`: Callback to receive agent events
-
-# Returns
-- A new `yiemAgent` instance with an active background task
-
-# Examples
-```julia
-julia> agent = yiemAgent(systemPrompt="You are a helpful assistant", model=my_model)
-yiemAgent(agentState(...), Channel(...), Channel(...), Channel(...), ..., ...)
-```
-"""
-function yiemAgent(
-  ; systemPrompt::String="You are helpful assistant.",
-    model=nothing,
-    tools::Vector{agentTool}=agentTool[],
-    messages::Vector{agentMessage}=agentMessage[],
-    prepareContext::Union{Function, Nothing}=nothing,
-    formatMsgForLLM::Function=defaultformatMsgForLLM,
-    llmCall::Function,
-    beforeToolCall::Union{Function, Nothing}=nothing,
-    afterToolCall::Union{Function, Nothing}=nothing,
-    # prepareNextTurn::Union{Function, Nothing}=nothing,
-    # prepareNextTurnWithContext::Union{Function, Nothing}=nothing,
-    sessionId::Union{String, Nothing}=nothing,
-    maxRetryDelayMs::Union{Int64, Nothing}=nothing,
-    parallelToolExecute::Bool=false,
-    agentEventSink::Function,
-  )
-  # Create channels: input (user -> agent), followUp (async queue), output (agent -> user)
-  inputChannel    = Channel(16)
-  followUp    = Channel(32)
-  outputChannel   = Channel(16)
-
-  # Create struct with a placeholder task, then spawn and replace it
-  agent = yiemAgent(
-    agentState(systemPrompt, model, tools, messages),
-    inputChannel,
-    followUp,
-    outputChannel,
-    nothing,  # placeholder — replaced below
-    prepareContext,
-    formatMsgForLLM,
-    llmCall,
-    beforeToolCall,
-    afterToolCall,
-    # prepareNextTurn,
-    # prepareNextTurnWithContext,
-    sessionId,
-    maxRetryDelayMs,
-    parallelToolExecute,
-    agentEventSink,
-  )
-
-  # Spawn the background loop and attach it
-  agent._agent_loop = @spawn _agent_loop(agent)
-
-  return agent
-end
-
-
-
-"""
-    preparedToolCall(tool, toolCall, args)
-
-Intermediate state between tool call validation and execution.
-Created after `prepareToolCall` succeeds; serves as the bridge to
-the execution phase. Keeping the resolved tool, original call
-metadata, and validated args together avoids repeated lookups and
-allows the execution phase to access all necessary data without
-carrying the full context through the call chain.
-
-# Fields
-- `tool::agentTool`: The resolved tool definition from the context
-- `toolCall::agentToolCall`: The original tool call from the assistant
-- `args::any`: Validated (and coerced) argument values
-
-# Examples
-```julia
-# After prepareToolCall succeeds, the agent holds a preparedToolCall
-    prep = preparedToolCall(
-    tool,  # agentTool found in context.tools
-    toolCall,  # {id: "call_1", name: "search_wine", arguments: "{\"query\": \"red wine\"}"}
-    validatedArgs  # Dict("query" => "red wine")
-)
-```
-"""
-struct preparedToolCall
-    tool::agentTool               # The resolved tool definition from the context
-    toolCall::agentToolCall       # The original tool call from the assistant
-    args::Any                      # Validated (and coerced) argument values
-end
-
-"""
-    immediateOutcome(result, isError)
-
-A tool call that was resolved without actual execution — either
-because the tool was not found, validation failed, or a
-`beforeToolCall` hook blocked the call. The result is produced
-immediately and emitted as a tool result message.
-
-Returning an outcome instead of throwing an exception is intentional:
-it lets the agent feed the error back to the LLM as a tool result so
-the model can recover — for example, by re-issuing a tool call with
-corrected arguments after a validation failure.
-
-# Fields
-- `result::agentToolResult`: The pre-computed tool result
-- `isError::Bool`: Whether this outcome represents an error
-
-# Examples
-```julia
-# Tool not found — immediate error
-immediateOutcome(
-    createErrorToolResult("Tool search_wine not found"),
-    true
-)
-
-# beforeToolCall hook blocked execution
-immediateOutcome(
-    createErrorToolResult("Tool execution was blocked"),
-    true
-)
-```
-"""
-struct immediateOutcome
-    result::agentToolResult         # The pre-computed tool result
-    isError::Bool                  # Whether this outcome represents an error
-end
-
-"""
-    executedOutcome(result, isError)
-
-A tool call that has been executed by `tool.execute()` but has not
-yet been through the `afterToolCall` hook. This intermediate state
-is necessary because the hook may mutate the result (content, usage,
-termination, error status). Keeping execution and finalization
-separate allows the hook to inspect the raw result and decide
-whether to transform it or replace it entirely.
-
-# Fields
-- `result::agentToolResult`: The tool's execution result
-- `isError::Bool`: Whether execution raised an error
-
-# Examples
-```julia
-# Successful execution
-executedOutcome(
-    agentToolResult([textContent("text", "Found 3 wines")], dict{any,any}(), dict{any,any}()),
-    false
-)
-
-# Execution error
-executedOutcome(
-    createErrorToolResult("Connection timeout"),
-    true
-)
-```
-"""
-struct executedOutcome
-    result::agentToolResult         # The tool's execution result
-    isError::Bool                  # Whether execution raised an error
-end
-
-"""
-    finalizedOutcome(toolCall, result, isError)
-
-The complete outcome of a tool call after both execution and the
-`afterToolCall` hook. This is the final form used to construct
-the `toolResultMessage` emitted to the agent loop.
-
-The three-phase design (prepare → execute → finalize) exists so
-that each phase has a single responsibility: preparation handles
-validation and gating, execution performs the actual work, and
-finalization applies post-processing hooks. This separation allows
-the agent loop to emit `toolExecutionEnd` events with the
-finalized data while keeping each phase independently testable
-and swappable.
-
-# Fields
-- `toolCall::agentToolCall`: The original tool call reference
-- `result::agentToolResult`: The final tool result (post-afterToolCall)
-- `isError::Bool`: Whether the call failed or was blocked
-
-# Examples
-```julia
-# Normal successful finalization
-finalizedOutcome(tc, agentToolResult(content, details, usage, false), false)
-
-# afterToolCall mutated result and set terminate
-finalizedOutcome(tc, agentToolResult(content, details, usage, true), false)
-```
-"""
-struct finalizedOutcome
-    toolCall::agentToolCall        # The original tool call reference
-    result::agentToolResult         # The final tool result (post-afterToolCall)
-    isError::Bool                  # Whether the call failed or was blocked
-end
-
-"""
-    agentToolCallBatch(messages, terminate)
-
-A batch of tool result messages from executing one or more tool calls.
-The `terminate` flag indicates whether all tools in the batch
-requested termination, which causes the agent loop to stop
-processing further turns.
-
-This flag is set by the tool implementation (not the end user) to
-signal that the agent should not call the LLM again. Typical use
-cases:
-
-  - Task completion: a tool like `deploy` or `submit` finishes its
-    work and returns `terminate: true` so the agent stops instead
-    of asking the LLM what to do next.
-  - Unrecoverable error: a tool hits a fatal condition (e.g.
-    database connection lost, auth token expired) and returns
-    `terminate: true` so the agent stops with an error message
-    rather than retrying.
-  - Async handoff: a tool triggers a long-running external
-    operation and wants the agent to stop now; the external system
-    will later resume the agent via `continue()`.
-
-If `terminate` is `false` (default), the agent loop feeds the tool
-results back to the LLM for another turn.
-
-# Fields
-- `messages::Vector{toolResultMessage}`: Tool result messages for this batch
-- `terminate::Bool`: Whether the batch should terminate the loop
-
-# Examples
-```julia
-# Batch of 3 tool results, no termination
-agentToolCallBatch(resultMessages, false)
-
-# All tools requested termination
-agentToolCallBatch(resultMessages, true)
-
-# Empty batch — terminate is false regardless
-agentToolCallBatch(toolResultMessage[], false)
-```
-"""
-struct agentToolCallBatch
-    messages::Vector{toolResultMessage}  # Tool result messages for this batch
-    terminate::Bool                      # Whether the batch should terminate the loop
-end
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-end # module type

src/types.jl

@@ -0,0 +1,588 @@
+"""
+    types.jl - Core types for AgentCore
+
+This module defines the fundamental types used throughout the AgentCore package.
+"""
+
+module Types
+
+using Dates
+using UUIDs
+using JSON3
+using Unicode
+
+# ============================================================================
+# Basic type aliases
+# ============================================================================
+
+const Timestamp = Int64
+
+# ============================================================================
+# Thinking level enum
+# ============================================================================
+
+@enum ThinkingLevel begin
+    THINKING_OFF = "off"
+    THINKING_MINIMAL = "minimal"
+    THINKING_LOW = "low"
+    THINKING_MEDIUM = "medium"
+    THINKING_HIGH = "high"
+    THINKING_XHIGH = "xhigh"
+    THINKING_MAX = "max"
+end
+
+# ============================================================================
+# Tool execution modes
+# ============================================================================
+
+@enum ToolExecutionMode begin
+    EXECUTION_SEQUENTIAL = "sequential"
+    EXECUTION_PARALLEL = "parallel"
+end
+
+# ============================================================================
+# Queue drain modes
+# ============================================================================
+
+@enum QueueMode begin
+    QUEUE_ALL = "all"
+    QUEUE_ONE_AT_A_TIME = "one-at-a-time"
+end
+
+# ============================================================================
+# Message content types
+# ============================================================================
+
+abstract type MessageContent end
+
+struct TextContent <: MessageContent
+    text::String
+end
+
+struct ImageContent <: MessageContent
+    data::String
+    mime_type::String
+end
+
+# ============================================================================
+# Message types
+# ============================================================================
+
+abstract type Message end
+
+struct UserMessage <: Message
+    role::String
+    content::Vector{MessageContent}
+    timestamp::Timestamp
+end
+
+struct AssistantMessage <: Message
+    role::String
+    content::Vector{MessageContent}
+    api::String
+    provider::String
+    model::String
+    usage::Usage
+    stop_reason::String
+    error_message::Union{String, Nothing}
+    timestamp::Timestamp
+end
+
+struct ToolResultMessage <: Message
+    role::String
+    tool_call_id::String
+    tool_name::String
+    content::Vector{MessageContent}
+    details::Any
+    usage::Union{Usage, Nothing}
+    added_tool_names::Union{Vector{String}, Nothing}
+    is_error::Bool
+    timestamp::Timestamp
+end
+
+# ============================================================================
+# Usage statistics
+# ============================================================================
+
+struct UsageCost
+    input::Float64
+    output::Float64
+    cache_read::Float64
+    cache_write::Float64
+    total::Float64
+end
+
+struct Usage
+    input::Int64
+    output::Int64
+    cache_read::Int64
+    cache_write::Int64
+    total_tokens::Int64
+    cost::UsageCost
+end
+
+# ============================================================================
+# Model types
+# ============================================================================
+
+struct ModelCost
+    input::Float64
+    output::Float64
+    cache_read::Float64
+    cache_write::Float64
+end
+
+struct Model{Api}
+    id::String
+    name::String
+    api::Api
+    provider::String
+    base_url::String
+    reasoning::Bool
+    input::Vector{String}
+    cost::ModelCost
+    context_window::Int64
+    max_tokens::Int64
+end
+
+# ============================================================================
+# Agent message union type
+# ============================================================================
+
+abstract type AgentMessage end
+
+# Custom message types can extend this via multiple dispatch
+struct CustomMessage <: AgentMessage
+    message::AgentMessage
+    custom_type::String
+end
+
+# ============================================================================
+# Tool types
+# ============================================================================
+
+struct AgentToolResult{T}
+    content::Vector{MessageContent}
+    details::T
+    usage::Union{Usage, Nothing}
+    added_tool_names::Union{Vector{String}, Nothing}
+    terminate::Union{Bool, Nothing}
+end
+
+struct AgentTool{TParameters, TDetails}
+    name::String
+    label::String
+    description::String
+    parameters::TParameters
+    execute::Function
+    prepare_arguments::Union{Function, Nothing}
+    execution_mode::Union{ToolExecutionMode, Nothing}
+end
+
+# ============================================================================
+# Agent context
+# ============================================================================
+
+struct AgentContext
+    system_prompt::String
+    messages::Vector{AgentMessage}
+    tools::Union{Vector{AgentTool}, Nothing}
+end
+
+# ============================================================================
+# Event types
+# ============================================================================
+
+abstract type AgentEvent end
+
+struct AgentStartEvent <: AgentEvent end
+struct AgentEndEvent <: AgentEvent
+    messages::Vector{AgentMessage}
+end
+struct TurnStartEvent <: AgentEvent end
+struct TurnEndEvent <: AgentEvent
+    message::AgentMessage
+    tool_results::Vector{ToolResultMessage}
+end
+struct MessageStartEvent <: AgentEvent
+    message::AgentMessage
+end
+struct MessageUpdateEvent <: AgentEvent
+    message::AgentMessage
+    assistant_message_event::Any
+end
+struct MessageEndEvent <: AgentEvent
+    message::AgentMessage
+end
+struct ToolExecutionStartEvent <: AgentEvent
+    tool_call_id::String
+    tool_name::String
+    args::Any
+end
+struct ToolExecutionUpdateEvent <: AgentEvent
+    tool_call_id::String
+    tool_name::String
+    args::Any
+    partial_result::Any
+end
+struct ToolExecutionEndEvent <: AgentEvent
+    tool_call_id::String
+    tool_name::String
+    result::Any
+    is_error::Bool
+end
+
+# ============================================================================
+# Assistant message event types
+# ============================================================================
+
+abstract type AssistantMessageEvent end
+
+struct StartEvent <: AssistantMessageEvent
+    partial::AssistantMessage
+end
+struct TextStartEvent <: AssistantMessageEvent
+    content_index::Int64
+    partial::AssistantMessage
+end
+struct TextDeltaEvent <: AssistantMessageEvent
+    content_index::Int64
+    delta::String
+    partial::AssistantMessage
+end
+struct TextEndEvent <: AssistantMessageEvent
+    content_index::Int64
+    content::String
+    partial::AssistantMessage
+end
+struct DoneEvent <: AssistantMessageEvent
+    reason::String
+    usage::Usage
+    message::AssistantMessage
+end
+struct ErrorEvent <: AssistantMessageEvent
+    reason::String
+    error_message::Union{String, Nothing}
+    usage::Usage
+    error::AssistantMessage
+end
+
+# ============================================================================
+# Agent state
+# ============================================================================
+
+mutable struct AgentState
+    system_prompt::String
+    model::Model
+    thinking_level::ThinkingLevel
+    tools::Vector{AgentTool}
+    messages::Vector{AgentMessage}
+    is_streaming::Bool
+    streaming_message::Union{AgentMessage, Nothing}
+    pending_tool_calls::Set{String}
+    error_message::Union{String, Nothing}
+
+    function AgentState(
+        system_prompt::String="",
+        model::Model=Model("", "", "unknown", "unknown", "", false, String[], ModelCost(0.0, 0.0, 0.0, 0.0), 0, 0),
+        thinking_level::ThinkingLevel=THINKING_OFF,
+        tools::Vector{AgentTool}=AgentTool[],
+        messages::Vector{AgentMessage}=AgentMessage[],
+    )
+        new(
+            system_prompt,
+            model,
+            thinking_level,
+            copy(tools),
+            copy(messages),
+            false,
+            nothing,
+            Set{String}(),
+            nothing,
+        )
+    end
+end
+
+# ============================================================================
+# Tool call types
+# ============================================================================
+
+struct ToolCall
+    type::String
+    id::String
+    name::String
+    arguments::Dict{String, Any}
+    partial_json::Union{String, Nothing}
+end
+
+# ============================================================================
+# Context transform types
+# ============================================================================
+
+struct PrepareNextTurnContext
+    message::AssistantMessage
+    tool_results::Vector{ToolResultMessage}
+    context::AgentContext
+    new_messages::Vector{AgentMessage}
+end
+
+struct AgentLoopTurnUpdate
+    context::Union{AgentContext, Nothing}
+    model::Union{Model, Nothing}
+    thinking_level::Union{ThinkingLevel, Nothing}
+end
+
+# ============================================================================
+# Before/After tool call types
+# ============================================================================
+
+struct BeforeToolCallContext
+    assistant_message::AssistantMessage
+    tool_call::ToolCall
+    args::Any
+    context::AgentContext
+end
+
+struct BeforeToolCallResult
+    block::Union{Bool, Nothing}
+    reason::Union{String, Nothing}
+end
+
+struct AfterToolCallContext
+    assistant_message::AssistantMessage
+    tool_call::ToolCall
+    args::Any
+    result::AgentToolResult
+    is_error::Bool
+    context::AgentContext
+end
+
+struct AfterToolCallResult
+    content::Union{Vector{MessageContent}, Nothing}
+    details::Union{Any, Nothing}
+    is_error::Union{Bool, Nothing}
+    usage::Union{Usage, Nothing}
+    terminate::Union{Bool, Nothing}
+end
+
+# ============================================================================
+# Stream function signature
+# ============================================================================
+
+const StreamFn = Function
+
+# ============================================================================
+# File types
+# ============================================================================
+
+struct FileKind
+    value::String
+end
+const FILE_KIND_FILE = FileKind("file")
+const FILE_KIND_DIRECTORY = FileKind("directory")
+const FILE_KIND_SYMLINK = FileKind("symlink")
+
+struct FileInfo
+    name::String
+    path::String
+    kind::FileKind
+    size::Int64
+    mtime_ms::Int64
+end
+
+struct FileError <: Exception
+    code::String
+    message::String
+    path::Union{String, Nothing}
+    cause::Union{Exception, Nothing}
+end
+
+struct ExecutionError <: Exception
+    code::String
+    message::String
+    cause::Union{Exception, Nothing}
+end
+
+struct CompactionError <: Exception
+    code::String
+    message::String
+    cause::Union{Exception, Nothing}
+end
+
+struct BranchSummaryError <: Exception
+    code::String
+    message::String
+    cause::Union{Exception, Nothing}
+end
+
+struct SessionError <: Exception
+    code::String
+    message::String
+    cause::Union{Exception, Nothing}
+end
+
+struct AgentHarnessError <: Exception
+    code::String
+    message::String
+    cause::Union{Exception, Nothing}
+end
+
+# ============================================================================
+# Session tree entry types
+# ============================================================================
+
+abstract type SessionTreeEntry end
+
+struct SessionTreeEntryBase
+    type::String
+    id::String
+    parent_id::Union{String, Nothing}
+    timestamp::String
+end
+
+struct MessageEntry <: SessionTreeEntry
+    base::SessionTreeEntryBase
+    message::AgentMessage
+end
+
+struct ThinkingLevelChangeEntry <: SessionTreeEntry
+    base::SessionTreeEntryBase
+    thinking_level::String
+end
+
+struct ModelChangeEntry <: SessionTreeEntry
+    base::SessionTreeEntryBase
+    provider::String
+    model_id::String
+end
+
+struct ActiveToolsChangeEntry <: SessionTreeEntry
+    base::SessionTreeEntryBase
+    active_tool_names::Vector{String}
+end
+
+struct CompactionEntry{T} <: SessionTreeEntry
+    base::SessionTreeEntryBase
+    summary::String
+    first_kept_entry_id::Union{String, Nothing}
+    tokens_before::Int64
+    retained_tail::Union{Vector{AgentMessage}, Nothing}
+    details::Union{T, Nothing}
+    usage::Union{Usage, Nothing}
+    from_hook::Bool
+end
+
+struct BranchSummaryEntry{T} <: SessionTreeEntry
+    base::SessionTreeEntryBase
+    from_id::String
+    summary::String
+    details::Union{T, Nothing}
+    usage::Union{Usage, Nothing}
+    from_hook::Bool
+end
+
+struct CustomEntry{T} <: SessionTreeEntry
+    base::SessionTreeEntryBase
+    custom_type::String
+    data::Union{T, Nothing}
+end
+
+struct CustomMessageEntry{T} <: SessionTreeEntry
+    base::SessionTreeEntryBase
+    custom_type::String
+    content::String
+    details::Union{T, Nothing}
+    display::Bool
+end
+
+struct LabelEntry <: SessionTreeEntry
+    base::SessionTreeEntryBase
+    target_id::String
+    label::Union{String, Nothing}
+end
+
+struct SessionInfoEntry <: SessionTreeEntry
+    base::SessionTreeEntryBase
+    name::Union{String, Nothing}
+end
+
+struct LeafEntry <: SessionTreeEntry
+    base::SessionTreeEntryBase
+    target_id::Union{String, Nothing}
+end
+
+# ============================================================================
+# Session context
+# ============================================================================
+
+struct SessionContext
+    messages::Vector{AgentMessage}
+    thinking_level::String
+    model::Union{Dict{String, String}, Nothing}
+    active_tool_names::Union{Vector{String}, Nothing}
+end
+
+# ============================================================================
+# Session stats
+# ============================================================================
+
+struct SessionStats
+    message_count::Int64
+    cached_tokens::Int64
+    uncached_tokens::Int64
+    total_tokens::Int64
+    cost_total::Float64
+end
+
+# ============================================================================
+# Session metadata
+# ============================================================================
+
+abstract type SessionMetadata end
+
+struct JsonlSessionMetadata <: SessionMetadata
+    id::String
+    created_at::String
+    cwd::String
+    path::String
+    parent_session_path::Union{String, Nothing}
+    metadata::Union{Dict{String, Any}, Nothing}
+end
+
+# ============================================================================
+# Session storage interface
+# ============================================================================
+
+abstract type SessionStorage{T<:SessionMetadata} end
+
+# ============================================================================
+# Session repo interface
+# ============================================================================
+
+abstract type SessionRepo<
+    TMetadata<:SessionMetadata,
+    TCreateOptions,
+    TListOptions
+> end
+
+# ============================================================================
+# Helper functions
+# ============================================================================
+
+function create_timestamp()::String
+    return string(Dates.now(Dates.UTC))
+end
+
+function uuidv7()::String
+    return string(UUIDs.uuid7())
+end
+
+function uuidstring()::String
+    return string(UUIDs.uuid4())
+end
+
+function tempname()::String
+    return tempname()
+end
+
+end

src/utils.jl

@@ -1,435 +0,0 @@
-module utils
-
-export clearhistory, availableWineToText, prepareContext, formatMsgForLLM, validateRequiredArgs, validateToolArguments, _userMessageToOpenAI, 
-        _assistantMessageToOpenAI, _toolResultMessageToOpenAI, _messageContentToBlocks
-
-using UUIDs, Dates, DataStructures, HTTP, JSON
-using GeneralUtils
-using ..type
-
-# ---------------------------------------------- 100 --------------------------------------------- #
-
-"""
-Clear agent chat history.
-
-Empties the conversation history, short-term memory, events log, and chatbox.
-
-# Arguments
-- `a::T`: An agent instance (subtype of `agent`)
-
-# Returns
-- `nothing`
-
-# Notes
-- Does not clear long-term memory; use `[PENDING] clear memory` when implemented.
-
-# Examples
-```jldoctest
-julia> YiemAgent.clearhistory(agent)
-```
-"""
-function clearhistory(a::T)  where {T<:agent}
-  # empty!(a.chathistory)
-  # empty!(a.memory["shortmem"])
-  # empty!(a.memory["events"])
-  # a.memory["chatbox"] = ""
-end
-
-
-"""
-Convert a vector of wine dictionaries to a formatted text string.
-
-# Arguments
-- `vecd::Vector`: A vector of dictionaries, each representing a wine with key-value pairs
-
-# Returns
-- A formatted string where each wine is numbered and each key-value pair is comma-separated
-  in the format: `"1) key1:value1,key2:value2 key3:value3 2) ..."`
-
-# Examples
-```jldoctest
-julia> vecd = [Dict("wine_name" => "Chateau A", "price" => "50")]
-julia> YiemAgent.availableWineToText(vecd)
-"1) wine_name:Chateau A,price:50 "
-```
-"""
-function availableWineToText(vecd::Vector)::String
-  # Initialize an empty string to hold the final text
-  rowtext = ""
-  # Loop through each dictionary in the input vector
-  for (i, d) in enumerate(vecd)
-    # Iterate over all key-value pairs in the dictionary
-    temp = []
-    for (k, v) in d
-      # Append the formatted string to the text variable
-      t = "$k:$v"
-      push!(temp, t)
-    end
-    _rowtext = join(temp, ',')
-    rowtext *= "$i) $_rowtext "
-  end
-  
-  return rowtext
-end
-
-
-
-"""
-    prepareContext(state::agentState) -> agentContext
-
-Prepares an `agentContext` from the given `agentState` for sending to
-the LLM. By default, it deep copies the system prompt, messages, and
-tools from `state` into a new `agentContext`. 
-
-Override this function to customize the context — such as filtering
-tools based on the user's intent, modifying the system prompt, injecting
-additional context (retrieved documents, current time, user preferences),
-or pruning and reordering messages before formatting and calling the LLM.
-
-# Arguments
-- `state::agentState`: The current agent state containing conversation history,
-  system prompt, tools, and other configuration
-
-# Returns
-- `agentContext`: An `agentContext` containing the prepared system prompt,
-  messages, and tools to be sent to the LLM
-
-# Examples
-```julia
-# Default: returns an agentContext with deep copies of system prompt, messages, and tools
-prepareContext(state).messages == deepcopy(state.messages)
-
-# Override to filter tools and inject system context:
-# function prepareContext(state::agentState)
-#   msgs = deepcopy(state.messages)
-#   sysPrompt = state.systemPrompt * "\\nCurrent time: $(now())"
-#   tools = filter(t -> contains(t.description, "wine"), state.tools)
-#   return agentContext(sysPrompt, msgs, tools)
-# end
-```
-"""
-function prepareContext(state::agentState)::agentContext
-  
-  #TODO filter tools from state.tools based on user intend in user message and tool description
-  filteredTools = state.tools
-
-  #TODO add tools to current system prompt
-  preparedSystemPrompt = state.systemPrompt
-  
-  #TODO add system prompt, adjust/modify and inject additional context into messages
-  preparedMessages = deepcopy(state.messages) # messages that will be send to LLM
-
-  agentCtx = agentContext(preparedSystemPrompt, preparedMessages, filteredTools)
-
-  return agentCtx
-end
-
-
-"""
-    formatMsgForLLM(ctx::agentContext) -> Dict{String, Any}
-
-Converts an `agentContext` into OpenAI-compatible message format
-ready to be sent to the LLM. The system prompt is converted into
-a system role message, followed by user, assistant, and tool result
-messages.
-
-This function can be overridden in `yiemAgent` to produce custom
-LLM message formats for different APIs/providers.
-
-# Arguments
-- `ctx::agentContext`: The prepared context containing system prompt,
-  messages, and tools
-
-# Returns
-- `Dict{String, Any}`: A dictionary with `"messages"` key containing
-  an array of OpenAI-format message dicts
-
-# Examples
-```julia
-# Default output:
-formatMsgForLLm(ctx) == Dict("messages" => [
-  Dict("role" => "system", "content" => [...]),
-  Dict("role" => "user", "content" => [...]),
-  Dict("role" => "assistant", "content" => [...]),
-  Dict("role" => "tool", "tool_call_id" => "...", "content" => [...]),
-])
-```
-"""
-function formatMsgForLLM(ctx::agentContext)::Dict{String, Any}
-
-  """ openai message format example
-  msg = Dict(
-    "model" => "gemma-4-E4B-it-UD-Q4_K_XL",
-    "messages" => [
-      Dict(
-        "role" => "system",
-        "content" => [
-          Dict("type" => "text", "text" => systemmsg),
-        ]
-      ),
-      Dict(
-        "role" => "user",
-        "content" => [
-          Dict("type" => "text", "text" => "Do you have something similar to the one in the image?"),
-          Dict(
-            "type" => "image_url",
-            "image_url" => Dict("url" => data_uri)
-          )
-        ]
-      ),
-      Dict(
-        "role" => "assistant",
-        "content" => [
-          Dict("type" => "text", "text" => "let me check."),
-        ]
-      ),
-      Dict(
-        "role" => "toolResult",
-        "content" => [
-          Dict("type" => "text", "text" => "name: Chateau Montelena ..."),
-        ]
-      ),
-    ],
-    "temperature" => 0.7
-  )
-  """
-
-  messages = Vector{Dict{String, Any}}()
-
-  # System prompt as system message
-  if !isempty(ctx.systemPrompt)
-    push!(messages, Dict(
-      "role" => "system",
-      "content" => [Dict("type" => "text", "text" => ctx.systemPrompt)]
-    ))
-  end
-
-  # Conversation messages
-  for msg in ctx.messages
-    if msg isa userMessage
-      push!(messages, _userMessageToOpenAI(msg))
-    elseif msg isa assistantMessage
-      push!(messages, _assistantMessageToOpenAI(msg))
-    elseif msg isa toolResultMessage
-      push!(messages, _toolResultMessageToOpenAI(msg))
-    end
-  end
-
-  return Dict("messages" => messages)
-end
-
-
-"""
-Convert a userMessage to OpenAI message format.
-"""
-function _userMessageToOpenAI(msg::userMessage)::Dict{String, Any}
-  return Dict(
-    "role" => "user",
-    "content" => _messageContentToBlocks(msg.content)
-  )
-end
-
-
-"""
-Convert an assistantMessage to OpenAI message format.
-"""
-function _assistantMessageToOpenAI(msg::assistantMessage)::Dict{String, Any}
-  return Dict(
-    "role" => "assistant",
-    "content" => _messageContentToBlocks(msg.content)
-  )
-end
-
-
-"""
-Convert a toolResultMessage to OpenAI message format.
-"""
-function _toolResultMessageToOpenAI(msg::toolResultMessage)::Dict{String, Any}
-  return Dict(
-    "role" => "tool",
-    "tool_call_id" => msg.toolCallId,
-    "content" => _messageContentToBlocks(msg.content)
-  )
-end
-
-
-"""
-Convert a vector of messageContent to OpenAI content blocks.
-
-Each textContent becomes a text block, each imageContent becomes
-an image_url block.
-"""
-function _messageContentToBlocks(contents::Vector{messageContent})::Vector{Dict{String, Any}}
-  blocks = Vector{Dict{String, Any}}()
-
-  for c in contents
-    if c isa textContent
-      push!(blocks, Dict("type" => "text", "text" => c.text))
-    elseif c isa imageContent
-      push!(blocks, Dict(
-        "type" => "image_url",
-        "image_url" => Dict(
-          "url" => "data:$(c.mimeType);base64,$(c.data)"
-        )
-      ))
-    end
-  end
-
-    return blocks
-end
-
-
-"""
-    validateRequiredArgs(args::Dict{String,Any}, inputSchema::Dict{String,Any}) -> Union{Nothing,String}
-
-Validates that all required fields listed in the tool's JSON Schema are present
-in `args`. Returns `nothing` if validation passes, or a descriptive error string
-listing the missing required fields.
-
-This is the default `validateRequiredArgs` hook. Tool authors can override it
-with a custom validation function that performs additional checks (e.g. type
-coercion, format validation, cross-field constraints).
-
-# Arguments
-- `args::Dict{String,Any}`: The arguments provided by the LLM
-- `inputSchema::Dict{String,Any}`: The tool's `inputSchema` (JSON Schema format)
-
-# Returns
-- `nothing` if all required args are present
-- `String` error message listing missing fields otherwise
-
-# Examples
-```julia
-schema = Dict("required" => ["city"])
-args = Dict{String,Any}()
-validateRequiredArgs(args, schema)  # => "Missing required arguments: city"
-
-args2 = Dict("city" => "Tokyo")
-validateRequiredArgs(args2, schema)  # => nothing
-```
-"""
-function validateRequiredArgs(args::Dict{String,Any}, inputSchema::Dict{String,Any})::Union{Nothing,String}
-    required = get(inputSchema, "required", Any[])
-    if isempty(required)
-        return nothing
-    end
-    
-    missing = String[]
-    for field in required
-        if !(field in keys(args))
-            push!(missing, field)
-        end
-    end
-    
-    if !isempty(missing)
-        return "Missing required arguments: $(join(missing, ", "))"
-    end
-    
-    return nothing
-end
-
-
-"""
-    validateToolArguments(tool::agentTool, prepared::agentToolCall) -> Dict{String,Any}
-
-Validates the prepared tool call arguments by calling the tool's
-`validateRequiredArgs` hook (or the default implementation). If validation
-fails, returns a modified `agentToolCall` with an empty arguments dict
-so downstream code can detect the failure. If the hook exists on the tool
-and returns an error string, that error is returned.
-
-This runs **before** the `beforeToolCall` hook, allowing the agent to
-reject invalid calls without invoking lifecycle callbacks or logging
-false `toolExecutionStart` events.
-
-# Arguments
-- `tool::agentTool`: The resolved tool definition
-- `prepared::agentToolCall`: The prepared tool call with potentially transformed arguments
-
-# Returns
-- `Dict{String,Any}`: The validated arguments if successful
-
-# Errors
-- Throws `ArgumentError` if validation fails — this is caught by `prepareToolCall`
-  and converted to an `immediateOutcome`
-
-# Examples
-```julia
-# With validateRequiredArgs hook set on the tool
-validateToolArguments(toolWithHook, tc)  # => validated args or throws
-
-# With default validation (nothing on tool)
-validateToolArguments(toolDefault, tc)  # => args or throws
-```
-"""
-function validateToolArguments(tool::agentTool, prepared::agentToolCall)::Dict{String,Any}
-    # Use default (2-arg: args + schema) or tool-specific hook (1-arg: args only)
-    if isnothing(tool.validateRequiredArgs)
-        result = validateRequiredArgs(prepared.arguments, tool.inputSchema)
-    else
-        result = tool.validateRequiredArgs(prepared.arguments)
-    end
-    
-    if result !== nothing
-        throw(ArgumentError(result))
-    end
-    
-    return prepared.arguments
-end
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-end # module util

src_OLD/OLD_type.jl

@@ -1,375 +0,0 @@
-module type
-  
-export agent, sommelier, companion, virtualcustomer, agentcontext
-
-using Dates, UUIDs, DataStructures, JSON, NATS
-using GeneralUtils
-
-# ---------------------------------------------- 100 --------------------------------------------- #
-
-
-mutable struct agentcontext
-  text2textInstructLLM::Function
-  getTextEmbedding::Function
-  executeSQL::Function
-  similarSQLVectorDB::Function
-  insertSQLVectorDB::Function
-  similarSommelierDecision::Function
-  insertSommelierDecision::Function
-  find_related_tables_for_user_question::Function
-  pg_conn_str::String
-  agentconfig::AbstractDict
-end
-
-abstract type agent end
-
-mutable struct sommelier <: agent
-  name::String  # agent name
-  id::String    # agent id
-  retailername::String
-  retailerid::String
-  tools::Dict   
-  maxHistoryMsg::Integer  # e.g. 21th and earlier messages will get summarized
-  chathistory::Vector{Dict{String, Any}}
-  memory::Dict{String, Any}
-  context::agentcontext 
-  llmFormatName::String
-end
- 
-""" A sommelier agent.
-
-# Arguments
-  - `context::agentcontext`
-    Application context containing shared functions for LLM, SQL, and vector database operations.
-
-# Keyword Arguments
-  - `name::String`
-    Agent's name. Default: `"Assistant"`
-  - `id::String`
-    Agent's ID. Default: generated UUID string.
-  - `retailername::String`
-    Retailer name associated with the sommelier. Default: `"retailer_name"`
-  - `maxHistoryMsg::Integer`
-    Maximum history messages. Default: `20`
-  - `chathistory::Vector{Dict{String, String}}`
-    Chat history. Default: empty vector.
-  - `llmFormatName::String`
-    LLM format name. Default: `"granite3"`
-
-# Return
-  - `sommelier`: An instantiated sommelier agent.
-
-# Example
-```julia
-julia> using YiemAgent
-julia> context = agentcontext(
-           text2textInstructLLM,
-           getTextEmbedding,
-           executeSQL,
-           similarSQLVectorDB,
-           insertSQLVectorDB,
-           similarSommelierDecision,
-           insertSommelierDecision
-       )
-julia> agent = sommelier(context, name="WineExpert", id="123", retailername="MyWineShop")
-```
-"""
-function sommelier(
-  context::agentcontext,  # agent functions, db connect and other context
-  ;
-  name::String= "Assistant",
-  id::String= string(uuid4()),
-  retailername::String= "not specified",
-  retailerid::String= "not specified",
-  maxHistoryMsg::Integer= 20,
-  chathistory::Vector{Dict{String, Any}} = Vector{Dict{String, Any}}(),
-  llmFormatName::String= "granite3"
-  )
-
-  tools = Dict( # update input format
-      "chatbox"=> Dict(
-        "description" => "<askbox tool description>Useful for when you need to ask the user for more context. Do not ask the user their own question.</askbox tool description>",
-        "input" => """<input>Input is a text in JSON format.</input><input example>{\"Q1\": \"How are you doing?\", \"Q2\": \"How may I help you?\"}</input example>""",
-        "output" => "" ,
-      ),
-      "winestock"=> Dict(
-        "description" => "<winestock tool description>A handy tool for searching wine in your inventory that match the user preferences.</winestock tool description>",
-        "input" => """<input>Input is a JSON-formatted string that contains a detailed and precise search query.</input><input example>{\"wine type\": \"rose\", \"price\": \"max 35\", \"sweetness level\": \"sweet\", \"intensity level\": \"light bodied\", \"Tannin level\": \"low\", \"Acidity level\": \"low\"}</input example>""",
-        "output" => """<output>Output are wines that match the search query in JSON format.""",
-      ),
-    )
-
-  """ Memory
-
-    Chat history use openai format as follow:
-
-    image1_path = "test/large_image.png"                           ---
-    image1_bytes = read(image1_path)                                 | this part must be done
-    image1_base64_string = base64encode(image1_bytes)                | in frontend 
-    mime_type = "image/png"                                          | not in agent code
-    data1_uri = "data:<mime_type>;base64,<image1_base64_string>"   ---
-
-    chathistory= [
-      Dict(
-        "role" => "system",
-        "content" => [
-          Dict("type" => "text", "text" => "You are a helpful assistant"),
-        ]
-      ),
-      Dict(
-        "role" => "user",
-        "content" => [
-          Dict("type" => "text", "text" => "<internal_context_for_assistant> 
-                                            LLM context here...
-                                            </internal_context_for_assistant>
-                                            Do you know this wine? Just give me brief intro."
-          ),
-          Dict(
-            "type" => "image_url",
-            "image_url" => Dict("url" => data1_uri)
-          ),
-        ]
-      ),
-    ]
-  
-  shortmem = Dict(
-    "1"=> Dict("plan"=> "...", "action_name"=> "...", "action_input"=> "...", "action_result"=> "..."),
-    "2"=> Dict("plan"=> "...", "action_name"=> "...", "action_input"=> "...", "action_result"=> "..."),
-    ...
-    )
-  """
-  memory = Dict{String, Any}(
-            "shortmem"=> OrderedDict{String, Any}(),
-            "scratchpad"=> "",
-            "recap"=> OrderedDict{String, Any}(),
-          )
-
-  newAgent = sommelier(
-    name,
-    id,
-    retailername,
-    retailerid,
-    tools,
-    maxHistoryMsg,
-    chathistory,
-    memory,
-    context,
-    llmFormatName
-    )
-  systemmsg = 
-    """
-    # store_policy
-      - Generally speaking, the store inventory has some wines from France, the United States, Australia, Spain, and Italy, but you won't know exactly until you check your inventory.
-      - If you found wines in the store's database, they are in stock.
-      - You can only recommend wines that are currently in our inventory
-      - Before searching the database for wine, ensure you have at least the following information: 1) budget, 2) wine type, and 3) occasion. Additional details are always helpful. If the user is unsure, provide relevant information and gather insights to make reasonable inferences.
-      - Ask the user one question at a time.
-      - Once the user has selected their wine, if you haven't already, ask the user whether they need any further assistance. Do not offer any additional services.
-      - Only end the conversation when the user explicitly intends to do so. When ending, ensure a polite farewell and an invitation to return in the future.
-      - Spicy foods should be paired only with light red wines.
-      - We do not sell organic, sustainable, gluten-free, and sulfite-free wine. Inform the user imediately if they are looking for these types of wines. Do not sell our wines as such.
-      - Gift box, gift card, and custom messages are available. Inform the user to contact our sales team.
-
-    # store_guidelines
-      - Greeting the customer warmly by ask them how could you help. Do not ask any other questions during this greeting.
-      - Customer may provide images for you to look up.
-      - Encourage the customer to explore different options and try new things.
-      - If you are unable to locate the desired item in the database after 2 attempts, it may not be available in your inventory. In such cases, inform the user that the item is unavailable and suggest an alternative instead.
-      - Your store carries only wine.
-      - Vintage 0 means non-vintage.
-      - Start searching the database as broadly as possible within the given information boundary to maximize the chances of finding. Avoid unnecessary parameters unless specified by the user. Refine the search subsequently.
-      - User usually ask for something similar. This means you should use the search term based on the profile they like.
-
-    # situation
-      You are having conversation with a customer.
-
-    # your role
-      Your name is $(newAgent.name). You are a helpful sommelier for website-based $(newAgent.retailername)'s wine store.
-
-    # objective
-      - Establish a connection with the customer by talking to them politely and showing your enthusiasm for their wine preferences.
-      - Provide relevant information and guide them to select the best wines only from your store's inventory that align with their preferences.
-
-    # your responsibility includes
-      - According to the store's policy and guidelines, and make an informed decision about what available_actions you need to use to achieve the objective.
-      - Keep the conversation with the customer going smoothly
-
-    # your responsibility does NOT includes
-      - Requesting the user to place an order, make a purchase, or confirm the order. These are the job of our sales team at the store.
-      - Processing sales orders or engaging in any other sales-related activities. These are the job of our sales team at the store.
-      - Answering questions or offering additional services beyond those related to your store's wine recommendations such as discounts, quantity, rewards programs, promotions, delivery options, shipping, boxes, gift wrapping, packaging, personalized messages or something similar. These are the job of our sales team at the store.
-
-    # you should then respond to the user with interleaving plan, action_name, action_input in JSON format
-      1) "plan", Based on the current situation, state a complete action plan to complete the task and rationale. Be specific.
-      2) "action_name", (Typically corresponds to the execution of the first step in your plan) Can be one of the available_actions name
-      3) "action_input", The input to the action you are about to perform according to your plan.
-      After the action is executed you gets "action_result". It is the output from the action you selected.
-
-    # available actions
-      "CHAT_BOX", which you can use to talk with the user. The input is dialogue you want to chat with the user according to your plan.
-      "SEARCH_WINE_DATABASE", allows you to search information about wines you want in your inventory's database. The input is strictly supported search term including: retailer_name, wine price, winery, name, vintage, region, country, type of wine, grape varietal, tasting notes, occasion, food pairing, intensity, tannin, sweetness, and acidity.
-        Example query 1: "Dry, full-bodied red wine from Burgundy, France. Grape varietal could be Merlot or Syrah. price 100 to 1000 USD."
-        Example query 2: "Red or white wine, medium tannin, price under 700 USD"
-        Example query 3: "white wine from Tuscany, Italy or Bordeaux, France
-      "WINE_PRESENTATION_GUIDELINE", which you can use to check the store guidelines about how to present wines you have found to the user. The input is "nothing" keyword. The output is the guidelines that you can follow.
-      "END_CONVER_GUIDELINE", which you can use to check the store guidelines about how to end the conversation with the user. The input is "nothing" keyword. The output is the guidelines that you can follow.
-    """
-    
-  system_msg = Dict(
-      "role" => "system",
-      "content" => [
-        Dict("type" => "text", "text" => systemmsg),
-      ]
-    )
-
-  push!(newAgent.chathistory, system_msg)
-  
-  return newAgent
-end
- 
-
-mutable struct virtualcustomer <: agent
-  name::String  # agent name
-  id::String    # agent id
-  systemmsg::String # system message
-  tools::Dict   
-  maxHistoryMsg::Integer  # e.g. 21th and earlier messages will get summarized
-  chathistory::Vector{Dict{String, Any}}
-  memory::Dict{String, Any}
-  context  # NamedTuple of functions 
-  llmFormatName::String
-end
-
-function virtualcustomer(
-  context,  # NamedTuple of functions
-  ;
-  name::String= "Assistant",
-  id::String= string(uuid4()),
-  maxHistoryMsg::Integer= 20,
-  chathistory::Vector{Dict{String, String}} = Vector{Dict{String, String}}(),
-  llmFormatName::String= "granite3",
-  systemmsg::String=
-    """
-      Your name: $name
-      Your sex: Female
-      Your role: You are a helpful assistant.
-      You should follow the following guidelines:
-      - Focus on the latest conversation.
-      - Your like to be short and concise.
-
-      Let's begin!
-      """,
-  )
-
-  tools = Dict( # update input format
-      "chatbox"=> Dict(
-        "description" => "<askbox tool description>Useful for when you need to ask the user for more context. Do not ask the user their own question.</askbox tool description>",
-        "input" => """<input>Input is a text in JSON format.</input><input example>{\"Q1\": \"How are you doing?\", \"Q2\": \"How may I help you?\"}</input example>""",
-        "output" => "" ,
-      ),
-    )
-
-  """ Memory
-    Ref: Chat prompt format is openai
-    chathistory = [
-      Dict(
-        "role" => "system",
-        "content" => [
-          Dict("type" => "text", "text" => system_msg),
-        ]
-      ),
-      Dict(
-        "role" => "user",
-        "content" => [
-          Dict("type" => "text", "text" => "Do you know this wine? Just give me brief intro."),
-          Dict(
-            "type" => "image_url",
-            "image_url" => Dict("url" => data1_uri)
-          )
-        ]
-      )
-    ]
-  """
-  memory = Dict{String, Any}(
-            "shortmem"=> OrderedDict{String, Any}(
-            ),
-            "scratchpad"=> "",
-            "events"=> Vector{Dict{String, Any}}(),
-            "state"=> Dict{String, Any}(
-            ),
-            "recap"=> OrderedDict{String, Any}(),
-          )
-
-  newAgent = virtualcustomer(
-    name,
-    id,
-    systemmsg,
-    tools,
-    maxHistoryMsg,
-    chathistory,
-    memory,
-    context,
-    llmFormatName
-    )
-  
-  return newAgent
-end
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-end # module type

test/chatting_with_agent.jl

@@ -0,0 +1,296 @@
+using Revise
+using JSON, JSON3, Dates, UUIDs, PrettyPrinting, LibPQ, Base64, DataFrames
+using YiemAgent, GeneralUtils
+using Base.Threads
+
+# ---------------------------------------------- 100 --------------------------------------------- #
+
+
+
+# load config
+config = JSON.parsefile("/appfolder/app/dev/YiemAgent/test/config.json")
+# config = copy(JSON.parsefile("../mountvolume/config.json"))
+
+
+function executeSQL(sql::T) where {T<:AbstractString}
+  host = config[:externalservice][:wineDB][:host]
+  port = config[:externalservice][:wineDB][:port]
+  dbname = config[:externalservice][:wineDB][:dbname]
+  user = config[:externalservice][:wineDB][:user]
+  password = config[:externalservice][:wineDB][:password]
+  DBconnection = LibPQ.Connection("host=$host port=$port dbname=$dbname user=$user password=$password")
+  result = LibPQ.execute(DBconnection, sql)
+  close(DBconnection)
+  return result
+end
+
+function executeSQLVectorDB(sql)
+  host = config[:externalservice][:SQLVectorDB][:host]
+  port = config[:externalservice][:SQLVectorDB][:port]
+  dbname = config[:externalservice][:SQLVectorDB][:dbname]
+  user = config[:externalservice][:SQLVectorDB][:user]
+  password = config[:externalservice][:SQLVectorDB][:password]
+  DBconnection = LibPQ.Connection("host=$host port=$port dbname=$dbname user=$user password=$password")
+  result = LibPQ.execute(DBconnection, sql)
+  close(DBconnection)
+  return result
+end
+
+function text2textInstructLLM(prompt::String; maxattempt::Integer=3, modelsize::String="medium",
+    llmkwargs=Dict(
+      :num_ctx => 32768,
+      :temperature => 0.1,
+    )
+  )
+  msgMeta = GeneralUtils.generate_msgMeta(
+    config[:externalservice][:loadbalancer][:mqtttopic];
+    msgPurpose="inference",
+    senderName="yiemagent",
+    senderId=sessionId,
+    receiverName="text2textinstruct_$modelsize",
+    mqttBrokerAddress=config[:mqttServerInfo][:broker],
+    mqttBrokerPort=config[:mqttServerInfo][:port],
+  )
+
+  outgoingMsg = Dict(
+    :msgMeta => msgMeta,
+    :payload => Dict(
+      :text => prompt,
+      :kwargs => llmkwargs
+    )
+  )
+
+  response = nothing
+  for attempts in 1:maxattempt
+    _response = GeneralUtils.sendReceiveMqttMsg(outgoingMsg; timeout=180, maxattempt=maxattempt)
+    payload = _response[:response]
+    if _response[:success] && payload[:text] !== nothing
+      response = _response[:response][:text]
+      break
+    else
+      println("\n<text2textInstructLLM()> attempt $attempts/$maxattempt failed ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+      pprintln(outgoingMsg)
+      println("</text2textInstructLLM()> attempt $attempts/$maxattempt failed ", @__FILE__, ":", @__LINE__, " $(Dates.now())\n")
+      sleep(3)
+    end
+  end
+
+  return response
+end
+
+# get text embedding from a LLM service
+function getEmbedding(text::T) where {T<:AbstractString}
+  msgMeta = GeneralUtils.generate_msgMeta(
+    config[:externalservice][:loadbalancer][:mqtttopic];
+    msgPurpose="embedding",
+    senderName="yiemagent",
+    senderId=sessionId,
+    receiverName="textembedding",
+    mqttBrokerAddress=config[:mqttServerInfo][:broker],
+    mqttBrokerPort=config[:mqttServerInfo][:port],
+  )
+
+  outgoingMsg = Dict(
+    :msgMeta => msgMeta,
+    :payload => Dict(
+      :text => [text] # must be a vector of string
+    )
+  )
+
+  response = GeneralUtils.sendReceiveMqttMsg(outgoingMsg; timeout=120, maxattempt=3)
+  embedding = response[:response][:embeddings]
+  return embedding
+end
+
+function findSimilarTextFromVectorDB(text::T1, tablename::T2, embeddingColumnName::T3,
+  vectorDB::Function; limit::Integer=1
+  )::DataFrame where {T1<:AbstractString, T2<:AbstractString, T3<:AbstractString}
+  # get embedding from LLM service
+  embedding = getEmbedding(text)[1]
+  # check whether there is close enough vector already store in vectorDB. if no, add, else skip 
+  sql = """
+        SELECT *, $embeddingColumnName <-> '$embedding' as distance
+        FROM $tablename
+        ORDER BY distance LIMIT $limit;
+        """
+  response = vectorDB(sql)
+  df = DataFrame(response)
+  return df
+end
+
+function similarSQLVectorDB(query; maxdistance::Integer=100)
+  tablename = "sqlllm_decision_repository"
+  # get embedding of the query
+  df = findSimilarTextFromVectorDB(query, tablename, 
+                                  "function_input_embedding", executeSQLVectorDB)
+  # println(df[1, [:id, :function_output]])
+  row, col = size(df)
+  distance = row == 0 ? Inf : df[1, :distance]  
+  # distance = 100  # CHANGE this is for testing only
+  if row != 0 && distance < maxdistance
+    # if there is usable SQL, return it.
+    output_b64 = df[1, :function_output_base64]   # pick the closest match
+    output_str = String(base64decode(output_b64))
+    rowid = df[1, :id]
+    println("\n~~~ found similar sql. row id $rowid, distance $distance ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+    return (dict=output_str, distance=distance)
+  else
+    println("\n~~~ similar sql not found, max distance $maxdistance ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+    return (dict=nothing, distance=nothing)
+  end
+end
+
+function insertSQLVectorDB(query::T1, SQL::T2; maxdistance::Integer=3) where {T1<:AbstractString, T2<:AbstractString}
+  tablename = "sqlllm_decision_repository"
+  # get embedding of the query
+  # query = state[:thoughtHistory][:question]
+  df = findSimilarTextFromVectorDB(query, tablename, 
+                                  "function_input_embedding", executeSQLVectorDB)
+  row, col = size(df)
+  distance = row == 0 ? Inf : df[1, :distance]
+  if row == 0 || distance > maxdistance # no close enough SQL stored in the database
+    query_embedding = getEmbedding(query)[1]
+    query = replace(query, "'" => "")
+    sql_base64 = base64encode(SQL)
+    sql_ = replace(SQL, "'" => "")
+
+    sql = """
+          INSERT INTO $tablename (function_input, function_output, function_output_base64, function_input_embedding) VALUES ('$query', '$sql_', '$sql_base64', '$query_embedding');
+          """
+    # println("\n~~~ added new decision to vectorDB ", @__FILE__, ":", @__LINE__, " $(Dates.now())")
+    # println(sql)
+    _ = executeSQLVectorDB(sql)
+  end
+end
+
+
+function similarSommelierDecision(recentevents::T1; maxdistance::Integer=3
+  )::Union{AbstractDict, Nothing} where {T1<:AbstractString}
+  tablename = "sommelier_decision_repository"
+  # find similar
+  println("\n~~~ search vectorDB for this: $recentevents ", @__FILE__, " ", @__LINE__)
+  df = findSimilarTextFromVectorDB(recentevents, tablename, 
+                                  "function_input_embedding", executeSQLVectorDB)
+  row, col = size(df)
+  distance = row == 0 ? Inf : df[1, :distance]
+  if row != 0 && distance < maxdistance
+    # if there is usable decision, return it.
+    rowid = df[1, :id]
+    println("\n~~~ found similar decision. row id $rowid, distance $distance ", @__FILE__, " ", @__LINE__)
+    output_b64 = df[1, :function_output_base64]   # pick the closest match
+    _output_str = String(base64decode(output_b64))
+    output = copy(JSON.parsefile(_output_str))
+    return output
+  else
+    println("\n~~~ similar decision not found, max distance $maxdistance ", @__FILE__, " ", @__LINE__)
+    return nothing
+  end
+end
+
+
+function insertSommelierDecision(recentevents::T1, decision::T2; maxdistance::Integer=5
+  ) where {T1<:AbstractString, T2<:AbstractDict}
+  tablename = "sommelier_decision_repository"
+  # find similar
+  df = findSimilarTextFromVectorDB(recentevents, tablename, 
+                                  "function_input_embedding", executeSQLVectorDB)
+  row, col = size(df)
+  distance = row == 0 ? Inf : df[1, :distance]
+  if row == 0 || distance > maxdistance # no close enough SQL stored in the database
+    recentevents_embedding = getEmbedding(recentevents)[1]
+    recentevents = replace(recentevents, "'" => "")
+    decision_json = JSON.json(decision)
+    decision_base64 = base64encode(decision_json)
+    decision = replace(decision_json, "'" => "")
+    
+    sql = """
+          INSERT INTO $tablename (function_input, function_output, function_output_base64, function_input_embedding) VALUES ('$recentevents', '$decision', '$decision_base64', '$recentevents_embedding');
+          """
+    println("\n~~~ added new decision to vectorDB ", @__FILE__, " ", @__LINE__)
+    println(sql)
+    _ = executeSQLVectorDB(sql)
+  else
+    println("~~~ similar decision previously cached, distance $distance ", @__FILE__, " ", @__LINE__)
+  end
+end
+
+
+sessionId = "12345"
+
+externalFunction = (
+    getEmbedding=getEmbedding,
+    text2textInstructLLM=text2textInstructLLM,
+    executeSQL=executeSQL,
+    similarSQLVectorDB=similarSQLVectorDB,
+    insertSQLVectorDB=insertSQLVectorDB,
+    similarSommelierDecision=similarSommelierDecision,
+    insertSommelierDecision=insertSommelierDecision,
+  )
+
+
+
+a = YiemAgent.sommelier(
+  externalFunction;
+  name="Ton",
+  id=sessionId,  # agent instance id
+  retailername="Yiem",
+)
+
+while true
+  print("\nyour respond: ")
+  user_answer = readline()
+  response = YiemAgent.conversation(agent;
+                                  userinput=Dict(:text=> user_answer),
+                                  maximumMsg=50)
+  println("\n$response")
+end
+
+
+# response = YiemAgent.conversation(a, Dict(:text=> "I want to get a French red wine under 100."))
+
+
+"""
+hello I want to get a bottle of red wine for my boss. I have a budget around 50 dollars. Show me some options.
+
+I have no idea about his wine taste but he likes spicy food.
+
+
+"""
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

test/prompttest.jl

@@ -0,0 +1,159 @@
+using Revise
+using YiemAgent, GeneralUtils, JSON3, DataStructures
+
+thoughtDict = OrderedDict(
+  :Question=> "Hello, I would like a get a bottle of wine",
+  :Thought_1=> "The customer wants to buy a bottle of wine, but we need more information about their preferences.",
+  :Action_1=> Dict(
+      :name=> "chatbox",
+      :input=> "What occasion are you buying the wine for?",
+    ),
+  :Observation_1=> "We are having a wedding pary this weekend.",
+
+  :Thought_2=> "A wedding party is a great occasion to have a good bottle of wine.",
+  :Action_2=> Dict(
+      :name=> "chatbox",
+      :input=> "What type of food will you be serving with the wine?",
+    ),
+  :Observation_2=> "I think it is Thai dishes",
+
+  :Thought_3=> "Now that I know the occasion and food, I need to ask about the budget.",
+  :Action_3=> Dict(
+      :name=> "chatbox",
+      :input=> "What is your budget for this wine?",
+    ),
+  :Observation_3=> "50 bucks",
+
+  :Thought_4=> "With a budget of \$50, we have a wide range of options. Now that I know it's a wedding party and Thai dishes, I need to ask about the type of wine they prefer.",
+  :Action_4=> Dict(
+      :name=> "chatbox",
+      :input=> "What type of wine are you looking for? (Red, White, Sparkling, Rose, Dessert, Fortified)",
+    ),
+  :Observation_4=> "Sparkling please.",
+
+  :Thought_5=> "Now that I know the occasion, food, budget and preferred type of wine, it's time to check our inventory for the best matching wine.",
+  :Action_5=> Dict(
+      :name=> "winestock",
+      :input=> "wine with budget \$50, Thai dishes, sparkling, wedding party",
+    ),
+  :Observation_5=> "I found the following wine in stock {1 : Zena Crown Vista, 2 : Schrader Cabernet Sauvignon}",
+  
+  :Thought_6=> "Now that I have all the information, it's time to recommend a wine that fits their preferences.",
+  :Action_6=> Dict(
+      :name=> "recommendation",
+      :input=> "I recommend Zena Crown Vista for its sparkling and affordable price.",
+    ),
+  :Observation_6=> "I don't like it. Do you have another option?",
+  )
+
+_thoughtJsonStr = JSON.json(thoughtDict)
+thoughtJsonStr = _thoughtJsonStr[1:end-1]  # remove } at the end
+# @show thoughtJsonStr
+
+_, latestThoughtIndice = GeneralUtils.findHighestIndexKey(thoughtDict, "Thought")
+nextThoughtIndice = latestThoughtIndice + 1
+
+_prompt = 
+"""
+You are a helpful sommelier working for a wine store. 
+Your goal is to reccommend the best wine from your inventory that match the user preferences.
+
+You must follow the following criteria:
+1) Get to know what occasion the user is buying wine for
+2) Get to know what food the user will have with wine
+3) Get to know how much the user willing to spend
+4) Get to know type of wine the user is looking for e.g. Red, White, Sparkling, Rose, Dessert, Fortified
+5) Get to know what characteristics of wine the user is looking for 
+    e.g. tannin, sweetness, intensity, acidity
+6) Check your inventory for the best wine that match the user preference
+7) Recommend wine to the user
+
+You should only respond with interleaving Thought, Action, Observation steps. 
+Thought can reason about the current situation, and Action can be three types:
+1) winestock[query], which you can use to find wine in your inventory. The more input data the better.
+2) chatbox[text], which you can use to interact with the user.
+3) recommendation[answer], which returns your wine reccommendation to the user. 
+
+You should only respond in JSON format as describe below:
+{
+  "Thought": "your reasoning",
+  "Action": {"name": "action to take", "input": "Action input"},
+  "Observation": "result of the action"
+}
+
+Here are some examples:
+{
+"Question": "I would like to buy a sedan with 8 seats.",
+"Thought_1": "Our showroom carries various vehicle model. But I'm not sure whether we have a models that fits the user demand, I need to check our inventory.",
+"Action_1": {"name": "inventory", "input": "sedan with 8 seats."},
+"Observation_1": "Several model has 8 seats. Available color are black, red green"
+}
+{
+  "Thought_2": "I have to ask the user what color he likes.",
+  "Action_2": {"name": "chatbox", "input": "Which color do you like?"}
+  "Observation_2": "I'll take black."
+}
+{
+  "Thought_3": "There is only one model that fits the user preference. It's Yiem model A",
+  "Action_3": {"name": "recommendation", "input": "I recommend a Yiem model A"}
+}
+
+Let's begin!
+
+$(JSON.json(thoughtDict))
+{Thought_$nextThoughtIndice
+"""
+
+prompt =  YiemAgent.formatLLMtext_llama3instruct("system", _prompt)
+@show prompt
+msgMeta = Dict(:requestResponse => nothing,
+  :msgPurpose => nothing,
+  :receiverId => nothing,
+  :getPost => nothing,
+  :msgId => "4c7111e0-c30e-44c3-8f85-1c8b3f03a8be",
+  :acknowledgestatus => nothing,
+  :replyToMsgId => nothing,
+  :msgFormatVersion => nothing,
+  :mqttServerInfo => Dict(:port => 1883, :broker => "mqtt.yiem.cc"),
+  :sendTopic => "/loadbalancer/requestingservice",
+  :receiverName => "text2textinstruct",
+  :replyTopic => nothing,
+  :senderName => "decisionMaker",
+  :senderSelfnote => nothing,
+  :senderId => "testingSessionID",
+  :timeStamp => "2024-05-04T08:06:23.561"
+  )
+
+outgoingMsg = Dict(
+  :msgMeta=> msgMeta,
+  :payload=> Dict(
+    :text=> prompt,
+  )
+)
+
+
+
+_response = GeneralUtils.sendReceiveMqttMsg(outgoingMsg)
+thoughtJsonStr = _response[:response][:text]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

test/prompttest_1.jl

@@ -0,0 +1,87 @@
+using Revise      # remove when this package is completed
+using YiemAgent, GeneralUtils, JSON3, MQTTClient, Dates, UUIDs, DataStructures
+using Base.Threads
+
+# ---------------------------------------------- 100 --------------------------------------------- #
+
+config = copy(JSON.parsefile("config.json"))
+
+instanceInternalTopic = config[:serviceInternalTopic][:mqtttopic] * "/1" 
+
+client, connection = MakeConnection(config[:mqttServerInfo][:broker], 
+                                    config[:mqttServerInfo][:port])
+
+receiveUserMsgChannel = Channel{Dict}(4)
+receiveInternalMsgChannel = Channel{Dict}(4)
+
+msgMeta = GeneralUtils.generate_msgMeta(
+  "N/A", 
+  replyTopic = config[:servicetopic][:mqtttopic]  # ask frontend reply to this instance_chat_topic
+  )  
+
+agentConfig = Dict(
+  :mqttServerInfo=> config[:mqttServerInfo],
+  :receivemsg=>  Dict(
+    :prompt=> config[:servicetopic][:mqtttopic], # topic to receive prompt i.e. frontend send msg to this topic
+    :internal=> instanceInternalTopic,
+  ),
+  :externalservice=> config[:externalservice],
+)
+
+# Instantiate an agent
+tools=Dict( # update input format
+    "askbox"=> Dict(
+      :description => "<askbox tool description>Useful for when you need to ask the user for more context. Do not ask the user their own question.</askbox tool description>",
+      :input => """<input>Input is a text in JSON format.</input><input example>{\"Q1\": \"How are you doing?\", \"Q2\": \"How may I help you?\"}</input example>""",
+      :output => "" ,
+      :func => nothing,
+    ),
+    # "winestock"=> Dict(
+    #   :description => "<winestock tool description>A handy tool for searching wine in your inventory that match the user preferences.</winestock tool description>",
+    #   :input => """<input>Input is a JSON-formatted string that contains a detailed and precise search query.</input><input example>{\"wine type\": \"rose\", \"price\": \"max 35\", \"sweetness level\": \"sweet\", \"intensity level\": \"light bodied\", \"Tannin level\": \"low\", \"Acidity level\": \"low\"}</input example>""",
+    #   :output => """<output>Output are wines that match the search query in JSON format.""",
+    #   :func => ChatAgent.winestock,
+    # ),
+    "finalanswer"=> Dict(
+      :description => "<tool description>Useful for when you are ready to recommend wines to the user.</tool description>",
+      :input => """<input format>{\"finalanswer\": \"some text\"}.</input format><input example>{\"finalanswer\": \"I recommend Zena Crown Vista\"}</input example>""",
+      :output => "" ,
+      :func => nothing,
+    ),
+  )
+
+  a = YiemAgent.sommelier(
+    receiveUserMsgChannel,
+    receiveInternalMsgChannel,
+    agentConfig, 
+    name= "assistant",
+    id= "testingSessionID",  # agent instance id
+    tools=tools,
+  )
+
+
+
+
+
+
+
+input =
+OrderedDict{String, Any}(:question => "Hello, I would like a get a bottle of wine", :thought_1 => "It's great that the user is looking for a bottle of wine. To give them a personalized recommendation, I need to know more about their preferences.", :action_1 => Dict{String, Any}(:name => "chatbox", :input => "What occasion are you planning to use this wine for?"), :observation_1 => "We are holding a wedding party", :thought_2 => "A wedding party is a great occasion for a special bottle of wine. I need to know what type of food will be served, and how much the user is willing to spend.", :action_2 => Dict{String, Any}(:name => "chatbox", :input => "What type of food will you be serving at the wedding?"), :observation_2 => "It will be Thai dishes.", :thought_3 => "The type of wine that pairs well with Thai dishes is usually a crisp and refreshing white wine, but I also need to consider the budget and personal preferences.", :action_3 => Dict{String, Any}(:name => "chatbox", :input => "How much are you willing to spend on this bottle of wine?"), :observation_3 => "I would spend up to 50 bucks.", :thought_4 => "I have a good idea of the occasion, food, and budget. Now I need to know what type of wine the user is looking for.", :action_4 => Dict{String, Any}(:name => "chatbox", :input => "What type of wine are you usually looking for? Red, White, Sparkling, Rose, Dessert or Fortified?"), :observation_4 => "I like full-bodied Red wine with low tannin.", :thought_5 => "Now that I have all the necessary information, I can start searching for a suitable wine in our inventory.", :action_5 => Dict{String, Any}(:name => "winestock", :input => "red wine with low tannins"), :observation_5 => "I found the following wines in our stock: \n{\n  1: El Enemigo Cabernet Franc 2019\n2: Tantara Chardonnay 2017\n\n}\n", :thought_6 => "Now that I have the information about the wine, it's time to make a recommendation.", :action_6 => Dict{String, Any}(:name => "recommendbox", :input => "El Enemigo Cabernet Franc 2019"), :observation_6 => "I don't like the one you recommend. I want dry wine.")
+
+
+result = YiemAgent.jsoncorrection(a, input)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

test/prompttest_2.jl

@@ -0,0 +1,119 @@
+using Revise
+using YiemAgent, GeneralUtils, JSON3, DataStructures, LibPQ
+using SQLLLM
+
+
+# _prompt = 
+# """
+# You are a helpful assistant.
+# answer the following question:
+# From the following CSV text:
+# "{\"tabledescription\":[\"The customer table stores information about customers. It includes details such as first name, last name, display name, username, password, gender, country, telephone number, email, birthdate, additional_search_term, other attributes (in JSON format) and a description.\",\"The wine table stores information about different wines. It includes details namely id, name, brand, manufacturer, region, country, wine_type, grape_variety, serving_temperature, intensity, sweetness, tannin, acidity, fizziness, additional_search_term, other attributes (in JSON format) and a description.\",\"The wine_food table represents the association between wines and food items. It estab" ⋯ 477 bytes ⋯ "ed to retailer names, usernames, passwords, addresses, contact persons, telephone numbers, email addresses, additional_search_term, other attributes (in JSON format) and a description.\",\"The retailer_wine table represents the relationship between retailers and wines. It stores information about the wines available from which retailers, including vintage, their price, and the currency.\",\"The retailer_food table represents the relationship between retailers and food items. It stores information about the food items available from which retailers, including their price and the currency.\"],\"tablename\":[\"customer\",\"wine\",\"wine_food\",\"food\",\"retailer\",\"retailer_wine\",\"retailer_food\"]}"
+# What is the description of table wine?
+# """
+
+# prompt =  YiemAgent.formatLLMtext_llama3instruct("system", _prompt)
+# @show prompt
+# msgMeta = Dict(:requestResponse => nothing,
+#   :msgPurpose => nothing,
+#   :receiverId => nothing,
+#   :getPost => nothing,
+#   :msgId => "4c7111e0-c30e-44c3-8f85-1c8b3f03a8be",
+#   :acknowledgestatus => nothing,
+#   :replyToMsgId => nothing,
+#   :msgFormatVersion => nothing,
+#   :mqttServerInfo => Dict(:port => 1883, :broker => "mqtt.yiem.cc"),
+#   :sendTopic => "/loadbalancer/requestingservice",
+#   :receiverName => "text2textinstruct",
+#   :replyTopic => nothing,
+#   :senderName => "decisionMaker",
+#   :senderSelfnote => nothing,
+#   :senderId => "testingSessionID",
+#   :timeStamp => "2024-05-04T08:06:23.561"
+#   )
+
+# outgoingMsg = Dict(
+#   :msgMeta=> msgMeta,
+#   :payload=> Dict(
+#     :text=> prompt,
+#   )
+# )
+
+
+
+# _response = GeneralUtils.sendReceiveMqttMsg(outgoingMsg)
+# result = _response[:response][:text]
+
+
+
+
+
+
+
+DBconnection = LibPQ.Connection("host=192.168.88.12 port=5432 dbname=yiem_wine_assistant user=yiem password=yiem@Postgres_0.0")
+
+tableinfo, df1, df2, df3 = SQLLLM.tableinfo(DBconnection, "wine")
+
+
+_prompt = 
+"""
+You are a helpful assistant helping to answer user question from a database table.
+
+$tableinfo
+
+Are there any chardonnay?
+"""
+
+prompt =  YiemAgent.formatLLMtext_llama3instruct("system", _prompt)
+@show prompt
+msgMeta = Dict(:requestResponse => nothing,
+  :msgPurpose => nothing,
+  :receiverId => nothing,
+  :getPost => nothing,
+  :msgId => "4c7111e0-c30e-44c3-8f85-1c8b3f03a8be",
+  :acknowledgestatus => nothing,
+  :replyToMsgId => nothing,
+  :msgFormatVersion => nothing,
+  :mqttServerInfo => Dict(:port => 1883, :broker => "mqtt.yiem.cc"),
+  :sendTopic => "/loadbalancer/requestingservice",
+  :receiverName => "text2textinstruct",
+  :replyTopic => nothing,
+  :senderName => "decisionMaker",
+  :senderSelfnote => nothing,
+  :senderId => "testingSessionID",
+  :timeStamp => "2024-05-04T08:06:23.561"
+  )
+
+outgoingMsg = Dict(
+  :msgMeta=> msgMeta,
+  :payload=> Dict(
+    :text=> prompt,
+  )
+)
+
+
+
+_response = GeneralUtils.sendReceiveMqttMsg(outgoingMsg)
+result2 = _response[:response][:text]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

test/runtests.jl

@@ -0,0 +1,223 @@
+
+using JSON, Dates, UUIDs, PrettyPrinting, Base64, NATS, HTTP
+using GeneralUtils, msghandler
+
+config = JSON.parsefile("./appconfig.json")
+
+agent_conn  = NATS.connect(config["nats_server_info"]["url"])
+
+function text2text_instruct_llm(sender_id::String, openai_msg::Dict{String, Any})
+  payloads = [("msg", openai_msg, "dictionary")]  # List of tuples
+  _, msg_envelope_json_str = msghandler.smartpack(
+        config["externalservice"]["servicesloadbalancer"]["nats"], 
+        payloads;
+        sender_id=sender_id,
+        msg_purpose="text2text",  
+        broker_url=config["nats_server_info"]["url"],
+        fileserver_url=config["externalservice"]["fileserver"]["url"])
+
+  reply = NATS.request(agent_conn, 
+                      config["externalservice"]["servicesloadbalancer"]["nats"], 
+                      msg_envelope_json_str, timeout=120)
+
+  incoming_env_json_str = String(reply.payload)
+  incoming_env = msghandler.smartunpack(incoming_env_json_str)
+  _llm_response = incoming_env["payloads"][1][2]
+  llm_response = _llm_response["choices"][1]["message"]["content"]
+  return llm_response
+end
+
+
+
+
+# 1. Read local file and encode to base64 string
+image1_path = "test/large_image.png"
+image1_bytes = read(image1_path)
+image1_base64_string = base64encode(image1_bytes)
+
+# 2. Match the MIME type according to your file extension (e.g., png, jpeg)
+mime_type = "image/png" 
+data1_uri = "data:$(mime_type);base64,$(image1_base64_string)"
+
+# 3. Construct payload with the Data URI
+openai_msg = Dict(
+  "model" => "gemma-4-E4B-it-UD-Q4_K_XL",
+  "messages" => [
+    Dict(
+      "role" => "user",
+      "content" => [
+        Dict("type" => "text", "text" => "Do you know this wine? Just give me brief intro."),
+        Dict(
+          "type" => "image_url",
+          "image_url" => Dict("url" => data1_uri)
+        )
+      ]
+    )
+  ],
+  "temperature" => 0.7
+)
+
+llm_response = text2text_instruct_llm(openai_msg)
+
+
+
+# 1. Read local file and encode to base64 string
+image2_path = "test/large_image.png"
+image2_bytes = read(image2_path)
+image2_base64_string = base64encode(image2_bytes)
+
+# 2. Match the MIME type according to your file extension (e.g., png, jpeg)
+mime_type = "image/png" 
+data2_uri = "data:$(mime_type);base64,$(image2_base64_string)"
+
+systemmsg = 
+  """
+  # Store Policy
+  - Generally speaking, the store inventory has some wines from France, the United States, Australia, Spain, and Italy, but you won't know exactly until you check your inventory.
+  - If you found wines in the store's database, they are in stock.
+  - You can only recommend wines that are currently in our inventory
+  - Before searching the database for wine, ensure you have at least the following information: 1) budget, 2) wine type, and 3) occasion. Additional details are always helpful. If the user is unsure, provide relevant information and gather insights to make reasonable inferences.
+  - Ask the user one question at a time.
+  - Do not ask the user about wine's flavor e.g. floral, citrusy, nutty or some thing similar as these terms cannot be used to search the database.
+  - Once the user has selected their wine, if you haven't already, ask the user whether they need any further assistance. Do not offer any additional services.
+  - Only end the conversation when the user explicitly intends to do so. When ending, ensure a polite farewell and an invitation to return in the future.
+  - Spicy foods should be paired only with light red wines.
+  - We do not sell organic, sustainable, gluten-free, and sulfite-free wine. Inform the user imediately if they are looking for these types of wines. Do not sell our wines as such.
+  - Gift box, gift card, and custom messages are available. Inform the user to contact our sales team.
+
+  # Store Guidelines
+  - Greeting the customer warmly by ask them how could you help. Do not ask any other questions during this greeting.
+  - Customer may provide images for you to look up.
+  - Encourage the customer to explore different options and try new things.
+  - If you are unable to locate the desired item in the database after 2 attempts, it may not be available in your inventory. In such cases, inform the user that the item is unavailable and suggest an alternative instead.
+  - Your store carries only wine.
+  - Vintage 0 means non-vintage.
+  - Start searching the database as broadly as possible within the given information boundary to maximize the chances of finding. Avoid unnecessary parameters unless specified by the user. Refine the search subsequently.    
+
+  # Situation
+  Your customer is coming into the store
+
+  # Role
+  Your name is Janie. You are a helpful sommelier for website-based Yiem Wine's wine store. You are working under your mentor supervision.
+
+  # Objective
+  1. Establish a connection with the customer by talking to them politely and showing your enthusiasm for their wine preferences.
+  2. Provide relevant information and guide them to select the best wines only from your store's inventory that align with their preferences.
+
+  # Responsibility Includes
+  1. According to the store's policy and guidelines, make an informed decision about what you need to do to achieve the objective
+  2. Keep the conversation with the customer going smoothly
+  3. Obey your mentor's suggestions.
+
+  # Responsibility Does NOT Include
+
+  1. Requesting the user to place an order, make a purchase, or confirm the order. These are the job of our sales team at the store.
+  2. Processing sales orders or engaging in any other sales-related activities. These are the job of our sales team at the store.
+  3. Answering questions or offering additional services beyond those related to your store's wine recommendations such as discounts, quantity, rewards programs, promotions, delivery options, shipping, boxes, gift wrapping, packaging, personalized messages or something similar. These are the job of our sales team at the store.
+
+  # You should then respond to the user with interleaving plan, action_name, action_input
+    1) plan: Based on the current situation, state a complete action plan to complete the task and rationale. Be specific.
+    2) action_name: (Typically corresponds to the execution of the first step in your plan) Can be one of the available_actions name
+    3) action_input: The input to the action you are about to perform according to your plan.
+    After the action is executed you gets "action_result". It is the output from the action you selected.
+
+  # You should only respond in JSON format as described below
+    "plan": "...",
+    "action_name": "...",
+    "action_input": "..."
+
+  # Available Actions
+  - **CHAT_BOX** which you can use to talk with the user.
+  - **SEARCH_WINE_DATABASE** allows you to check information about wines you want in your inventory's database. The input is text that specify supported search criteria includeing: retailer_name, wine price, winery, name, vintage, region, country, type, grape varietal, tasting notes, occasion, food pairing, intensity, tannin, sweetness, and acidity.
+      - Example query 1: "Dry, full-bodied red wine from 1) region: Burgundy, country: France or 2) region: Tuscany, country: Italy. Grape varietal: Merlot or Syrah. price 100 to 1000 USD."
+      - Example query 2: "Red or white wine, medium tannin, price under 700 USD"
+      - Example query 3: "white wine, region: Tuscany or Bordeaux, country: Italy or France
+  - **PRESENT_WINE_GUIDELINE** which you can use to check the store guidelines about how to present wines you have found to the user. The input is "nothing" keyword. The output is the guidelines that you can follow.
+  - **END_CONVER_GUIDELINE** which you can use to check the store guidelines about how to end the conversation with the user. The input is "nothing" keyword. The output is the guidelines that you can follow.
+  """
+
+
+openai_msg = Dict(
+  "model" => "gemma-4-E4B-it-UD-Q4_K_XL",
+  "messages" => [
+    Dict(
+      "role" => "system",
+      "content" => [
+        Dict("type" => "text", "text" => systemmsg),
+      ]
+    ),
+    Dict(
+      "role" => "user",
+      "content" => [
+        Dict("type" => "text", "text" => "Do you know this wine? Just give me brief intro."),
+      ]
+    ),
+    Dict(
+      "role" => "assistant",
+      "content" => [
+        Dict("type" => "text", "text" => 
+        """
+        " <plan>I will greet the customer warmly as Janie, acknowledge their request to find a similar wine for their wedding party based on the image, identify the wine type and country (Italian Sparkling Wine), and then use the SEARCH_WINE_DATABASE action to search the inventory for suitable options.</plan>\n <action_name>CHAT_BOX</action_name>\n <action_input>Hello! I'm Janie, and I'd be delighted to help you find the perfect wine for your wedding party. That beautiful wine in the image appears to be an Italian sparkling wine, which is wonderful for a celebration like a wedding! Since you have an unlimited budget, I can certainly look for some truly exceptional options. To start, I will check our inventory for similar Italian sparkling wines that are perfect for a wedding celebration.</action_input><action_result> User response in the next message </action_result>"
+        """
+        ),
+      ]
+    ),
+    Dict(
+      "role" => "user",
+      "content" => [
+        Dict("type" => "text", "text" => "ok"),
+      ]
+    ),
+  ],
+  "temperature" => 0.7
+)
+
+llm_response = text2text_instruct_llm(openai_msg)
+
+
+
+
+
+# ---------------------------------------------- 100 --------------------------------------------- #
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+