120 lines
11 KiB
Markdown
120 lines
11 KiB
Markdown
# Solution Design: YiemAgent Agent Framework
|
|
|
|
## 1. Problem Decomposition
|
|
|
|
| Problem | Description | User Impact |
|
|
|---------|-------------|-------------|
|
|
| P-001 | Developers need a chatbot framework in Julia that handles LLM calls and tool calls without writing boilerplate | Must write separate request/response handling, tool call parsing, and loop management code for each agent |
|
|
| P-002 | Tool discovery is manual and repetitive | Must register every tool individually; no automatic discovery from MCP-compatible servers |
|
|
| P-003 | Multiple LLM providers have different response formats | Must handle `message.tool_calls` (LMStudio/vLLM) vs `content` blocks (OpenAI API) separately |
|
|
| P-004 | Tool execution needs lifecycle hooks for observability and control | No built-in before/after hooks for authorization, logging, or result sanitization |
|
|
| P-005 | Agent state management is error-prone | Must handle message history, tool registration, and pending tool call tracking manually |
|
|
|
|
## 2. Solution Approach
|
|
|
|
**Approach**: "Build a channel-based agent system where the LLM talks to tools automatically, tools talk to MCP servers automatically, and you compose agents by injecting `llmCall` and `mcpServer` functions — the framework owns the loop, the state, and the format conversion."
|
|
|
|
**Key Principles**:
|
|
- **Channel-based async processing**: Each agent owns three channels (`inputChannel`, `followUpChannel`, `outputChannel`); a background `@spawn` task runs the agent loop, processing messages in priority order
|
|
- **Tool call loop**: The agent calls `llmCall()`, extracts tool calls, executes them (sequentially or in parallel), appends results, and calls `llmCall()` again — repeating until no tool calls remain
|
|
- **Pluggable hooks**: `prepareContext`, `formatMsgForLLM`, `beforeToolCall`, `afterToolCall`, and `llmCall` are all injectable functions — the framework provides sensible defaults, but you override what you need
|
|
- **MCP-native tool discovery**: Tools are discovered at startup via MCP `tools/list` and can be discovered at runtime via the `listTools` builtin tool (idempotent registration with pagination support)
|
|
- **Format-agnostic LLM responses**: `_extractToolCalls()` handles both OpenAI content blocks and LMStudio/vLLM nested message format, allowing the same agent to work with any model provider
|
|
|
|
## 3. Decision Rationale
|
|
|
|
| Decision ID | Decision | Rationale | Alternatives Rejected |
|
|
|-------------|----------|-----------|----------------------|
|
|
| SD-001 | Use channel-based async processing with `@spawn` task | Provides non-blocking message queues; `inputChannel` blocks if full (backpressure); `followUpChannel` allows async follow-ups while agent is busy | Threading with shared state — race conditions; callback-based async — hard to compose |
|
|
| SD-002 | Background `_agentLoop` task runs message dispatch loop | Isolates message processing from construction; agent can be created and used concurrently | Synchronous agent.run(message) — blocks caller; manual task management — boilerplate |
|
|
| SD-003 | `_processMessage` runs `llmCall()` in an inner loop (while loop) until no tool calls remain | Handles multi-turn tool use transparently — agent decides when to stop based on LLM response | External loop in caller code — duplicates agent loop logic; single-turn only — no multi-step tool use |
|
|
| SD-004 | MCP server integration via callable struct `mcpServer(method, args...)` | MCP is model-agnostic protocol; callable struct allows type-stable single dispatch; supports NATS transport (struct has `natsConn`, `topic`, `senderID` fields) | Ad-hoc HTTP client for MCP — breaks NATS-only transports; tight coupling to specific transport |
|
|
| SD-005 | Tool call lifecycle: `prepareToolCall()` → `executePreparedToolCall()` → `finalizeExecutedToolCall()` — three phases, never throw errors | Preparation validates args and runs `beforeToolCall` hook (can block); execution calls `tool.execute()`; finalization runs `afterToolCall` hook. Each phase returns a result type, not exceptions — agent loop catches all and builds error tool result | `try/catch` in loop — loses context; throwing errors breaks tool result message contract |
|
|
| SD-006 | Message types are Julia structs with keyword constructors (`userMessage(content=[...])`) | Type stable; keyword constructors provide defaults (`role="user"`, `timestamp=now()`); supports both positional and keyword usage | `Dict{String, Any}` for messages — lose type safety, docgen, IDE support |
|
|
| SD-007 | `eventSink` function passed to all deep lifecycle calls | Provides hooks for external monitoring (logging, metrics, NATS pub/sub) without modifying core logic | Built-in `println` — can't disable or redirect; no metrics integration |
|
|
| SD-008 | `abortSignal` struct with `.aborted::Bool` field passed to `prepareToolCall()` and hooks | Simple mutable state for interrupting long-running tool calls (no async cancellation — just flag check) | Threading with `Task.cancel()` — Julia thread model doesn't support it; `@async` spawn — complex cleanup |
|
|
| SD-009 | `_extractToolCalls()` supports both `message.tool_calls` and `content[{type="tool_calls"}]` formats | LMStudio.jl/vLLM vs OpenAI API have different response shapes; framework must work with both | Separate parser per provider — maintenance burden; require provider wrapper — adds indirection |
|
|
|
|
## 4. High-Level Component Diagram
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
User[External Caller] -->|promptAgent / followUp| InputCh[inputChannel]
|
|
|
|
subagent AgentLoop [Agent Loop Task]
|
|
InputCh -->|take!| Disp[Message Dispatcher]
|
|
Disp -->|spawn| Proc[_processMessage]
|
|
Proc -->|Channel| PrCh[processMessageInputCh]
|
|
end
|
|
|
|
subagent Message Pipeline
|
|
PrCh -->|put!| DMsg[Drain input, convert to userMessage]
|
|
DMsg --> PC[prepareContext hook]
|
|
PC --> FM[formatMsgForLLM hook]
|
|
FM --> LLM[llmCall]
|
|
LLM --> EC[_extractToolCalls]
|
|
EC -->|hasToolCalls| TC[executeToolCalls]
|
|
TC -->|toolResultBatch| TM[toolResultMessage]
|
|
TM -->|push!| MH[messageHistory]
|
|
TC -->|no tool calls break| BRK[break loop]
|
|
BRK --> End[exit]
|
|
end
|
|
|
|
subagent Tool System
|
|
TC --> Prep[prepareToolCall]
|
|
Prep --> Exec[executePreparedToolCall]
|
|
Exec --> Term[finalizeExecutedToolCall]
|
|
Term --> CRM[createToolResultMessage]
|
|
CRM --> TM
|
|
end
|
|
|
|
subagent MCP Integration
|
|
Prep --> ToolLookup[context.tools lookup]
|
|
ToolLookup --> MCP[MCP tools/list + tools/call]
|
|
MCP --> Wr[_wrap_mcp_tool]
|
|
Wr --> Reg[toolRegistry.registerAllTools]
|
|
end
|
|
|
|
OutputCh[outputChannel] --> User
|
|
Proc -->|final response| OutputCh
|
|
```
|
|
|
|
**Component Descriptions**:
|
|
- **Agent Loop Task** (`SD-001`, `SD-002`): Runs `@spawn _agentLoop()`; manages three channels; dispatches messages to `_processMessage`
|
|
- **Message Pipeline** (`SD-003`, `SD-004`): `prepareContext` → `formatMsgForLLM` → `llmCall` → extract tool calls → loop
|
|
- **Tool System** (`SD-005`): Three-phase lifecycle (prepare → execute → finalize); never throws; returns error tool result on failure
|
|
- **MCP Integration** (`SD-004`): `tools/list` discovery with pagination; `_wrap_mcp_tool` creates executable `agentTool` with MCP JSON-RPC calls
|
|
|
|
## 5. Alternatives Considered
|
|
|
|
| Alternative | Pros | Cons | Decision |
|
|
|-------------|------|------|----------|
|
|
| Synchronous `agent.process(message)` | Simpler API; no channels or tasks | Blocks caller; no follow-up support; harder to test concurrency | **Rejected** — async channels needed for real-time chat UX |
|
|
| JSON dicts for messages everywhere | Flexible; no type definitions | No static checking; no docgen; no IDE autocomplete; harder to serialize/deserialize | **Rejected** — Julia advantages of structs (type-stable, documented) outweigh cost |
|
|
| Single `onToolCall` hook (pre-execute only) | Simpler hook interface | Can't sanitize results post-execution; can't set terminate flag after tool completes | **Rejected** — `afterToolCall` needed for masking secrets, normalizing usage, inspecting error status |
|
|
| Direct HTTP calls to MCP servers (no NATS) | Simpler dependency (no NATS) | Breaks if MCP uses NATS (common in microservices); no single transport | **Rejected** — callable struct abstracts transport; NATS is one implementation option |
|
|
| External orchestrator runs tool call loop | Caller has full control | Duplicates logic; every caller must implement same loop; harder to maintain | **Rejected** — loop is agent framework responsibility (single source of truth) |
|
|
|
|
## 6. Risk Assessment
|
|
|
|
| Risk | Impact | Probability | Mitigation |
|
|
|------|--------|-------------|------------|
|
|
| `llmCall` throws (network timeout, API error) | Agent loop exits with error in `eventSink` | Medium | `try/catch` in `_processMessage` and `_agentLoop`; eventSink receives error string |
|
|
| Tool `prepareArguments` or `validateRequiredArgs` hook throws | `prepareToolCall` catches and returns `immediateOutcome` error | Low | Hook errors caught; error tool result sent to LLM for recovery |
|
|
| MCP server unreachable | Tool discovery returns 0 tools; `listTools` still available | Medium | `register_mcp_tools` returns 0 on error; tool execution calls catch errors |
|
|
| Long-running tool call hangs | Caller blocks on `outputChannel`; no built-in timeout | Medium | `eventSink` passes abort signal; caller sets `signal.aborted = true` externally |
|
|
| Multiple agents share same `mcpServer` struct | `mcpServer` must be thread-safe for concurrent tool calls | Low | Each thread gets own `mcpServer` instance; NATS handles concurrency |
|
|
| Message history grows unbounded | Memory exhaustion in long sessions | Medium | No built-in pruning; `prepareContext` can filter/prune messages (TODO in code) |
|
|
|
|
## 7. Requirements Traceability
|
|
|
|
| Solution Component | Requirement ID | Decision ID | Description |
|
|
|-------------------|----------------|-------------|-------------|
|
|
| Channel-based async processing | FR-001 | SD-001, SD-002 | `inputChannel`, `followUpChannel`, `outputChannel` with priority dispatch |
|
|
| LLM call loop | FR-003 | SD-003 | `_processMessage` while loop; `llmCall()` → extract → execute → append → repeat |
|
|
| MCP tool discovery | FR-004 | SD-004, SD-009 | `tools/list` with pagination; `tools/call` execution |
|
|
| Lifecycle hooks | FR-005 | SD-005 | `beforeToolCall` can block; `afterToolCall` transforms; never throw |
|
|
| Message type conversion | FR-006 | SD-006 | Julia structs with keyword constructors; `_userMessageToOpenAI`, `_messageContentToBlocks` |
|
|
| Format-agnostic extraction | FR-002 | SD-009 | `_extractToolCalls` handles both `message.tool_calls` and `content` blocks |
|
|
| Event sink observability | NFR-201, NFR-103 | SD-007 | `eventSink` passed to all lifecycle steps; logs state machine transitions |
|