Files
pi_harness/packages/agent/learning/06-AGENTHARNESS-REFERENCE.md
2026-07-29 10:59:18 +07:00

804 lines
20 KiB
Markdown

# AgentHarness Reference
## Overview
`AgentHarness` is the **high-level API** that wraps the core agent with session management, persistence, branching, and tool context binding.
---
## Key Differences: Agent vs AgentHarness
| Feature | Agent (Core) | AgentHarness |
|---------|-------------|--------------|
| **Session Persistence** | No | Yes (JSONL/Memory) |
| **Branching** | No | Yes |
| **Context Compaction** | No | Yes |
| **Tool Context** | Manual | Automatic binding |
| **Skills/Templates** | Manual | Built-in |
| **State Management** | Manual | Automatic |
| **Event Hooks** | Basic | Rich system |
---
## AgentHarness Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ AgentHarness │
├─────────────────────────────────────────────────────────────┤
│ State │
│ ├─ Session (persistence) │
│ ├─ Model │
│ ├─ ThinkingLevel │
│ ├─ Tools (Map) │
│ ├─ ActiveTools (string[]) │
│ └─ SystemPrompt (string or function) │
│ │
│ Queues │
│ ├─ steerQueue (messages to interrupt agent) │
│ ├─ followUpQueue (messages after agent stops) │
│ └─ nextTurnQueue (messages for next turn) │
│ │
│ Hooks │
│ ├─ before_agent_start │
│ ├─ context │
│ ├─ tool_call │
│ ├─ tool_result │
│ ├─ session_before_compact │
│ ├─ session_before_tree │
│ ├─ before_provider_request │
│ └─ before_provider_payload │
│ │
│ Methods │
│ ├─ prompt() - Run new conversation │
│ ├─ skill() - Execute skill │
│ ├─ promptFromTemplate() - Run template │
│ ├─ steer() - Interrupt agent │
│ ├─ followUp() - Queue message │
│ ├─ compact() - Compress context │
│ ├─ navigateTree() - Branch session │
│ └─ subscribe() - Add event listener │
└─────────────────────────────────────────────────────────────┘
```
---
## Core Concepts
### 1. Session
The session holds **conversation history as a tree**:
```typescript
interface Session<TMetadata extends SessionMetadata = SessionMetadata> {
readonly id: string;
readonly storage: SessionStorage<TMetadata>;
getMetadata(): Promise<TMetadata>;
getLeafId(): Promise<string>;
getEntry(id: string): Promise<SessionTreeEntry | undefined>;
getBranch(): Promise<SessionTreeEntry[]>;
buildContext(options?: SessionContextBuildOptions): Promise<SessionContext>;
appendMessage(message: AgentMessage): Promise<string>;
appendModelChange(provider: string, modelId: string): Promise<string>;
appendThinkingLevelChange(thinkingLevel: ThinkingLevel): Promise<string>;
appendActiveToolsChange(activeToolNames: string[]): Promise<string>;
appendCompaction(...): Promise<string>;
appendBranchSummary(...): Promise<string>;
fork(targetId: string): Promise<Session>;
}
```
### 2. Resources
Skills and prompt templates available to the agent:
```typescript
interface AgentHarnessResources<TSkill = Skill, TPromptTemplate = PromptTemplate> {
skills?: TSkill[];
promptTemplates?: TPromptTemplate[];
}
interface Skill {
name: string;
description: string;
content: string;
filePath: string;
disableModelInvocation?: boolean;
}
interface PromptTemplate {
name: string;
description?: string;
content: string;
}
```
### 3. Tool Context
Context passed to all tool executions:
```typescript
interface ToolContext {
userId: string;
environment: "dev" | "staging" | "prod";
// ... custom properties
}
// Zero-arg function for dynamic context
type ToolContextProvider<TContext> = () => TContext | Promise<TContext>;
```
---
## AgentHarness API
### Constructor
```typescript
constructor(options: AgentHarnessOptions<TContext, TSkill, TPromptTemplate, TTool>)
```
**Options**:
```typescript
interface AgentHarnessOptions<TContext, TSkill, TPromptTemplate, TTool> {
session: Session; // Session storage
models: Models; // LLM provider
resources?: AgentHarnessResources<TSkill, TPromptTemplate>;
streamOptions?: AgentHarnessStreamOptions;
retry?: RetryPolicy;
// System prompt
systemPrompt?:
| string // Static string
| AgentHarnessSystemPrompt<TContext, TSkill, TPromptTemplate, TTool>; // Dynamic function
// Tool context
toolContext?: AgentHarnessToolContextSource<TContext>;
// Tools
tools?: TTool[];
// Active tools
activeToolNames?: string[];
// Model and thinking
model: Model<any>;
thinkingLevel?: ThinkingLevel;
// Queue modes
steeringMode?: QueueMode;
followUpMode?: QueueMode;
}
```
**Example**:
```typescript
const harness = new AgentHarness({
session: memorySession,
models: models,
resources: {
skills: [weatherSkill, gitSkill],
promptTemplates: [summaryTemplate]
},
systemPrompt: async ({ session, model, activeTools, resources }) => {
const sessionMetadata = await session.getMetadata();
const toolsList = activeTools.map(t => t.name).join(", ");
return `You are an AI assistant with access to tools: ${toolsList}.
Current session: ${sessionMetadata.id}
Date: ${new Date().toISOString()}
Available skills:
${resources.skills?.map(s => `- ${s.name}: ${s.description}`).join("\n")}
`;
},
toolContext: { userId: "user123", environment: "prod" },
tools: [weatherTool, gitTool, readFileTool],
activeToolNames: ["weather", "git"],
model: gpt4Model,
thinkingLevel: "medium"
});
```
### System Prompt
**Static string**:
```typescript
systemPrompt: "You are a helpful assistant."
```
**Dynamic function**:
```typescript
systemPrompt: async ({
session,
model,
thinkingLevel,
activeTools,
resources
}) => {
const metadata = await session.getMetadata();
return `System: ${metadata.id}
Model: ${model.id}
Date: ${new Date().toISOString()}
Active tools: ${activeTools.map(t => t.name).join(", ")}
`;
};
```
---
## Main Methods
### `prompt()`
Run a new prompt:
```typescript
async prompt(text: string, options?: { images?: ImageContent[] }): Promise<AssistantMessage>
```
**Flow**:
1. Validate harness is idle
2. Create turn state (context, tools, system prompt)
3. Emit `before_agent_start` hook
4. Run agent loop with prompt
5. Return assistant message
**Example**:
```typescript
const message = await harness.prompt("What's the weather in London?");
console.log(message.content); // Assistant response
```
### `skill()`
Execute a named skill:
```typescript
async skill(name: string, additionalInstructions?: string): Promise<AssistantMessage>
```
**Example**:
```typescript
const message = await harness.skill("git", "Also create a PR for the changes");
// Skill content injected into prompt
```
### `promptFromTemplate()`
Execute a prompt template:
```typescript
async promptFromTemplate(
name: string,
args: string[] = []
): Promise<AssistantMessage>
```
**Example**:
```typescript
// Template: "Fix the following error: {{0}}"
const message = await harness.promptFromTemplate("fix_error", ["TypeError: x is undefined"]);
```
### `steer()`
Interrupt agent mid-execution:
```typescript
async steer(text: string, options?: { images?: ImageContent[] }): Promise<void>
```
**Example**:
```typescript
await harness.prompt("Write a long report...");
// While agent is working...
await harness.steer("Wait, change focus to climate change");
// Agent continues with new instructions
```
### `followUp()`
Queue message for after agent stops:
```typescript
async followUp(text: string, options?: { images?: ImageContent[] }): Promise<void>
```
**Example**:
```typescript
await harness.prompt("Analyze this data...");
// Agent finishes...
await harness.followUp("Now create a summary");
// Agent continues with summary request
```
### `nextTurn()`
Queue message for next turn (doesn't interrupt current turn):
```typescript
async nextTurn(text: string, options?: { images?: ImageContent[] }): Promise<void>
```
**Difference from `steer()`**:
- `steer()`: Interrupts immediately
- `nextTurn()`: Waits for current turn to finish
### `compact()`
Compress conversation history:
```typescript
async compact(customInstructions?: string): Promise<CompactResult>
```
**Returns**:
```typescript
interface CompactResult {
summary: string;
firstKeptEntryId?: string;
tokensBefore: number;
usage?: Usage;
retainedTail?: AgentMessage[];
details?: unknown;
}
```
**Example**:
```typescript
const result = await harness.compact();
console.log(`Compressed from ${result.tokensBefore} tokens to summary`);
```
### `navigateTree()`
Navigate conversation tree (branching):
```typescript
async navigateTree(
targetId: string,
options?: {
summarize?: boolean;
customInstructions?: string;
replaceInstructions?: boolean;
label?: string;
}
): Promise<NavigateTreeResult>
```
**Returns**:
```typescript
interface NavigateTreeResult {
cancelled: boolean;
editorText?: string; // If target is user message
summaryEntry?: BranchSummaryEntry;
}
```
**Example**:
```typescript
// Navigate to earlier point in conversation
const result = await harness.navigateTree("entry_abc123", { summarize: true });
// Create branch from current point
const newHarness = createNewHarness();
await newHarness.navigateTree("entry_xyz789");
```
---
## State Management
### Model
```typescript
getModel(): Model<any>;
async setModel(model: Model<any>): Promise<void>;
```
**Example**:
```typescript
console.log(harness.getModel().id); // "gpt-4"
await harness.setModel(gpt4oModel);
```
### Thinking Level
```typescript
getThinkingLevel(): ThinkingLevel;
async setThinkingLevel(level: ThinkingLevel): Promise<void>;
```
**Levels**: `"off"`, `"minimal"`, `"low"`, `"medium"`, `"high"`, `"xhigh"`, `"max"`
**Example**:
```typescript
await harness.setThinkingLevel("high"); // More reasoning for complex tasks
```
### Tools
```typescript
getTools(): TTool[];
getActiveTools(): TTool[];
async setTools(tools: TTool[], activeToolNames?: string[]): Promise<void>;
async setActiveTools(toolNames: string[]): Promise<void>;
```
**Example**:
```typescript
// Add new tool
await harness.setTools([...harness.getTools(), newTool]);
// Change active tools
await harness.setActiveTools(["read", "write"]);
```
### Resources
```typescript
getResources(): AgentHarnessResources<TSkill, TPromptTemplate>;
async setResources(resources: AgentHarnessResources<TSkill, TPromptTemplate>): Promise<void>;
```
**Example**:
```typescript
await harness.setResources({
skills: [...harness.getResources().skills, newSkill]
});
```
---
## Queue Management
### Steering Queue
```typescript
getSteeringMode(): QueueMode;
async setSteeringMode(mode: QueueMode): Promise<void>;
```
**Modes**:
- `"all"`: Drain all queued messages at once
- `"one-at-a-time"`: Drain one message at a time
### Follow-up Queue
```typescript
getFollowUpMode(): QueueMode;
async setFollowUpMode(mode: QueueMode): Promise<void>;
```
### Queue Helpers
```typescript
// Clear all queued messages
harness.clearAllQueues();
// Check if queues have pending messages
harness.hasQueuedMessages(); // boolean
```
---
## Event Handling
### Subscribe to All Events
```typescript
subscribe(
listener: (event: AgentHarnessEvent<TSkill, TPromptTemplate>, signal?: AbortSignal) => Promise<void> | void
): () => void;
```
**Event types**:
```typescript
type AgentHarnessEvent<TSkill, TPromptTemplate> =
// Agent events (forwarded from core agent)
| { type: "agent_start" }
| { type: "agent_end"; messages: AgentMessage[] }
| { type: "turn_start" }
| { type: "turn_end"; message: AgentMessage; toolResults: ToolResultMessage[] }
| { type: "message_start"; message: AgentMessage }
| { type: "message_update"; message: AgentMessage; assistantMessageEvent: AssistantMessageEvent }
| { type: "message_end"; message: AgentMessage }
| { type: "tool_execution_start"; toolCallId: string; toolName: string; args: any }
| { type: "tool_execution_update"; toolCallId: string; toolName: string; args: any; partialResult: any }
| { type: "tool_execution_end"; toolCallId: string; toolName: string; result: any; isError: boolean }
// Harness-specific events
| { type: "before_agent_start"; ... }
| { type: "context"; messages: AgentMessage[] }
| { type: "tool_call"; ... }
| { type: "tool_result"; ... }
| { type: "session_before_compact"; ... }
| { type: "session_before_tree"; ... }
| { type: "before_provider_request"; ... }
| { type: "before_provider_payload"; ... }
| { type: "after_provider_response"; ... }
| { type: "save_point"; ... }
| { type: "settled"; ... }
| { type: "model_update"; ... }
| { type: "thinking_level_update"; ... }
| { type: "tools_update"; ... }
| { type: "resources_update"; ... }
| { type: "session_compact"; ... }
| { type: "session_tree"; ... }
| { type: "queue_update"; ... }
| { type: "retry_scheduled"; ... }
| { type: "retry_attempt_start"; ... }
| { type: "retry_finished"; ... }
| { type: "abort"; clearedSteer: UserMessage[]; clearedFollowUp: UserMessage[] };
```
**Example**:
```typescript
const unsubscribe = harness.subscribe(async (event, signal) => {
if (event.type === "message_end") {
console.log("Message:", event.message.role);
}
if (event.type === "agent_end") {
console.log("Conversation complete");
}
if (event.type === "tool_execution_end") {
console.log("Tool:", event.toolName, "completed");
}
});
```
### Subscribe to Specific Events
```typescript
on<TType extends keyof AgentHarnessEventResultMap>(
type: TType,
handler: (event: Extract<AgentHarnessOwnEvent, { type: TType }>) => Promise<AgentHarnessEventResultMap[TType]> | AgentHarnessEventResultMap[TType]
): () => void;
```
**Example**:
```typescript
// Handle tool calls
harness.on("tool_call", async ({ toolCallId, toolName, input }) => {
console.log(`Tool ${toolName} called with:`, input);
return undefined; // Allow execution
});
// Handle tool results
harness.on("tool_result", async ({ toolName, content, isError }) => {
console.log(`Tool ${toolName} result:`, isError ? "Error" : "Success");
return undefined; // Use default result
});
// Modify system prompt
harness.on("before_agent_start", async ({ systemPrompt }) => {
return {
systemPrompt: `${systemPrompt}\n\nRemember to be concise.`
};
});
```
---
## Session Persistence
### Append Message
```typescript
async appendMessage(message: AgentMessage): Promise<void>;
```
**Example**:
```typescript
// Manually add message to session
await harness.appendMessage({
role: "user",
content: [{ type: "text", text: "Custom message" }],
timestamp: Date.now()
});
```
### Flush Pending Writes
```typescript
async abort(): Promise<AbortResult>
```
**Returns**:
```typescript
interface AbortResult {
clearedSteer: UserMessage[];
clearedFollowUp: UserMessage[];
}
```
**Example**:
```typescript
const result = await harness.abort();
console.log(`Cleared ${result.clearedSteer.length} steering messages`);
```
### Wait for Idle
```typescript
async waitForIdle(): Promise<void>;
```
**Example**:
```typescript
await harness.prompt("Do something...");
await harness.waitForIdle(); // Wait for completion
console.log("Done");
```
---
## Error Handling
### Error Codes
```typescript
type AgentHarnessErrorCode =
| "busy" // Agent is already processing
| "invalid_state" // Invalid state for operation
| "invalid_argument" // Invalid arguments
| "session" // Session error
| "hook" // Hook error
| "auth" // Authentication error
| "compaction" // Compaction error
| "branch_summary" // Branch summary error
| "unknown"; // Unknown error
```
### Error Handling Pattern
```typescript
try {
await harness.prompt("Do something");
} catch (error) {
if (error instanceof AgentHarnessError) {
switch (error.code) {
case "busy":
console.log("Agent busy, try again later");
break;
case "compaction":
console.log("Compaction failed:", error.message);
break;
case "hook":
console.log("Hook error:", error.cause?.message);
break;
default:
console.log("Error:", error.message);
}
}
}
```
---
## Advanced Patterns
### 1. Dynamic System Prompt
```typescript
systemPrompt: async ({ session, model, activeTools, resources }) => {
const metadata = await session.getMetadata();
// Customize based on session type
if (metadata.type === "coding") {
return `You are a coding assistant. Use tools: ${activeTools.map(t => t.name).join(", ")}`;
} else if (metadata.type === "writing") {
return `You are a writing assistant. Focus on clarity and style.`;
}
return "You are a helpful assistant.";
}
```
### 2. Conditional Tool Activation
```typescript
// Enable tools based on user request
harness.on("before_agent_start", async ({ prompt }) => {
if (prompt.includes("weather")) {
return {
messages: [{ role: "user", content: [{ type: "text", text: "Enable weather tool" }] }]
};
}
return undefined;
});
```
### 3. Session Branching
```typescript
async function exploreAlternative(harness: AgentHarness, prompt: string): Promise<AssistantMessage> {
// Get current leaf
const leafId = await harness.session.getLeafId();
// Create branch
const branchSession = await harness.session.fork(leafId);
const branchHarness = new AgentHarness({
...harnessOptions,
session: branchSession
});
// Run alternative
return await branchHarness.prompt(prompt);
}
```
### 4. Custom Compaction
```typescript
harness.on("session_before_compact", async ({ preparation }) => {
// Skip compaction for short sessions
if (preparation.tokensBefore < 1000) {
return { cancel: true };
}
// Provide custom summary
return {
compaction: {
summary: "User asked about X, Y, Z and assistant provided guidance.",
tokensBefore: preparation.tokensBefore,
firstKeptEntryId: preparation.firstKeptEntry.id,
details: { manual: true }
}
};
});
```
### 5. Tool Execution Logging
```typescript
harness.on("tool_call", async ({ toolName, input }) => {
console.log(`[TOOL_CALL] ${toolName}:`, JSON.stringify(input, null, 2));
return undefined;
});
harness.on("tool_result", async ({ toolName, content, isError }) => {
console.log(`[TOOL_RESULT] ${toolName}:`, isError ? "❌" : "✅");
return undefined;
});
```
---
## Summary
**AgentHarness provides**:
- Session persistence and tree navigation
- Built-in tool context binding
- Rich hook system for customization
- Skills and prompt templates
- Context compaction and branching
**Key methods**:
- `prompt()` - Main interaction
- `steer()` / `followUp()` - Queue management
- `compact()` - Context management
- `navigateTree()` - Branching
**Key patterns**:
- Dynamic system prompts
- Conditional tool activation
- Session branching for experimentation
- Hook-based customization