update
This commit is contained in:
@@ -1,57 +1,211 @@
|
||||
# ── executeToolCalls() Julia pseudo code ──────────────────────────
|
||||
# Full call stack from runLoop → executeToolCalls → prepare → execute → finalize → emit
|
||||
|
||||
"""
|
||||
preparedToolCall(tool, toolCall, args)
|
||||
|
||||
Intermediate state between tool call validation and execution.
|
||||
This struct is created after `prepareToolCall` succeeds and serves
|
||||
as the bridge to the execution phase. Keeping the resolved tool,
|
||||
original call metadata, and validated args together avoids repeated
|
||||
lookups and allows the execution phase to access all necessary data
|
||||
without carrying the full context through the call chain.
|
||||
"""
|
||||
struct preparedToolCall
|
||||
tool::AgentTool
|
||||
toolCall::AgentToolCall
|
||||
args::Any
|
||||
tool::agentTool # The resolved tool definition from the context
|
||||
toolCall::agentToolCall # The original tool call from the assistant
|
||||
args::any # Validated (and coerced) argument values
|
||||
end
|
||||
|
||||
"""
|
||||
immediateOutcome(result, isError)
|
||||
|
||||
A tool call that was resolved without actual execution — either
|
||||
because the tool was not found, validation failed, or a
|
||||
`beforeToolCall` hook blocked the call. The result is produced
|
||||
immediately and emitted as a tool result message.
|
||||
|
||||
Returning an outcome instead of throwing an exception is intentional:
|
||||
it lets the agent feed the error back to the LLM as a tool result so
|
||||
the model can recover — for example, by re-issuing a tool call with
|
||||
corrected arguments after a validation failure.
|
||||
"""
|
||||
struct immediateOutcome
|
||||
result::AgentToolResult
|
||||
isError::Bool
|
||||
result::agentToolResult # The pre-computed tool result
|
||||
isError::bool # Whether this outcome represents an error
|
||||
end
|
||||
|
||||
"""
|
||||
executedOutcome(result, isError)
|
||||
|
||||
A tool call that has been executed by `tool.execute()` but has not
|
||||
yet been through the `afterToolCall` hook. This intermediate state
|
||||
is necessary because the hook may mutate the result (content, usage,
|
||||
termination, error status). Keeping execution and finalization separate
|
||||
allows the hook to inspect the raw result and decide whether to
|
||||
transform it or replace it entirely.
|
||||
"""
|
||||
struct executedOutcome
|
||||
result::AgentToolResult
|
||||
isError::Bool
|
||||
result::agentToolResult # The tool's execution result
|
||||
isError::bool # Whether execution raised an error
|
||||
end
|
||||
|
||||
"""
|
||||
finalizedOutcome(toolCall, result, isError)
|
||||
|
||||
The complete outcome of a tool call after both execution and the
|
||||
`afterToolCall` hook. This is the final form that is used to
|
||||
construct the `toolResultMessage` emitted to the agent loop.
|
||||
|
||||
The three-phase design (prepare → execute → finalize) exists so that
|
||||
each phase has a single responsibility: preparation handles validation
|
||||
and gating, execution performs the actual work, and finalization
|
||||
applies post-processing hooks. This separation allows the agent loop
|
||||
to emit `tool_execution_end` events with the finalized data while
|
||||
keeping each phase independently testable and swappable.
|
||||
"""
|
||||
struct finalizedOutcome
|
||||
toolCall::AgentToolCall
|
||||
result::AgentToolResult
|
||||
isError::Bool
|
||||
toolCall::agentToolCall # The original tool call reference
|
||||
result::agentToolResult # The final tool result (post-afterToolCall)
|
||||
isError::bool # Whether the call failed or was blocked
|
||||
end
|
||||
|
||||
"""
|
||||
ToolCallBatch(messages, terminate)
|
||||
|
||||
A batch of tool result messages from executing one or more tool calls.
|
||||
The `terminate` flag indicates whether all tools in the batch requested
|
||||
termination, which causes the agent loop to stop processing further turns.
|
||||
|
||||
This flag is set by the tool implementation (not the end user) to signal
|
||||
that the agent should not call the LLM again. Typical use cases:
|
||||
|
||||
- Task completion: a tool like `deploy` or `submit` finishes its work and
|
||||
returns `terminate: true` so the agent stops instead of asking the LLM
|
||||
what to do next.
|
||||
- Unrecoverable error: a tool hits a fatal condition (e.g. database
|
||||
connection lost, auth token expired) and returns `terminate: true` so
|
||||
the agent stops with an error message rather than retrying.
|
||||
- Async handoff: a tool triggers a long-running external operation and
|
||||
wants the agent to stop now; the external system will later resume the
|
||||
agent via `continue()`.
|
||||
|
||||
If `terminate` is `false` (default), the agent loop feeds the tool results
|
||||
back to the LLM for another turn.
|
||||
"""
|
||||
struct toolCallBatch
|
||||
messages::Vector{ToolResultMessage}
|
||||
terminate::Bool
|
||||
messages::vector{toolResultMessage} # Tool result messages for this batch
|
||||
terminate::bool # Whether the batch should terminate the loop
|
||||
end
|
||||
|
||||
# ── helpers ─────────────────────────────────────────────────────
|
||||
"""
|
||||
createErrorToolResult(msg)
|
||||
|
||||
function createErrorToolResult(msg::String)::AgentToolResult
|
||||
return AgentToolResult([TextContent("text", msg)], Dict{Any,Any}())
|
||||
Builds an `agentToolResult` containing a single text content item
|
||||
with the provided error message and an empty details dictionary.
|
||||
Used when a tool call cannot be executed due to errors.
|
||||
|
||||
Returning a result instead of throwing ensures that errors at any
|
||||
point in the tool call pipeline are fed back to the LLM as a tool
|
||||
result message. This allows the model to see the error and decide
|
||||
whether to retry, re-issue the call with different arguments, or
|
||||
report failure to the user.
|
||||
"""
|
||||
function createErrorToolResult(msg::String)::agentToolResult
|
||||
return agentToolResult([textContent("text", msg)], dict{any,any}())
|
||||
end
|
||||
|
||||
function createToolResultMessage(f::finalizedOutcome)::ToolResultMessage
|
||||
return ToolResultMessage(
|
||||
"""
|
||||
createToolResultMessage(f)
|
||||
|
||||
Constructs a `toolResultMessage` from a `finalizedOutcome`.
|
||||
Normalizes missing content to an empty array and includes
|
||||
the `addedToolNames` field only when the tool dynamically
|
||||
registered new tools during execution.
|
||||
|
||||
This conversion is necessary because the tool result is an
|
||||
`agentToolResult` used by tool implementations, while the agent
|
||||
loop consumes `toolResultMessage` objects that become part of the
|
||||
conversation history. The message format includes metadata like
|
||||
timestamp and tool call ID that the raw result does not carry,
|
||||
and it is the object emitted via `message_start`/`message_end`
|
||||
events so the LLM receives the result as a proper assistant/user
|
||||
message in the context window.
|
||||
"""
|
||||
function createToolResultMessage(f::finalizedOutcome)::toolResultMessage
|
||||
return toolResultMessage(
|
||||
"toolResult", f.toolCall.id, f.toolCall.name,
|
||||
f.result.content, f.result.details, f.result.usage,
|
||||
get(f.result, :addedToolNames, String[]), f.isError, now_millis()
|
||||
get(f.result, :addedToolNames, string[]), f.isError, nowMillis()
|
||||
)
|
||||
end
|
||||
|
||||
function shouldTerminate(batches::Vector{finalizedOutcome})::Bool
|
||||
"""
|
||||
shouldTerminate(finalizedCalls)
|
||||
|
||||
Returns `true` only when every finalized call in the batch has
|
||||
`result.terminate == true`. All tools must agree — if any tool
|
||||
did not request termination, the agent continues. This prevents
|
||||
a single tool that happens to set `terminate: true` (e.g. for
|
||||
metadata purposes) from accidentally stopping the agent when
|
||||
other tools in the batch did not intend to terminate.
|
||||
"""
|
||||
function shouldTerminate(batches::vector{finalizedOutcome})::bool
|
||||
return !isempty(batches) && all(b -> b.result.terminate, batches)
|
||||
end
|
||||
|
||||
# ── per-call preparation ────────────────────────────────────────
|
||||
"""
|
||||
prepareToolCallArguments(tool, toolCall)
|
||||
|
||||
Calls the tool's optional `prepareArguments` hook to transform
|
||||
the raw argument values from the LLM before schema validation.
|
||||
If the tool has no hook or the hook returns the same object
|
||||
reference, the original call is returned unchanged.
|
||||
|
||||
This hook allows tools to normalize arguments that the LLM may
|
||||
have produced in a non-standard format — for example, converting
|
||||
a date string to a timestamp, expanding a short file path to an
|
||||
absolute path, or normalizing casing. It runs before schema
|
||||
validation so the validator sees the normalized form rather than
|
||||
raw LLM output.
|
||||
"""
|
||||
function prepareToolCallArguments(tool::agentTool, toolCall::agentToolCall)::agentToolCall
|
||||
if tool.prepareArguments === nothing
|
||||
return toolCall
|
||||
end
|
||||
prepared = tool.prepareArguments(toolCall.arguments)
|
||||
if prepared == toolCall.arguments
|
||||
return toolCall
|
||||
end
|
||||
return merge(toolCall, dict(:arguments => prepared))
|
||||
end
|
||||
|
||||
"""
|
||||
prepareToolCall(context, assistantMsg, toolCall, config, signal)
|
||||
|
||||
Resolves the tool by name, prepares and validates its arguments,
|
||||
and runs the `beforeToolCall` hook. Returns a `preparedToolCall`
|
||||
if successful or an `immediateOutcome` if the tool is not found,
|
||||
validation fails, the hook blocks execution, or the signal is
|
||||
aborted. Errors during preparation are caught and returned as
|
||||
immediate error outcomes so the agent loop can feed them back
|
||||
to the model.
|
||||
|
||||
The key design decision here is that preparation never throws.
|
||||
Every failure path returns an `immediateOutcome` with an error
|
||||
result. This ensures the agent loop always receives a valid tool
|
||||
result message for every tool call the assistant requested,
|
||||
regardless of whether preparation succeeded. The LLM can then
|
||||
use the error message to decide whether to retry with different
|
||||
arguments or acknowledge the failure.
|
||||
"""
|
||||
function prepareToolCall(
|
||||
context::AgentContext,
|
||||
assistantMsg::AssistantMessage,
|
||||
toolCall::AgentToolCall,
|
||||
config::AgentLoopConfig,
|
||||
signal::Union{Nothing,AbortSignal},
|
||||
)::Union{preparedToolCall,immediateOutcome}
|
||||
context::agentContext,
|
||||
assistantMsg::assistantMessage,
|
||||
toolCall::agentToolCall,
|
||||
config::agentLoopConfig,
|
||||
signal::union{nothing,abortSignal},
|
||||
)::union{preparedToolCall,immediateOutcome}
|
||||
|
||||
tool = find(t -> t.name == toolCall.name, context.tools)
|
||||
if tool === nothing
|
||||
@@ -60,13 +214,13 @@ function prepareToolCall(
|
||||
|
||||
try
|
||||
# 1. prepare arguments (tool-specific transform)
|
||||
preparedArgs = prepareToolCallArguments(tool, toolCall)
|
||||
validatedArgs = validateToolArguments(tool, preparedArgs)
|
||||
prepared = prepareToolCallArguments(tool, toolCall)
|
||||
validatedArgs = validateToolArguments(tool, prepared)
|
||||
|
||||
# 2. beforeToolCall hook
|
||||
if config.before_tool_call !== nothing
|
||||
before = config.before_tool_call(
|
||||
AssistantMsgCtx(assistantMsg, toolCall, validatedArgs, context), signal
|
||||
# 2. beforeToolCall hook — can block
|
||||
if config.beforeToolCall !== nothing
|
||||
before = config.beforeToolCall(
|
||||
assistantMsgCtx(assistantMsg, toolCall, validatedArgs, context), signal
|
||||
)
|
||||
if signal !== nothing && signal.aborted
|
||||
return immediateOutcome(createErrorToolResult("Operation aborted"), true)
|
||||
@@ -85,13 +239,32 @@ end
|
||||
|
||||
# ── per-call execution ──────────────────────────────────────────
|
||||
|
||||
"""
|
||||
executePreparedToolCall(prep, signal, emit)
|
||||
|
||||
Executes the tool by calling `tool.execute()` with the validated
|
||||
arguments, the abort signal, and a callback for streaming partial
|
||||
results. Emits `toolExecutionUpdate` events for each partial
|
||||
result batch. Waits for all pending update events to settle before
|
||||
returning. Catches execution errors and returns them as an error
|
||||
outcome. The `accepting` guard prevents emitting updates after
|
||||
the call has finished.
|
||||
|
||||
Long-running tools (e.g. file uploads, model training, web scraping)
|
||||
may take seconds or minutes. The streaming update mechanism allows
|
||||
UI listeners and other consumers to show progress in real time rather
|
||||
than waiting for the entire call to complete. The `accepting` guard
|
||||
ensures that if the tool's execute function yields after emitting
|
||||
updates but before returning, no duplicate or stale updates are
|
||||
emitted after the result has already been captured.
|
||||
"""
|
||||
function executePreparedToolCall(
|
||||
prep::preparedToolCall,
|
||||
signal::Union{Nothing,AbortSignal},
|
||||
emit::AgentEventSink,
|
||||
signal::union{nothing,abortSignal},
|
||||
emit::agentEventSink,
|
||||
)::executedOutcome
|
||||
|
||||
updateEvents = Promise[]
|
||||
updateEvents = promise[]
|
||||
accepting = true
|
||||
|
||||
try
|
||||
@@ -100,7 +273,7 @@ function executePreparedToolCall(
|
||||
partialResult -> begin
|
||||
if accepting
|
||||
push!(updateEvents,
|
||||
emit(ToolExecUpdateEvent(prep.toolCall.id, prep.toolCall.name,
|
||||
emit(toolExecUpdateEvent(prep.toolCall.id, prep.toolCall.name,
|
||||
prep.toolCall.arguments, partialResult)))
|
||||
end
|
||||
end
|
||||
@@ -117,29 +290,56 @@ end
|
||||
|
||||
# ── per-call finalization ───────────────────────────────────────
|
||||
|
||||
"""
|
||||
finalizeExecutedToolCall(context, assistantMsg, prep, executed, config, signal)
|
||||
|
||||
Runs the `afterToolCall` hook on the executed result, allowing
|
||||
the consumer to mutate the result content, details, usage,
|
||||
termination flag, or error status. Catches errors from the
|
||||
hook and converts them to error outcomes. Returns a
|
||||
`finalizedOutcome` that is used to construct the tool result
|
||||
message.
|
||||
|
||||
The `afterToolCall` hook exists as a post-processing step that
|
||||
runs after every tool call regardless of success or failure.
|
||||
Common use cases include:
|
||||
|
||||
- Masking sensitive data from result content before the LLM
|
||||
sees it (e.g. removing API keys from error messages).
|
||||
- Normalizing usage tracking data into a consistent format.
|
||||
- Inspecting the result and deciding to flip `terminate: true`
|
||||
based on business logic (e.g. "if deployment failed, stop
|
||||
the agent rather than retrying").
|
||||
- Wrapping an error result in a friendlier message for the LLM
|
||||
to understand.
|
||||
|
||||
If the hook itself throws, the error is caught and the result
|
||||
becomes an error outcome. This ensures the tool pipeline never
|
||||
breaks due to a buggy hook.
|
||||
"""
|
||||
function finalizeExecutedToolCall(
|
||||
context::AgentContext,
|
||||
assistantMsg::AssistantMessage,
|
||||
context::agentContext,
|
||||
assistantMsg::assistantMessage,
|
||||
prep::preparedToolCall,
|
||||
executed::executedOutcome,
|
||||
config::AgentLoopConfig,
|
||||
signal::Union{Nothing,AbortSignal},
|
||||
config::agentLoopConfig,
|
||||
signal::union{nothing,abortSignal},
|
||||
)::finalizedOutcome
|
||||
|
||||
result = executed.result
|
||||
isError = executed.isError
|
||||
|
||||
if config.afterToolCalls !== nothing
|
||||
if config.afterToolCall !== nothing
|
||||
try
|
||||
after = config.afterToolCalls(
|
||||
AfterCtx(assistantMsg, prep.toolCall, prep.args, result, isError, context), signal
|
||||
after = config.afterToolCall(
|
||||
afterCtx(assistantMsg, prep.toolCall, prep.args, result, isError, context), signal
|
||||
)
|
||||
if after !== nothing
|
||||
result = merge(result, Dict(:content=>get(after,:content,result.content),
|
||||
result = merge(result, dict(:content=>get(after,:content,result.content),
|
||||
:details=>get(after,:details,result.details),
|
||||
:usage=>get(after,:usage,result.usage),
|
||||
:terminate=>get(after,:terminate,result.terminate)))
|
||||
isError = get(after, :is_error, isError)
|
||||
isError = get(after, :isError, isError)
|
||||
end
|
||||
catch err
|
||||
result = createErrorToolResult(sprint(showerror, err))
|
||||
@@ -150,27 +350,59 @@ function finalizeExecutedToolCall(
|
||||
return finalizedOutcome(prep.toolCall, result, isError)
|
||||
end
|
||||
|
||||
function emitToolExecutionEnd(finalized::finalizedOutcome, emit::AgentEventSink)
|
||||
emit(ToolExecEndEvent(finalized.toolCall.id, finalized.toolCall.name,
|
||||
"""
|
||||
emitToolExecutionEnd(finalized, emit)
|
||||
|
||||
Emits the `toolExecutionEnd` event with the finalized outcome,
|
||||
signalling to listeners that the tool call has completed.
|
||||
|
||||
This event is part of the tool execution lifecycle:
|
||||
`tool_execution_start` → (zero or more `tool_execution_update` events) →
|
||||
`tool_execution_end`. Listeners (such as the TUI or logging systems)
|
||||
use this lifecycle to track individual tool calls. The event carries
|
||||
the final result so listeners have all the data they need without
|
||||
requiring external state lookups.
|
||||
"""
|
||||
function emitToolExecutionEnd(finalized::finalizedOutcome, emit::agentEventSink)
|
||||
emit(toolExecEndEvent(finalized.toolCall.id, finalized.toolCall.name,
|
||||
finalized.result, finalized.isError))
|
||||
end
|
||||
|
||||
# ── sequential execution ────────────────────────────────────────
|
||||
|
||||
"""
|
||||
executeToolCallsSequential(context, assistantMsg, toolCalls, config, signal, emit)
|
||||
|
||||
Executes tool calls one at a time in the order they appear. For each
|
||||
call: emits `toolExecutionStart`, runs `prepareToolCall`,
|
||||
then either resolves the immediate outcome or executes/finalizes
|
||||
the prepared call. Emits `toolExecutionEnd` and creates the tool
|
||||
result message before proceeding to the next call. Respects the
|
||||
abort signal — if aborted, remaining calls are skipped. Returns
|
||||
a batch with `terminate` determined by whether all results set
|
||||
the termination flag.
|
||||
|
||||
Sequential execution is required when tool calls have implicit
|
||||
dependencies — for example, a `create_database` tool must complete
|
||||
before `create_table` can reference it. It is also the safer
|
||||
default because it prevents race conditions when multiple tools
|
||||
share state (e.g. writing to the same file or API rate limits).
|
||||
Use parallel only when you are confident the tools are independent.
|
||||
"""
|
||||
function executeToolCallsSequential(
|
||||
context::AgentContext,
|
||||
assistantMsg::AssistantMessage,
|
||||
toolCalls::Vector{AgentToolCall},
|
||||
config::AgentLoopConfig,
|
||||
signal::Union{Nothing,AbortSignal},
|
||||
emit::AgentEventSink,
|
||||
)::toolCallBatch
|
||||
context::agentContext,
|
||||
assistantMsg::assistantMessage,
|
||||
toolCalls::vector{agentToolCall},
|
||||
config::agentLoopConfig,
|
||||
signal::union{nothing,abortSignal},
|
||||
emit::agentEventSink,
|
||||
)::agentToolCallBatch
|
||||
|
||||
finalizedCalls = finalizedOutcome[]
|
||||
messages = ToolResultMessage[]
|
||||
messages = toolResultMessage[]
|
||||
|
||||
for tc in toolCalls
|
||||
emit(ToolExecStartEvent(tc.id, tc.name, tc.arguments))
|
||||
emit(toolExecStartEvent(tc.id, tc.name, tc.arguments))
|
||||
|
||||
prep = prepareToolCall(context, assistantMsg, tc, config, signal)
|
||||
|
||||
@@ -195,20 +427,40 @@ end
|
||||
|
||||
# ── parallel execution ──────────────────────────────────────────
|
||||
|
||||
function executeToolCallsParallel(
|
||||
context::AgentContext,
|
||||
assistantMsg::AssistantMessage,
|
||||
toolCalls::Vector{AgentToolCall},
|
||||
config::AgentLoopConfig,
|
||||
signal::Union{Nothing,AbortSignal},
|
||||
emit::AgentEventSink,
|
||||
)::toolCallBatch
|
||||
"""
|
||||
executeToolCallsParallel(context, assistantMsg, toolCalls, config, signal, emit)
|
||||
|
||||
# Each entry: finalizedOutcome (already done) or Task → finalizedOutcome (pending)
|
||||
entries = Union{finalizedOutcome,Task{finalizedOutcome}}[]
|
||||
Prepares all tool calls concurrently and spawns a task for each
|
||||
prepared call. Immediate outcomes are resolved instantly. Task
|
||||
entries are collected in order, then `fetch`ed to await all
|
||||
concurrent executions. Tool result messages are created from
|
||||
finalized outcomes in order and returned as a batch. Respects
|
||||
the abort signal — if aborted during preparation, remaining
|
||||
calls are skipped. Finalization order preserves the original
|
||||
call order.
|
||||
|
||||
Parallel execution is appropriate when the assistant requests
|
||||
independent tools — for example, reading multiple files, querying
|
||||
separate databases, or making independent API calls. It reduces
|
||||
wall-clock time compared to sequential execution. The tradeoff is
|
||||
that parallel calls can overwhelm external resources (rate limits,
|
||||
connection pools, disk I/O). Finalization preserves the original
|
||||
call order so tool result messages appear in the same order the
|
||||
assistant requested them, regardless of which call finishes first.
|
||||
"""
|
||||
function executeToolCallsParallel(
|
||||
context::agentContext,
|
||||
assistantMsg::assistantMessage,
|
||||
toolCalls::vector{agentToolCall},
|
||||
config::agentLoopConfig,
|
||||
signal::union{nothing,abortSignal},
|
||||
emit::agentEventSink,
|
||||
)::agentToolCallBatch
|
||||
|
||||
entries = union{finalizedOutcome,task{finalizedOutcome}}[]
|
||||
|
||||
for tc in toolCalls
|
||||
emit(ToolExecStartEvent(tc.id, tc.name, tc.arguments))
|
||||
emit(toolExecStartEvent(tc.id, tc.name, tc.arguments))
|
||||
|
||||
prep = prepareToolCall(context, assistantMsg, tc, config, signal)
|
||||
|
||||
@@ -217,8 +469,7 @@ function executeToolCallsParallel(
|
||||
emitToolExecutionEnd(finalized, emit)
|
||||
push!(entries, finalized)
|
||||
else
|
||||
# spawn lazy computation task
|
||||
task = Task() do
|
||||
task = task() do
|
||||
executed = executePreparedToolCall(prep, signal, emit)
|
||||
finalized = finalizeExecutedToolCall(context, assistantMsg, prep, executed, config, signal)
|
||||
emitToolExecutionEnd(finalized, emit)
|
||||
@@ -233,14 +484,13 @@ function executeToolCallsParallel(
|
||||
end
|
||||
end
|
||||
|
||||
# Wait for all tasks, collect in order
|
||||
finalizedCalls = finalizedOutcome[]
|
||||
for entry in entries
|
||||
outcome = entry isa Task ? fetch(entry) : entry
|
||||
outcome = entry isa task ? fetch(entry) : entry
|
||||
push!(finalizedCalls, outcome)
|
||||
end
|
||||
|
||||
messages = ToolResultMessage[]
|
||||
messages = toolResultMessage[]
|
||||
for f in finalizedCalls
|
||||
push!(messages, createToolResultMessage(f))
|
||||
end
|
||||
@@ -248,25 +498,99 @@ function executeToolCallsParallel(
|
||||
return toolCallBatch(messages, shouldTerminate(finalizedCalls))
|
||||
end
|
||||
|
||||
# ── dispatcher ──────────────────────────────────────────────────
|
||||
|
||||
"""
|
||||
executeToolCalls(context, assistantMsg, toolCalls, config, signal, emit)
|
||||
|
||||
Dispatches to sequential or parallel execution. Uses sequential mode
|
||||
when `config.toolExecution == "sequential"` or when any of the
|
||||
tool calls reference a tool with `executionMode: "sequential"`.
|
||||
Otherwise uses parallel execution. This is the entry point called
|
||||
from `streamAssistantResponse` in the agent loop.
|
||||
|
||||
The sequential mode takes priority over parallel because it is the
|
||||
safe default. If even one tool in a batch is marked sequential, all
|
||||
tools execute sequentially — this prevents a single dependent tool
|
||||
from racing with an otherwise independent one. The per-tool
|
||||
`executionMode` allows fine-grained control (e.g. most tools are
|
||||
parallel but a specific write tool is sequential), while the config-level
|
||||
`toolExecution` provides a global override.
|
||||
"""
|
||||
function executeToolCalls(
|
||||
context::AgentContext,
|
||||
assistantMsg::AssistantMessage,
|
||||
toolCalls::Vector{AgentToolCall},
|
||||
config::AgentLoopConfig,
|
||||
signal::Union{Nothing,AbortSignal},
|
||||
emit::AgentEventSink,
|
||||
)::toolCallBatch
|
||||
context::agentContext,
|
||||
assistantMsg::assistantMessage,
|
||||
toolCalls::vector{agentToolCall},
|
||||
config::agentLoopConfig,
|
||||
signal::union{nothing,abortSignal},
|
||||
emit::agentEventSink,
|
||||
)::agentToolCallBatch
|
||||
|
||||
# Check if any tool is marked sequential, or config forces sequential
|
||||
hasSequential = any(tc ->
|
||||
any(t -> t.name == tc.name && get(t.executionMode, "parallel") == "sequential",
|
||||
context.tools), toolCalls)
|
||||
|
||||
if config.tool_execution == "sequential" || hasSequential
|
||||
if config.toolExecution == "sequential" || hasSequential
|
||||
return executeToolCallsSequential(context, assistantMsg, toolCalls, config, signal, emit)
|
||||
else
|
||||
return executeToolCallsParallel(context, assistantMsg, toolCalls, config, signal, emit)
|
||||
end
|
||||
end
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,537 +0,0 @@
|
||||
https://github.com/andyz245/LanguageAgentTreeSearch/blob/main/hotpot/hotpot.py
|
||||
|
||||
|
||||
standard_prompt = '''
|
||||
Write a coherent passage of 4 short paragraphs. The end sentence of each paragraph must be: {input}
|
||||
'''
|
||||
|
||||
reflection_prompt = '''You are an advanced reasoning agent that can improve based on self refection. You will be given a previous reasoning trial in which you were given access to an Docstore API environment and a question to answer. You were unsuccessful in answering the question either because you guessed the wrong answer with Finish[<answer>], or you used up your set number of reasoning steps. In a few sentences, Diagnose a possible reason for failure and devise a new, concise, high level plan that aims to mitigate the same failure. Use complete sentences.
|
||||
Here are some examples:
|
||||
Previous Trial:
|
||||
Question: The Rome Protocols were signed by three Prime Ministers one of which was assassinated as part of what?
|
||||
Thought 1: I need to search Rome Protocols, find the three Prime Ministers, then find what they were assassinated as part of.
|
||||
Action 1: Search[Rome Protocols]
|
||||
Observation 1: The Rome Protocols were a series of three international agreements concluded in Rome on 17 March 1934 between the governments of Austria, Hungary and Italy. They were signed by Italian Prime Minister Benito Mussolini, Austrian Chancellor Engelbert Dollfuss and Hungarian Prime Minister Gyula Gömbös. All of the protocols went into effect on 12 July 1934 and were registered in League of Nations Treaty Series on 12 December 1934.
|
||||
Thought 2: The three Prime Ministers were Benito Mussolini, Engelbert Dollfuss, and Gyula Gömbös. I need to search Benito Mussolini and find what he was assassinated as part of.
|
||||
Action 2: Search[Benito Mussolini]
|
||||
Observation 2: Benito Amilcare Andrea Mussolini (UK: MU(U)SS-ə-LEE-nee, US: MOOSS-, Italian: [beˈniːto aˈmilkare anˈdrɛːa mussoˈliːni]; 29 July 1883 – 28 April 1945) was an Italian politician and journalist who founded and led the National Fascist Party (PNF). He was Prime Minister of Italy from the March on Rome in 1922 until his deposition in 1943, as well as "Duce" of Italian fascism from the establishment of the Italian Fasces of Combat in 1919 until his summary execution in 1945 by Italian partisans. As dictator of Italy and principal founder of fascism, Mussolini inspired and supported the international spread of fascist movements during the inter-war period.Mussolini was originally a socialist politician and a journalist at the Avanti! newspaper. In 1912, he became a member of the National Directorate of the Italian Socialist Party (PSI), but he was expelled from the PSI for advocating military intervention in World War I, in opposition to the party's stance on neutrality. In 1914, Mussolini founded a new journal, Il Popolo d'Italia, and served in the Royal Italian Army during the war until he was wounded and discharged in 1917. Mussolini denounced the PSI, his views now centering on Italian nationalism instead of socialism, and later founded the fascist movement which came to oppose egalitarianism and class conflict, instead advocating "revolutionary nationalism" transcending class lines. On 31 October 1922, following the March on Rome (28–30 October), Mussolini was appointed prime minister by King Victor Emmanuel III, becoming the youngest individual to hold the office up to that time. After removing all political opposition through his secret police and outlawing labor strikes, Mussolini and his followers consolidated power through a series of laws that transformed the nation into a one-party dictatorship. Within five years, Mussolini had established dictatorial authority by both legal and illegal means and aspired to create a totalitarian state. In 1929, Mussolini signed the Lateran Treaty with the Holy See to establish Vatican City.
|
||||
Mussolini's foreign policy aimed to restore the ancient grandeur of the Roman Empire by expanding Italian colonial possessions and the fascist sphere of influence. In the 1920s, he ordered the Pacification of Libya, instructed the bombing of Corfu over an incident with Greece, established a protectorate over Albania, and incorporated the city of Fiume into the Italian state via agreements with Yugoslavia. In 1936, Ethiopia was conquered following the Second Italo-Ethiopian War and merged into Italian East Africa (AOI) with Eritrea and Somalia. In 1939, Italian forces annexed Albania. Between 1936 and 1939, Mussolini ordered the successful Italian military intervention in Spain in favor of Francisco Franco during the Spanish Civil War. Mussolini's Italy initially tried to avoid the outbreak of a second global war, sending troops at the Brenner Pass to delay Anschluss and taking part in the Stresa Front, the Lytton Report, the Treaty of Lausanne, the Four-Power Pact and the Munich Agreement. However, Italy then alienated itself from Britain and France by aligning with Germany and Japan. Germany invaded Poland on 1 September 1939, resulting in declarations of war by France and the UK and the start of World War II.
|
||||
On 10 June 1940, Mussolini decided to enter the war on the Axis side. Despite initial success, the subsequent Axis collapse on multiple fronts and eventual Allied invasion of Sicily made Mussolini lose the support of the population and members of the Fascist Party. As a consequence, early on 25 July 1943, the Grand Council of Fascism passed a motion of no confidence in Mussolini; later that day King Victor Emmanuel III dismissed him as head of government and had him placed in custody, appointing Pietro Badoglio to succeed him as Prime Minister. After the king agreed to an armistice with the Allies, on 12 September 1943 Mussolini was rescued from captivity in the Gran Sasso raid by German paratroopers and Waffen-SS commandos led by Major Otto-Harald Mors. Adolf Hitler, after meeting with the rescued former dictator, then put Mussolini in charge of a puppet regime in northern Italy, the Italian Social Republic (Italian: Repubblica Sociale Italiana, RSI), informally known as the Salò Republic, causing a civil war. In late April 1945, in the wake of near total defeat, Mussolini and his mistress Clara Petacci attempted to flee to Switzerland, but both were captured by Italian communist partisans and summarily executed by firing squad on 28 April 1945 near Lake Como. The bodies of Mussolini and his mistress were then taken to Milan, where they were hung upside down at a service station to publicly confirm their demise.
|
||||
Thought 3: Benito Mussolini was assassinated as part of the collapse of the Axis powers during World War II. So the Rome Protocols were signed by three Prime Ministers one of which was assassinated as part of World War II.
|
||||
Action 3: Finish[World War II]
|
||||
|
||||
Reflection: I searched one of the prime ministers involved in the signing, then attemted to answer right away. I should have searched each of the prime ministers, then looked up 'death' on each of their pages in order to get more information before answering.
|
||||
|
||||
Previous Trial:
|
||||
Question: Kam Heskin plays Paige Morgan in a 2004 film directed by who?
|
||||
Thought 1: I need to search Kam Heskin and Paige Morgan, find the 2004 film they are in, then find the director of the film.
|
||||
Action 1: Search[Kam Heskin]
|
||||
Observation 1: Kam Erika Heskin (born May 8, 1973) is an American actress. She began her career playing Caitlin Deschanel on the NBC daytime soap opera Sunset Beach (1998–1999), before appearing in films Planet of the Apes (2001) and Catch Me If You Can (2002). Heskin went on in 2003 to play Elizabeth Bennet in an independent film Pride & Prejudice: A Latter-Day Comedy, Robin in Charmed (2004), and Paige Morgan in The Prince and Me film franchise (2006–2010).
|
||||
Thought 2: Kam Heskin plays Paige Morgan in The Prince and Me film franchise. I need to search The Prince and Me film franchise and find the director of the 2004 film.
|
||||
Action 2: Search[The Prince and Me film franchise]
|
||||
Observation 2: Could not find [The Prince and Me film franchise]. Similar: ['Kam Heskin', 'Prince of Persia', 'Harry Potter and the Half-Blood Prince (film)', 'Prince of Persia: The Sands of Time (film)', 'The Chronicles of Narnia (film series)', 'Majestic Prince (franchise)', 'Kingsman (franchise)', 'The Little Prince (2015 film)', 'Shrek (franchise)', 'Harry Potter (film series)']
|
||||
Thought 3: I need to search The Prince and Me (2004 film) instead.
|
||||
Action 3: Search[The Prince and Me (2004 film)]
|
||||
Observation 3: Could not find [The Prince and Me (2004 film)]. Similar: ['The Prince & Me', 'The Prince & Me 2: The Royal Wedding', 'Kam Heskin', 'Prince of Darkness (film)', 'Prince of Persia: The Sands of Time (film)', 'Rob Knox', 'Alexander (2004 film)', 'Prince of Persia', 'Brooklynn Prince', 'Prince (musician)']
|
||||
Thought 4: I need to search The Prince & Me (2004 film).
|
||||
Action 4: Search[The Prince & Me (2004 film)]
|
||||
Observation 4: Could not find [The Prince & Me (2004 film)]. Similar: ['The Prince & Me', 'The Prince & Me 2: The Royal Wedding', 'Prince of Darkness (film)', 'Prince of Persia: The Sands of Time (film)', 'Rob Knox', 'Alexander (2004 film)', 'Prince (musician)', 'Prince of Persia', 'Kam Heskin', 'Brooklynn Prince']
|
||||
Thought 5: I need to search The Prince & Me (2004 film) instead.
|
||||
Action 5: Search[The Prince & Me (2004 film)]
|
||||
Observation 5: Could not find [The Prince & Me (2004 film)]. Similar: ['The Prince & Me', 'The Prince & Me 2: The Royal Wedding', 'Prince of Darkness (film)', 'Prince of Persia: The Sands of Time (film)', 'Rob Knox', 'Alexander (2004 film)', 'Prince (musician)', 'Prince of Persia', 'Kam Heskin', 'Brooklynn Prince']
|
||||
Thought 6: I need to search The Prince & Me (2004 film).
|
||||
Action 6: Search[The Prince & Me (2004 film)]
|
||||
Observation 6: Could not find [The Prince & Me (2004 film)]. Similar: ['The Prince & Me', 'The Prince & Me 2: The Royal Wedding', 'Prince of Darkness (film)', 'Prince of Persia: The Sands of Time (film)', 'Rob Knox', 'Alexander (2004 film)', 'Prince (musician)', 'Prince of Persia', 'Kam Heskin', 'Brooklynn Prince']
|
||||
|
||||
Reflection: I got stuck in a loop where I kept trying to search 'The Prince & Me (2004 film)' but the page could not be found. Instead I should have tried to search the similar results that had a similar name to see and they were made in 2004.
|
||||
|
||||
Previous Trial:
|
||||
Question: The Oberoi family is part of a hotel company that has a head office in what city?
|
||||
Thought 1: I need to search the Oberoi family and find the city where the hotel company's head office is located.
|
||||
Action 1: Search[Oberoi family]
|
||||
Observation 1: The Oberoi Group is a luxury hotel group with its head office in New Delhi, India.[2] Founded in 1934, the company owns and operates 32 luxury hotels and two river cruise ships in 7 countries, primarily under its Oberoi Hotels & Resorts and Trident brands.[3] The group also operates The Oberoi Centre for Learning and Development, which is regarded as one of Asia's top institutions for hospitality education.[4]. The foundations of the Oberoi Group date back to 1934 when The Rai Bahadur Mohan Singh Oberoi, the founder of the group, bought two properties: the Maidens in Delhi and the Clarke's in Shimla.[5] In the following years Oberoi, assisted by his two sons, Tilak Raj Singh Oberoi and Prithvi Raj Singh Oberoi (P.R.S. Oberoi
|
||||
Thought 2: The Oberoi Group has its head office in New Delhi, India. I just need to finish with New Delhi.
|
||||
Action 2: Finish[New Delhi]
|
||||
Observation 2: Episode finished, reward = 0
|
||||
|
||||
Reflection: I appeared to have retrieved the correct information about The Oberoi Family and the location of it's head office, and provided a corresponding answer. However this answer does not exactly match the ground truth answer so I should try a different wording, such as Delhi.
|
||||
|
||||
Previous trial:
|
||||
{trajectory}Reflection:'''
|
||||
|
||||
cot_prompt = '''
|
||||
Solve a question answering task with interleaving Thought, Action, Observation steps. Thought can reason about the current situation, and Action can be three types:
|
||||
(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
|
||||
(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
|
||||
(3) Finish[answer], which returns the answer and finishes the task.
|
||||
After each observation, provide the next Thought and next Action. Here are some examples:
|
||||
|
||||
Question: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?
|
||||
Thought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.
|
||||
Action 1: Search[Colorado orogeny]
|
||||
Observation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas. This took place from 1780 to 1650 million years ago (Mya), during the Paleoproterozoic (Statherian Period). It is recorded in the Colorado orogen, a >500-km-wide belt of oceanic arc rock that extends southward into New Mexico. The Colorado orogeny was likely part of the larger Yavapai orogeny.
|
||||
Thought 2: It does not mention the eastern sector of the Colorado orogeny. I need to look up eastern sector.
|
||||
Action 2: Lookup[eastern sector]
|
||||
Observation 2: (Result 1 / 1) The Colorado orogen, formerly called the Colorado province, is a >500-km-wide belt of oceanic arc rock (1.78\u20131.65 Ga) that extends southward into New Mexico and composes a major part of the Proterozoic provinces of southwestern United States. This transcontinental collisional event occurred during the Paleoproterozoic (Statherian Period).[1] The Wyoming sector of the Colorado orogeny was formerly called the Medicine Bow orogeny. The eastern sector extends into the High Plains and is called the Central Plains orogeny. The boundary between the Colorado orogeny and the Wyoming craton is the Cheyenne belt, a 5-km-wide mylonitic shear zone that verges northward. The Cheyenne belt transects and cuts off the south edge of the older Trans-Hudson orogeny.[2]
|
||||
Thought 3: The eastern sector of Colorado orogeny extends into the High Plains, so High Plains is the area. I need to search High Plains and find its elevation range.
|
||||
Action 3: Search[High Plains]Observation 3: High Plains refers to one of two distinct land regions:
|
||||
Thought 4: I need to search High Plains (United States).
|
||||
Action 4: Search[High Plains (United States)]
|
||||
Observation 4: The High Plains are a subregion of the Great Plains, mainly in the Western United States, but also partly in the Midwest states of Nebraska, Kansas, and South Dakota, generally encompassing the western part of the Great Plains before the region reaches the Rocky Mountains. The High Plains are located in eastern Montana, southeastern Wyoming, southwestern South Dakota, western Nebraska, eastern Colorado, western Kansas, eastern New Mexico, western Oklahoma, and to just south of the Texas Panhandle.[4] The southern region of the Western High Plains ecology region contains the geological formation known as Llano Estacado which can be seen from a short distance or on satellite maps.[5] From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]
|
||||
Thought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.
|
||||
Action 5: Finish[1,800 to 7,000 ft]
|
||||
|
||||
Question: Guitars for Wounded Warriors is an album that was recorded in the village in which New York county?
|
||||
Thought 1: I need to search Guitars for Wounded Warriors, then find in which New York county it was recorded.
|
||||
Action 1: Search[Guitars for Wounded Warriors]
|
||||
Observation 1: Guitars for Wounded Warriors is the sixth solo studio album by guitarist Brian Tarquin, released in August 2014 by Cleopatra Records. In 2016, it received a Best Compilation Album nomination by the Independent Music Awards. All tracks were recorded at Tarquin's Jungle Room Studios in New Paltz (village), New York. Being moved by the lack of support for returning veterans through his life Tarquin decided to compose a heavy instrumental rock album as a way to show his appreciation to all veterans. So once again he enlisted top notch players to guest with him on the album, Billy Sheehan on Black Hawk, Ron \"Bumblefoot\" Thal and Reb Beach on Taliban Terror, Steve Morse on Freedom, Chris Poland on Charlie Surfs and Hunting, Chuck Loeb on Escape Kabul, Hal Lindes on Sand & Blood, Gary Hoey on 5 Klicks To Hell and Baghdad, Alex De Rosso Dokken on Broken Arrow, and The Flyin' Ryan Brothers on Surgical Strike. The entire album was engineered, produced and composed by Tarquin especially for each one of the guest guitarists. Partial proceeds are donated to the Fisher House Foundation from sales.
|
||||
Thought 2: Guitars for Wounded Warriors was recorded in New Paltz (village), New York. I need to search New Paltz (village), New York and find which county it is in
|
||||
Action 2: Search[New Paltz (village), New York]
|
||||
Observation 2: New Paltz is a village in Ulster County located in the U.S. state of New York. It is approximately 80 miles (130 km) north of New York City and 70 miles (110 km) south of Albany. The population was 7,324 at the 2020 census.[3]
|
||||
Thought 3: New Paltz (village), New York is in Ulster County, so the answer is Ulster County.
|
||||
Action 3: Finish[Ulster County]
|
||||
|
||||
Question: Musician and satirist Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?
|
||||
Thought 1: The question simplifies to "The Simpsons" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.
|
||||
Action 1: Search[Milhouse]
|
||||
Observation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.
|
||||
Thought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up "named after".
|
||||
Action 2: Lookup[named after]
|
||||
Observation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous.
|
||||
Thought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.
|
||||
Action 3: Finish[President Richard Nixon]
|
||||
|
||||
{input}
|
||||
'''
|
||||
|
||||
cot_prompt_short = '''
|
||||
Solve a question answering task with interleaving Thought, Action, Observation steps. Thought can reason about the current situation, and Action can be three types:
|
||||
(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
|
||||
(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
|
||||
(3) Finish[answer], which returns the answer and finishes the task.
|
||||
After each observation, provide the next Thought and next Action. Here are some examples:
|
||||
|
||||
Question: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?
|
||||
Thought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.
|
||||
Action 1: Search[Colorado orogeny]
|
||||
Observation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas. This took place from 1780 to 1650 million years ago (Mya), during the Paleoproterozoic (Statherian Period). It is recorded in the Colorado orogen, a >500-km-wide belt of oceanic arc rock that extends southward into New Mexico. The Colorado orogeny was likely part of the larger Yavapai orogeny.
|
||||
Thought 2: It does not mention the eastern sector of the Colorado orogeny. I need to look up eastern sector.
|
||||
Action 2: Lookup[eastern sector]
|
||||
Observation 2: (Result 1 / 1) The Colorado orogen, formerly called the Colorado province, is a >500-km-wide belt of oceanic arc rock (1.78\u20131.65 Ga) that extends southward into New Mexico and composes a major part of the Proterozoic provinces of southwestern United States. This transcontinental collisional event occurred during the Paleoproterozoic (Statherian Period).[1] The Wyoming sector of the Colorado orogeny was formerly called the Medicine Bow orogeny. The eastern sector extends into the High Plains and is called the Central Plains orogeny. The boundary between the Colorado orogeny and the Wyoming craton is the Cheyenne belt, a 5-km-wide mylonitic shear zone that verges northward. The Cheyenne belt transects and cuts off the south edge of the older Trans-Hudson orogeny.[2]
|
||||
Thought 3: The eastern sector of Colorado orogeny extends into the High Plains, so High Plains is the area. I need to search High Plains and find its elevation range.
|
||||
Action 3: Search[High Plains]Observation 3: High Plains refers to one of two distinct land regions:
|
||||
Thought 4: I need to search High Plains (United States).
|
||||
Action 4: Search[High Plains (United States)]
|
||||
Observation 4: The High Plains are a subregion of the Great Plains, mainly in the Western United States, but also partly in the Midwest states of Nebraska, Kansas, and South Dakota, generally encompassing the western part of the Great Plains before the region reaches the Rocky Mountains. The High Plains are located in eastern Montana, southeastern Wyoming, southwestern South Dakota, western Nebraska, eastern Colorado, western Kansas, eastern New Mexico, western Oklahoma, and to just south of the Texas Panhandle.[4] The southern region of the Western High Plains ecology region contains the geological formation known as Llano Estacado which can be seen from a short distance or on satellite maps.[5] From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]
|
||||
Thought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.
|
||||
Action 5: Finish[1,800 to 7,000 ft]
|
||||
|
||||
Question: Musician and satirist Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?
|
||||
Thought 1: The question simplifies to "The Simpsons" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.
|
||||
Action 1: Search[Milhouse]
|
||||
Observation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.
|
||||
Thought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up "named after".
|
||||
Action 2: Lookup[named after]
|
||||
Observation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous.
|
||||
Thought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.
|
||||
Action 3: Finish[President Richard Nixon]
|
||||
|
||||
{input}
|
||||
'''
|
||||
|
||||
cot_prompt_feedback_short = '''You are also an advanced reasoning agent that can improve based on self refection. Solve a question answering task with interleaving Thought, Action, Observation steps. Thought can reason about the current situation, and Action can be three types:
|
||||
(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
|
||||
(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
|
||||
(3) Finish[answer], which returns the answer and finishes the task.
|
||||
After each observation, provide the next Thought and next Action. Here are some examples:
|
||||
|
||||
Question: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?
|
||||
Thought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.
|
||||
Action 1: Search[Colorado orogeny]
|
||||
Observation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas. This took place from 1780 to 1650 million years ago (Mya), during the Paleoproterozoic (Statherian Period). It is recorded in the Colorado orogen, a >500-km-wide belt of oceanic arc rock that extends southward into New Mexico. The Colorado orogeny was likely part of the larger Yavapai orogeny.
|
||||
Thought 2: It does not mention the eastern sector of the Colorado orogeny. I need to look up eastern sector.
|
||||
Action 2: Lookup[eastern sector]
|
||||
Observation 2: (Result 1 / 1) The Colorado orogen, formerly called the Colorado province, is a >500-km-wide belt of oceanic arc rock (1.78\u20131.65 Ga) that extends southward into New Mexico and composes a major part of the Proterozoic provinces of southwestern United States. This transcontinental collisional event occurred during the Paleoproterozoic (Statherian Period).[1] The Wyoming sector of the Colorado orogeny was formerly called the Medicine Bow orogeny. The eastern sector extends into the High Plains and is called the Central Plains orogeny. The boundary between the Colorado orogeny and the Wyoming craton is the Cheyenne belt, a 5-km-wide mylonitic shear zone that verges northward. The Cheyenne belt transects and cuts off the south edge of the older Trans-Hudson orogeny.[2]
|
||||
Thought 3: The eastern sector of Colorado orogeny extends into the High Plains, so High Plains is the area. I need to search High Plains and find its elevation range.
|
||||
Action 3: Search[High Plains]Observation 3: High Plains refers to one of two distinct land regions:
|
||||
Thought 4: I need to search High Plains (United States).
|
||||
Action 4: Search[High Plains (United States)]
|
||||
Observation 4: The High Plains are a subregion of the Great Plains, mainly in the Western United States, but also partly in the Midwest states of Nebraska, Kansas, and South Dakota, generally encompassing the western part of the Great Plains before the region reaches the Rocky Mountains. The High Plains are located in eastern Montana, southeastern Wyoming, southwestern South Dakota, western Nebraska, eastern Colorado, western Kansas, eastern New Mexico, western Oklahoma, and to just south of the Texas Panhandle.[4] The southern region of the Western High Plains ecology region contains the geological formation known as Llano Estacado which can be seen from a short distance or on satellite maps.[5] From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]
|
||||
Thought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.
|
||||
Action 5: Finish[1,800 to 7,000 ft]
|
||||
|
||||
Question: Musician and satirist Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?
|
||||
Thought 1: The question simplifies to "The Simpsons" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.
|
||||
Action 1: Search[Milhouse]
|
||||
Observation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.
|
||||
Thought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up "named after".
|
||||
Action 2: Lookup[named after]
|
||||
Observation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous.
|
||||
Thought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.
|
||||
Action 3: Finish[President Richard Nixon]
|
||||
|
||||
You have attempted to answer the following question before and failed. The following reflection(s) give a plan to avoid failing to answer the question in the same way you did previously. Use them to improve your strategy of correctly answering the given question.
|
||||
|
||||
{trajectories}
|
||||
|
||||
{input}
|
||||
'''
|
||||
|
||||
cot_prompt_feedback = '''You are also an advanced reasoning agent that can improve based on self refection. Solve a question answering task with interleaving Thought, Action, Observation steps. Thought can reason about the current situation, and Action can be three types:
|
||||
(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
|
||||
(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
|
||||
(3) Finish[answer], which returns the answer and finishes the task.
|
||||
After each observation, provide the next Thought and next Action. Here are some examples:
|
||||
|
||||
Question: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?
|
||||
Thought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.
|
||||
Action 1: Search[Colorado orogeny]
|
||||
Observation 1: The Colorado orogeny was an episode of mountain building (an orogeny) in Colorado and surrounding areas. This took place from 1780 to 1650 million years ago (Mya), during the Paleoproterozoic (Statherian Period). It is recorded in the Colorado orogen, a >500-km-wide belt of oceanic arc rock that extends southward into New Mexico. The Colorado orogeny was likely part of the larger Yavapai orogeny.
|
||||
Thought 2: It does not mention the eastern sector of the Colorado orogeny. I need to look up eastern sector.
|
||||
Action 2: Lookup[eastern sector]
|
||||
Observation 2: (Result 1 / 1) The Colorado orogen, formerly called the Colorado province, is a >500-km-wide belt of oceanic arc rock (1.78\u20131.65 Ga) that extends southward into New Mexico and composes a major part of the Proterozoic provinces of southwestern United States. This transcontinental collisional event occurred during the Paleoproterozoic (Statherian Period).[1] The Wyoming sector of the Colorado orogeny was formerly called the Medicine Bow orogeny. The eastern sector extends into the High Plains and is called the Central Plains orogeny. The boundary between the Colorado orogeny and the Wyoming craton is the Cheyenne belt, a 5-km-wide mylonitic shear zone that verges northward. The Cheyenne belt transects and cuts off the south edge of the older Trans-Hudson orogeny.[2]
|
||||
Thought 3: The eastern sector of Colorado orogeny extends into the High Plains, so High Plains is the area. I need to search High Plains and find its elevation range.
|
||||
Action 3: Search[High Plains]Observation 3: High Plains refers to one of two distinct land regions:
|
||||
Thought 4: I need to search High Plains (United States).
|
||||
Action 4: Search[High Plains (United States)]
|
||||
Observation 4: The High Plains are a subregion of the Great Plains, mainly in the Western United States, but also partly in the Midwest states of Nebraska, Kansas, and South Dakota, generally encompassing the western part of the Great Plains before the region reaches the Rocky Mountains. The High Plains are located in eastern Montana, southeastern Wyoming, southwestern South Dakota, western Nebraska, eastern Colorado, western Kansas, eastern New Mexico, western Oklahoma, and to just south of the Texas Panhandle.[4] The southern region of the Western High Plains ecology region contains the geological formation known as Llano Estacado which can be seen from a short distance or on satellite maps.[5] From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m).[3]
|
||||
Thought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.
|
||||
Action 5: Finish[1,800 to 7,000 ft]
|
||||
|
||||
Question: Guitars for Wounded Warriors is an album that was recorded in the village in which New York county?
|
||||
Thought 1: I need to search Guitars for Wounded Warriors, then find in which New York county it was recorded.
|
||||
Action 1: Search[Guitars for Wounded Warriors]
|
||||
Observation 1: Guitars for Wounded Warriors is the sixth solo studio album by guitarist Brian Tarquin, released in August 2014 by Cleopatra Records. In 2016, it received a Best Compilation Album nomination by the Independent Music Awards. All tracks were recorded at Tarquin's Jungle Room Studios in New Paltz (village), New York. Being moved by the lack of support for returning veterans through his life Tarquin decided to compose a heavy instrumental rock album as a way to show his appreciation to all veterans. So once again he enlisted top notch players to guest with him on the album, Billy Sheehan on Black Hawk, Ron \"Bumblefoot\" Thal and Reb Beach on Taliban Terror, Steve Morse on Freedom, Chris Poland on Charlie Surfs and Hunting, Chuck Loeb on Escape Kabul, Hal Lindes on Sand & Blood, Gary Hoey on 5 Klicks To Hell and Baghdad, Alex De Rosso Dokken on Broken Arrow, and The Flyin' Ryan Brothers on Surgical Strike. The entire album was engineered, produced and composed by Tarquin especially for each one of the guest guitarists. Partial proceeds are donated to the Fisher House Foundation from sales.
|
||||
Thought 2: Guitars for Wounded Warriors was recorded in New Paltz (village), New York. I need to search New Paltz (village), New York and find which county it is in
|
||||
Action 2: Search[New Paltz (village), New York]
|
||||
Observation 2: New Paltz is a village in Ulster County located in the U.S. state of New York. It is approximately 80 miles (130 km) north of New York City and 70 miles (110 km) south of Albany. The population was 7,324 at the 2020 census.[3]
|
||||
Thought 3: New Paltz (village), New York is in Ulster County, so the answer is Ulster County.
|
||||
Action 3: Finish[Ulster County]
|
||||
|
||||
Question: Musician and satirist Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?
|
||||
Thought 1: The question simplifies to "The Simpsons" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.
|
||||
Action 1: Search[Milhouse]
|
||||
Observation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.
|
||||
Thought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up "named after".
|
||||
Action 2: Lookup[named after]
|
||||
Observation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous.
|
||||
Thought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.
|
||||
Action 3: Finish[President Richard Nixon]
|
||||
|
||||
You have attempted to answer the following question before and failed, either because your reasoning for the answer was incorrect or the phrasing of your response did not exactly match the answer. The following reflection(s) give a plan to avoid failing to answer the question in the same way you did previously. Use them to improve your strategy of correctly answering the given question.
|
||||
|
||||
{trajectories}
|
||||
When providing the thought and action for the current trial, that into account these failed trajectories and make sure not to repeat the same mistakes and incorrect answers.
|
||||
|
||||
{input}
|
||||
'''
|
||||
|
||||
vote_prompt = '''Analyze the trajectories of a solution to a question answering task. The trajectories are labeled by pairs of thoughts that can reason about the current situation and actions that can be three types:
|
||||
(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
|
||||
(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
|
||||
(3) Finish[answer], which returns the answer and finishes the task.
|
||||
|
||||
Given a question and a list of trajectories, decide which trajectory is most promising. Analyze each trajectory in detail and consider possible errors, then conclude in the last line "The best trajectory is {s}", where s the integer id of the trajectory.
|
||||
'''
|
||||
|
||||
compare_prompt = '''Analyze the trajectories of a solution to a question answering task. The trajectories are labeled by pairs of thoughts that can reason about the current situation and actions that can be three types:
|
||||
(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
|
||||
(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
|
||||
(3) Finish[answer], which returns the answer and finishes the task.
|
||||
|
||||
Briefly analyze the correctness of the following two trajectories. Conclude in the last line "The more correct trajectory is 1", "The more correct trajectory is 2", or "The two trajectories are similarly correct".
|
||||
'''
|
||||
|
||||
score_prompt = '''Analyze the trajectories of a solution to a question answering task. The trajectories are labeled by pairs of thoughts that can reason about the current situation and actions that can be three types:
|
||||
(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
|
||||
(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
|
||||
(3) Finish[answer], which returns the answer and finishes the task.
|
||||
|
||||
Given a question and a trajectory, analyze the following trajectory, then at the last line conclude "Thus the correctness score is {s}", where s is an integer from 1 to 10.
|
||||
'''
|
||||
|
||||
value_prompt = '''Analyze the trajectories of a solution to a question answering task. The trajectories are labeled by environmental observations about the situation, thoughts that can reason about the current situation and actions that can be three types:
|
||||
(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
|
||||
(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
|
||||
(3) Finish[answer], which returns the answer and finishes the task.
|
||||
|
||||
Given a question and a trajectory, evaluate its correctness and provide your reasoning and analysis in detail. Focus on the latest thought, action, and observation. Incomplete trajectories can be correct if the thoughts and actions so far are correct, even if the answer is not found yet. Do not generate additional thoughts or actions. Then at the last line conclude "Thus the correctness score is {s}", where s is an integer from 1 to 10.
|
||||
|
||||
Question: Which other Mexican Formula One race car driver has held the podium besides the Force India driver born in 1990?
|
||||
Thought 1: I need to search for a list of Mexican Formula One race car drivers, then eliminate the one who drove for Force India and was born in 1990.
|
||||
Action 1: Search[Formula One drivers from Mexico]
|
||||
Observation 1: There have been six Formula One drivers from Mexico who have taken part in races since the championship began in 1950. Sergio P\u00e9rez is the most successful, having won six Grands Prix. Since 2021, P\u00e9rez drives for the Red Bull team.. P\u00e9rez is the only Mexican F1 driver to achieve a pole position. He took his first pole position at the 2022 Saudi Arabian Grand Prix, in his 219th Grand Prix meeting (a record for the most Grands Prix before a pole).
|
||||
Thought 2: Sergio P\u00e9rez, born in 1990 and drove for Force India, is not the Mexican driver we are looking for. The passage does not give me enough information about other Mexican drivers who have achieved a podium finish. I need to look up more information.
|
||||
Action 2: Lookup[Formula One drivers from Mexico]
|
||||
Observation 2: No more results.
|
||||
Thought 3: The available information doesn't allow me to determine which other Mexican Formula One race car driver, besides Sergio P\u00e9rez, has held the podium. With the search function failing to provide a full list of Mexican Formula One drivers, I need to return different search results. Let's try searching for Mexican Grand Prix winners, as they would have held the podium.
|
||||
Action 3: Search[Mexican Grand Prix winners]
|
||||
Observation 3: Could not find Mexican Grand Prix winners. Similar: ['Mexican Grand Prix', 'List of Formula One Grand Prix winners', '1990 Mexican Grand Prix', '2018 Mexican Grand Prix', '2019 Mexican Grand Prix'].
|
||||
Thought 4: Given the difficulties in finding a direct list of Mexican Grand Prix winners, I should search for the Mexican Grand Prix to get a broader picture of the race's history. This might include winners.
|
||||
Action 4: Search[Mexican Grand Prix]
|
||||
Thus the correctness score is 3
|
||||
|
||||
Question: The football manager who recruited David Beckham managed Manchester United during what timeframe?Question: The football manager who recruited David Beckham managed Manchester United during what timeframe?
|
||||
Thought 1: I need to search for the football manager who recruited David Beckham, and then find out the timeframe in which he managed Manchester United.
|
||||
Action 1: Search[David Beckham's football manager]
|
||||
Observation 1: Could not find David Beckham's football manager. Similar: ['David Beckham', 'Inter Miami CF (redirect from Miami Beckham United)', 'Victoria Beckham', 'The Beckham Experiment', 'Miami Freedom Park'].
|
||||
Thought 2: I couldn't find the information on David Beckham's football manager. I need to search Manchester United and find out the timeframe of the manager.
|
||||
Action 2: Search[Manchester United]
|
||||
Observation 2: Manchester United Football Club, commonly referred to as Man United (often stylised as Man Utd), or simply United, is a professional football club based in Old Trafford, Greater Manchester, England. The club competes in the Premier League, the top division in the English football league system. Nicknamed the Red Devils, they were founded as Newton Heath LYR Football Club in 1878, but changed their name to Manchester United in 1902. After a spell playing in Clayton, Manchester, the club moved to their current stadium, Old Trafford, in 1910.. Domestically, Manchester United have won a record 20 league titles, 12 FA Cups, six League Cups and a record 21 FA Community Shields.
|
||||
Thought 3: The information about David Beckham's football manager is not available. I will search for Manchester United's managers to find the timeframe.
|
||||
Action 3: Search[Manchester United managers]
|
||||
Observation 3: Manchester United Football Club is a professional football club based in Old Trafford, Greater Manchester, England. The club was formed in Newton Heath in 1878 as Newton Heath LYR F.C., and played their first competitive match in October 1886, when they entered the First Round of the 1886–87 FA Cup. The club was renamed Manchester United F.C. in 1902, and moved to Old Trafford in 1910.. From the beginning of the club's official managerial records in 1892 to the start of the 2022–23 season, Manchester United have had 23 full-time managers.
|
||||
Thought 4: The information about David Beckham's football manager is not available. The question asks for the timeframe when the manager managed Manchester United, so I need to find a list of Manchester United managers.
|
||||
Action 4: Search[list of Manchester United managers]
|
||||
Observation 4: Manchester United Football Club is a professional football club based in Old Trafford, Greater Manchester, England. The club was formed in Newton Heath in 1878 as Newton Heath LYR F.C., and played their first competitive match in October 1886, when they entered the First Round of the 1886–87 FA Cup. The club was renamed Manchester United F.C. in 1902, and moved to Old Trafford in 1910.. From the beginning of the club's official managerial records in 1892 to the start of the 2022–23 season, Manchester United have had 23 full-time managers.
|
||||
Thus the correctness score is 4
|
||||
|
||||
Question: Which magazine was started first Arthur's Magazine or First for Women?
|
||||
Thought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.
|
||||
Action 1: Search[Arthur's Magazine]
|
||||
Observation 1: Arthur's Magazine (1844\u00e2\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. Edited by Timothy Shay Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others.[1][2] In May 1846 it was merged into Godey's Lady's Book.[3]
|
||||
Thus the correctness score is 10
|
||||
|
||||
{input}
|
||||
'''
|
||||
|
||||
value_prompt_feedback = '''Analyze the trajectories of a solution to a question answering task. The trajectories are labeled by environmental observations about the situation, thoughts that can reason about the current situation and actions that can be three types:
|
||||
(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
|
||||
(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
|
||||
(3) Finish[answer], which returns the answer and finishes the task.
|
||||
|
||||
Given a question and a trajectory, evaluate its correctness and provide your reasoning and analysis in detail. Focus on the latest thought, action, and observation. Incomplete trajectories can be correct if the thoughts and actions so far are correct, even if the answer is not found yet. Do not generate additional thoughts or actions. Then at the last line conclude "Thus the correctness score is {s}", where s is an integer from 1 to 10.
|
||||
|
||||
Question: Which other Mexican Formula One race car driver has held the podium besides the Force India driver born in 1990?
|
||||
Thought 1: I need to search for a list of Mexican Formula One race car drivers, then eliminate the one who drove for Force India and was born in 1990.
|
||||
Action 1: Search[Formula One drivers from Mexico]
|
||||
Observation 1: There have been six Formula One drivers from Mexico who have taken part in races since the championship began in 1950. Sergio P\u00e9rez is the most successful, having won six Grands Prix. Since 2021, P\u00e9rez drives for the Red Bull team.. P\u00e9rez is the only Mexican F1 driver to achieve a pole position. He took his first pole position at the 2022 Saudi Arabian Grand Prix, in his 219th Grand Prix meeting (a record for the most Grands Prix before a pole).
|
||||
Thought 2: Sergio P\u00e9rez, born in 1990 and drove for Force India, is not the Mexican driver we are looking for. The passage does not give me enough information about other Mexican drivers who have achieved a podium finish. I need to look up more information.
|
||||
Action 2: Lookup[Formula One drivers from Mexico]
|
||||
Observation 2: No more results.
|
||||
Thought 3: The available information doesn't allow me to determine which other Mexican Formula One race car driver, besides Sergio P\u00e9rez, has held the podium. With the search function failing to provide a full list of Mexican Formula One drivers, I need to return different search results. Let's try searching for Mexican Grand Prix winners, as they would have held the podium.
|
||||
Action 3: Search[Mexican Grand Prix winners]
|
||||
Observation 3: Could not find Mexican Grand Prix winners. Similar: ['Mexican Grand Prix', 'List of Formula One Grand Prix winners', '1990 Mexican Grand Prix', '2018 Mexican Grand Prix', '2019 Mexican Grand Prix'].
|
||||
Thought 4: Given the difficulties in finding a direct list of Mexican Grand Prix winners, I should search for the Mexican Grand Prix to get a broader picture of the race's history. This might include winners.
|
||||
Action 4: Search[Mexican Grand Prix]
|
||||
Thus the correctness score is 3
|
||||
|
||||
Question: Which magazine was started first Arthur's Magazine or First for Women?
|
||||
Thought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.
|
||||
Action 1: Search[Arthur's Magazine]
|
||||
Observation 1: Arthur's Magazine (1844\u00e2\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. Edited by Timothy Shay Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others.[1][2] In May 1846 it was merged into Godey's Lady's Book.[3]
|
||||
Thus the correctness score is 10
|
||||
|
||||
{trajectories}
|
||||
Question: Musician and satirist Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?
|
||||
Thought 1: The question simplifies to "The Simpsons" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.
|
||||
Action 1: Search[Milhouse]
|
||||
Observation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.
|
||||
Thought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up "named after".
|
||||
Action 2: Lookup[named after]
|
||||
Observation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous.
|
||||
Thought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.
|
||||
Action 3: Finish[President Richard Nixon]
|
||||
Thus the correctness score is 10
|
||||
|
||||
{input}
|
||||
'''
|
||||
|
||||
value_prompt_reasoning = '''You are an advanced reasoning agent that can improve based on self refection. Analyze the trajectories of your previous solutions to a question answering task. The trajectories are labeled by environmental observations about the situation, thoughts that can reason about the current situation and actions that can be three types:
|
||||
(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
|
||||
(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
|
||||
(3) Finish[answer], which returns the answer and finishes the task.
|
||||
|
||||
Given a question and a trajectory, evaluate its correctness and provide your reasoning and analysis in detail. Focus on the latest thought, action, and observation. Incomplete trajectories can be correct if the thoughts and actions so far are correct, even if the answer is not found yet. Do not generate additional thoughts or actions. Then at the last line conclude "Thus the correctness score is {s}", where s is an integer from 1 to 10.
|
||||
|
||||
Question: Which other Mexican Formula One race car driver has held the podium besides the Force India driver born in 1990?
|
||||
Thought 1: I need to search for a list of Mexican Formula One race car drivers, then eliminate the one who drove for Force India and was born in 1990.
|
||||
Action 1: Search[Formula One drivers from Mexico]
|
||||
Observation 1: There have been six Formula One drivers from Mexico who have taken part in races since the championship began in 1950. Sergio P\u00e9rez is the most successful, having won six Grands Prix. Since 2021, P\u00e9rez drives for the Red Bull team.. P\u00e9rez is the only Mexican F1 driver to achieve a pole position. He took his first pole position at the 2022 Saudi Arabian Grand Prix, in his 219th Grand Prix meeting (a record for the most Grands Prix before a pole).
|
||||
Thought 2: Sergio P\u00e9rez, born in 1990 and drove for Force India, is not the Mexican driver we are looking for. The passage does not give me enough information about other Mexican drivers who have achieved a podium finish. I need to look up more information.
|
||||
Action 2: Lookup[Formula One drivers from Mexico]
|
||||
Observation 2: No more results.
|
||||
Thought 3: The available information doesn't allow me to determine which other Mexican Formula One race car driver, besides Sergio P\u00e9rez, has held the podium. With the search function failing to provide a full list of Mexican Formula One drivers, I need to return different search results. Let's try searching for Mexican Grand Prix winners, as they would have held the podium.
|
||||
Action 3: Search[Mexican Grand Prix winners]
|
||||
Observation 3: Could not find Mexican Grand Prix winners. Similar: ['Mexican Grand Prix', 'List of Formula One Grand Prix winners', '1990 Mexican Grand Prix', '2018 Mexican Grand Prix', '2019 Mexican Grand Prix'].
|
||||
Thought 4: Given the difficulties in finding a direct list of Mexican Grand Prix winners, I should search for the Mexican Grand Prix to get a broader picture of the race's history. This might include winners.
|
||||
Action 4: Search[Mexican Grand Prix]
|
||||
This trajectory is incorrect as my search should be related to Mexican Formula One race car drivers, not winners of the Mexican Grand Prix, a seperate event. A better search would have been for the List of Formula One Grand Prix winners, as suggested.
|
||||
Thus the correctness score is 3
|
||||
|
||||
Question: Which magazine was started first Arthur's Magazine or First for Women?
|
||||
Thought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.
|
||||
Action 1: Search[Arthur's Magazine]
|
||||
Observation 1: Arthur's Magazine (1844\u00e2\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. Edited by Timothy Shay Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others.[1][2] In May 1846 it was merged into Godey's Lady's Book.[3]
|
||||
This trajectory is correct as it is reasonable to search for the first magazine provided in the question. It is also better to have simple searches corresponding to a single entity, making this the best action.
|
||||
Thus the correctness score is 10
|
||||
|
||||
Question: The football manager who recruited David Beckham managed Manchester United during what timeframe?Question: The football manager who recruited David Beckham managed Manchester United during what timeframe?
|
||||
Thought 1: I need to search for the football manager who recruited David Beckham, and then find out the timeframe in which he managed Manchester United.
|
||||
Action 1: Search[David Beckham's football manager]
|
||||
Observation 1: Could not find David Beckham's football manager. Similar: ['David Beckham', 'Inter Miami CF (redirect from Miami Beckham United)', 'Victoria Beckham', 'The Beckham Experiment', 'Miami Freedom Park'].
|
||||
Thought 2: I couldn't find the information on David Beckham's football manager. I need to search Manchester United and find out the timeframe of the manager.
|
||||
Action 2: Search[Manchester United]
|
||||
Observation 2: Manchester United Football Club, commonly referred to as Man United (often stylised as Man Utd), or simply United, is a professional football club based in Old Trafford, Greater Manchester, England. The club competes in the Premier League, the top division in the English football league system. Nicknamed the Red Devils, they were founded as Newton Heath LYR Football Club in 1878, but changed their name to Manchester United in 1902. After a spell playing in Clayton, Manchester, the club moved to their current stadium, Old Trafford, in 1910.. Domestically, Manchester United have won a record 20 league titles, 12 FA Cups, six League Cups and a record 21 FA Community Shields.
|
||||
Thought 3: The information about David Beckham's football manager is not available. I will search for Manchester United's managers to find the timeframe.
|
||||
Action 3: Search[Manchester United managers]
|
||||
Observation 3: Manchester United Football Club is a professional football club based in Old Trafford, Greater Manchester, England. The club was formed in Newton Heath in 1878 as Newton Heath LYR F.C., and played their first competitive match in October 1886, when they entered the First Round of the 1886–87 FA Cup. The club was renamed Manchester United F.C. in 1902, and moved to Old Trafford in 1910.. From the beginning of the club's official managerial records in 1892 to the start of the 2022–23 season, Manchester United have had 23 full-time managers.
|
||||
Thought 4: The information about David Beckham's football manager is not available. The question asks for the timeframe when the manager managed Manchester United, so I need to find a list of Manchester United managers.
|
||||
Action 4: Search[list of Manchester United managers]
|
||||
Observation 4: Manchester United Football Club is a professional football club based in Old Trafford, Greater Manchester, England. The club was formed in Newton Heath in 1878 as Newton Heath LYR F.C., and played their first competitive match in October 1886, when they entered the First Round of the 1886–87 FA Cup. The club was renamed Manchester United F.C. in 1902, and moved to Old Trafford in 1910.. From the beginning of the club's official managerial records in 1892 to the start of the 2022–23 season, Manchester United have had 23 full-time managers.
|
||||
This trajectory is mostly incorrect as I was not able to retrieve the relevant information to answer the question. The searches are too broad and not relevant to the specific player recruited, David Beckham. The last two actions are also too similar. It would be better to first search for David Beckham and lookup information related to his manager.
|
||||
Thus the correctness score is 4
|
||||
|
||||
Question: Musician and satirist Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?
|
||||
Thought 1: The question simplifies to "The Simpsons" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.
|
||||
Action 1: Search[Milhouse]
|
||||
Observation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.
|
||||
Thought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up "named after".
|
||||
Action 2: Lookup[named after]
|
||||
Observation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous.
|
||||
Thought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.
|
||||
Action 3: Finish[President Richard Nixon]
|
||||
This trajectory is correct as all of my thoughts and actions are correct. It makes sense to search for Milhouse first as it is the central subject of the question. It is also correct to directly look up the relevant information in the article, instead of trying another search.
|
||||
Thus the correctness score is 10
|
||||
|
||||
{input}
|
||||
'''
|
||||
|
||||
value_prompt_reasoning_feedback = '''Analyze the trajectories of a solution to a question answering task. The trajectories are labeled by environmental observations about the situation, thoughts that can reason about the current situation and actions that can be three types:
|
||||
(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
|
||||
(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
|
||||
(3) Finish[answer], which returns the answer and finishes the task.
|
||||
|
||||
Given a question and a trajectory, evaluate its correctness and provide your reasoning and analysis in detail. Focus on the latest thought, action, and observation. Incomplete trajectories can be correct if the thoughts and actions so far are correct, even if the answer is not found yet. Do not generate additional thoughts or actions. Then at the last line conclude "Thus the correctness score is {s}", where s is an integer from 1 to 10.
|
||||
|
||||
Question: Which other Mexican Formula One race car driver has held the podium besides the Force India driver born in 1990?
|
||||
Thought 1: I need to search for a list of Mexican Formula One race car drivers, then eliminate the one who drove for Force India and was born in 1990.
|
||||
Action 1: Search[Formula One drivers from Mexico]
|
||||
Observation 1: There have been six Formula One drivers from Mexico who have taken part in races since the championship began in 1950. Sergio P\u00e9rez is the most successful, having won six Grands Prix. Since 2021, P\u00e9rez drives for the Red Bull team.. P\u00e9rez is the only Mexican F1 driver to achieve a pole position. He took his first pole position at the 2022 Saudi Arabian Grand Prix, in his 219th Grand Prix meeting (a record for the most Grands Prix before a pole).
|
||||
Thought 2: Sergio P\u00e9rez, born in 1990 and drove for Force India, is not the Mexican driver we are looking for. The passage does not give me enough information about other Mexican drivers who have achieved a podium finish. I need to look up more information.
|
||||
Action 2: Lookup[Formula One drivers from Mexico]
|
||||
Observation 2: No more results.
|
||||
Thought 3: The available information doesn't allow me to determine which other Mexican Formula One race car driver, besides Sergio P\u00e9rez, has held the podium. With the search function failing to provide a full list of Mexican Formula One drivers, I need to return different search results. Let's try searching for Mexican Grand Prix winners, as they would have held the podium.
|
||||
Action 3: Search[Mexican Grand Prix winners]
|
||||
Observation 3: Could not find Mexican Grand Prix winners. Similar: ['Mexican Grand Prix', 'List of Formula One Grand Prix winners', '1990 Mexican Grand Prix', '2018 Mexican Grand Prix', '2019 Mexican Grand Prix'].
|
||||
Thought 4: Given the difficulties in finding a direct list of Mexican Grand Prix winners, I should search for the Mexican Grand Prix to get a broader picture of the race's history. This might include winners.
|
||||
Action 4: Search[Mexican Grand Prix]
|
||||
This trajectory is incorrect as my search should be related to Mexican Formula One race car drivers, not winners it is reasonable to checkof the Mexican Grand Prix, a seperate event. A better search would have been for the List of Formula One Grand Prix winners, as suggested.
|
||||
Thus the correctness score is 3
|
||||
|
||||
Question: Which magazine was started first Arthur's Magazine or First for Women?
|
||||
Thought 1: I need to search Arthur's Magazine and First for Women, and find which was started first.
|
||||
Action 1: Search[Arthur's Magazine]
|
||||
Observation 1: Arthur's Magazine (1844\u00e2\u0080\u00931846) was an American literary periodical published in Philadelphia in the 19th century. Edited by Timothy Shay Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others.[1][2] In May 1846 it was merged into Godey's Lady's Book.[3]
|
||||
This trajectory is correct as it is reasonable to search for the first magazine provided in the question. It is also better to have simple searches corresponding to a single entity, making this the best action.
|
||||
Thus the correctness score is 10
|
||||
|
||||
Question: The football manager who recruited David Beckham managed Manchester United during what timeframe?Question: The football manager who recruited David Beckham managed Manchester United during what timeframe?
|
||||
Thought 1: I need to search for the football manager who recruited David Beckham, and then find out the timeframe in which he managed Manchester United.
|
||||
Action 1: Search[David Beckham's football manager]
|
||||
Observation 1: Could not find David Beckham's football manager. Similar: ['David Beckham', 'Inter Miami CF (redirect from Miami Beckham United)', 'Victoria Beckham', 'The Beckham Experiment', 'Miami Freedom Park'].
|
||||
Thought 2: I couldn't find the information on David Beckham's football manager. I need to search Manchester United and find out the timeframe of the manager.
|
||||
Action 2: Search[Manchester United]
|
||||
Observation 2: Manchester United Football Club, commonly referred to as Man United (often stylised as Man Utd), or simply United, is a professional football club based in Old Trafford, Greater Manchester, England. The club competes in the Premier League, the top division in the English football league system. Nicknamed the Red Devils, they were founded as Newton Heath LYR Football Club in 1878, but changed their name to Manchester United in 1902. After a spell playing in Clayton, Manchester, the club moved to their current stadium, Old Trafford, in 1910.. Domestically, Manchester United have won a record 20 league titles, 12 FA Cups, six League Cups and a record 21 FA Community Shields.
|
||||
Thought 3: The information about David Beckham's football manager is not available. I will search for Manchester United's managers to find the timeframe.
|
||||
Action 3: Search[Manchester United managers]
|
||||
Observation 3: Manchester United Football Club is a professional football club based in Old Trafford, Greater Manchester, England. The club was formed in Newton Heath in 1878 as Newton Heath LYR F.C., and played their first competitive match in October 1886, when they entered the First Round of the 1886–87 FA Cup. The club was renamed Manchester United F.C. in 1902, and moved to Old Trafford in 1910.. From the beginning of the club's official managerial records in 1892 to the start of the 2022–23 season, Manchester United have had 23 full-time managers.
|
||||
Thought 4: The information about David Beckham's football manager is not available. The question asks for the timeframe when the manager managed Manchester United, so I need to find a list of Manchester United managers.
|
||||
Action 4: Search[list of Manchester United managers]
|
||||
Observation 4: Manchester United Football Club is a professional football club based in Old Trafford, Greater Manchester, England. The club was formed in Newton Heath in 1878 as Newton Heath LYR F.C., and played their first competitive match in October 1886, when they entered the First Round of the 1886–87 FA Cup. The club was renamed Manchester United F.C. in 1902, and moved to Old Trafford in 1910.. From the beginning of the club's official managerial records in 1892 to the start of the 2022–23 season, Manchester United have had 23 full-time managers.
|
||||
This trajectory is mostly incorrect as I was not able to retrieve the relevant information to answer the question. The searches are too broad and not relevant to the specific player recruited, David Beckham. The last two actions are also too similar. It would be better to first search for David Beckham and lookup information related to his manager.
|
||||
Thus the correctness score is 4
|
||||
|
||||
Question: Musician and satirist Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?
|
||||
Thought 1: The question simplifies to "The Simpsons" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.
|
||||
Action 1: Search[Milhouse]
|
||||
Observation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.
|
||||
Thought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up "named after".
|
||||
Action 2: Lookup[named after]
|
||||
Observation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous.
|
||||
Thought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.
|
||||
Action 3: Finish[President Richard Nixon]
|
||||
This trajectory is correct as all of my thoughts and actions are correct. It makes sense to search for Milhouse first as it is the central subject of the question. It is also correct to directly look up the relevant information in the article, instead of trying another search.
|
||||
Thus the correctness score is 10
|
||||
|
||||
{trajectories}
|
||||
|
||||
{input}
|
||||
'''
|
||||
|
||||
value_prompt_reasoning_feedback_short = '''Analyze the trajectories of a solution to a question answering task. The trajectories are labeled by environmental observations about the situation, thoughts that can reason about the current situation and actions that can be three types:
|
||||
(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
|
||||
(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
|
||||
(3) Finish[answer], which returns the answer and finishes the task.
|
||||
|
||||
Given a question and a trajectory, evaluate its correctness and provide your reasoning and analysis in detail. Focus on the latest thought, action, and observation. Incomplete trajectories can be correct if the thoughts and actions so far are correct, even if the answer is not found yet. Do not generate additional thoughts or actions. Then at the last line conclude "Thus the correctness score is {s}", where s is an integer from 1 to 10.
|
||||
|
||||
Question: Which other Mexican Formula One race car driver has held the podium besides the Force India driver born in 1990?
|
||||
Thought 1: I need to search for a list of Mexican Formula One race car drivers, then eliminate the one who drove for Force India and was born in 1990.
|
||||
Action 1: Search[Formula One drivers from Mexico]
|
||||
Observation 1: There have been six Formula One drivers from Mexico who have taken part in races since the championship began in 1950. Sergio P\u00e9rez is the most successful, having won six Grands Prix. Since 2021, P\u00e9rez drives for the Red Bull team.. P\u00e9rez is the only Mexican F1 driver to achieve a pole position. He took his first pole position at the 2022 Saudi Arabian Grand Prix, in his 219th Grand Prix meeting (a record for the most Grands Prix before a pole).
|
||||
Thought 2: Sergio P\u00e9rez, born in 1990 and drove for Force India, is not the Mexican driver we are looking for. The passage does not give me enough information about other Mexican drivers who have achieved a podium finish. I need to look up more information.
|
||||
Action 2: Lookup[Formula One drivers from Mexico]
|
||||
Observation 2: No more results.
|
||||
Thought 3: The available information doesn't allow me to determine which other Mexican Formula One race car driver, besides Sergio P\u00e9rez, has held the podium. With the search function failing to provide a full list of Mexican Formula One drivers, I need to return different search results. Let's try searching for Mexican Grand Prix winners, as they would have held the podium.
|
||||
Action 3: Search[Mexican Grand Prix winners]
|
||||
Observation 3: Could not find Mexican Grand Prix winners. Similar: ['Mexican Grand Prix', 'List of Formula One Grand Prix winners', '1990 Mexican Grand Prix', '2018 Mexican Grand Prix', '2019 Mexican Grand Prix'].
|
||||
Thought 4: Given the difficulties in finding a direct list of Mexican Grand Prix winners, I should search for the Mexican Grand Prix to get a broader picture of the race's history. This might include winners.
|
||||
Action 4: Search[Mexican Grand Prix]
|
||||
This trajectory is incorrect as my search should be related to Mexican Formula One race car drivers, not winners of the Mexican Grand Prix, a seperate event. A better search would have been for the List of Formula One Grand Prix winners, as suggested.
|
||||
Thus the correctness score is 3
|
||||
|
||||
Question: Musician and satirist Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?
|
||||
Thought 1: The question simplifies to "The Simpsons" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.
|
||||
Action 1: Search[Milhouse]
|
||||
Observation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.
|
||||
Thought 2: The paragraph does not tell who Milhouse is named after, maybe I can look up "named after".
|
||||
Action 2: Lookup[named after]
|
||||
Observation 2: (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose middle name was Milhous.
|
||||
Thought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.
|
||||
Action 3: Finish[President Richard Nixon]
|
||||
This trajectory is correct as all of my thoughts and actions are correct. It makes sense to search for Milhouse first as it is the central subject of the question. It is also correct to directly look up the relevant information in the article, instead of trying another search.
|
||||
Thus the correctness score is 10
|
||||
|
||||
{trajectories}
|
||||
|
||||
{input}
|
||||
'''
|
||||
|
||||
rap_prompt = '''
|
||||
Solve a question answering task with interleaving Thought and Action steps. Thought can reason about the current situation, and Action can be three types:
|
||||
(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
|
||||
(2) Lookup[keyword], which returns the next sentence containing keyword in the current passage.
|
||||
(3) Finish[answer], which returns the answer and finishes the task.
|
||||
Provide the next Thought and next Action. Here are some examples:
|
||||
|
||||
Question: What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?
|
||||
Thought 1: I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area.
|
||||
Action 1: Search[Colorado orogeny]
|
||||
Thought 2: Then I need to look up eastern sector.
|
||||
Action 2: Lookup[eastern sector]
|
||||
Thought 3: The eastern sector of Colorado orogeny extends into the High Plains, so High Plains is the area. I need to search High Plains and find its elevation range.
|
||||
Action 3: Search[High Plains]
|
||||
Thought 4: I need to search High Plains (United States).
|
||||
Action 4: Search[High Plains (United States)]
|
||||
Thought 5: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer is 1,800 to 7,000 ft.
|
||||
Action 5: Finish[1,800 to 7,000 ft]
|
||||
|
||||
Question: Guitars for Wounded Warriors is an album that was recorded in the village in which New York county?
|
||||
Thought 1: I need to search Guitars for Wounded Warriors, then find in which New York county it was recorded.
|
||||
Action 1: Search[Guitars for Wounded Warriors]
|
||||
Thought 2: I need to search New Paltz (village), New York and find which county it is in
|
||||
Action 2: Search[New Paltz (village), New York]
|
||||
Thought 3: New Paltz (village), New York is in Ulster County, so the answer is Ulster County.
|
||||
Action 3: Finish[Ulster County]
|
||||
|
||||
Question: Musician and satirist Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?
|
||||
Thought 1: The question simplifies to "The Simpsons" character Milhouse is named after who. I only need to search Milhouse and find who it is named after.
|
||||
Action 1: Search[Milhouse]
|
||||
Thought 2: I can look up "named after" for finding the specific individual Milhouse is named after.
|
||||
Action 2: Lookup[named after]
|
||||
Thought 3: Milhouse was named after U.S. president Richard Nixon, so the answer is Richard Nixon.
|
||||
Action 3: Finish[President Richard Nixon]
|
||||
|
||||
{input}
|
||||
'''
|
||||
@@ -191,6 +191,8 @@ function _process_message(agent::yiemAgent, msg)::assistantMessage
|
||||
|
||||
# call executeToolCalls()
|
||||
|
||||
# save toolResults to agent._state.messages
|
||||
|
||||
# else
|
||||
# break out of while loop
|
||||
|
||||
|
||||
+10
-1
@@ -295,7 +295,7 @@ function agentState(
|
||||
end
|
||||
|
||||
|
||||
struct toolCall # A tool invocation from the LLM
|
||||
struct agentToolCall # A tool invocation from the LLM
|
||||
type::String # Always "function"
|
||||
id::String # Unique tool call identifier
|
||||
name::String # Tool name
|
||||
@@ -528,6 +528,15 @@ end
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user