update
This commit is contained in:
@@ -650,12 +650,11 @@ function evaluator(state::T1, text2textInstructLLM::Function;
|
||||
Trajectories: ...
|
||||
|
||||
You should then respond to the user with:
|
||||
1) Trajectory_evaluation:
|
||||
- Analyze the trajectories of a solution to answer the user's original question.
|
||||
Then given a question and a trajectory, evaluate its correctness and provide your reasoning and
|
||||
analysis in detail. Focus on the latest thought, action, and observation.
|
||||
Incomplete trajectories can be correct if the thoughts and actions so far are correct,
|
||||
even if the answer is not found yet. Do not generate additional thoughts or actions.
|
||||
1) Trajectory_evaluation: Analyze the trajectory of a solution to answer the user's original question.
|
||||
- Evaluate the correctness of each section and the overall trajectory based on the given question.
|
||||
- Provide detailed reasoning and analysis, focusing on the latest thought, action, and observation.
|
||||
- Incomplete trajectories are acceptable if the thoughts and actions up to that point are correct, even if the final answer isn't reached.
|
||||
- Do not generate additional thoughts or actions.
|
||||
2) Answer_evaluation:
|
||||
- Focus only on the matter mentioned in the question and comprehensively analyze how the latest observation's details addresses the question
|
||||
- State your rationale
|
||||
@@ -738,14 +737,14 @@ function evaluator(state::T1, text2textInstructLLM::Function;
|
||||
state[:isterminal] = true
|
||||
|
||||
# user score as reward because different answers hold different value for the user.
|
||||
state[:reward] = responsedict[:score]
|
||||
state[:reward] = responsedict[:score]
|
||||
|
||||
#add to vectorDB
|
||||
# add to vectorDB
|
||||
if addSQLVectorDB !== nothing
|
||||
addSQLVectorDB(state)
|
||||
end
|
||||
end
|
||||
println("~~~ 5 Evaluator() ", @__FILE__, " ", @__LINE__)
|
||||
println("~~~ Evaluator() ", @__FILE__, " ", @__LINE__)
|
||||
pprintln(Dict(responsedict))
|
||||
|
||||
return responsedict[:score]
|
||||
|
||||
Reference in New Issue
Block a user