This commit is contained in:
2026-06-30 12:11:18 +07:00
parent e866daa3f5
commit 73f769d13b
+15 -8
View File
@@ -13,7 +13,7 @@ MCTS is a powerful search algorithm that balances exploration and exploitation t
However, in many real-world problems, **rewards are sparse**—they only come at the final state. This creates two critical problems:
1. **Value estimation delay** — Rewards must propagate backward through many layers before affecting early decisions
2. **Exploration inefficiency** — Without intermediate signals, MCTS explores randomly until it偶然 discovers a reward
2. **Exploration inefficiency** — Without intermediate signals, MCTS explores randomly until it discovers a reward
### How LLMs Fix the Sparse Reward Problem
@@ -138,13 +138,13 @@ function transition(state::Dict, args::NamedTuple)
end
# Define transition arguments
transition_args = (param1 = "value1", param2 = "value2")
transitionargs = (param1 = "value1", param2 = "value2")
# Run MCTS
result = runMCTS(
initialstate,
transition,
transition_args;
transitionargs;
maxiterations = 10,
explorationweight = 1.0,
maxSimulationDepth = 3
@@ -162,9 +162,9 @@ high_value_states = result.highValueStateList
```julia
# With custom parameters
result = runMCTS(
initialState,
initialstate,
transition_func,
transition_args;
transitionargs;
horizontalSampleExpansionPhase = 5, # More children during expansion
horizontalSampleSimulationPhase = 3, # Sample 3 children during simulation
maxSimulationDepth = 5, # Deeper search
@@ -203,12 +203,19 @@ Search for the best action to take for a given state and task.
#### `simulateThenBackpropagate(node, transition, transitionargs; kwargs...)`
Run simulation from a node and backpropagate the reward.
Run simulation from a node and backpropagate the reward. Returns `nothing`.
**Keyword Arguments:**
- `maxSimulationDepth::Integer=3` — Maximum simulation depth
- `horizontalSampleSimulationPhase::Integer=3` — Children per simulation node
- `saveSimulatedNode::Bool=false` — Keep simulation nodes
- `multithread::Bool=false` — Enable multithreading
- `highValueState` — Channel to store high-value states
### Utility Functions
- `UCTselect(node, w)` — Select node using UCT score
- `dictify(x; keytype=Any, stringkey=false)` — Convert JSON.Object/OrderedDict to plain Dict
- `dictify(x; keytype=Any)` — Convert JSON.Object/OrderedDict to plain Dict
### MCTS Node Structure
@@ -255,7 +262,7 @@ Where:
| Scenario | Alternative approach |
|----------|---------------------|
| **Dense rewards available** | Use pure RL with reward shaping |
| **Simple决策 problems** | Classical search (DFS, BFS) is sufficient |
| **Simple decision problems** | Classical search (DFS, BFS) is sufficient |
| **Real-time constraints** | LLM calls may be too slow; use pre-trained value function |
| **No LLM access** | Use pure MCTS with hand-designed heuristics |