ton/LLMMCTS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

update

Browse Source
This commit is contained in:
narawat
2026-06-30 11:58:20 +07:00
parent f21274d6d8
commit e866daa3f5
1 changed files with 101 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+101 -53
View File
@@ -7,60 +7,41 @@ LLMMCTS implements Monte Carlo Tree Search (MCTS) for Large Language Model (LLM)
## Why LLM + MCTS?
Integrating LLMs with MCTS creates a powerful planning system that combines:
- **LLM reasoning**: Understand complex tasks, generate creative solutions, and reason about states
- **MCTS optimization**: Systematically explore solution spaces and find optimal trajectories
MCTS is a powerful search algorithm that balances exploration and exploitation through the UCT formula:
\[ \text{UCT}(s,a) = Q(s,a) + c \sqrt{\frac{\ln N(s)}{N(s,a)}} \]
### The Sparse Reward Problem in Traditional MCTS
However, in many real-world problems, **rewards are sparse**—they only come at the final state. This creates two critical problems:
In traditional reinforcement learning and MCTS, **sparse rewards** are a fundamental challenge that severely limits performance:
1. **Value estimation delay** — Rewards must propagate backward through many layers before affecting early decisions
2. **Exploration inefficiency** — Without intermediate signals, MCTS explores randomly until it偶然 discovers a reward
| Challenge | Traditional MCTS | LLM-MCTS Integration |
|-----------|------------------|----------------------|
| **Reward signal** | Only at terminal states (e.g., win/loss) | **Pseudo-rewards at every state** via LLM |
| **Value estimation** | Relies on Monte Carlo sampling (high variance, slow convergence) | LLM provides **progress value** as heuristic |
| **Sample efficiency** | Low—requires many samples to discover reward | High—LLM guides search toward promising regions |
| **Exploration strategy** | Blind exploration until reward discovered | LLM suggests promising actions to try |
### How LLMs Fix the Sparse Reward Problem
#### Why Sparse Rewards Are Problematic
1. **High variance in value estimates** — With sparse rewards, Monte Carlo estimates have high variance because few samples contribute to each node's value
2. **Slow learning** — Rewards must propagate backward through many layers before affecting early decisions, requiring many iterations
3. **Poor exploration** — Without intermediate signals, MCTS explores randomly until it偶然 discovers a reward, wasting computation
4. **Local optima** — Without guidance, MCTS may get stuck in suboptimal regions of the search space
#### How LLM-MCTS Solves This
**Progress Value vs. Reward:**
- `progressvalue` — LLM's estimate of how close we are to solving the task (pseudo-reward)
- `statevalue` — Actual cumulative reward from Monte Carlo simulations
- `reward` — Immediate reward from environment (may be sparse, only at terminal states)
**The three-tier value system:**
LLMs provide **reasoning and pseudo-reward** to guide the solution search process:
- **Reasoning** — LLM understands task structure and generates promising candidate actions
- **Pseudo-reward** — LLM estimates state quality at every node (not just terminal states)
In the code, this is represented by the `progressvalue` field:
```julia
# LLM provides progress value at every node (dense, fast guidance)
progressvalue = llm_estimate(state) # Heuristic, available immediately
# MCTS computes statevalue via simulation (sparse but accurate)
statevalue = monte_carlo_average(simulations) # Accurate but expensive
# Environment provides immediate reward (may be sparse)
reward = environment_reward(state) # Only at terminal states
progressvalue = llm_reasoning_estimate(state)
```
**How MCTS uses these values together:**
The LLM evaluates how close the current state is to solving the task, providing dense guidance even when the environment only gives rewards at the end.
| Phase | Which value used | Why |
|-------|------------------|-----|
| Node selection (UCT) | `progressvalue` + `statevalue` | Dense guidance for fast exploration |
| Simulation | `statevalue` + `reward` | Accurate long-term estimates |
| Backpropagation | `reward` | Ground truth updates |
### The Three-Tier Value System
**Why this matters:** LLMs provide dense `progressvalue` guidance at every node, allowing MCTS to focus computation on promising trajectories. The simulation phase confirms these estimates with accurate `statevalue` computed via Monte Carlo. This combination solves the sparse reward problem while maintaining accuracy.
LLMMCTS combines LLM guidance with MCTS search using three complementary value signals:
| Field | Source | Purpose |
|-------|--------|---------|
| `progressvalue` | LLM heuristic | Estimate of how close we are to solving; used for fast node selection |
| `statevalue` | Monte Carlo simulation | Actual cumulative reward from simulations; accurate but expensive to compute |
| `reward` | Environment | Immediate reward from environment (may be sparse, only at terminal states) |
**Why this matters:**
- `progressvalue` enables MCTS to explore promising branches quickly without waiting for terminal rewards
- `statevalue` provides accurate long-term estimates through Monte Carlo simulations
- `reward` supplies ground truth for backpropagation updates
### Benefits of LLM-MCTS Integration
@@ -74,6 +55,37 @@ reward = environment_reward(state) # Only at terminal states
| **Configurable depth** | MCTS depth controls planning horizon; LLM handles long-term reasoning at each step |
| **Parallel exploration** | MCTS naturally supports parallel simulation; LLM generates diverse candidate actions |
### MCTS Node Structure
```julia
MCTSNode(
nodekey::String,
state::Dict,
visits::Integer,
progressvalue::Number,
statevalue::Number,
reward::Number,
isterminal::Bool,
parent::Union{MCTSNode, Nothing},
children::Dict{String, MCTSNode},
etc::Dict{String, Any}
)
```
### Understanding `progressvalue`, `statevalue`, and `reward`
| Field | Source | Purpose |
|-------|--------|---------|
| `progressvalue` | LLM heuristic | Estimate of how close we are to solving; used for fast node selection |
| `statevalue` | Monte Carlo simulation | Actual cumulative reward from simulations; accurate but expensive to compute |
| `reward` | Environment | Immediate reward (may be sparse, only at terminal states) |
**Why this matters:** In traditional MCTS, sparse rewards force extensive exploration. Here, LLM provides dense `progressvalue` guidance at every node, while `statevalue` (computed via simulation) provides accurate long-term estimates. MCTS balances both via UCT:
- High `progressvalue` → explored early (fast guidance)
- High `statevalue` → exploited once confirmed (accurate value)
## Contributing
## Overview
### Key Features
@@ -215,17 +227,53 @@ MCTSNode(
)
```
### Understanding `progressvalue`, `statevalue`, and `reward`
### How UCT Uses progressvalue and statevalue
| Field | Source | Purpose |
|-------|--------|---------|
| `progressvalue` | LLM heuristic | Estimate of how close we are to solving; used for fast node selection |
| `statevalue` | Monte Carlo simulation | Actual cumulative reward from simulations; accurate but expensive to compute |
| `reward` | Environment | Immediate reward (may be sparse, only at terminal states) |
The UCT formula selects children using both value signals:
\[ \text{UCT}(s,a) = Q(s,a) + c \sqrt{\frac{\ln N(s)}{N(s,a)}} \]
**Why this matters:** In traditional MCTS, sparse rewards force extensive exploration. Here, LLM provides dense `progressvalue` guidance at every node, while `statevalue` (computed via simulation) provides accurate long-term estimates. MCTS balances both via UCT:
- High `progressvalue` → explored early (fast guidance)
- High `statevalue` → exploited once confirmed (accurate value)
Where:
- **Exploitation term** (`Q(s,a)`) — Uses `progressvalue` for fast guidance, refined by `statevalue` as simulations accumulate
- **Exploration term** — Encourages visiting less-explored branches, even those with high `progressvalue` but low visit count
**Selection priority:**
1. Nodes with high `progressvalue` and low `visits` → explored first (fast guidance)
2. Nodes with high `statevalue` confirmed by simulations → exploited once reliable
3. Balance determined by `explorationweight` parameter
### When to Use LLM-MCTS
| Scenario | Why LLM-MCTS is suitable |
|----------|--------------------------|
| **Sparse reward environments** | Rewards only at terminal states (e.g., game win, code execution success) |
| **Complex reasoning tasks** | Tasks requiring multi-step planning (math, coding, tool use) |
| **High branching factor** | Many possible actions; LLM filters to promising candidates |
| **Need sample efficiency** | Limited budget for environment interactions |
### When Not to Use LLM-MCTS
| Scenario | Alternative approach |
|----------|---------------------|
| **Dense rewards available** | Use pure RL with reward shaping |
| **Simple决策 problems** | Classical search (DFS, BFS) is sufficient |
| **Real-time constraints** | LLM calls may be too slow; use pre-trained value function |
| **No LLM access** | Use pure MCTS with hand-designed heuristics |
### Performance Characteristics
| Metric | Typical range |
|--------|---------------|
| **Sample efficiency** | 5-10x fewer samples than pure Monte Carlo |
| **LLM calls per iteration** | 1-5 (depends on `horizontalSample*` settings) |
| **Convergence time** | Scales with depth × LLM latency |
| **Memory usage** | O(branching_factor^depth) for tree storage |
### Limitations
- **LLM latency** — Each node expansion requires an LLM call; can be slow for large trees
- **LLM cost** — Each LLM invocation has financial cost; monitor usage
- **Heuristic quality** — Poor LLM pseudo-rewards lead to suboptimal search
- **Determinism** — LLM outputs are stochastic; use temperature=0 for reproducibility
## Contributing