From e866daa3f5ebcb21e0225255ea5d029f56377d8e Mon Sep 17 00:00:00 2001 From: narawat Date: Tue, 30 Jun 2026 11:58:20 +0700 Subject: [PATCH] update --- README.md | 154 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 101 insertions(+), 53 deletions(-) diff --git a/README.md b/README.md index 4d0cfa4..39cbc70 100644 --- a/README.md +++ b/README.md @@ -7,60 +7,41 @@ LLMMCTS implements Monte Carlo Tree Search (MCTS) for Large Language Model (LLM) ## Why LLM + MCTS? -Integrating LLMs with MCTS creates a powerful planning system that combines: -- **LLM reasoning**: Understand complex tasks, generate creative solutions, and reason about states -- **MCTS optimization**: Systematically explore solution spaces and find optimal trajectories +MCTS is a powerful search algorithm that balances exploration and exploitation through the UCT formula: +\[ \text{UCT}(s,a) = Q(s,a) + c \sqrt{\frac{\ln N(s)}{N(s,a)}} \] -### The Sparse Reward Problem in Traditional MCTS +However, in many real-world problems, **rewards are sparse**—they only come at the final state. This creates two critical problems: -In traditional reinforcement learning and MCTS, **sparse rewards** are a fundamental challenge that severely limits performance: +1. **Value estimation delay** — Rewards must propagate backward through many layers before affecting early decisions +2. **Exploration inefficiency** — Without intermediate signals, MCTS explores randomly until it偶然 discovers a reward -| Challenge | Traditional MCTS | LLM-MCTS Integration | -|-----------|------------------|----------------------| -| **Reward signal** | Only at terminal states (e.g., win/loss) | **Pseudo-rewards at every state** via LLM | -| **Value estimation** | Relies on Monte Carlo sampling (high variance, slow convergence) | LLM provides **progress value** as heuristic | -| **Sample efficiency** | Low—requires many samples to discover reward | High—LLM guides search toward promising regions | -| **Exploration strategy** | Blind exploration until reward discovered | LLM suggests promising actions to try | +### How LLMs Fix the Sparse Reward Problem -#### Why Sparse Rewards Are Problematic - -1. **High variance in value estimates** — With sparse rewards, Monte Carlo estimates have high variance because few samples contribute to each node's value - -2. **Slow learning** — Rewards must propagate backward through many layers before affecting early decisions, requiring many iterations - -3. **Poor exploration** — Without intermediate signals, MCTS explores randomly until it偶然 discovers a reward, wasting computation - -4. **Local optima** — Without guidance, MCTS may get stuck in suboptimal regions of the search space - -#### How LLM-MCTS Solves This - -**Progress Value vs. Reward:** -- `progressvalue` — LLM's estimate of how close we are to solving the task (pseudo-reward) -- `statevalue` — Actual cumulative reward from Monte Carlo simulations -- `reward` — Immediate reward from environment (may be sparse, only at terminal states) - -**The three-tier value system:** +LLMs provide **reasoning and pseudo-reward** to guide the solution search process: +- **Reasoning** — LLM understands task structure and generates promising candidate actions +- **Pseudo-reward** — LLM estimates state quality at every node (not just terminal states) +In the code, this is represented by the `progressvalue` field: ```julia -# LLM provides progress value at every node (dense, fast guidance) -progressvalue = llm_estimate(state) # Heuristic, available immediately - -# MCTS computes statevalue via simulation (sparse but accurate) -statevalue = monte_carlo_average(simulations) # Accurate but expensive - -# Environment provides immediate reward (may be sparse) -reward = environment_reward(state) # Only at terminal states +progressvalue = llm_reasoning_estimate(state) ``` -**How MCTS uses these values together:** +The LLM evaluates how close the current state is to solving the task, providing dense guidance even when the environment only gives rewards at the end. -| Phase | Which value used | Why | -|-------|------------------|-----| -| Node selection (UCT) | `progressvalue` + `statevalue` | Dense guidance for fast exploration | -| Simulation | `statevalue` + `reward` | Accurate long-term estimates | -| Backpropagation | `reward` | Ground truth updates | +### The Three-Tier Value System -**Why this matters:** LLMs provide dense `progressvalue` guidance at every node, allowing MCTS to focus computation on promising trajectories. The simulation phase confirms these estimates with accurate `statevalue` computed via Monte Carlo. This combination solves the sparse reward problem while maintaining accuracy. +LLMMCTS combines LLM guidance with MCTS search using three complementary value signals: + +| Field | Source | Purpose | +|-------|--------|---------| +| `progressvalue` | LLM heuristic | Estimate of how close we are to solving; used for fast node selection | +| `statevalue` | Monte Carlo simulation | Actual cumulative reward from simulations; accurate but expensive to compute | +| `reward` | Environment | Immediate reward from environment (may be sparse, only at terminal states) | + +**Why this matters:** +- `progressvalue` enables MCTS to explore promising branches quickly without waiting for terminal rewards +- `statevalue` provides accurate long-term estimates through Monte Carlo simulations +- `reward` supplies ground truth for backpropagation updates ### Benefits of LLM-MCTS Integration @@ -74,6 +55,37 @@ reward = environment_reward(state) # Only at terminal states | **Configurable depth** | MCTS depth controls planning horizon; LLM handles long-term reasoning at each step | | **Parallel exploration** | MCTS naturally supports parallel simulation; LLM generates diverse candidate actions | +### MCTS Node Structure + +```julia +MCTSNode( + nodekey::String, + state::Dict, + visits::Integer, + progressvalue::Number, + statevalue::Number, + reward::Number, + isterminal::Bool, + parent::Union{MCTSNode, Nothing}, + children::Dict{String, MCTSNode}, + etc::Dict{String, Any} +) +``` + +### Understanding `progressvalue`, `statevalue`, and `reward` + +| Field | Source | Purpose | +|-------|--------|---------| +| `progressvalue` | LLM heuristic | Estimate of how close we are to solving; used for fast node selection | +| `statevalue` | Monte Carlo simulation | Actual cumulative reward from simulations; accurate but expensive to compute | +| `reward` | Environment | Immediate reward (may be sparse, only at terminal states) | + +**Why this matters:** In traditional MCTS, sparse rewards force extensive exploration. Here, LLM provides dense `progressvalue` guidance at every node, while `statevalue` (computed via simulation) provides accurate long-term estimates. MCTS balances both via UCT: +- High `progressvalue` → explored early (fast guidance) +- High `statevalue` → exploited once confirmed (accurate value) + +## Contributing + ## Overview ### Key Features @@ -215,17 +227,53 @@ MCTSNode( ) ``` -### Understanding `progressvalue`, `statevalue`, and `reward` +### How UCT Uses progressvalue and statevalue -| Field | Source | Purpose | -|-------|--------|---------| -| `progressvalue` | LLM heuristic | Estimate of how close we are to solving; used for fast node selection | -| `statevalue` | Monte Carlo simulation | Actual cumulative reward from simulations; accurate but expensive to compute | -| `reward` | Environment | Immediate reward (may be sparse, only at terminal states) | +The UCT formula selects children using both value signals: +\[ \text{UCT}(s,a) = Q(s,a) + c \sqrt{\frac{\ln N(s)}{N(s,a)}} \] -**Why this matters:** In traditional MCTS, sparse rewards force extensive exploration. Here, LLM provides dense `progressvalue` guidance at every node, while `statevalue` (computed via simulation) provides accurate long-term estimates. MCTS balances both via UCT: -- High `progressvalue` → explored early (fast guidance) -- High `statevalue` → exploited once confirmed (accurate value) +Where: +- **Exploitation term** (`Q(s,a)`) — Uses `progressvalue` for fast guidance, refined by `statevalue` as simulations accumulate +- **Exploration term** — Encourages visiting less-explored branches, even those with high `progressvalue` but low visit count + +**Selection priority:** +1. Nodes with high `progressvalue` and low `visits` → explored first (fast guidance) +2. Nodes with high `statevalue` confirmed by simulations → exploited once reliable +3. Balance determined by `explorationweight` parameter + +### When to Use LLM-MCTS + +| Scenario | Why LLM-MCTS is suitable | +|----------|--------------------------| +| **Sparse reward environments** | Rewards only at terminal states (e.g., game win, code execution success) | +| **Complex reasoning tasks** | Tasks requiring multi-step planning (math, coding, tool use) | +| **High branching factor** | Many possible actions; LLM filters to promising candidates | +| **Need sample efficiency** | Limited budget for environment interactions | + +### When Not to Use LLM-MCTS + +| Scenario | Alternative approach | +|----------|---------------------| +| **Dense rewards available** | Use pure RL with reward shaping | +| **Simple决策 problems** | Classical search (DFS, BFS) is sufficient | +| **Real-time constraints** | LLM calls may be too slow; use pre-trained value function | +| **No LLM access** | Use pure MCTS with hand-designed heuristics | + +### Performance Characteristics + +| Metric | Typical range | +|--------|---------------| +| **Sample efficiency** | 5-10x fewer samples than pure Monte Carlo | +| **LLM calls per iteration** | 1-5 (depends on `horizontalSample*` settings) | +| **Convergence time** | Scales with depth × LLM latency | +| **Memory usage** | O(branching_factor^depth) for tree storage | + +### Limitations + +- **LLM latency** — Each node expansion requires an LLM call; can be slow for large trees +- **LLM cost** — Each LLM invocation has financial cost; monitor usage +- **Heuristic quality** — Poor LLM pseudo-rewards lead to suboptimal search +- **Determinism** — LLM outputs are stochastic; use temperature=0 for reproducibility ## Contributing