update

2026-06-30 13:28:53 +07:00
parent da96c34e58
commit 6cd58ffd14
1 changed files with 90 additions and 0 deletions
@@ -18,6 +18,96 @@ The package addresses the sparse reward problem in MCTS by using LLMs to provide
 | `statevalue` | Monte Carlo simulation | Actual cumulative reward from simulations; accurate but expensive to compute |
 | `reward` | Environment | Immediate reward from environment (may be sparse, only at terminal states) |

+### How reward and progressvalue Work Together
+
+The package uses two distinct value signals that serve different purposes during MCTS search:
+
+#### `reward` - Environment Feedback
+- **Source**: Environment only
+- **When**: Only at terminal states (sparse) or immediate action outcome
+- **Purpose**: Ground truth for backpropagation updates
+- **Usage**: 
+  - Accumulated during simulation rollouts (line 287 in mcts.jl)
+  - Used to update `statevalue` via backpropagation
+  - Final metric for determining solution quality
+
+#### `progressvalue` - LLM Heuristic
+- **Source**: LLM's reasoning about state quality
+- **When**: At every node expansion (dense guidance)
+- **Purpose**: Fast node selection without waiting for terminal rewards
+- **Usage**:
+  - Passed to `_expand()` from `transition()` result (line 237 in mcts.jl)
+  - Used as fallback in UCT when child has zero visits (util.jl:58)
+  - Combined with `reward` in `selectChildNode()` (line 171 in mcts.jl)
+
+#### The Selection Hierarchy
+
+```
+1. UCT SELECTION (interface.jl → runMCTS → UCTselect)
+   └── Uses: statevalue + exploration_weight * sqrt(ln(parent_visits) / child_visits)
+   └── Purpose: Balance exploration vs exploitation during tree traversal
+
+2. CHILD SELECTION (mcts.jl → selectChildNode)
+   └── Uses: progressvalue + reward
+   └── Purpose: Pick best child after expansion during simulation rollout
+
+3. BEST NODE SELECTION (mcts.jl → selectBestNextNode)
+   └── If statevalue sum > 0: statevalue / visits
+   └── Else: progressvalue + reward
+   └── Purpose: Determine final best trajectory after MCTS completes
+```
+
+#### Data Flow Example
+
+```
+Expansion Phase (LLM call):
+┌────────────────────────────────────────────────────────────────────────┐
+│ transition(state, args) → {                                            │
+│   newNodeKey: "abc-123",                                               │
+│   newstate: { reward: 0, isterminal: false },                          │
+│   progressvalue: 7.5   ← LLM estimates this state is promising        │
+│ }                                                                       │
+└────────────────────────────────────────────────────────────────────────┘
+                                    ↓
+New Node Created:
+MCTSNode(
+  nodekey = "abc-123",
+  state = { reward: 0, isterminal: false },
+  visits = 0,
+  progressvalue = 7.5,   ← LLM heuristic (fast, initial guidance)
+  statevalue = 0,        ← Will be updated after simulation
+  reward = 0,            ← Immediate environment reward
+  ...
+)
+
+Simulation Phase:
+┌────────────────────────────────────────────────────────────────────────┐
+│ simulate() rolls out 3 levels:                                         │
+│ Level 1: reward += 0  (non-terminal)                                   │
+│ Level 2: reward += 3  (intermediate reward)                            │
+│ Level 3: reward += 10 (terminal state)                                 │
+│ ─────────────────────────────────────────────────────────────────────  │
+│ Total simTrajectoryReward = 13                                         │
+└────────────────────────────────────────────────────────────────────────┘
+                                    ↓
+Backpropagation:
+┌────────────────────────────────────────────────────────────────────────┐
+│ Update all ancestors with simTrajectoryReward = 13:                    │
+│ node.visits += 1                                                       │
+│ node.statevalue = (old_statevalue * (visits-1) + 13) / visits         │
+│ reward *= 0.9 (discount for future rewards)                            │
+└────────────────────────────────────────────────────────────────────────┘
+```
+
+#### Why Two Value Systems?
+
+| Problem | Traditional MCTS | LLMMCTS Solution |
+|---------|-----------------|------------------|
+| **Sparse rewards** | Must explore blindly until terminal | LLM provides `progressvalue` at every node |
+| **Slow learning** | Need many terminal outcomes | Dense `progressvalue` enables fast guidance |
+| **Exploration cost** | Random expansion is inefficient | LLM filters to promising candidates |
+| **Value uncertainty** | Hand-designed heuristics may be wrong | `statevalue` (from simulation) confirms LLM estimates |
+
 ### Module Structure

 ```