diff --git a/workprocess.md b/workprocess.md index 24aa6c6..6893af2 100644 --- a/workprocess.md +++ b/workprocess.md @@ -18,6 +18,96 @@ The package addresses the sparse reward problem in MCTS by using LLMs to provide | `statevalue` | Monte Carlo simulation | Actual cumulative reward from simulations; accurate but expensive to compute | | `reward` | Environment | Immediate reward from environment (may be sparse, only at terminal states) | +### How reward and progressvalue Work Together + +The package uses two distinct value signals that serve different purposes during MCTS search: + +#### `reward` - Environment Feedback +- **Source**: Environment only +- **When**: Only at terminal states (sparse) or immediate action outcome +- **Purpose**: Ground truth for backpropagation updates +- **Usage**: + - Accumulated during simulation rollouts (line 287 in mcts.jl) + - Used to update `statevalue` via backpropagation + - Final metric for determining solution quality + +#### `progressvalue` - LLM Heuristic +- **Source**: LLM's reasoning about state quality +- **When**: At every node expansion (dense guidance) +- **Purpose**: Fast node selection without waiting for terminal rewards +- **Usage**: + - Passed to `_expand()` from `transition()` result (line 237 in mcts.jl) + - Used as fallback in UCT when child has zero visits (util.jl:58) + - Combined with `reward` in `selectChildNode()` (line 171 in mcts.jl) + +#### The Selection Hierarchy + +``` +1. UCT SELECTION (interface.jl → runMCTS → UCTselect) + └── Uses: statevalue + exploration_weight * sqrt(ln(parent_visits) / child_visits) + └── Purpose: Balance exploration vs exploitation during tree traversal + +2. CHILD SELECTION (mcts.jl → selectChildNode) + └── Uses: progressvalue + reward + └── Purpose: Pick best child after expansion during simulation rollout + +3. BEST NODE SELECTION (mcts.jl → selectBestNextNode) + └── If statevalue sum > 0: statevalue / visits + └── Else: progressvalue + reward + └── Purpose: Determine final best trajectory after MCTS completes +``` + +#### Data Flow Example + +``` +Expansion Phase (LLM call): +┌────────────────────────────────────────────────────────────────────────┐ +│ transition(state, args) → { │ +│ newNodeKey: "abc-123", │ +│ newstate: { reward: 0, isterminal: false }, │ +│ progressvalue: 7.5 ← LLM estimates this state is promising │ +│ } │ +└────────────────────────────────────────────────────────────────────────┘ + ↓ +New Node Created: +MCTSNode( + nodekey = "abc-123", + state = { reward: 0, isterminal: false }, + visits = 0, + progressvalue = 7.5, ← LLM heuristic (fast, initial guidance) + statevalue = 0, ← Will be updated after simulation + reward = 0, ← Immediate environment reward + ... +) + +Simulation Phase: +┌────────────────────────────────────────────────────────────────────────┐ +│ simulate() rolls out 3 levels: │ +│ Level 1: reward += 0 (non-terminal) │ +│ Level 2: reward += 3 (intermediate reward) │ +│ Level 3: reward += 10 (terminal state) │ +│ ───────────────────────────────────────────────────────────────────── │ +│ Total simTrajectoryReward = 13 │ +└────────────────────────────────────────────────────────────────────────┘ + ↓ +Backpropagation: +┌────────────────────────────────────────────────────────────────────────┐ +│ Update all ancestors with simTrajectoryReward = 13: │ +│ node.visits += 1 │ +│ node.statevalue = (old_statevalue * (visits-1) + 13) / visits │ +│ reward *= 0.9 (discount for future rewards) │ +└────────────────────────────────────────────────────────────────────────┘ +``` + +#### Why Two Value Systems? + +| Problem | Traditional MCTS | LLMMCTS Solution | +|---------|-----------------|------------------| +| **Sparse rewards** | Must explore blindly until terminal | LLM provides `progressvalue` at every node | +| **Slow learning** | Need many terminal outcomes | Dense `progressvalue` enables fast guidance | +| **Exploration cost** | Random expansion is inefficient | LLM filters to promising candidates | +| **Value uncertainty** | Hand-designed heuristics may be wrong | `statevalue` (from simulation) confirms LLM estimates | + ### Module Structure ```