From da96c34e58510ac9e7a21dc5e2fce1acfaea63bd Mon Sep 17 00:00:00 2001 From: narawat Date: Tue, 30 Jun 2026 13:07:46 +0700 Subject: [PATCH] update readme --- workprocess.md | 966 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 917 insertions(+), 49 deletions(-) diff --git a/workprocess.md b/workprocess.md index 41fed5d..24aa6c6 100644 --- a/workprocess.md +++ b/workprocess.md @@ -22,11 +22,11 @@ The package addresses the sparse reward problem in MCTS by using LLMs to provide ``` src/ -├── type.jl # MCTSNode struct definition -├── util.jl # UCT selection utility function +├── type.jl # MCTSNode struct definition (MCTSNode type) +├── util.jl # UCT selection utility function (UCTselect) ├── mcts.jl # Core MCTS operations (select, expand, simulate, backpropagate) -├── interface.jl # High-level interface (runMCTS) -└── LLMMCTS.jl # Main package entry point +├── interface.jl # High-level interface (runMCTS, simulateThenBackpropagate) +└── LLMMCTS.jl # Main package entry point (module exports) ``` ## Data Flow @@ -48,62 +48,895 @@ MCTSNode( ) ``` -### 2. Main Workflow (interface.jl → runMCTS) +### 2. MCTS Search Process: Rollout and Exploration + +LLMMCTS explores the solution space through iterative MCTS cycles, each consisting of four phases: **Selection**, **Expansion**, **Simulation**, and **Backpropagation**. + +#### The Four Phases of MCTS ``` -runMCTS() -├── Initialize root node -└── For each iteration (1 to maxiterations): - ├── Traverse tree using UCT select (until leaf) - │ └── UCTselect() → balances exploration/exploitation - ├── Check if leaf is terminal - │ ├── Yes → backpropagate reward - │ └── No → expand and simulate - │ ├── expand() → generate child nodes (horizontal sampling) - │ └── simulateThenBackpropagate() - │ ├── simulate() → rollout to max depth - │ └── backpropagate() → update statistics up to root - └── Early stop check (if provided) +Phase 1: SELECTION +└── Start from root node +└── Use UCT (Upper Confidence Bound for Trees) to select child nodes + ├── UCT formula: Q(s,a) + c * sqrt(ln(N(s)) / N(s,a)) + ├── Exploitation term (Q): Uses progressvalue or statevalue + └── Exploration term: Prefers less-visited nodes +└── Continue until reaching a leaf node + +Phase 2: EXPANSION +└── If leaf is not terminal: + ├── Apply transition function multiple times (horizontal sampling) + ├── Generate new child nodes with LLM assistance + ├── Each child represents a potential next state + └── Dejavu detection prevents duplicate states + +Phase 3: SIMULATION (Rollout) +└── From expanded leaf node: + ├── Perform rollout up to maxSimulationDepth + ├── At each level: + │ ├── Accumulate reward from current node + │ ├── Check if terminal state reached + │ ├── Expand node to generate children + │ └── Select best child using progressvalue + reward + └── Return cumulative trajectory reward and terminal state + +Phase 4: BACKPROPAGATION +└── Update statistics along the path to root: + ├── Increment visit count for each node + ├── Update statevalue (running average of rewards) + ├── Apply discount to future rewards + └── Propagate reward upward ``` -### 3. Node Selection (util.jl → UCTselect) +#### Detailed Expansion Process (mcts.jl → expand/_expand) -Uses Upper Confidence Bound for Trees formula: +**Horizontal Sampling**: At each node, the algorithm generates multiple child nodes by applying the transition function several times. + +``` +Current Node (state S) +├── transition(S, args) → Child 1 (state S1, progressvalue=P1) +├── transition(S, args) → Child 2 (state S2, progressvalue=P2) +├── transition(S, args) → Child 3 (state S3, progressvalue=P3) +└── ... (horizontalSample times) + +Each child is evaluated by: +- progressvalue: LLM's estimate of state quality (fast, heuristic) +- reward: Immediate environment reward +``` + +**Dejavu Detection**: Before adding a child node, the algorithm checks if the `newNodeKey` already exists in the current node's children. If it does, the node is skipped to avoid duplicates. This handles semantically equivalent states. + +#### Detailed Simulation Process (mcts.jl → simulate) + +The simulation phase performs a rollout from a given node to explore the solution space vertically: + +``` +Input: Starting node, transition function, maxSimulationDepth + +simTrajectoryReward = 0 +terminalstate = nothing + +for depth in 1:maxSimulationDepth: + # Step 1: Accumulate current node's reward + simTrajectoryReward += node.reward + + # Step 2: Check if terminal state + if node.isterminal: + terminalstate = node.state + break + + # Step 3: Expand node horizontally (generate children) + expand(node, transition, transitionargs; + horizontalSample=horizontalSample, + multithread=multithread) + + # Step 4: Select best child for next step + node = selectChildNode(node) # Uses progressvalue + reward + +Return: (simTrajectoryReward, terminalstate) +``` + +**Why this matters**: The simulation phase estimates the value of a node by looking ahead `maxSimulationDepth` levels. Each rollout provides an estimate of the cumulative reward that can be obtained from the current state. + +#### Detailed Backpropagation Process (mcts.jl → backpropagate) + +After simulation, rewards are propagated back up the tree to update all visited nodes: + +``` +Input: Starting node (leaf from simulation), cumulative trajectory reward + +while !isroot(node): + # Update visit count + node.visits += 1 + + # Update statevalue (running average) + # Formula: new_avg = (old_avg * (n-1) + new_value) / n + node.statevalue = (node.statevalue * (node.visits-1) + simTrajectoryReward) / node.visits + + # Discount reward for future states + # Future rewards are less certain, so they receive lower weight + simTrajectoryReward *= discountRewardCoeff + + # Move to parent + node = node.parent +``` + +**Discounting rationale**: Rewards further from the current state are discounted because: +1. Future rewards are uncertain +2. The longer the horizon, the more opportunities for suboptimal decisions +3. This creates a more realistic value estimate + +#### Selection Phase: UCT (util.jl → UCTselect) + +The UCT formula balances exploration vs. exploitation: ``` UCT(s,a) = Q(s,a) + c * sqrt(ln(N(s)) / N(s,a)) -``` Where: -- **Exploitation term** (`Q(s,a)`) — Uses `progressvalue` for fast guidance, refined by `statevalue` -- **Exploration term** — Encourages visiting less-explored branches +- Q(s,a) = childNode.statevalue (exploitation: current estimate) +- c = explorationweight (controls exploration vs exploitation) +- N(s) = parent.visits (total visits to parent node) +- N(s,a) = childNode.visits (visits to this specific child) +``` -### 4. Expansion (mcts.jl → expand/_expand) +**Behavior**: +- If `childNode.visits = 0`: Exploration term becomes undefined, so use `progressvalue` as fallback +- If `childNode.visits` is low: Exploration term is high → encourages visiting unexplored branches +- If `childNode.visits` is high: Exploration term approaches 0 → exploits known high-value nodes -Generates child nodes by applying the transition function multiple times (horizontal sampling). +**Selection priority**: +1. **High `progressvalue` + low visits**: Explored first (fast LLM guidance) +2. **High `statevalue` + high visits**: Exploited once confirmed (accurate value) +3. **Balance**: Controlled by `explorationweight` parameter -**Dejavu detection**: Checks for semantically equivalent states to avoid duplicate nodes. +### 3. Main Workflow (interface.jl → runMCTS) -### 5. Simulation (mcts.jl → simulate) +``` +runMCTS(initialstate, transition, args) +├── Initialize root node with initialstate +│ └── root.visits = 0, root.statevalue = 0, root.children = {} +│ +├── For iteration 1 to maxiterations: +│ │ +│ ├── PHASE 1: SELECTION +│ │ ├── Start at root node +│ │ ├── While not leaf node: +│ │ │ ├── Apply UCTselect() with explorationweight +│ │ │ │ ├── UCT = statevalue + w * sqrt(ln(parent_visits) / child_visits) +│ │ │ │ └── Select child with highest UCT score +│ │ │ └── node = selected child +│ │ └── node is now a leaf node +│ │ +│ ├── PHASE 2: TERMINAL CHECK +│ │ ├── If node.isterminal == true: +│ │ │ ├── If node.state[:reward] >= 8: +│ │ │ │ └── Store in highValueState channel +│ │ │ └── backpropagate(node, node.reward) +│ │ │ └── Update all ancestors up to root +│ │ └── Else (non-terminal leaf): +│ │ └── Continue to EXPANSION +│ │ +│ ├── PHASE 3: EXPANSION +│ │ ├── expand(node, transition, transitionargs; +│ │ │ horizontalSample=horizontalSampleExpansionPhase) +│ │ │ +│ │ ├── For each child generated: +│ │ │ ├── newNode = transition(current_state, args) +│ │ │ ├── newNodeKey = result[:newNodeKey] +│ │ │ ├── newstate = result[:newstate] +│ │ │ ├── progressvalue = result[:progressvalue] +│ │ │ └── Dejavu check: skip if newNodeKey already exists +│ │ │ +│ │ └── Each child gets initialized with: +│ │ ├── visits = 0 +│ │ ├── statevalue = 0 (no simulations yet) +│ │ ├── progressvalue = LLM estimate +│ │ ├── reward = newstate[:reward] +│ │ └── parent = current node +│ │ +│ ├── PHASE 4: SIMULATION + BACKPROPAGATION +│ │ ├── If multithread == true: +│ │ │ └── Spawn parallel simulateThenBackpropagate() for each child +│ │ │ +│ │ └── For each leafNode in node.children: +│ │ ├── simulateThenBackpropagate(leafNode, ...) +│ │ │ ├── simulate(leafNode, ...): +│ │ │ │ └── Rollout up to maxSimulationDepth +│ │ │ │ └── Returns (simTrajectoryReward, terminalstate) +│ │ │ │ +│ │ │ ├── If terminalstate[:reward] >= 8: +│ │ │ │ └── Store in highValueState channel +│ │ │ │ +│ │ │ └── backpropagate(leafNode, simTrajectoryReward): +│ │ │ └── Update visits and statevalue for all ancestors +│ │ │ +│ │ └── If saveSimulatedNode == false: +│ │ └── Clear children (free memory for next iteration) +│ │ +│ └── PHASE 5: EARLY STOP CHECK +│ ├── If earlystop(node.state) == true: +│ │ └── Break out of iteration loop +│ └── Continue to next iteration +│ +└── After all iterations, select best result: + ├── bestNextState = selectBestNextNode(root) + │ └── Uses statevalue/visits or progressvalue + reward + │ + ├── bestTerminalState = selectBestTrajectoryNode(root) + │ └── Follows optimal trajectory to leaf + │ + └── highValueStateList = collect from highValueState channel +``` -Performs rollout from a node up to `maxSimulationDepth`: -1. Accumulate current node's reward -2. Check for terminal state -3. Expand node (generate children) -4. Select best child (using `progressvalue + reward`) -5. Repeat until max depth reached +### 7. Selection Phase Details (util.jl → UCTselect) -Returns cumulative trajectory reward and terminal state (if any). +**Purpose**: Select the best child node using UCT formula that balances exploration and exploitation. + +**Process**: + +``` +UCTselect(node, w) + +Input: Parent node, exploration weight w +Output: Child node with highest UCT score + +maxUCT = -Inf +selectedNode = nothing + +for each childNode in node.children: + ┌── Calculate UCT value + │ └── if childNode.visits != 0: + │ ├── weightedterm = w * sqrt(ln(node.visits) / childNode.visits) + │ ├── UCTvalue = childNode.statevalue + weightedterm + │ └── Exploration term encourages low-visited children + │ + │ else: # childNode.visits == 0 + │ ├── UCTvalue = childNode.progressvalue + │ └── No exploration term (division by zero) + │ + ├── Compare with current max + │ └── if UCTvalue > maxUCT: + │ ├── maxUCT = UCTvalue + │ └── selectedNode = childNode + +return selectedNode +``` + +**Behavior analysis**: + +| Scenario | UCT Value | Behavior | +|----------|-----------|----------| +| Child never visited (`visits=0`) | `progressvalue` | Strong exploration → will be tried | +| Child visited often, high reward | High `statevalue`, low exploration | Exploitation dominates | +| Child visited often, low reward | Low `statevalue`, low exploration | Exploitation avoids this node | +| Child visited rarely, moderate reward | Moderate `statevalue`, high exploration | May explore further | + +### 8. Expansion Phase Details (mcts.jl → expand/_expand) + +**Purpose**: Generate new child nodes by applying the transition function multiple times. + +**Process**: + +``` +expand(node, transition, transitionargs; + horizontalSample=3, multithread=false) + +Input: Node to expand, transition function, arguments +Output: node.children populated with child nodes + +if multithread == true: + └── @sync for i in 1 to horizontalSample: + └── @spawn _expand(node, transition, transitionargs) + └── Run expansion in parallel threads +else: + └── for i in 1 to horizontalSample: + └── _expand(node, transition, transitionargs) + └── Sequential expansion + +_expand(node, transition, transitionargs) + +Input: Single node to expand +Output: One child node added to node.children + +result = transition(node.state, transitionargs) + └── LLM generates next state: + └── response = llm_call(state[:thoughtHistory], args.prompt) + └── Parse response into new state structure + +newNodeKey = result[:newNodeKey] +newstate = result[:newstate] +progressvalue = result[:progressvalue] + +if newNodeKey ∉ keys(node.children): + ┌── Create new MCTSNode + │ └── newNode = MCTSNode( + │ ├── nodekey = newNodeKey + │ ├── state = newstate + │ ├── visits = 0 + │ ├── progressvalue = progressvalue + │ ├── statevalue = 0 (no simulations yet) + │ ├── reward = newstate[:reward] + │ ├── isterminal = newstate[:isterminal] + │ ├── parent = node + │ ├── children = {} + │ └── etc = {} + │ + └── node.children[newNodeKey] = newNode +``` + +**Dejavu detection**: Before adding a child, check if `newNodeKey` already exists in `node.children`. If so, skip to avoid duplicate states. + +**Example**: Expanding a node with `horizontalSample=3` +``` +Current Node: "Math problem: Solve x^2 = 16" + └── Attempt 1 (LLM): + └── Thought: "Take square root of both sides" + └── Action: "x = sqrt(16)" + └── New state: "Solved: x = 4" + └── newNodeKey = "abc-123" + └── Added to children: "abc-123" → Node + + └── Attempt 2 (LLM): + └── Thought: "Consider negative root" + └── Action: "x = -sqrt(16)" + └── New state: "Solved: x = -4" + └── newNodeKey = "def-456" + └── Added to children: "def-456" → Node + + └── Attempt 3 (LLM): + └── Thought: "Check both solutions" + └── Action: "Verify x=4 and x=-4" + └── New state: "Verified: x=4, x=-4" + └── newNodeKey = "ghi-789" + └── Added to children: "ghi-789" → Node + +Result: Node has 3 children with different solution approaches +``` + +### 9. Backpropagation Phase Details (mcts.jl → backpropagate) + +**Purpose**: Update the statistics of all nodes along the simulation path with the observed reward. + +**Process**: + +``` +backpropagate(node, simTrajectoryReward; + discountRewardCoeff=0.9) + +Input: Starting node (leaf from simulation), cumulative reward +Output: Updates visits and statevalue for all ancestors + +while !isroot(node): + ┌── Update visit count + │ └── node.visits += 1 + │ + ├── Update statevalue (running average formula) + │ │ + │ ├── Current average: node.statevalue + │ ├── Current count: node.visits - 1 + │ ├── New value to add: simTrajectoryReward + │ │ + │ └── Formula: new_avg = (old_avg * (n-1) + new_value) / n + │ node.statevalue = (node.statevalue * (node.visits-1) + simTrajectoryReward) / node.visits + │ + ├── Apply discount to future reward + │ └── simTrajectoryReward *= discountRewardCoeff + │ └── Reward gets 10% smaller for each level up + │ + └── Move to parent + └── node = node.parent + +# After loop, root node still needs update +# (root node's parent is nothing, so loop stops but root is still updated) +``` + +**Example**: Backpropagating reward=20 with discount=0.9 +``` +Depth 3 (leaf): Node A2b2 + └── visits = 1, statevalue = 20/1 = 20.0 + └── reward for parent = 20 * 0.9 = 18.0 + +Depth 2: Node A2b + └── visits = 1, statevalue = (0*0 + 18)/1 = 18.0 + └── reward for parent = 18 * 0.9 = 16.2 + +Depth 1: Node A2 + └── visits = 1, statevalue = (0*0 + 16.2)/1 = 16.2 + └── reward for parent = 16.2 * 0.9 = 14.58 + +Depth 0 (root): Node A + └── visits = 1, statevalue = (0*0 + 14.58)/1 = 14.58 +``` + +**Why discounting matters**: Future rewards are discounted because: +1. **Uncertainty**: The longer the horizon, the more uncertain the outcome +2. **Temporal credit assignment**: Immediate rewards should have higher value +3. **Realistic evaluation**: A reward 10 steps away is worth less than immediate reward + +### 10. Solution Space Exploration Strategy + +LLMMCTS explores the solution space using a combination of **guided exploration** and **systematic exploitation**: + +#### Exploration vs. Exploitation Trade-off + +``` +Iteration 1: Initial exploration (all nodes have visits=0) +├── UCTselect() uses progressvalue for all children +├── LLM provides guidance: which states seem promising? +├── High progressvalue nodes get visited first +└── Results stored in statevalue + +Iterations 2-5: Early exploitation +├── Some nodes have higher statevalue confirmed by simulations +├── UCT balances: +│ ├── Exploitation: High statevalue nodes +│ └── Exploration: Nodes with low visits (high UCT exploration term) +└── Nodes with high progressvalue + high statevalue dominate + +Iterations 6+: Refinement +├── Well-visited nodes have reliable statevalue estimates +├── Poor nodes (low statevalue) are explored less +├── Still explore unvisited branches (UCT exploration term) +└── Tree converges to optimal trajectory +``` + +#### Horizontal vs. Vertical Exploration + +**Horizontal exploration** (expansion): +- **Parameter**: `horizontalSample` +- **Purpose**: Generate multiple candidate next states from current state +- **Method**: Apply transition function multiple times +- **LLM role**: Each call generates a different thought/action + +**Vertical exploration** (simulation): +- **Parameter**: `maxSimulationDepth` +- **Purpose**: Evaluate the long-term value of a state +- **Method**: Rollout from current node to leaf +- **Reward accumulation**: Sum all rewards along trajectory + +#### Dejavu Detection and State Pruning + +**Dejavu detection** (mcts.jl → _expand): + +```julia +if newNodeKey ∉ keys(node.children) + newNode = MCTSNode(...) + node.children[newNodeKey] = newNode +end +``` + +**Why it matters**: +- Prevents infinite loops in cyclic state spaces +- Handles semantically equivalent states as duplicates +- Reduces tree size and memory usage +- Improves search efficiency + +**Example**: +``` +Current Node: "Problem solving step 5" +├── Attempt 1: Generates "Next step: Apply formula A" +├── Attempt 2: Generates "Next step: Apply formula A" (same as attempt 1) +│ └── Dejavu detected: skip duplicate +└── Attempt 3: Generates "Next step: Apply formula B" +``` + +### 11. Complete MCTS Iteration Flow + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ MCTS ITERATION 1 │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ 1. SELECTION (UCT-based traversal): │ +│ Root (visits=1) │ +│ └── UCTselect() on root (has no children yet, so uses progressvalue) │ +│ └── Select child with highest progressvalue │ +│ └── Continue until reaching leaf node │ +│ │ +│ 2. EXPANSION (horizontal sampling): │ +│ Leaf node │ +│ └── expand(horizontalSample=3) │ +│ └── Generate 3 child nodes via LLM transition │ +│ └── Each child gets: progressvalue, reward, parent=leaf │ +│ │ +│ 3. SIMULATION (vertical rollout): │ +│ Each child node │ +│ └── simulate(maxSimulationDepth=3) │ +│ └── Rollout 3 levels deep, accumulating rewards │ +│ └── Return (simTrajectoryReward, terminalstate) │ +│ │ +│ 4. BACKPROPAGATION (update statistics): │ +│ simTrajectoryReward │ +│ └── backpropagate() up to root │ +│ └── Update visits and statevalue for all ancestors │ +│ └── Apply discount to future rewards │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ MCTS ITERATION 2 │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ 1. SELECTION: │ +│ Root (visits=2) │ +│ └── UCTselect() now considers: │ +│ ├── statevalue (from iteration 1) │ +│ └── exploration term (low visits on unexpanded branches) │ +│ └── May select different path than iteration 1 │ +│ │ +│ 2. EXPANSION: │ +│ └── May expand different node or same node │ +│ └── New children added to node.children │ +│ │ +│ 3. SIMULATION: │ +│ └── Different rollout trajectory │ +│ └── New reward estimate added to statistics │ +│ │ +│ 4. BACKPROPAGATION: │ +│ └── Statistics updated with new information │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ MCTS ITERATION 3 to N │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ Pattern repeats, with increasingly informed selection: │ +│ │ +│ • Nodes with high statevalue (confirmed by many simulations) │ +│ → Exploited (selected frequently) │ +│ │ +│ • Nodes with low visits but promising progressvalue │ +│ → Explored (UCT exploration term encourages tries) │ +│ │ +│ • Tree grows: more branches explored, more statistics accumulated │ +│ │ +│ • Best trajectory emerges from accumulated statistics │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +``` +simulate(node, transition, transitionargs; + maxSimulationDepth=3, horizontalSample=3, multithread=false) + +Input: Starting node at depth 0 +Output: (simTrajectoryReward, terminalstate) + +simTrajectoryReward = 0.0 +terminalstate = nothing + +for depth in 1 to maxSimulationDepth: + ┌── Step 1: Accumulate reward + │ └── simTrajectoryReward += node.reward + │ + ├── Step 2: Check for terminal state + │ └── if node.isterminal: + │ └── terminalstate = deepcopy(node.state) + │ break + │ + ├── Step 3: Expand node (horizontal sampling) + │ └── expand(node, transition, transitionargs; + │ horizontalSample=horizontalSample) + │ └── For i in 1 to horizontalSample: + │ └── _expand(node, transition, transitionargs) + │ ├── result = transition(node.state, args) + │ ├── newNodeKey = result[:newNodeKey] + │ ├── newstate = result[:newstate] + │ ├── progressvalue = result[:progressvalue] + │ └── if newNodeKey not in node.children: + │ └── Create new MCTSNode with: + │ ├── state = newstate + │ ├── progressvalue = progressvalue + │ ├── reward = newstate[:reward] + │ ├── parent = node + │ └── children = {} + │ + └── Step 4: Select best child for next iteration + └── node = selectChildNode(node) + └── Uses: progressvalue + reward (no UCT here) + +return (simTrajectoryReward, terminalstate) +``` + +**Example**: With `maxSimulationDepth=3` and `horizontalSample=3`: +``` +Depth 0: Node A (reward=2) + └── Expand 3 children: A1, A2, A3 + └── Select A2 (highest progressvalue + reward) + +Depth 1: Node A2 (reward=3) + └── Expand 3 children: A2a, A2b, A2c + └── Select A2b (highest progressvalue + reward) + +Depth 2: Node A2b (reward=5) + └── Expand 3 children: A2b1, A2b2, A2b3 + └── Select A2b2 (highest progressvalue + reward) + +Depth 3: Node A2b2 (reward=10, isterminal=true) + └── Terminal state reached! + +Return: (simTrajectoryReward=20, terminalstate=A2b2.state) + = (2 + 3 + 5 + 10, ...) +``` ### 6. Backpropagation (mcts.jl → backpropagate) -Updates statistics along the path from leaf to root: +**Purpose**: Update the statistics of all nodes along the simulation path with the observed reward. + +**Process**: ``` -node.visits += 1 -node.statevalue = (node.statevalue * (node.visits-1) + simTrajectoryReward) / node.visits -simTrajectoryReward *= discountRewardCoeff # Discount future rewards -move to parent node +backpropagate(node, simTrajectoryReward; + discountRewardCoeff=0.9) + +Input: Starting node (leaf from simulation), cumulative reward +Output: Updates visits and statevalue for all ancestors + +while !isroot(node): + ┌── Update visit count + │ └── node.visits += 1 + │ + ├── Update statevalue (running average formula) + │ │ + │ ├── Current average: node.statevalue + │ ├── Current count: node.visits - 1 + │ ├── New value to add: simTrajectoryReward + │ │ + │ └── Formula: new_avg = (old_avg * (n-1) + new_value) / n + │ node.statevalue = (node.statevalue * (node.visits-1) + simTrajectoryReward) / node.visits + │ + ├── Apply discount to future reward + │ └── simTrajectoryReward *= discountRewardCoeff + │ └── Reward gets 10% smaller for each level up + │ + └── Move to parent + └── node = node.parent + +# After loop, root node still needs update +# (root node's parent is nothing, so loop stops but root is still updated) +``` + +**Example**: Backpropagating reward=20 with discount=0.9 +``` +Depth 3 (leaf): Node A2b2 + └── visits = 1, statevalue = 20/1 = 20.0 + └── reward for parent = 20 * 0.9 = 18.0 + +Depth 2: Node A2b + └── visits = 1, statevalue = (0*0 + 18)/1 = 18.0 + └── reward for parent = 18 * 0.9 = 16.2 + +Depth 1: Node A2 + └── visits = 1, statevalue = (0*0 + 16.2)/1 = 16.2 + └── reward for parent = 16.2 * 0.9 = 14.58 + +Depth 0 (root): Node A + └── visits = 1, statevalue = (0*0 + 14.58)/1 = 14.58 +``` + +**Why discounting matters**: Future rewards are discounted because: +1. **Uncertainty**: The longer the horizon, the more uncertain the outcome +2. **Temporal credit assignment**: Immediate rewards should have higher value +3. **Realistic evaluation**: A reward 10 steps away is worth less than immediate reward + +### 7. Solution Space Exploration Strategy + +LLMMCTS explores the solution space using a combination of **guided exploration** and **systematic exploitation**: + +#### Exploration vs. Exploitation Trade-off + +``` +Iteration 1: Initial exploration (all nodes have visits=0) +├── UCTselect() uses progressvalue for all children +├── LLM provides guidance: which states seem promising? +├── High progressvalue nodes get visited first +└── Results stored in statevalue + +Iterations 2-5: Early exploitation +├── Some nodes have higher statevalue confirmed by simulations +├── UCT balances: +│ ├── Exploitation: High statevalue nodes +│ └── Exploration: Nodes with low visits (high UCT exploration term) +└── Nodes with high progressvalue + high statevalue dominate + +Iterations 6+: Refinement +├── Well-visited nodes have reliable statevalue estimates +├── Poor nodes (low statevalue) are explored less +├── Still explore unvisited branches (UCT exploration term) +└── Tree converges to optimal trajectory +``` + +#### Horizontal vs. Vertical Exploration + +**Horizontal exploration** (expansion): +- **Parameter**: `horizontalSample` +- **Purpose**: Generate multiple candidate next states from current state +- **Method**: Apply transition function multiple times +- **LLM role**: Each call generates a different thought/action + +**Vertical exploration** (simulation): +- **Parameter**: `maxSimulationDepth` +- **Purpose**: Evaluate the long-term value of a state +- **Method**: Rollout from current node to leaf +- **Reward accumulation**: Sum all rewards along trajectory + +#### Dejavu Detection and State Pruning + +**Dejavu detection** (mcts.jl → _expand): + +```julia +if newNodeKey ∉ keys(node.children) + newNode = MCTSNode(...) + node.children[newNodeKey] = newNode +end +``` + +**Why it matters**: +- Prevents infinite loops in cyclic state spaces +- Handles semantically equivalent states as duplicates +- Reduces tree size and memory usage +- Improves search efficiency + +**Example**: +``` +Current Node: "Problem solving step 5" +├── Attempt 1: Generates "Next step: Apply formula A" +├── Attempt 2: Generates "Next step: Apply formula A" (same as attempt 1) +│ └── Dejavu detected: skip duplicate +└── Attempt 3: Generates "Next step: Apply formula B" +``` + +### 8. Selection Phase Details (util.jl → UCTselect) + +### 9. High-Level Interface (interface.jl → runMCTS) + +**Purpose**: Execute the complete MCTS search algorithm with all phases. + +**Process**: + +``` +runMCTS(initialstate, transition, transitionargs; + horizontalSampleExpansionPhase=3, + horizontalSampleSimulationPhase=3, + maxSimulationDepth=3, + maxiterations=10, + explorationweight=1.0, + earlystop=nothing, + saveSimulatedNode=false, + multithread=false) + +Output: NamedTuple with: + ├── root: Complete MCTS tree + ├── bestNextState: Best immediate next state + ├── bestTerminalState: Best final state along optimal trajectory + └── highValueStateList: List of high-value terminal states (reward >= 8) + +Step-by-step execution: + └── root = MCTSNode("root", initialstate, 0, 0, 0, 0, false, nothing, {}, {}) + └── highValueState = Channel{Any}(100) + + for iteration = 1 to maxiterations: + ┌── node = root + │ └── node.visits += 1 # Increment root visits + │ + │ ┌── PHASE 1: SELECTION (until leaf) + │ │ while !isleaf(node): + │ │ └── node = UCTselect(node, explorationweight) + │ │ + │ ├── PHASE 2: TERMINAL CHECK + │ │ if node.isterminal: + │ │ ├── if node.state[:reward] >= 8: + │ │ │ └── put!(highValueState, deepcopy(node.state)) + │ │ └── backpropagate(node, node.reward) + │ │ └── Update statistics up to root + │ │ └── continue to next iteration + │ │ + │ └── PHASE 3: EXPANSION + │ ├── expand(node, transition, transitionargs; + │ │ horizontalSample=horizontalSampleExpansionPhase, + │ │ multithread=multithread) + │ │ + │ └── For each leafNode in node.children: + │ └── simulateThenBackpropagate(leafNode, ...) + │ ├── simulate() → rollout and get reward + │ ├── If terminal state with reward >= 8: + │ │ └── Store in highValueState + │ └── backpropagate() → update statistics + │ + └── PHASE 4: EARLY STOP CHECK + └── if earlystop !== nothing && earlystop(node.state): + └── break # Exit iteration loop + + ┌── After all iterations, select best result: + │ ├── bestNextState = selectBestNextNode(root) + │ │ └── Uses statevalue/visits or progressvalue + reward + │ │ + │ ├── bestTerminalState = selectBestTrajectoryNode(root) + │ │ └── Follows optimal trajectory to leaf + │ │ + │ └── highValueStateList = collect from channel + │ └── while !isempty(highValueState): + │ push!(highValueStateList, take!(highValueState)) + │ + └── return (root=root, bestNextState=..., bestTerminalState=..., highValueStateList=...) +``` + +### 10. Complete MCTS Iteration Flow + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ MCTS ITERATION 1 │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ 1. SELECTION (UCT-based traversal): │ +│ Root (visits=1) │ +│ └── UCTselect() on root (has no children yet, so uses progressvalue) │ +│ └── Select child with highest progressvalue │ +│ └── Continue until reaching leaf node │ +│ │ +│ 2. EXPANSION (horizontal sampling): │ +│ Leaf node │ +│ └── expand(horizontalSample=3) │ +│ └── Generate 3 child nodes via LLM transition │ +│ └── Each child gets: progressvalue, reward, parent=leaf │ +│ │ +│ 3. SIMULATION (vertical rollout): │ +│ Each child node │ +│ └── simulate(maxSimulationDepth=3) │ +│ └── Rollout 3 levels deep, accumulating rewards │ +│ └── Return (simTrajectoryReward, terminalstate) │ +│ │ +│ 4. BACKPROPAGATION (update statistics): │ +│ simTrajectoryReward │ +│ └── backpropagate() up to root │ +│ └── Update visits and statevalue for all ancestors │ +│ └── Apply discount to future rewards │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ MCTS ITERATION 2 │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ 1. SELECTION: │ +│ Root (visits=2) │ +│ └── UCTselect() now considers: │ +│ ├── statevalue (from iteration 1) │ +│ └── exploration term (low visits on unexpanded branches) │ +│ └── May select different path than iteration 1 │ +│ │ +│ 2. EXPANSION: │ +│ └── May expand different node or same node │ +│ └── New children added to node.children │ +│ │ +│ 3. SIMULATION: │ +│ └── Different rollout trajectory │ +│ └── New reward estimate added to statistics │ +│ │ +│ 4. BACKPROPAGATION: │ +│ └── Statistics updated with new information │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────┐ +│ MCTS ITERATION 3 to N │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ Pattern repeats, with increasingly informed selection: │ +│ │ +│ • Nodes with high statevalue (confirmed by many simulations) │ +│ → Exploited (selected frequently) │ +│ │ +│ • Nodes with low visits but promising progressvalue │ +│ → Explored (UCT exploration term encourages tries) │ +│ │ +│ • Tree grows: more branches explored, more statistics accumulated │ +│ │ +│ • Best trajectory emerges from accumulated statistics │ +│ │ +└─────────────────────────────────────────────────────────────────────────────┘ ``` ## Key Functions @@ -126,14 +959,23 @@ move to parent node | Function | Purpose | |----------|---------| -| `runMCTS(initialstate, transition, args; kwargs...)` | Main MCTS search function | -| `simulateThenBackpropagate(node, ...)` | Run simulation and backpropagate reward | +| `runMCTS(initialstate, transition, args; kwargs...)` | Main MCTS search function that executes all four phases across multiple iterations | +| `simulateThenBackpropagate(node, ...)` | Run simulation and backpropagate reward for a single node | ### Utilities (util.jl) | Function | Purpose | |----------|---------| | `UCTselect(node, w)` | Select node using UCT score with exploration weight `w` | +| `selectBestNextNode(node)` | Select best child based on `statevalue/visits` or `progressvalue + reward` | +| `selectBestTrajectoryNode(node)` | Traverse down tree to find highest-value leaf | +| `selectChildNode(node)` | Select child with highest `progressvalue + reward` | +| `expand(node, ...)` | Generate child nodes using transition function | +| `_expand(node, ...)` | Helper to create single child node | +| `simulate(node, ...)` | Perform rollout simulation | +| `backpropagate(node, reward)` | Update statistics up to root | +| `isleaf(node)` | Check if node has no children | +| `isroot(node)` | Check if node is root (`nodekey == "root"`) | ## Configuration Parameters @@ -141,14 +983,23 @@ move to parent node | Parameter | Default | Description | |-----------|---------|-------------| -| `horizontalSampleExpansionPhase` | 3 | Children per node during expansion phase | -| `horizontalSampleSimulationPhase` | 3 | Children per node during simulation | -| `maxSimulationDepth` | 3 | Maximum depth during rollout | -| `maxiterations` | 10 | Number of MCTS iterations | -| `explorationweight` | 1.0 | UCT exploration weight (higher = more exploration) | -| `earlystop` | nothing | Optional function to check early stopping condition | -| `saveSimulatedNode` | false | Whether to keep nodes created during simulation | -| `multithread` | false | Enable parallel simulation | +| `horizontalSampleExpansionPhase` | 3 | Number of child nodes generated at each expansion (horizontal sampling) | +| `horizontalSampleSimulationPhase` | 3 | Number of child nodes generated during simulation rollout | +| `maxSimulationDepth` | 3 | Maximum depth of simulation rollout (vertical exploration) | +| `maxiterations` | 10 | Number of MCTS iterations (each iteration: selection → expansion → simulation → backpropagation) | +| `explorationweight` | 1.0 | UCT exploration weight (controls exploration vs exploitation balance) | +| `earlystop` | nothing | Optional function to check early stopping condition (takes node.state, returns bool) | +| `saveSimulatedNode` | false | Whether to keep nodes created during simulation (true = keep for analysis, false = free memory) | +| `multithread` | false | Enable parallel simulation across child nodes (true = use Julia threads) | + +### How Parameters Affect Search + +| Parameter | Low Value | High Value | Trade-off | +|-----------|-----------|------------|-----------| +| `horizontalSample` | Fast, less exploration | Slow, more exploration | More children = better coverage but more LLM calls | +| `maxSimulationDepth` | Quick estimates, short horizon | Slow, long horizon | Deeper rollouts = more accurate but expensive | +| `maxiterations` | Quick search | Thorough search | More iterations = better solution but slower | +| `explorationweight` | Greedy (exploitation) | Aggressive (exploration) | Higher = explores more novel paths | ## Usage Pattern @@ -209,6 +1060,23 @@ high_value_states = result.highValueStateList - **Heuristic quality**: Poor LLM pseudo-rewards lead to suboptimal search - **Determinism**: LLM outputs are stochastic; use temperature=0 for reproducibility +## Known Issues + +### Bug: Variable name mismatch in interface.jl:94 + +The code references `highrewardNode` but the channel is named `highValueState`. This causes a `UndefVarError` when a terminal state with reward >= 8 is encountered. + +**Current code (buggy)**: +```julia +highValueState = Channel{Any}(100) +# ... +if node.state[:reward] >= 8 + put!(highrewardNode, deepcopy(node.state)) # BUG: should be highValueState +end +``` + +**Expected behavior**: Store high-value terminal states in the `highValueState` channel. + ## Related Files - `src/type.jl`: MCTSNode struct definition