Upload folder using huggingface_hub

Project.md

@@ -153,6 +153,100 @@ It uses AST inspection to detect:
 
 This is not the task grader. It is the direct-review helper.
 
+### Reward System
+
+The reward system is **dynamic and multi-component**, designed to provide meaningful feedback at every step of the agent's learning process.
+
+#### Reward Architecture
+
+The system computes rewards using **6 independent components**:
+
+1. **Progress Reward** (max +0.25)
+   - Awarded when the agent improves the score from one step to the next
+   - Formula: `min(PROGRESS_SCALE * score_delta, 0.25)`
+   - Encourages continuous improvement
+
+2. **Syntax Reward** (max +0.35)
+   - One-time bonus awarded for fixing syntax errors (first time compiling)
+   - Applied once per episode when code transitions from uncompilable to compilable
+   - Acknowledges the critical first step of making code valid
+
+3. **Test Reward** (max +0.20)
+   - Based on improvement in test pass rate
+   - Computed as: `min(TEST_PASS_REWARD_SCALE * test_improvement_fraction, 0.20)`
+   - Rewards incremental progress on passing more tests
+
+4. **Quality Reward** (max +0.15)
+   - Based on AST-detected code quality metrics
+   - Rewards improvements in code structure, readability, and best practices
+   - Uses deterministic grader feedback
+
+5. **Stagnation Penalty** (−0.10)
+   - Applied when the agent takes action but code doesn't change
+   - Encourages the agent to edit the code rather than analyze repeatedly
+   - Configurable via `STAGNATION_PENALTY` constant
+
+6. **Regression Penalty** (scale −0.20)
+   - Applied when score decreases from previous step
+   - Formula: `REGRESSION_PENALTY_SCALE * abs(score_delta)`
+   - Discourages actions that make code worse
+
+#### Reward Constants
+
+Defined at the top of `server/env.py`:
+
+```python
+SYNTAX_FIX_BONUS = 0.35          # One-time syntax reward
+TEST_PASS_REWARD_SCALE = 0.30    # Per test improvement
+QUALITY_BONUS_SCALE = 0.15       # Code quality improvement
+PROGRESS_SCALE = 0.25             # Score improvement
+COMPLETION_BONUS = 0.50           # Full correctness bonus
+INVALID_ACTION_PENALTY = 0.15     # For unsupported actions
+STAGNATION_PENALTY = 0.10         # For unchanged code
+REGRESSION_PENALTY_SCALE = 0.20   # For score decline
+TIMEOUT_PENALTY = 0.15            # For execution timeout
+```
+
+#### Final Reward Computation
+
+The final reward is:
+
+```
+total = progress + syntax + test + quality - stagnation - regression
+final_reward = clamp(total, -1.0, +1.0)
+```
+
+The result is always between −1.0 and +1.0, providing bounded, interpretable feedback.
+
+#### RewardDetails: Transparent Feedback
+
+Every reward is returned as a `RewardDetails` object with these fields:
+
+- `value`: The scalar reward for this step
+- `syntax_reward`: Contribution from syntax fixes
+- `test_reward`: Contribution from test improvements
+- `quality_bonus`: Contribution from code quality
+- `progress_delta`: Contribution from score improvement
+- `stagnation_penalty`: Penalty for unchanged code
+- `regression_penalty`: Penalty for score decline
+- `prev_score` / `curr_score`: Score before and after the action
+- `code_changed`: Whether the action modified the code
+- `reason`: Human-readable explanation of the reward
+
+This transparency is crucial for:
+- Debugging agent behavior
+- Understanding what drives reward
+- Tuning the constants
+- Training supervised models on reward components
+
+#### Why This Design Helps Agents Learn
+
+1. **Non-Constant**: Different actions produce different rewards, enabling meaningful gradient signals
+2. **Progressive**: Early bonuses (syntax) are high; later improvements are smaller, promoting efficiency
+3. **Transparent**: Detailed component breakdown helps agents understand what matters
+4. **Bounded**: Clamping to [−1, 1] prevents reward hacking and explosion
+5. **Balanced**: Positive and negative signals teach precision and recall together
+
 ### `server/code_review_environment.py`
 
 This is the environment core.

REWARD_SYSTEM_GUIDE.md

@@ -0,0 +1,206 @@
+# Reward System Implementation Guide
+
+This document shows how the reward system is implemented in code and how to use it.
+
+## Module Documentation
+
+The reward system architecture is documented at the module level:
+
+```python
+import server.env
+print(server.env.__doc__)
+```
+
+Output shows all 6 reward components and the final computation formula.
+
+## Reward Constants
+
+All reward constants are defined in `server/env.py` (lines 57-87):
+
+```python
+# Component 1: Score improvement reward
+PROGRESS_SCALE = 0.25
+
+# Component 2: Syntax/compilation fix reward
+SYNTAX_FIX_BONUS = 0.35
+
+# Component 3: Test improvement reward
+TEST_PASS_REWARD_SCALE = 0.30
+
+# Component 4: Code quality reward
+QUALITY_BONUS_SCALE = 0.15
+
+# Component 5: Stagnation penalty
+STAGNATION_PENALTY = 0.10
+
+# Component 6: Regression penalty
+REGRESSION_PENALTY_SCALE = 0.20
+
+# One-time completion bonus
+COMPLETION_BONUS = 0.50
+
+# Invalid/error penalties
+INVALID_ACTION_PENALTY = 0.15
+TIMEOUT_PENALTY = 0.15
+```
+
+To tune the reward system, edit these constants and re-test.
+
+## RewardDetails Model Documentation
+
+Located in `models.py` (lines 26-80):
+
+```python
+from models import RewardDetails
+print(RewardDetails.__doc__)
+```
+
+Shows all 15 fields and their meanings:
+- `value`: Final scalar reward [-1.0, +1.0]
+- `progress_delta`: Score improvement component
+- `syntax_reward`: Syntax fix bonus
+- `test_reward`: Test improvement bonus
+- `quality_bonus`: Code quality improvement
+- `stagnation_penalty`: Unchanged code penalty
+- `regression_penalty`: Score decline penalty
+- `reason`: Human-readable explanation
+- `prev_score`, `curr_score`: Score before/after
+- `code_changed`: Whether code was modified
+
+## Core Computation Method
+
+The main reward computation is in `_compute_reward_components()` (server/env.py, lines 507-703):
+
+```python
+def _compute_reward_components(
+    self,
+    curr_score: float,
+    prev_score: float,
+    curr_grade: TaskGrade,
+    code_changed: bool,
+    prev_grade_score: float = 0.0,
+) -> dict:
+    """Compute all six reward components and return combined result."""
+```
+
+### What It Does
+
+1. **Initializes** empty component dict
+2. **Computes each component**:
+   - Progress: Score improvement scaled by PROGRESS_SCALE
+   - Syntax: One-time bonus if first compile
+   - Test: Test pass rate improvement scaled by TEST_PASS_REWARD_SCALE
+   - Quality: Code quality improvement scaled by QUALITY_BONUS_SCALE
+   - Stagnation: Penalty if code unchanged
+   - Regression: Penalty if score decreased
+3. **Combines**: Sums positives, subtracts negatives
+4. **Clamps**: Bounds result to [-1.0, +1.0]
+
+### Key Design Decisions
+
+- **Monotonic tracking**: Best test rate and quality in episode are tracked
+- **One-time bonuses**: Syntax reward awarded once per episode
+- **Scale capping**: Each component has a maximum (e.g., progress max +0.25)
+- **Timeout handling**: Special penalty instead of score-based
+- **Clamping**: Final reward bounded for numerical stability
+
+## Debug Logging
+
+When `verbose=True`, the environment prints detailed debug output via `_log_debug_step()`:
+
+```python
+env = PythonCodeReviewEnvironment(verbose=True)
+obs = env.reset()
+obs = env.step(action)
+```
+
+Output format:
+```
+Step  1 | Score: 0.698 | Delta: +0.698 | Reward: +0.4239 | Changed: False
+         | Progress=+0.174 | Quality=+0.149 | Stagnation=+0.100
+         | Reason: Syntax error detected: '(' was never closed
+```
+
+Shows:
+- Step number
+- Current score and delta from previous
+- Final reward value
+- Whether code changed
+- Non-zero components only
+- Human-readable reason
+
+## Example: Full Episode with Rewards
+
+```python
+from server.env import PythonCodeReviewEnvironment
+from models import PythonCodeReviewAction
+
+env = PythonCodeReviewEnvironment(verbose=True)
+obs = env.reset(task_id='syntax-fix-easy')
+
+# Step 1: Analyze (no code change)
+action = PythonCodeReviewAction(action_type='analyze_code')
+obs = env.step(action)
+print(f"Reward 1: {obs.reward_details.value:.4f}")
+
+# Step 2: Edit with fix
+code = 'x = 1; y = 2; print(x + y)'
+action = PythonCodeReviewAction(action_type='edit_code', code=code)
+obs = env.step(action)
+print(f"Reward 2: {obs.reward_details.value:.4f}")
+
+# Step 3: Submit
+action = PythonCodeReviewAction(action_type='submit_solution')
+obs = env.step(action)
+print(f"Final Reward: {obs.reward_details.value:.4f}")
+```
+
+## Interpreting Rewards
+
+### Positive Rewards (+0 to +1.0)
+- **+0.5 - +1.0**: Major progress (syntax fix, many tests passing)
+- **+0.2 - +0.5**: Good progress (score improvement, test gains)
+- **+0.0 - +0.2**: Small progress (quality improvement, minor gains)
+
+### Negative Rewards (−1.0 to −0)
+- **−0.1 - 0**: Stagnation (analyzed without changing code)
+- **−0.2 - −0.1**: Slight regression (small score drop)
+- **−0.5 - −0.2**: Major regression (significant score drop)
+- **−1.0 - −0.5**: Invalid action or timeout
+
+## Tuning the Reward System
+
+### For Faster Early Learning
+↑ Increase `SYNTAX_FIX_BONUS` and `COMPLETION_BONUS`
+
+### To Encourage Editing Over Analysis
+↑ Increase `STAGNATION_PENALTY`
+
+### To Reward Test Improvements More
+↑ Increase `TEST_PASS_REWARD_SCALE`
+
+### To Penalize Mistakes More
+↑ Increase `REGRESSION_PENALTY_SCALE`
+
+### To Balance All Components
+Adjust the Scale constants (all in range 0.15-0.35 for stability)
+
+## Accessing Documentation Programmatically
+
+```python
+from server.env import PythonCodeReviewEnvironment
+from models import RewardDetails
+import server.env
+
+# Module-level architecture
+print(server.env.__doc__)
+
+# RewardDetails fields
+print(RewardDetails.__doc__)
+
+# One method
+env = PythonCodeReviewEnvironment()
+help(env._compute_reward_components)
+```
+
+All major functions and classes have comprehensive docstrings.

models.py

@@ -24,24 +24,61 @@ class HistoryEntry(BaseModel):
 
 
 class RewardDetails(BaseModel):
-    """Detailed reward breakdown for transparency."""
-
-    value: float = Field(..., description="Net scalar reward for this step")
-    syntax_reward: float = Field(default=0.0, description="Bonus for fixing syntax")
-    test_reward: float = Field(default=0.0, description="Reward from passing tests")
-    quality_bonus: float = Field(default=0.0, description="Bonus for code quality improvements")
-    correctness_bonus: float = Field(default=0.0, description="Bonus for full correctness")
-    progress_delta: float = Field(default=0.0, description="Reward from score improvement")
-    stagnation_penalty: float = Field(default=0.0, description="Penalty for code not changing")
-    regression_penalty: float = Field(default=0.0, description="Penalty for score decline")
-    invalid_action_penalty: float = Field(default=0.0, description="Penalty for invalid actions")
-    timeout_penalty: float = Field(default=0.0, description="Penalty for timeouts")
-    reason: str = Field(..., description="Explanation of reward")
+    """Detailed reward breakdown for transparent agent feedback.
     
-    # Debug info
+    The reward system is dynamic and multi-component, with 6 independent sources:
+    
+    1. Progress Reward (max +0.25)
+       - Awarded for score improvement from previous step
+       - Formula: min(PROGRESS_SCALE * score_delta, 0.25)
+       - Encourages continuous improvement
+    
+    2. Syntax Reward (max +0.35)
+       - One-time bonus for fixing syntax errors (first compile)
+       - Applied when code transitions from uncompilable to compilable
+       - Acknowledges the critical first step of valid code
+    
+    3. Test Reward (max +0.20)
+       - Based on improvement in test pass rate
+       - Formula: min(TEST_PASS_REWARD_SCALE * test_improvement, 0.20)
+       - Rewards incremental test progress
+    
+    4. Quality Reward (max +0.15)
+       - Based on AST-detected code quality metrics
+       - Rewards improvements in structure, readability, best practices
+       - Uses deterministic grader feedback
+    
+    5. Stagnation Penalty (−0.10)
+       - Applied when agent acts but code doesn't change
+       - Encourages editing rather than repeated analysis
+       - Configurable via STAGNATION_PENALTY constant
+    
+    6. Regression Penalty (scale −0.20)
+       - Applied when score decreases from previous step
+       - Formula: REGRESSION_PENALTY_SCALE * abs(score_delta)
+       - Discourages actions that make code worse
+    
+    Final Reward: clamp(progress + syntax + test + quality - stagnation - regression, -1.0, +1.0)
+    
+    The result is always bounded in [-1.0, +1.0], providing interpretable feedback for learning.
+    """
+
+    value: float = Field(..., description="Net scalar reward for this step (bounded in [-1.0, +1.0])")
+    syntax_reward: float = Field(default=0.0, description="Bonus for fixing syntax errors (max +0.35)")
+    test_reward: float = Field(default=0.0, description="Reward from test improvements (max +0.20)")
+    quality_bonus: float = Field(default=0.0, description="Bonus for code quality improvements (max +0.15)")
+    correctness_bonus: float = Field(default=0.0, description="Bonus for full correctness (max +0.50)")
+    progress_delta: float = Field(default=0.0, description="Reward from score improvement (max +0.25)")
+    stagnation_penalty: float = Field(default=0.0, description="Penalty for unchanged code (−0.10)")
+    regression_penalty: float = Field(default=0.0, description="Penalty for score decline (scale −0.20)")
+    invalid_action_penalty: float = Field(default=0.0, description="Penalty for invalid actions (−0.15)")
+    timeout_penalty: float = Field(default=0.0, description="Penalty for execution timeout (−0.15)")
+    reason: str = Field(..., description="Human-readable explanation of the reward")
+    
+    # Debug information for transparency
     prev_score: float = Field(default=0.0, description="Score before this step")
     curr_score: float = Field(default=0.0, description="Score after this step")
-    code_changed: bool = Field(default=False, description="Whether code was modified")
+    code_changed: bool = Field(default=False, description="Whether the action modified the code")
 
 
 class PythonCodeReviewAction(Action):

server/env.py

@@ -1,4 +1,44 @@
-"""Core OpenEnv environment for Python code review and repair tasks."""
+"""Core OpenEnv environment for Python code review and repair tasks.
+
+REWARD SYSTEM ARCHITECTURE
+==========================
+
+The environment implements a dynamic, multi-component reward system to provide
+meaningful feedback at every step of agent learning. 
+
+Six independent reward components are computed and combined:
+
+1. PROGRESS REWARD (max +0.25)
+   - Awarded for score improvement: min(PROGRESS_SCALE * score_delta, 0.25)
+   - Encourages continuous improvement on the task
+   
+2. SYNTAX REWARD (max +0.35)
+   - One-time bonus when code first becomes compilable
+   - Acknowledges the critical step of creating valid code
+   
+3. TEST REWARD (max +0.20)
+   - Based on test pass rate improvement
+   - Formula: min(TEST_PASS_REWARD_SCALE * test_improvement, 0.20)
+   
+4. QUALITY REWARD (max +0.15)
+   - Based on AST-detected code quality improvements
+   - Rewards better structure, readability, best practices
+   
+5. STAGNATION PENALTY (−0.10)
+   - Applied when agent acts but code doesn't change
+   - Encourages editing rather than repeated analysis
+   
+6. REGRESSION PENALTY (scale −0.20)
+   - Applied when score declines: REGRESSION_PENALTY_SCALE * abs(score_delta)
+   - Discourages actions that make code worse
+
+FINAL REWARD
+Final reward = clamp(progress + syntax + test + quality - stagnation - regression, -1.0, +1.0)
+
+Always bounded in [-1.0, +1.0] for interpretability and learning stability.
+
+See RewardDetails in models.py for all fields returned with each reward.
+"""
 
 from __future__ import annotations
 
@@ -21,22 +61,55 @@ from models import (
 from tasks import TaskSpec, get_task, list_task_descriptors, list_task_summaries, task_ids
 
 
-# Reward shaping constants (balanced for meaningful variation)
-SYNTAX_FIX_BONUS = 0.35  # One-time for fixing syntax
-TEST_PASS_REWARD_SCALE = 0.30  # Per test improvement
-QUALITY_BONUS_SCALE = 0.15  # Code quality improvement
-PROGRESS_SCALE = 0.25  # Score improvement
-COMPLETION_BONUS = 0.50  # Full correctness (one-time)
+# ============================================================================
+# REWARD SHAPING CONSTANTS
+# ============================================================================
+# These constants control the reward magnitude for each component.
+# Tuning these values changes agent learning incentives.
+
+# Component 1: Score improvement reward
+PROGRESS_SCALE = 0.25
+"""Scale for progress rewards. Higher = more reward for score improvement."""
+
+# Component 2: Syntax/compilation fix reward
+SYNTAX_FIX_BONUS = 0.35
+"""One-time bonus for first time code compiles."""
+
+# Component 3: Test improvement reward
+TEST_PASS_REWARD_SCALE = 0.30
+"""Scale for test pass rate rewards."""
+
+# Component 4: Code quality reward
+QUALITY_BONUS_SCALE = 0.15
+"""Scale for code quality improvements (AST-based)."""
+
+# Component 5: Stagnation penalty
+STAGNATION_PENALTY = 0.10
+"""Penalty when action is taken but code unchanged."""
+
+# Component 6: Regression penalty
+REGRESSION_PENALTY_SCALE = 0.20
+"""Scale for penalties when score declines."""
+
+# One-time completion bonus
+COMPLETION_BONUS = 0.50
+"""Bonus for fully correct solution."""
+
+# Invalid/error penalties
 INVALID_ACTION_PENALTY = 0.15
-STAGNATION_PENALTY = 0.10  # If code unchanged but action taken
-REGRESSION_PENALTY_SCALE = 0.20  # Per 0.1 score decline
+"""Penalty for unsupported action types."""
+
 TIMEOUT_PENALTY = 0.15
+"""Penalty for execution timeout."""
 
 
 class PythonCodeReviewEnvironment(
     Environment[PythonCodeReviewAction, PythonCodeReviewObservation, PythonCodeReviewState]
 ):
-    """Production-style environment for reviewing and fixing Python code."""
+    """Production-style environment for reviewing and fixing Python code.
+    
+    Implements OpenEnv compatibility and dynamic multi-component reward system.
+    """
 
     SUPPORTS_CONCURRENT_SESSIONS = True
 
@@ -433,7 +506,67 @@ class PythonCodeReviewEnvironment(
         code_changed: bool,
         prev_grade_score: float = 0.0,
     ) -> dict:
-        """Compute all reward components and return as dict."""
+        """Compute all six reward components and return combined result.
+        
+        This method is the core of the reward system. It evaluates agent progress
+        across multiple dimensions and provides transparent, component-wise feedback.
+        
+        REWARD COMPONENTS (6 total):
+        ============================
+        
+        1. PROGRESS REWARD (positive, max +0.25)
+           - Awarded when score improves from previous step
+           - Formula: min(PROGRESS_SCALE * score_delta, 0.25)
+           - Why: Encourages monotonic improvement
+        
+        2. SYNTAX REWARD (positive, max +0.35)
+           - One-time bonus when code first compiles
+           - Transition: uncompilable → compilable
+           - Why: Acknowledges critical first step of valid code
+        
+        3. TEST REWARD (positive, max +0.20)
+           - Based on improvement in test pass rate
+           - Formula: min(TEST_PASS_REWARD_SCALE * test_improvement, 0.20)
+           - Tracks best test rate seen in episode (monotonic)
+           - Why: Rewards incremental progress on passing tests
+        
+        4. QUALITY REWARD (positive, max +0.15)
+           - Based on AST-detected code quality metrics
+           - Computed by deterministic grader (syntax_score, quality_score)
+           - Tracks best quality seen in episode (monotonic)
+           - Why: Teaches code structure and maintainability
+        
+        5. STAGNATION PENALTY (negative, −0.10)
+           - Applied when action is taken but code doesn't change
+           - Exception: No penalty if code has compile errors (still debugging)
+           - Why: Encourages editing over repeated analysis
+        
+        6. REGRESSION PENALTY (negative, scale −0.20)
+           - Applied when score decreases from previous step
+           - Formula: REGRESSION_PENALTY_SCALE * abs(score_delta)
+           - Special case: Timeout returns fixed TIMEOUT_PENALTY (−0.15)
+           - Why: Discourages actions that make code worse
+        
+        FINAL REWARD:
+        =============
+        total = progress + syntax + test + quality - stagnation - regression
+        final_reward = clamp(total, -1.0, +1.0)
+        
+        The result is always bounded for interpretability and stability.
+        
+        Args:
+            curr_score: Current score after action (0.0 to 1.0)
+            prev_score: Score from previous step (0.0 to 1.0)
+            curr_grade: TaskGrade object with detailed metrics
+            code_changed: Boolean, whether the action modified code
+            prev_grade_score: Previous syntax_score for detecting first compile
+        
+        Returns:
+            dict with keys: "progress", "syntax", "test", "quality", 
+                           "stagnation", "regression", "total"
+                 All values are floats, with total clamped to [-1.0, +1.0]
+        """
+        # Initialize all components to zero
         components = {
             "progress": 0.0,
             "syntax": 0.0,
@@ -444,44 +577,83 @@ class PythonCodeReviewEnvironment(
             "total": 0.0,
         }
         
-        # 1. Progress reward: score improvement
+        # ====================================================================
+        # COMPONENT 1: PROGRESS REWARD
+        # ====================================================================
+        # Reward score improvement. Encourages continuous progress towards goal.
         score_delta = curr_score - prev_score
         if score_delta > 0:
+            # Scale improvement by constant, cap at 0.25 to prevent dominance
             components["progress"] = min(PROGRESS_SCALE * score_delta, 0.25)
         
-        # 2. Syntax reward: one-time bonus for first time compiling
+        # ====================================================================
+        # COMPONENT 2: SYNTAX REWARD
+        # ====================================================================
+        # One-time bonus for fixing syntax errors and making code compilable.
+        # This is tracked per episode with _syntax_reward_awarded flag.
         if not self._syntax_reward_awarded and curr_grade.syntax_score >= 0.99:
+            # Only award if transitioning from non-compilable to compilable
             if prev_grade_score < 0.99:
                 components["syntax"] = SYNTAX_FIX_BONUS
                 self._syntax_reward_awarded = True
         
-        # 3. Test reward: improvement in test pass rate
+        # ====================================================================
+        # COMPONENT 3: TEST REWARD
+        # ====================================================================
+        # Reward improvement in test pass rate. Track best rate seen this episode.
         if curr_grade.tests_total > 0:
+            # Fraction of visible tests currently passing
             curr_test_frac = curr_grade.tests_passed / curr_grade.tests_total
+            # Improvement since best rate seen in episode
             test_delta = curr_test_frac - self._best_visible_test_fraction
+            
             if test_delta > 0:
+                # Scale improvement, cap at 0.20 to prevent dominance
                 components["test"] = min(TEST_PASS_REWARD_SCALE * test_delta, 0.20)
-                self._best_visible_test_fraction = max(self._best_visible_test_fraction, curr_test_frac)
-        
-        # 4. Quality reward: code quality improvement
+                # Update best rate seen in this episode (monotonic)
+                self._best_visible_test_fraction = max(
+                    self._best_visible_test_fraction, curr_test_frac
+                )
+        
+        # ====================================================================
+        # COMPONENT 4: QUALITY REWARD
+        # ====================================================================
+        # Reward improvements in code quality (AST-based metrics from grader).
+        # Track best quality metric seen in this episode.
         quality_delta = curr_grade.quality_score - self._best_quality_score
         if quality_delta > 0:
+            # Scale improvement, cap at 0.15 to prevent dominance
             components["quality"] = min(QUALITY_BONUS_SCALE * quality_delta, 0.15)
-            self._best_quality_score = max(self._best_quality_score, curr_grade.quality_score)
+            # Update best quality seen in this episode (monotonic)
+            self._best_quality_score = max(
+                self._best_quality_score, curr_grade.quality_score
+            )
         
-        # 5. Stagnation penalty: code not changed (except analyze_code)
+        # ====================================================================
+        # COMPONENT 5: STAGNATION PENALTY
+        # ====================================================================
+        # Penalize when agent acts but doesn't change code (except during debugging).
+        # Exception: No penalty if code still has compile errors (debugging mode).
         if not code_changed and not (curr_grade.details.get("compile_error") == ""):
             components["stagnation"] = -STAGNATION_PENALTY
         
-        # 6. Regression penalty: score decreased
+        # ====================================================================
+        # COMPONENT 6: REGRESSION PENALTY
+        # ====================================================================
+        # Penalize when score decreases (regression).
+        # Special case: Timeout incurs fixed penalty instead of score-based.
         if score_delta < 0:
+            # Scale penalty by magnitude of regression
             components["regression"] = REGRESSION_PENALTY_SCALE * abs(score_delta)
         
-        # 7. Timeout penalty
+        # Timeout gets special fixed penalty
         if curr_grade.timed_out:
             components["regression"] = -TIMEOUT_PENALTY
         
-        # Compute total and clamp to [-1.0, 1.0]
+        # ====================================================================
+        # FINAL REWARD COMPUTATION
+        # ====================================================================
+        # Combine all components: sum positives, subtract negatives, clamp to [-1, 1]
         total = (
             components["progress"]
             + components["syntax"]
@@ -490,6 +662,8 @@ class PythonCodeReviewEnvironment(
             - components["stagnation"]
             - components["regression"]
         )
+        
+        # Clamp to [-1.0, +1.0] for bounded, interpretable rewards
         components["total"] = max(-1.0, min(1.0, round(total, 6)))
         
         return components
@@ -524,7 +698,39 @@ class PythonCodeReviewEnvironment(
         self._state.history.append(entry)
 
     def _log_debug_step(self, reward: RewardDetails) -> None:
-        """Log step details for debugging."""
+        """Log step details for debugging and agent understanding.
+        
+        When verbose=True during initialization, this method prints detailed 
+        information about each step, including:
+        
+        - Step number in episode
+        - Score before and after (and delta)
+        - Final reward value (bounded in [-1.0, +1.0])
+        - Whether code was modified
+        - Component breakdown (only non-zero components shown)
+        - Human-readable reason/explanation
+        
+        This output is designed to help:
+        - Monitor agent learning trajectory
+        - Debug why rewards are what they are
+        - Verify reward system is functioning correctly
+        - Understand what agent actions are incentivized
+        
+        Example output:
+        -----
+        Step  1 | Score: 0.698 | Delta: +0.698 | Reward: +0.4239 | Changed: False
+                 | Progress=+0.174 | Quality=+0.149 | Stagnation=+0.100
+                 | Reason: Syntax error detected: '(' was never closed
+        
+        Step  2 | Score: 1.000 | Delta: +0.302 | Reward: +0.6006 | Changed: True
+                 | Progress=+0.250 | Syntax=+0.350
+                 | Reason: Code updated.
+        -----
+        
+        Args:
+            reward: RewardDetails object containing all reward information
+        """
+        # Print main step summary line
         print(
             f"\nStep {self._state.step_count:2d} | "
             f"Score: {reward.curr_score:.3f} | "
@@ -533,7 +739,7 @@ class PythonCodeReviewEnvironment(
             f"Changed: {reward.code_changed}"
         )
         
-        # Log components if non-zero
+        # Build list of all reward components (only show non-zero)
         components = [
             ("Progress", reward.progress_delta),
             ("Syntax", reward.syntax_reward),
@@ -543,9 +749,12 @@ class PythonCodeReviewEnvironment(
             ("Regression", -reward.regression_penalty),
         ]
         
+        # Filter to only non-zero components for clarity
         non_zero = [f"{name}={val:+.3f}" for name, val in components if abs(val) > 0.001]
         if non_zero:
             print(f"         | {' | '.join(non_zero)}")
+        
+        # Print human-readable explanation
         print(f"         | Reason: {reward.reason}")