Update TODO.md: evidence-based Phase 2 overhaul with confidence ratings, literature citations, coordinated experiment sequence

TODO.md

@@ -1,8 +1,11 @@
 # NeuroGolf Solver — Roadmap
 
-> Current: v4.3 · 50 arc-gen validated · ~670 LB · Target: 3000+
+> Current: v5.0 · 50 arc-gen validated (v4 baseline) · ~670 LB · Target: 3000+
 > Philosophy: **Research → Design → Experiment → Analyze → Research** loop until confirmed score increase.
 > Rule: **NEVER claim a feature works without full arc-gen validation on representative tasks.**
+> Updated: 2026-04-26 — Phase 2 overhauled with literature-backed experiment plan.
+
+---
 
 ## Phase 1: Cheap Wins (est +400 pts → ~1100)
 
@@ -33,58 +36,184 @@
 
 ---
 
-## Phase 2: Fix Arc-Gen Survival (est +100-150 tasks → ~2000-2500)
-
-> **This is the #1 blocker.** We solve 307 locally but only 50 survive arc-gen.
-> Research (Bartlett et al., Belkin et al., arXiv:2306.13185) shows:
-> - Our patch covariance has LOW effective rank (~10-40) vs n≈600 patches
-> - This is CATASTROPHIC overfitting regime, NOT benign
-> - Ridge/LOOCV λ tuning CANNOT fix this — theory predicts failure
-
-### 2a: Skip Interpolation Threshold Kernels
-- [ ] **Remove ks=5,7,9 from conv fitting** — these are at/near double descent peak
-  - Try ks list: [1, 3, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29]
-  - Rationale: ks=5 (p=490, n≈600) is worst-case. ks=1 (p=10) is safe. ks=29 (p=8410) is overparameterized but at least past the peak.
-- [ ] **Validate**: Full 400 arc-gen. Compare arc-gen survival rate vs v4.
-  - Accept only if survival rate improves by >10% (5+ more tasks).
-
-### 2b: PCA Dimensionality Reduction Before lstsq
-- [ ] **PCA pre-processing**: project patch matrix P to top-k components (k=15-25 matching effective rank)
-  - Fit PCA on training patches, transform both P and test patches, then lstsq on reduced space
-  - Ensures p_reduced << n, avoiding interpolation regime entirely
-- [ ] **Validate**: Test on 20 tasks that currently fail arc-gen at ks=7,9.
-  - Compare: raw lstsq vs PCA+lstsq. Measure arc-gen pass rate.
-  - Accept only if >20% of previously-failing tasks now pass.
-
-### 2c: Gradient Descent with Early Stopping (Alternative to lstsq)
-- [ ] **Iterative solver**: Adam on conv weights, early stop at ~95% train accuracy (don't interpolate)
-  - Implicit ℓ₁-like regularization — theory predicts better generalization than explicit Ridge
-  - Use small model: ks=3 single-layer or ks=(3,1) two-layer
-- [ ] **Validate**: Same 20 failing tasks. Compare lstsq vs early-stopping GD.
-  - Accept only if >15% improvement in arc-gen survival.
-
-### 2d: Lasso / Sparse Regression
-- [ ] **Replace np.linalg.lstsq with sklearn.linear_model.Lasso**
-  - α tuning via cross-validation on training data
-  - Matches sparse signal structure of one-hot patches
-- [ ] **Validate**: Same 20 failing tasks. Compare lstsq vs Lasso.
-  - Accept only if >15% improvement.
-
-### 2e: PyTorch Multi-Seed with Arc-Gen Training (GPU Required)
-- [ ] **Train Conv→ReLU→Conv on train+test+arc-gen** (all available examples matching grid size)
-  - Multi-seed (0,7,42), 3000 steps, lr=0.03, early stopping on arc-gen loss
-  - ks=(3,1) or (5,1) two-layer
-  - **Ternary snap**: after training, snap weights to {-1,0,1}, re-validate on arc-gen
-- [ ] **Validate**: Run on 50 tasks. Compare arc-gen survival vs lstsq baseline.
-  - Needs GPU (T4 minimum). CPU too slow for 400×3 seeds.
-  - Accept only if >10% improvement AND total runtime <12hr Kaggle limit.
-
-### 2f: Generate More ARC-GEN Data
-- [ ] **Use ARC-GEN generator** (github.com/google/ARC-GEN) to produce 1000+ examples/task
-  - More fitting data = more constraints, but ONLY helps if we avoid interpolation regime
-  - Combine with PCA or GD — lstsq with more rows still overfits if p > n
-- [ ] **Validate**: Test on 20 tasks with 1000 vs 250 arc-gen examples.
-  - Compare arc-gen survival. Accept only if >10% improvement.
+## Phase 2: Fix Arc-Gen Survival — THE #1 BLOCKER
+
+> **Status:** 307 solved locally, only 50 survive arc-gen. ~250 tasks affected by lstsq overfitting.
+> **Root cause confirmed by literature:** catastrophic overfitting at the interpolation threshold (p ≈ n).
+> **Strategy:** Coordinated experiment sequence — each builds on the previous. Do NOT test in isolation.
+
+### The Problem (with numbers from conv.py)
+
+Current `_lstsq_conv()` runs naked `np.linalg.lstsq(P, T_oh, rcond=None)` — zero regularization.
+Kernel search is `[1, 3, 5, 7, 9, 11, ...]`, stops at **first ks that interpolates training**.
+
+| Kernel | p (features) | n (patches, 7×7 grid, 4 ex) | p/n | Regime |
+|--------|-------------|------------------------------|-----|--------|
+| ks=1 | 10 | 196 | 0.05 | ✅ Safe underparameterized |
+| ks=3 | 90 | 196 | 0.46 | ✅ Underparameterized |
+| **ks=5** | **250** | **196** | **1.27** | **❌ INTERPOLATION THRESHOLD** |
+| **ks=7** | **490** | **196** | **2.50** | **❌ PAST THRESHOLD — WORST CASE** |
+| **ks=9** | **810** | **196** | **4.13** | **⚠️ Near peak** |
+| ks=11 | 1210 | 196 | 6.17 | Overparameterized |
+| ks=29 | 8410 | 196 | 42.9 | Heavily overparameterized |
+
+The solver accepts ks=5 (which perfectly interpolates training via minimum-norm solution) and **never tries ks=11+** which might actually generalize.
+
+### Literature Backing
+
+| Paper | arxiv | Key Finding for Us |
+|-------|-------|--------------------|
+| Nakkiran et al. 2019 (NeurIPS) | `1912.02292` | Test error peaks at p≈n (interpolation threshold). This is exactly where ks=5,7 sit. Skipping these eliminates the peak entirely. |
+| Segert 2023 | `2311.11093` | Truncated SVD / PCA regression achieves flatter loss basins than Ridge. Optimal for low-rank covariance (our case: effective rank ~10-40). |
+| Zhou & Ge 2023 (NeurIPS) | `2302.00257` | L1 (Lasso) achieves near-minimax for sparse β*. L2 (Ridge) cannot. ARC one-hot patches are sparse (3-5 of 10 channels active). |
+| Liu et al. 2023 | `2302.01088` | More fitting rows help ONLY with regularization. Without it, adding rows near threshold can *hurt* (sample-wise non-monotonicity). |
+| Ali et al. 2019 | — | GD with early stopping ≡ Ridge with λ=1/(2t). Since Ridge is suboptimal here, GD early stopping is also suboptimal. |
+| Liao & Gu 2024 (CompressARC) | `2512.06104` | ARC solvers that generalize use regularized fitting (MDL/KL), not ridgeless interpolation. Direct evidence regularization is needed. |
+
+### Coordinated Experiment Sequence
+
+> Run in order. Each experiment keeps wins from previous experiments.
+> **Goal: highest-scoring valid model per task**, not first-valid.
+
+---
+
+#### Exp 0: Baseline Measurement ⬜
+> *Prerequisite for all other experiments.*
+
+- [ ] Run current v5 on all 400 tasks with full arc-gen validation
+- [ ] Record per-task: (a) solver used, (b) ks chosen, (c) arc-gen pass/fail, (d) score, (e) p/n ratio
+- [ ] Identify the ~250 tasks that fail arc-gen — classify by ks and p/n ratio
+- **Exit criteria:** Have baseline numbers to compare against
+
+---
+
+#### Exp 1: Skip ks=5,7,9 ⬜ — Confidence: **90%**
+> *1 line change per solver. Eliminates the interpolation threshold peak entirely.*
+
+**Evidence:** Nakkiran 2019 (`1912.02292`) proves test error peaks at p≈n. Our ks=5 (p=250, n≈196) is textbook worst-case. Removing these kernel sizes cannot make things worse — if a task *needs* ks=5 to solve, it was going to fail arc-gen anyway because it's in the catastrophic regime.
+
+**10% doubt:** Some tasks with large grids (21×21, 16 examples → n≈7056) have p/n < 1 even at ks=7. For those, ks=7 is safe. But the solver can't distinguish these cases without computing p/n per-task.
+
+- [ ] Change ks list in all 4 conv solvers: `[1, 3, 5, 7, 9, 11, ...]` → `[1, 3, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29]`
+  - Files: `conv.py` — `solve_conv_fixed`, `solve_conv_variable`, `solve_conv_diffshape`, `solve_conv_var_diff`
+- [ ] Run all 400 with arc-gen. Compare survival rate vs Exp 0.
+- **Accept if:** arc-gen survival improves by ≥5 tasks
+- **Expected:** +10-30 tasks
+
+---
+
+#### Exp 2: Best-of-N Model Selection ⬜ — Confidence: **85%**
+> *Structural change to solve loop. Return highest-scoring valid model, not first-valid.*
+
+**Evidence:** Pure engineering — no theoretical uncertainty. Currently `conv.py` iterates ks smallest-first and returns the first that passes validation. This means if ks=3 AND ks=13 both pass arc-gen, we might return ks=13 (tried first due to bias loop) when ks=3 scores higher (lower MACs). Score = max(1, 25 - ln(cost)), so smaller models always score higher.
+
+**15% doubt:** Runtime. Trying all 12 kernel sizes × 2 bias options × arc-gen validation = up to 24 candidates per task. May blow the 12hr Kaggle time budget. Mitigate: set tighter per-candidate timeout, parallelize validation.
+
+- [ ] Refactor `solve_conv_*` to return **list of (model, ks, cost)** candidates instead of first-valid
+- [ ] Refactor `solve_task` to collect all candidates (analytical + all conv variants), pick cheapest valid
+- [ ] Add static cost estimation to pick cheapest before saving
+- [ ] Run all 400. Compare total score vs Exp 1.
+- **Accept if:** total score improves by ≥3% on existing solved tasks
+- **Expected:** +5-15 score points on tasks already solved (better model selection)
+
+---
+
+#### Exp 3: PCA / Truncated SVD Before lstsq ⬜ — Confidence: **75%**
+> *~30 lines in `_lstsq_conv()`. Maps every ks into effective underparameterized regime.*
+
+**Evidence:** Segert 2023 (`2311.11093`) — PCA regression is provably better than Ridge for low-rank covariance. Our patch covariance has effective rank ~10-40 (few active colors in one-hot encoding). Truncating to top-k components removes the ~(p-40) pure noise dimensions that ridgeless lstsq amplifies into catastrophic overfitting.
+
+**25% doubt:** The variance threshold (99%) might be wrong for some tasks. Too aggressive truncation (k too small) kills signal. Too little (k too large) doesn't fix the problem. Need per-task adaptive k based on singular value gap.
+
+**Implementation:**
+```python
+def _lstsq_pcr(P, T_oh, var_threshold=0.99):
+    U, s, Vt = np.linalg.svd(P, full_matrices=False)
+    cumvar = np.cumsum(s**2) / np.sum(s**2)
+    k = np.searchsorted(cumvar, var_threshold) + 1
+    k = max(k, 5)  # floor: keep at least 5 components
+    P_red = U[:, :k] * s[:k]  # n × k, always k << n
+    w_red = np.linalg.lstsq(P_red, T_oh, rcond=None)[0]
+    w_full = Vt[:k].T @ w_red  # back to p-dim for ONNX weights
+    return w_full
+```
+
+- [ ] Add `_lstsq_pcr()` to `conv.py` alongside existing `_lstsq_conv()`
+- [ ] Use PCA path for all ks where p/n > 0.5 (safe margin below threshold)
+- [ ] Keep raw lstsq for ks=1,3 where p << n (PCA unnecessary, adds cost)
+- [ ] Try var_threshold in {0.95, 0.99, 0.999} — pick best per arc-gen survival
+- [ ] Run all 400. Compare survival rate vs Exp 1.
+- **Accept if:** arc-gen survival improves by ≥10 tasks vs Exp 1
+- **Expected:** +15-40 tasks (the big win — makes previously-impossible ks usable)
+
+---
+
+#### Exp 4: Increase Arc-Gen Fitting Cap ⬜ — Confidence: **60%**
+> *1 line change. Only works WITH regularization (PCA) in place.*
+
+**Evidence:** Liu et al. 2023 (`2302.01088`) — more fitting rows help only in regularized regime. With PCA, more rows = more constraints in the reduced k-dimensional space = better conditioning. Without PCA, adding rows to underdetermined lstsq doesn't fix the fundamental problem.
+
+**40% doubt:** Arc-gen examples may have different grid sizes (filtered out by `get_exs_for_fitting`). The cap increase only helps if enough same-shape arc-gen examples exist.
+
+- [ ] Change `get_exs_for_fitting()`: cap 10 → 50
+- [ ] Change `get_exs_for_fitting_variable()`: cap 20 → 100
+- [ ] Run all 400. Compare vs Exp 3.
+- **Accept if:** arc-gen survival improves by ≥3 tasks vs Exp 3
+- **Expected:** +5-15 tasks (modest but free)
+
+---
+
+#### Exp 5: Lasso (L1) for Large Kernels ⬜ — Confidence: **55%**
+> *~15 lines + sklearn dependency. Better than L2 for sparse signals, but slower and fragile.*
+
+**Evidence:** Zhou & Ge 2023 (`2302.00257`) — L1 achieves near-minimax O(σ²·s·log(d/s)/n) for sparse β*, vs Ridge's Ω(‖β*‖²). ARC one-hot patches are sparse (3-5 of 10 channels active). Segert 2023 Section A.10: Lasso competitive with PCA/Nuclear in sparse regime, wins ~40% of cases.
+
+**45% doubt:** (1) Lasso is slow — coordinate descent vs single SVD. (2) `alpha` tuning via CV with only 4-6 training examples is fragile. (3) Need `MultiTaskLassoCV` for 10-column one-hot target, not scalar `LassoCV`. (4) sklearn adds a dependency (not available on Kaggle by default — need to verify).
+
+- [ ] Add `_lstsq_lasso()` using `sklearn.linear_model.MultiTaskLassoCV`
+- [ ] Use Lasso only for ks≥11 where p > n even after PCA (complement, not replacement)
+- [ ] Verify sklearn available on Kaggle runtime
+- [ ] Run all 400. Compare vs Exp 3+4.
+- **Accept if:** arc-gen survival improves by ≥5 tasks vs Exp 3+4
+- **Expected:** +5-10 tasks (incremental over PCA)
+
+---
+
+#### Exp 6 [DEPRIORITIZED]: GD with Early Stopping ⬜ — Confidence: **40%**
+> *Moved to backlog. Literature shows GD early stopping ≡ Ridge (Ali et al. 2019), which is suboptimal for our low-rank regime. Only revisit if Exp 1-5 plateau.*
+
+- [ ] Only attempt if Exp 1-5 combined yield <80 arc-gen validated tasks
+- [ ] If attempted: use `sklearn.linear_model.SGDRegressor` with `early_stopping=True`
+
+---
+
+#### Exp 7 [DEPRIORITIZED]: PyTorch Multi-Seed (GPU Required) ⬜ — Confidence: **50%**
+> *Needs GPU, slow, complex. Only after simpler fixes validated.*
+
+- [!] Blocked on GPU availability
+- [ ] Only attempt if Exp 1-5 combined yield <100 arc-gen validated tasks
+
+---
+
+#### Exp 8 [DEPRIORITIZED]: Generate More ARC-GEN Data ⬜ — Confidence: **45%**
+> *Only useful WITH regularization in place (Exp 3+). Without it, more rows can *hurt* (Nakkiran 2019 sample-wise non-monotonicity).*
+
+- [ ] Only attempt after Exp 4 to see if cap increase helped
+- [ ] If yes: generate 1000+ examples/task using ARC-GEN generator
+
+---
+
+### Phase 2 Combined Projection
+
+| Scenario | Expected arc-gen tasks | LB estimate | Confidence |
+|----------|----------------------|-------------|------------|
+| Exp 1 alone (skip ks) | 60-80 | ~800-1000 | 90% |
+| Exp 1+2 (skip + best-of-N) | 60-80 tasks, better scores | ~900-1100 | 85% |
+| Exp 1+2+3 (+ PCA) | 90-130 | ~1200-1700 | 70% |
+| Exp 1+2+3+4 (+ more arc-gen) | 100-140 | ~1400-1900 | 60% |
+| Full stack 1-5 (+ Lasso) | 110-150 | ~1600-2200 | 50% |
+
+**The big win is the Exp 1+2+3 stack.** Skip bad kernels, pick best model, PCA regularization. If those three work, we roughly double or triple the LB score.
 
 ---
 
@@ -119,27 +248,23 @@
 - [ ] **Validate**: Run on all 400 models. Compare total score before/after.
   - Accept if total score improves by >2%.
 
-### 4b: Best-of-N Model Selection
-- [ ] **For each task**: generate multiple candidates (different ks, bias/no-bias, PCA vs raw, etc.)
-  - Keep cheapest valid one
-- [ ] **Validate**: Full 400 run. Compare total score vs single-candidate selection.
-  - Accept if total score improves by >3%.
-
-### 4c: Official Scoring Alignment
+### 4b: Official Scoring Alignment
 - [ ] **Use `neurogolf_utils.score_network()`** — `onnx_tool` for exact cost matching
   - Our static profiler may diverge on edge cases
 - [ ] **Validate**: Compare static profiler vs onnx_tool on 50 random models.
   - Accept if divergence >5% and fix profiler.
 
+> **Note:** Best-of-N model selection moved to Phase 2 Exp 2 — it's part of the core overfitting fix, not just optimization.
+
 ---
 
 ## BLENDING — EXPLICITLY EXCLUDED
 
 > **User's competitive philosophy**: "I am writing my own models no blending. This is major flaw in the competition loophole."
 
-- [ ] ~~Blend pipeline~~ — **NOT DONE. Not our strategy.**
-- [ ] ~~Upload submission.zip as Kaggle dataset~~ — **NOT DONE.**
-- [ ] ~~Attach public datasets (24 sources)~~ — **NOT DONE.**
+- ~~Blend pipeline~~ — **NOT DONE. Not our strategy.**
+- ~~Upload submission.zip as Kaggle dataset~~ — **NOT DONE.**
+- ~~Attach public datasets (24 sources)~~ — **NOT DONE.**
 
 Competitive intelligence on blending stays in LEARNING.md "What Others Do" section only.
 
@@ -149,9 +274,10 @@ Competitive intelligence on blending stays in LEARNING.md "What Others Do" secti
 
 | Date | Experiment | Tasks Tested | Result | Decision |
 |------|-----------|-------------|--------|----------|
-| 2026-04-24 | v4.2 baseline | 400 | 50 arc-gen, ~670 LB | Keep |
+| 2026-04-24 | v4.2 baseline | 400 | 50 arc-gen, ~670 LB | Keep as baseline |
 | 2026-04-25 | v5 untested code | 10 | 3/10 FAILED arc-gen | **REVERTED** |
 | 2026-04-25 | LOOCV Ridge theory | 0 | Never tested — theory predicts failure | **NOT IMPLEMENTED** |
+| 2026-04-26 | v5.0 refactor | TBD | Running on Kaggle | **AWAITING RESULTS** |
 
 ---
 
@@ -159,16 +285,20 @@ Competitive intelligence on blending stays in LEARNING.md "What Others Do" secti
 
 | Symbol | Meaning |
 |--------|---------|
-| `[ ]` | Not started — need research/design first |
+| `⬜` / `[ ]` | Not started — designed, ready to implement |
 | `[~]` | In progress — experiment running |
 | `[x]` | Done — validated with arc-gen on ≥20 tasks, confirmed score increase |
 | `[!]` | Blocked — needs prerequisite or resource (e.g., GPU) |
 | `[-]` | Rejected — tested, did not improve arc-gen survival or score |
 
-## Research Queue (Next 3 Papers to Read)
+## Research Queue (Papers Read ✅ / To Read)
 
-1. **arXiv:2302.00257** — "Benign overfitting in ridge regression..." (Lasso vs Ridge in sparse regimes)
-2. **Belkin et al. (2019) PNAS** — "Reconciling modern machine-learning practice..." (double descent, interpolation threshold)
-3. **CITE NEEDED** — ARC-AGI solver papers from NeurIPS 2024 / ICML 2024 workshops
+1. ✅ **Nakkiran et al. 2019** (`1912.02292`) — Double descent, interpolation threshold peak at p≈n
+2. ✅ **Segert 2023** (`2311.11093`) — Truncated SVD/PCA > Ridge for low-rank covariance
+3. ✅ **Zhou & Ge 2023** (`2302.00257`) — L1 near-minimax for sparse signals, L2 fails
+4. ✅ **Liu et al. 2023** (`2302.01088`) — More rows help only with regularization
+5. ✅ **Liao & Gu 2024** (`2512.06104`) — CompressARC: regularization enables ARC generalization
+6. ✅ **Ali et al. 2019** — GD early stopping ≡ Ridge (therefore suboptimal here)
+7. [ ] **ARC Prize 2025 Technical Report** (`2601.10904`) — competition landscape, top approaches
 
 > Loop: Research → Design → Experiment → Analyze → Research → ... until score increases.