Upload TODO.md

TODO.md

@@ -1,304 +1 @@
-# NeuroGolf Solver — Roadmap
-
-> Current: v5.0 · 50 arc-gen validated (v4 baseline) · ~670 LB · Target: 3000+
-> Philosophy: **Research → Design → Experiment → Analyze → Research** loop until confirmed score increase.
-> Rule: **NEVER claim a feature works without full arc-gen validation on representative tasks.**
-> Updated: 2026-04-26 — Phase 2 overhauled with literature-backed experiment plan.
-
----
-
-## Phase 1: Cheap Wins (est +400 pts → ~1100)
-
-### 1a: Opset 17 Slice-Based Analytical Solvers (~0 cost)
-- [ ] **Convert ALL analytical solvers to opset 17** — not just new ones
-  - Rotation: `Crop → Transpose → Slice(step=-1)` = ~0 cost (was ~165K)
-  - Flip: `Crop → Slice(step=-1)` = ~0 cost (was ~165K)
-  - Transpose: `Crop → Transpose(perm)` = ~0 cost (was ~36K)
-  - Pad nodes: all must use opset 17 tensor-based `pads` input (not attribute)
-  - Affected solvers: s_tile, s_upscale, s_concat, s_concat_enhanced, s_kronecker, s_diagonal_tile, s_shift, s_mirror_h, s_mirror_v, s_quad_mirror, s_fixed_crop, s_spatial_gather, s_varshape_spatial_gather
-- [ ] **Validate**: Full 400 arc-gen run. Compare analytical task count vs v4.
-  - Target: ~25 analytical tasks scoring ~25 pts each (was ~15)
-  - Accept only if >10% improvement in analytical category total score.
-
-### 1b: Composition Detectors
-- [ ] **Identify actual tasks** that are rotation+recolor, flip+recolor, transpose+recolor
-  - Scan 400 tasks: apply rotate → check if color_map solves, etc.
-  - Only implement solvers for combinations that exist in dataset
-- [ ] **Build composition solver** — chain analytical + color_map as single ONNX graph
-- [ ] **Validate**: Full 400 arc-gen. Count new tasks solved. Accept only if >0 new tasks.
-
-### 1c: Channel Reduction Wrapper
-- [ ] **Design for Gather compatibility** — current Reshape hardcodes [1,10,900]
-  - Option A: Add Conv1x1(10→N) before + Conv1x1(N→10) after for conv-based models
-  - Option B: Use Slice to extract active channels + Gather remapping for pure spatial transforms
-- [ ] **Validate**: Pick 5 tasks with <5 colors. Compare score with/without wrapper.
-  - Accept only if >5% score improvement per task AND arc-gen still passes.
-
----
-
-## Phase 2: Fix Arc-Gen Survival — THE #1 BLOCKER
-
-> **Status:** 307 solved locally, only 50 survive arc-gen. ~250 tasks affected by lstsq overfitting.
-> **Root cause confirmed by literature:** catastrophic overfitting at the interpolation threshold (p ≈ n).
-> **Strategy:** Coordinated experiment sequence — each builds on the previous. Do NOT test in isolation.
-
-### The Problem (with numbers from conv.py)
-
-Current `_lstsq_conv()` runs naked `np.linalg.lstsq(P, T_oh, rcond=None)` — zero regularization.
-Kernel search is `[1, 3, 5, 7, 9, 11, ...]`, stops at **first ks that interpolates training**.
-
-| Kernel | p (features) | n (patches, 7×7 grid, 4 ex) | p/n | Regime |
-|--------|-------------|------------------------------|-----|--------|
-| ks=1 | 10 | 196 | 0.05 | ✅ Safe underparameterized |
-| ks=3 | 90 | 196 | 0.46 | ✅ Underparameterized |
-| **ks=5** | **250** | **196** | **1.27** | **❌ INTERPOLATION THRESHOLD** |
-| **ks=7** | **490** | **196** | **2.50** | **❌ PAST THRESHOLD — WORST CASE** |
-| **ks=9** | **810** | **196** | **4.13** | **⚠️ Near peak** |
-| ks=11 | 1210 | 196 | 6.17 | Overparameterized |
-| ks=29 | 8410 | 196 | 42.9 | Heavily overparameterized |
-
-The solver accepts ks=5 (which perfectly interpolates training via minimum-norm solution) and **never tries ks=11+** which might actually generalize.
-
-### Literature Backing
-
-| Paper | arxiv | Key Finding for Us |
-|-------|-------|--------------------|
-| Nakkiran et al. 2019 (NeurIPS) | `1912.02292` | Test error peaks at p≈n (interpolation threshold). This is exactly where ks=5,7 sit. Skipping these eliminates the peak entirely. |
-| Segert 2023 | `2311.11093` | Truncated SVD / PCA regression achieves flatter loss basins than Ridge. Optimal for low-rank covariance (our case: effective rank ~10-40). |
-| Zhou & Ge 2023 (NeurIPS) | `2302.00257` | L1 (Lasso) achieves near-minimax for sparse β*. L2 (Ridge) cannot. ARC one-hot patches are sparse (3-5 of 10 channels active). |
-| Liu et al. 2023 | `2302.01088` | More fitting rows help ONLY with regularization. Without it, adding rows near threshold can *hurt* (sample-wise non-monotonicity). |
-| Ali et al. 2019 | — | GD with early stopping ≡ Ridge with λ=1/(2t). Since Ridge is suboptimal here, GD early stopping is also suboptimal. |
-| Liao & Gu 2024 (CompressARC) | `2512.06104` | ARC solvers that generalize use regularized fitting (MDL/KL), not ridgeless interpolation. Direct evidence regularization is needed. |
-
-### Coordinated Experiment Sequence
-
-> Run in order. Each experiment keeps wins from previous experiments.
-> **Goal: highest-scoring valid model per task**, not first-valid.
-
----
-
-#### Exp 0: Baseline Measurement ⬜
-> *Prerequisite for all other experiments.*
-
-- [ ] Run current v5 on all 400 tasks with full arc-gen validation
-- [ ] Record per-task: (a) solver used, (b) ks chosen, (c) arc-gen pass/fail, (d) score, (e) p/n ratio
-- [ ] Identify the ~250 tasks that fail arc-gen — classify by ks and p/n ratio
-- **Exit criteria:** Have baseline numbers to compare against
-
----
-
-#### Exp 1: Skip ks=5,7,9 ⬜ — Confidence: **90%**
-> *1 line change per solver. Eliminates the interpolation threshold peak entirely.*
-
-**Evidence:** Nakkiran 2019 (`1912.02292`) proves test error peaks at p≈n. Our ks=5 (p=250, n≈196) is textbook worst-case. Removing these kernel sizes cannot make things worse — if a task *needs* ks=5 to solve, it was going to fail arc-gen anyway because it's in the catastrophic regime.
-
-**10% doubt:** Some tasks with large grids (21×21, 16 examples → n≈7056) have p/n < 1 even at ks=7. For those, ks=7 is safe. But the solver can't distinguish these cases without computing p/n per-task.
-
-- [ ] Change ks list in all 4 conv solvers: `[1, 3, 5, 7, 9, 11, ...]` → `[1, 3, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29]`
-  - Files: `conv.py` — `solve_conv_fixed`, `solve_conv_variable`, `solve_conv_diffshape`, `solve_conv_var_diff`
-- [ ] Run all 400 with arc-gen. Compare survival rate vs Exp 0.
-- **Accept if:** arc-gen survival improves by ≥5 tasks
-- **Expected:** +10-30 tasks
-
----
-
-#### Exp 2: Best-of-N Model Selection ⬜ — Confidence: **85%**
-> *Structural change to solve loop. Return highest-scoring valid model, not first-valid.*
-
-**Evidence:** Pure engineering — no theoretical uncertainty. Currently `conv.py` iterates ks smallest-first and returns the first that passes validation. This means if ks=3 AND ks=13 both pass arc-gen, we might return ks=13 (tried first due to bias loop) when ks=3 scores higher (lower MACs). Score = max(1, 25 - ln(cost)), so smaller models always score higher.
-
-**15% doubt:** Runtime. Trying all 12 kernel sizes × 2 bias options × arc-gen validation = up to 24 candidates per task. May blow the 12hr Kaggle time budget. Mitigate: set tighter per-candidate timeout, parallelize validation.
-
-- [ ] Refactor `solve_conv_*` to return **list of (model, ks, cost)** candidates instead of first-valid
-- [ ] Refactor `solve_task` to collect all candidates (analytical + all conv variants), pick cheapest valid
-- [ ] Add static cost estimation to pick cheapest before saving
-- [ ] Run all 400. Compare total score vs Exp 1.
-- **Accept if:** total score improves by ≥3% on existing solved tasks
-- **Expected:** +5-15 score points on tasks already solved (better model selection)
-
----
-
-#### Exp 3: PCA / Truncated SVD Before lstsq ⬜ — Confidence: **75%**
-> *~30 lines in `_lstsq_conv()`. Maps every ks into effective underparameterized regime.*
-
-**Evidence:** Segert 2023 (`2311.11093`) — PCA regression is provably better than Ridge for low-rank covariance. Our patch covariance has effective rank ~10-40 (few active colors in one-hot encoding). Truncating to top-k components removes the ~(p-40) pure noise dimensions that ridgeless lstsq amplifies into catastrophic overfitting.
-
-**25% doubt:** The variance threshold (99%) might be wrong for some tasks. Too aggressive truncation (k too small) kills signal. Too little (k too large) doesn't fix the problem. Need per-task adaptive k based on singular value gap.
-
-**Implementation:**
-```python
-def _lstsq_pcr(P, T_oh, var_threshold=0.99):
-    U, s, Vt = np.linalg.svd(P, full_matrices=False)
-    cumvar = np.cumsum(s**2) / np.sum(s**2)
-    k = np.searchsorted(cumvar, var_threshold) + 1
-    k = max(k, 5)  # floor: keep at least 5 components
-    P_red = U[:, :k] * s[:k]  # n × k, always k << n
-    w_red = np.linalg.lstsq(P_red, T_oh, rcond=None)[0]
-    w_full = Vt[:k].T @ w_red  # back to p-dim for ONNX weights
-    return w_full
-```
-
-- [ ] Add `_lstsq_pcr()` to `conv.py` alongside existing `_lstsq_conv()`
-- [ ] Use PCA path for all ks where p/n > 0.5 (safe margin below threshold)
-- [ ] Keep raw lstsq for ks=1,3 where p << n (PCA unnecessary, adds cost)
-- [ ] Try var_threshold in {0.95, 0.99, 0.999} — pick best per arc-gen survival
-- [ ] Run all 400. Compare survival rate vs Exp 1.
-- **Accept if:** arc-gen survival improves by ≥10 tasks vs Exp 1
-- **Expected:** +15-40 tasks (the big win — makes previously-impossible ks usable)
-
----
-
-#### Exp 4: Increase Arc-Gen Fitting Cap ⬜ — Confidence: **60%**
-> *1 line change. Only works WITH regularization (PCA) in place.*
-
-**Evidence:** Liu et al. 2023 (`2302.01088`) — more fitting rows help only in regularized regime. With PCA, more rows = more constraints in the reduced k-dimensional space = better conditioning. Without PCA, adding rows to underdetermined lstsq doesn't fix the fundamental problem.
-
-**40% doubt:** Arc-gen examples may have different grid sizes (filtered out by `get_exs_for_fitting`). The cap increase only helps if enough same-shape arc-gen examples exist.
-
-- [ ] Change `get_exs_for_fitting()`: cap 10 → 50
-- [ ] Change `get_exs_for_fitting_variable()`: cap 20 → 100
-- [ ] Run all 400. Compare vs Exp 3.
-- **Accept if:** arc-gen survival improves by ≥3 tasks vs Exp 3
-- **Expected:** +5-15 tasks (modest but free)
-
----
-
-#### Exp 5: Lasso (L1) for Large Kernels ⬜ — Confidence: **55%**
-> *~15 lines + sklearn dependency. Better than L2 for sparse signals, but slower and fragile.*
-
-**Evidence:** Zhou & Ge 2023 (`2302.00257`) — L1 achieves near-minimax O(σ²·s·log(d/s)/n) for sparse β*, vs Ridge's Ω(‖β*‖²). ARC one-hot patches are sparse (3-5 of 10 channels active). Segert 2023 Section A.10: Lasso competitive with PCA/Nuclear in sparse regime, wins ~40% of cases.
-
-**45% doubt:** (1) Lasso is slow — coordinate descent vs single SVD. (2) `alpha` tuning via CV with only 4-6 training examples is fragile. (3) Need `MultiTaskLassoCV` for 10-column one-hot target, not scalar `LassoCV`. (4) sklearn adds a dependency (not available on Kaggle by default — need to verify).
-
-- [ ] Add `_lstsq_lasso()` using `sklearn.linear_model.MultiTaskLassoCV`
-- [ ] Use Lasso only for ks≥11 where p > n even after PCA (complement, not replacement)
-- [ ] Verify sklearn available on Kaggle runtime
-- [ ] Run all 400. Compare vs Exp 3+4.
-- **Accept if:** arc-gen survival improves by ≥5 tasks vs Exp 3+4
-- **Expected:** +5-10 tasks (incremental over PCA)
-
----
-
-#### Exp 6 [DEPRIORITIZED]: GD with Early Stopping ⬜ — Confidence: **40%**
-> *Moved to backlog. Literature shows GD early stopping ≡ Ridge (Ali et al. 2019), which is suboptimal for our low-rank regime. Only revisit if Exp 1-5 plateau.*
-
-- [ ] Only attempt if Exp 1-5 combined yield <80 arc-gen validated tasks
-- [ ] If attempted: use `sklearn.linear_model.SGDRegressor` with `early_stopping=True`
-
----
-
-#### Exp 7 [DEPRIORITIZED]: PyTorch Multi-Seed (GPU Required) ⬜ — Confidence: **50%**
-> *Needs GPU, slow, complex. Only after simpler fixes validated.*
-
-- [!] Blocked on GPU availability
-- [ ] Only attempt if Exp 1-5 combined yield <100 arc-gen validated tasks
-
----
-
-#### Exp 8 [DEPRIORITIZED]: Generate More ARC-GEN Data ⬜ — Confidence: **45%**
-> *Only useful WITH regularization in place (Exp 3+). Without it, more rows can *hurt* (Nakkiran 2019 sample-wise non-monotonicity).*
-
-- [ ] Only attempt after Exp 4 to see if cap increase helped
-- [ ] If yes: generate 1000+ examples/task using ARC-GEN generator
-
----
-
-### Phase 2 Combined Projection
-
-| Scenario | Expected arc-gen tasks | LB estimate | Confidence |
-|----------|----------------------|-------------|------------|
-| Exp 1 alone (skip ks) | 60-80 | ~800-1000 | 90% |
-| Exp 1+2 (skip + best-of-N) | 60-80 tasks, better scores | ~900-1100 | 85% |
-| Exp 1+2+3 (+ PCA) | 90-130 | ~1200-1700 | 70% |
-| Exp 1+2+3+4 (+ more arc-gen) | 100-140 | ~1400-1900 | 60% |
-| Full stack 1-5 (+ Lasso) | 110-150 | ~1600-2200 | 50% |
-
-**The big win is the Exp 1+2+3 stack.** Skip bad kernels, pick best model, PCA regularization. If those three work, we roughly double or triple the LB score.
-
----
-
-## Phase 3: Hard Tasks — Hash Matchers & Pattern Detectors (est +20-50 tasks → ~2500-3000)
-
-### 3a: Hash-Based Matcher Builder
-- [ ] **Generic hash matcher**: flatten input → MatMul(hash_weights) → match → apply stored delta
-  - Requires opset 17 (ScatterND)
-  - Works for ANY task where all examples fit in 1.44MB model
-  - Build `build_hash_matcher(task_data) → onnx_bytes`
-- [ ] **Validate**: Identify 10 tasks that no solver handles. Test hash matcher on them.
-  - Accept if it solves ≥2 tasks that are currently unsolved.
-
-### 3b: Run-Length / Gap Pattern Detector
-- [ ] **Depthwise conv to detect runs of N, gap patterns** — like task096 in public notebooks
-  - Template for "count and classify" tasks
-- [ ] **Validate**: Find tasks with run-length structure. Test detector.
-  - Accept if it solves ≥2 new tasks.
-
-### 3c: Per-Task LLM Rescue
-- [ ] **For ~20 hardest tasks**: feed task JSON + Python solution to LLM → get ONNX builder
-  - Priority: gravity, flood fill, outline extraction, pattern counting
-- [ ] **Validate**: Build 5 rescue models. Arc-gen validate. Accept if ≥3 pass.
-
----
-
-## Phase 4: Score Optimization (est +200-500 pts on existing tasks)
-
-### 4a: ONNX Optimizer Pass
-- [ ] **`onnxoptimizer.optimize()`** with dead-code elimination, identity removal
-  - Top notebooks do this; can shrink models 5-20%
-- [ ] **Validate**: Run on all 400 models. Compare total score before/after.
-  - Accept if total score improves by >2%.
-
-### 4b: Official Scoring Alignment
-- [ ] **Use `neurogolf_utils.score_network()`** — `onnx_tool` for exact cost matching
-  - Our static profiler may diverge on edge cases
-- [ ] **Validate**: Compare static profiler vs onnx_tool on 50 random models.
-  - Accept if divergence >5% and fix profiler.
-
-> **Note:** Best-of-N model selection moved to Phase 2 Exp 2 — it's part of the core overfitting fix, not just optimization.
-
----
-
-## BLENDING — EXPLICITLY EXCLUDED
-
-> **User's competitive philosophy**: "I am writing my own models no blending. This is major flaw in the competition loophole."
-
-- ~~Blend pipeline~~ — **NOT DONE. Not our strategy.**
-- ~~Upload submission.zip as Kaggle dataset~~ — **NOT DONE.**
-- ~~Attach public datasets (24 sources)~~ — **NOT DONE.**
-
-Competitive intelligence on blending stays in LEARNING.md "What Others Do" section only.
-
----
-
-## Experiment Log
-
-| Date | Experiment | Tasks Tested | Result | Decision |
-|------|-----------|-------------|--------|----------|
-| 2026-04-24 | v4.2 baseline | 400 | 50 arc-gen, ~670 LB | Keep as baseline |
-| 2026-04-25 | v5 untested code | 10 | 3/10 FAILED arc-gen | **REVERTED** |
-| 2026-04-25 | LOOCV Ridge theory | 0 | Never tested — theory predicts failure | **NOT IMPLEMENTED** |
-| 2026-04-26 | v5.0 refactor | TBD | Running on Kaggle | **AWAITING RESULTS** |
-
----
-
-## Status Key
-
-| Symbol | Meaning |
-|--------|---------|
-| `⬜` / `[ ]` | Not started — designed, ready to implement |
-| `[~]` | In progress — experiment running |
-| `[x]` | Done — validated with arc-gen on ≥20 tasks, confirmed score increase |
-| `[!]` | Blocked — needs prerequisite or resource (e.g., GPU) |
-| `[-]` | Rejected — tested, did not improve arc-gen survival or score |
-
-## Research Queue (Papers Read ✅ / To Read)
-
-1. ✅ **Nakkiran et al. 2019** (`1912.02292`) — Double descent, interpolation threshold peak at p≈n
-2. ✅ **Segert 2023** (`2311.11093`) — Truncated SVD/PCA > Ridge for low-rank covariance
-3. ✅ **Zhou & Ge 2023** (`2302.00257`) — L1 near-minimax for sparse signals, L2 fails
-4. ✅ **Liu et al. 2023** (`2302.01088`) — More rows help only with regularization
-5. ✅ **Liao & Gu 2024** (`2512.06104`) — CompressARC: regularization enables ARC generalization
-6. ✅ **Ali et al. 2019** — GD early stopping ≡ Ridge (therefore suboptimal here)
-7. [ ] **ARC Prize 2025 Technical Report** (`2601.10904`) — competition landscape, top approaches
-
-> Loop: Research → Design → Experiment → Analyze → Research → ... until score increases.
+placeholder