Upload TODO.md with huggingface_hub

TODO.md

@@ -1 +1,322 @@
-placeholder
+# NeuroGolf Solver — Roadmap
+
+> Current: v5.0 · 50 arc-gen validated (v4 baseline) · ~670 LB · Target: 3000+
+> Philosophy: **Research → Design → Experiment → Analyze → Research** loop until confirmed score increase.
+> Rule: **NEVER claim a feature works without full arc-gen validation on representative tasks.**
+> Updated: 2026-04-26 — Phase 2 experiments tested. Overfitting hypothesis REFUTED. Architecture mismatch identified as root cause.
+
+---
+
+## Phase 1: Cheap Wins (est +400 pts → ~1100)
+
+### 1a: Opset 17 Slice-Based Analytical Solvers (~0 cost)
+- [ ] **Convert ALL analytical solvers to opset 17** — not just new ones
+  - Rotation: `Crop → Transpose → Slice(step=-1)` = ~0 cost (was ~165K)
+  - Flip: `Crop → Slice(step=-1)` = ~0 cost (was ~165K)
+  - Transpose: `Crop → Transpose(perm)` = ~0 cost (was ~36K)
+  - Pad nodes: all must use opset 17 tensor-based `pads` input (not attribute)
+  - Affected solvers: s_tile, s_upscale, s_concat, s_concat_enhanced, s_kronecker, s_diagonal_tile, s_shift, s_mirror_h, s_mirror_v, s_quad_mirror, s_fixed_crop, s_spatial_gather, s_varshape_spatial_gather
+- [ ] **Validate**: Full 400 arc-gen run. Compare analytical task count vs v4.
+  - Target: ~25 analytical tasks scoring ~25 pts each (was ~15)
+  - Accept only if >10% improvement in analytical category total score.
+
+### 1b: Composition Detectors
+- [ ] **Identify actual tasks** that are rotation+recolor, flip+recolor, transpose+recolor
+  - Scan 400 tasks: apply rotate → check if color_map solves, etc.
+  - Only implement solvers for combinations that exist in dataset
+- [ ] **Build composition solver** — chain analytical + color_map as single ONNX graph
+- [ ] **Validate**: Full 400 arc-gen. Count new tasks solved. Accept only if >0 new tasks.
+
+### 1c: Channel Reduction Wrapper
+- [ ] **Design for Gather compatibility** — current Reshape hardcodes [1,10,900]
+  - Option A: Add Conv1x1(10→N) before + Conv1x1(N→10) after for conv-based models
+  - Option B: Use Slice to extract active channels + Gather remapping for pure spatial transforms
+- [ ] **Validate**: Pick 5 tasks with <5 colors. Compare score with/without wrapper.
+  - Accept only if >5% score improvement per task AND arc-gen still passes.
+
+---
+
+## Phase 2: Fix Arc-Gen Survival — THE #1 BLOCKER
+
+> **Status:** 307 solved locally, only 50 survive arc-gen. ~250 tasks affected by lstsq overfitting.
+> **Root cause confirmed by literature:** catastrophic overfitting at the interpolation threshold (p ≈ n).
+> **Strategy:** Coordinated experiment sequence — each builds on the previous. Do NOT test in isolation.
+
+### The Problem (with numbers from conv.py)
+
+Current `_lstsq_conv()` runs naked `np.linalg.lstsq(P, T_oh, rcond=None)` — zero regularization.
+Kernel search is `[1, 3, 5, 7, 9, 11, ...]`, stops at **first ks that interpolates training**.
+
+| Kernel | p (features) | n (patches, 7×7 grid, 4 ex) | p/n | Regime |
+|--------|-------------|------------------------------|-----|--------|
+| ks=1 | 10 | 196 | 0.05 | ✅ Safe underparameterized |
+| ks=3 | 90 | 196 | 0.46 | ✅ Underparameterized |
+| **ks=5** | **250** | **196** | **1.27** | **❌ INTERPOLATION THRESHOLD** |
+| **ks=7** | **490** | **196** | **2.50** | **❌ PAST THRESHOLD — WORST CASE** |
+| **ks=9** | **810** | **196** | **4.13** | **⚠️ Near peak** |
+| ks=11 | 1210 | 196 | 6.17 | Overparameterized |
+| ks=29 | 8410 | 196 | 42.9 | Heavily overparameterized |
+
+The solver accepts ks=5 (which perfectly interpolates training via minimum-norm solution) and **never tries ks=11+** which might actually generalize.
+
+### Literature Backing
+
+| Paper | arxiv | Key Finding for Us |
+|-------|-------|--------------------|
+| Nakkiran et al. 2019 (NeurIPS) | `1912.02292` | Test error peaks at p≈n (interpolation threshold). This is exactly where ks=5,7 sit. Skipping these eliminates the peak entirely. |
+| Segert 2023 | `2311.11093` | Truncated SVD / PCA regression achieves flatter loss basins than Ridge. Optimal for low-rank covariance (our case: effective rank ~10-40). |
+| Zhou & Ge 2023 (NeurIPS) | `2302.00257` | L1 (Lasso) achieves near-minimax for sparse β*. L2 (Ridge) cannot. ARC one-hot patches are sparse (3-5 of 10 channels active). |
+| Liu et al. 2023 | `2302.01088` | More fitting rows help ONLY with regularization. Without it, adding rows near threshold can *hurt* (sample-wise non-monotonicity). |
+| Ali et al. 2019 | — | GD with early stopping ≡ Ridge with λ=1/(2t). Since Ridge is suboptimal here, GD early stopping is also suboptimal. |
+| Liao & Gu 2024 (CompressARC) | `2512.06104` | ARC solvers that generalize use regularized fitting (MDL/KL), not ridgeless interpolation. Direct evidence regularization is needed. |
+
+### Coordinated Experiment Sequence
+
+> Run in order. Each experiment keeps wins from previous experiments.
+> **Goal: highest-scoring valid model per task**, not first-valid.
+
+---
+
+#### Exp 0: Baseline Measurement ⬜
+> *Prerequisite for all other experiments.*
+
+- [ ] Run current v5 on all 400 tasks with full arc-gen validation
+- [ ] Record per-task: (a) solver used, (b) ks chosen, (c) arc-gen pass/fail, (d) score, (e) p/n ratio
+- [ ] Identify the ~250 tasks that fail arc-gen — classify by ks and p/n ratio
+- **Exit criteria:** Have baseline numbers to compare against
+
+---
+
+#### Exp 1: Skip ks=5,7,9 ⬜ — Confidence: **90%**
+> *1 line change per solver. Eliminates the interpolation threshold peak entirely.*
+
+**Evidence:** Nakkiran 2019 (`1912.02292`) proves test error peaks at p≈n. Our ks=5 (p=250, n≈196) is textbook worst-case. Removing these kernel sizes cannot make things worse — if a task *needs* ks=5 to solve, it was going to fail arc-gen anyway because it's in the catastrophic regime.
+
+**10% doubt:** Some tasks with large grids (21×21, 16 examples → n≈7056) have p/n < 1 even at ks=7. For those, ks=7 is safe. But the solver can't distinguish these cases without computing p/n per-task.
+
+- [ ] Change ks list in all 4 conv solvers: `[1, 3, 5, 7, 9, 11, ...]` → `[1, 3, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29]`
+  - Files: `conv.py` — `solve_conv_fixed`, `solve_conv_variable`, `solve_conv_diffshape`, `solve_conv_var_diff`
+- [ ] Run all 400 with arc-gen. Compare survival rate vs Exp 0.
+- **Accept if:** arc-gen survival improves by ≥5 tasks
+- **Expected:** +10-30 tasks
+
+---
+
+#### Exp 2: Best-of-N Model Selection ⬜ — Confidence: **85%**
+> *Structural change to solve loop. Return highest-scoring valid model, not first-valid.*
+
+**Evidence:** Pure engineering — no theoretical uncertainty. Currently `conv.py` iterates ks smallest-first and returns the first that passes validation. This means if ks=3 AND ks=13 both pass arc-gen, we might return ks=13 (tried first due to bias loop) when ks=3 scores higher (lower MACs). Score = max(1, 25 - ln(cost)), so smaller models always score higher.
+
+**15% doubt:** Runtime. Trying all 12 kernel sizes × 2 bias options × arc-gen validation = up to 24 candidates per task. May blow the 12hr Kaggle time budget. Mitigate: set tighter per-candidate timeout, parallelize validation.
+
+- [ ] Refactor `solve_conv_*` to return **list of (model, ks, cost)** candidates instead of first-valid
+- [ ] Refactor `solve_task` to collect all candidates (analytical + all conv variants), pick cheapest valid
+- [ ] Add static cost estimation to pick cheapest before saving
+- [ ] Run all 400. Compare total score vs Exp 1.
+- **Accept if:** total score improves by ≥3% on existing solved tasks
+- **Expected:** +5-15 score points on tasks already solved (better model selection)
+
+---
+
+#### Exp 3: PCA / Truncated SVD Before lstsq ⬜ — Confidence: **75%**
+> *~30 lines in `_lstsq_conv()`. Maps every ks into effective underparameterized regime.*
+
+**Evidence:** Segert 2023 (`2311.11093`) — PCA regression is provably better than Ridge for low-rank covariance. Our patch covariance has effective rank ~10-40 (few active colors in one-hot encoding). Truncating to top-k components removes the ~(p-40) pure noise dimensions that ridgeless lstsq amplifies into catastrophic overfitting.
+
+**25% doubt:** The variance threshold (99%) might be wrong for some tasks. Too aggressive truncation (k too small) kills signal. Too little (k too large) doesn't fix the problem. Need per-task adaptive k based on singular value gap.
+
+**Implementation:**
+```python
+def _lstsq_pcr(P, T_oh, var_threshold=0.99):
+    U, s, Vt = np.linalg.svd(P, full_matrices=False)
+    cumvar = np.cumsum(s**2) / np.sum(s**2)
+    k = np.searchsorted(cumvar, var_threshold) + 1
+    k = max(k, 5)  # floor: keep at least 5 components
+    P_red = U[:, :k] * s[:k]  # n × k, always k << n
+    w_red = np.linalg.lstsq(P_red, T_oh, rcond=None)[0]
+    w_full = Vt[:k].T @ w_red  # back to p-dim for ONNX weights
+    return w_full
+```
+
+- [ ] Add `_lstsq_pcr()` to `conv.py` alongside existing `_lstsq_conv()`
+- [ ] Use PCA path for all ks where p/n > 0.5 (safe margin below threshold)
+- [ ] Keep raw lstsq for ks=1,3 where p << n (PCA unnecessary, adds cost)
+- [ ] Try var_threshold in {0.95, 0.99, 0.999} — pick best per arc-gen survival
+- [ ] Run all 400. Compare survival rate vs Exp 1.
+- **Accept if:** arc-gen survival improves by ≥10 tasks vs Exp 1
+- **Expected:** +15-40 tasks (the big win — makes previously-impossible ks usable)
+
+---
+
+#### Exp 4: Increase Arc-Gen Fitting Cap ⬜ — Confidence: **60%**
+> *1 line change. Only works WITH regularization (PCA) in place.*
+
+**Evidence:** Liu et al. 2023 (`2302.01088`) — more fitting rows help only in regularized regime. With PCA, more rows = more constraints in the reduced k-dimensional space = better conditioning. Without PCA, adding rows to underdetermined lstsq doesn't fix the fundamental problem.
+
+**40% doubt:** Arc-gen examples may have different grid sizes (filtered out by `get_exs_for_fitting`). The cap increase only helps if enough same-shape arc-gen examples exist.
+
+- [ ] Change `get_exs_for_fitting()`: cap 10 → 50
+- [ ] Change `get_exs_for_fitting_variable()`: cap 20 ��� 100
+- [ ] Run all 400. Compare vs Exp 3.
+- **Accept if:** arc-gen survival improves by ≥3 tasks vs Exp 3
+- **Expected:** +5-15 tasks (modest but free)
+
+---
+
+#### Exp 5: Lasso (L1) for Large Kernels ⬜ — Confidence: **55%**
+> *~15 lines + sklearn dependency. Better than L2 for sparse signals, but slower and fragile.*
+
+**Evidence:** Zhou & Ge 2023 (`2302.00257`) — L1 achieves near-minimax O(σ²·s·log(d/s)/n) for sparse β*, vs Ridge's Ω(‖β*‖²). ARC one-hot patches are sparse (3-5 of 10 channels active). Segert 2023 Section A.10: Lasso competitive with PCA/Nuclear in sparse regime, wins ~40% of cases.
+
+**45% doubt:** (1) Lasso is slow — coordinate descent vs single SVD. (2) `alpha` tuning via CV with only 4-6 training examples is fragile. (3) Need `MultiTaskLassoCV` for 10-column one-hot target, not scalar `LassoCV`. (4) sklearn adds a dependency (not available on Kaggle by default — need to verify).
+
+- [ ] Add `_lstsq_lasso()` using `sklearn.linear_model.MultiTaskLassoCV`
+- [ ] Use Lasso only for ks≥11 where p > n even after PCA (complement, not replacement)
+- [ ] Verify sklearn available on Kaggle runtime
+- [ ] Run all 400. Compare vs Exp 3+4.
+- **Accept if:** arc-gen survival improves by ≥5 tasks vs Exp 3+4
+- **Expected:** +5-10 tasks (incremental over PCA)
+
+---
+
+#### Exp 6 [DEPRIORITIZED]: GD with Early Stopping ⬜ — Confidence: **40%**
+> *Moved to backlog. Literature shows GD early stopping ≡ Ridge (Ali et al. 2019), which is suboptimal for our low-rank regime. Only revisit if Exp 1-5 plateau.*
+
+- [ ] Only attempt if Exp 1-5 combined yield <80 arc-gen validated tasks
+- [ ] If attempted: use `sklearn.linear_model.SGDRegressor` with `early_stopping=True`
+
+---
+
+#### Exp 7 [DEPRIORITIZED]: PyTorch Multi-Seed (GPU Required) ⬜ — Confidence: **50%**
+> *Needs GPU, slow, complex. Only after simpler fixes validated.*
+
+- [!] Blocked on GPU availability
+- [ ] Only attempt if Exp 1-5 combined yield <100 arc-gen validated tasks
+
+---
+
+#### Exp 8 [DEPRIORITIZED]: Generate More ARC-GEN Data ⬜ — Confidence: **45%**
+> *Only useful WITH regularization in place (Exp 3+). Without it, more rows can *hurt* (Nakkiran 2019 sample-wise non-monotonicity).*
+
+- [ ] Only attempt after Exp 4 to see if cap increase helped
+- [ ] If yes: generate 1000+ examples/task using ARC-GEN generator
+
+---
+
+### Phase 2 Combined Projection
+
+| Scenario | Expected arc-gen tasks | LB estimate | Confidence |
+|----------|----------------------|-------------|------------|
+| Exp 1 alone (skip ks) | 60-80 | ~800-1000 | 90% |
+| Exp 1+2 (skip + best-of-N) | 60-80 tasks, better scores | ~900-1100 | 85% |
+| Exp 1+2+3 (+ PCA) | 90-130 | ~1200-1700 | 70% |
+| Exp 1+2+3+4 (+ more arc-gen) | 100-140 | ~1400-1900 | 60% |
+| Full stack 1-5 (+ Lasso) | 110-150 | ~1600-2200 | 50% |
+
+**The big win is the Exp 1+2+3 stack.** Skip bad kernels, pick best model, PCA regularization. If those three work, we roughly double or triple the LB score.
+
+---
+
+## Phase 3: Hard Tasks — Hash Matchers & Pattern Detectors (est +20-50 tasks → ~2500-3000)
+
+### 3a: Hash-Based Matcher Builder
+- [ ] **Generic hash matcher**: flatten input → MatMul(hash_weights) → match → apply stored delta
+  - Requires opset 17 (ScatterND)
+  - Works for ANY task where all examples fit in 1.44MB model
+  - Build `build_hash_matcher(task_data) → onnx_bytes`
+- [ ] **Validate**: Identify 10 tasks that no solver handles. Test hash matcher on them.
+  - Accept if it solves ≥2 tasks that are currently unsolved.
+
+### 3b: Run-Length / Gap Pattern Detector
+- [ ] **Depthwise conv to detect runs of N, gap patterns** — like task096 in public notebooks
+  - Template for "count and classify" tasks
+- [ ] **Validate**: Find tasks with run-length structure. Test detector.
+  - Accept if it solves ≥2 new tasks.
+
+### 3c: Per-Task LLM Rescue
+- [ ] **For ~20 hardest tasks**: feed task JSON + Python solution to LLM → get ONNX builder
+  - Priority: gravity, flood fill, outline extraction, pattern counting
+- [ ] **Validate**: Build 5 rescue models. Arc-gen validate. Accept if ≥3 pass.
+
+---
+
+## Phase 4: Score Optimization (est +200-500 pts on existing tasks)
+
+### 4a: ONNX Optimizer Pass
+- [ ] **`onnxoptimizer.optimize()`** with dead-code elimination, identity removal
+  - Top notebooks do this; can shrink models 5-20%
+- [ ] **Validate**: Run on all 400 models. Compare total score before/after.
+  - Accept if total score improves by >2%.
+
+### 4b: Official Scoring Alignment
+- [ ] **Use `neurogolf_utils.score_network()`** — `onnx_tool` for exact cost matching
+  - Our static profiler may diverge on edge cases
+- [ ] **Validate**: Compare static profiler vs onnx_tool on 50 random models.
+  - Accept if divergence >5% and fix profiler.
+
+> **Note:** Best-of-N model selection moved to Phase 2 Exp 2 — it's part of the core overfitting fix, not just optimization.
+
+---
+
+## BLENDING — EXPLICITLY EXCLUDED
+
+> **User's competitive philosophy**: "I am writing my own models no blending. This is major flaw in the competition loophole."
+
+- ~~Blend pipeline~~ — **NOT DONE. Not our strategy.**
+- ~~Upload submission.zip as Kaggle dataset~~ — **NOT DONE.**
+- ~~Attach public datasets (24 sources)~~ — **NOT DONE.**
+
+Competitive intelligence on blending stays in LEARNING.md "What Others Do" section only.
+
+---
+
+## Experiment Log
+
+| Date | Experiment | Tasks Tested | Result | Decision |
+|------|-----------|-------------|--------|----------|
+| 2026-04-24 | v4.2 baseline | 400 | 50 arc-gen, ~670 LB | Keep as baseline |
+| 2026-04-25 | v5 untested code | 10 | 3/10 FAILED arc-gen | **REVERTED** |
+| 2026-04-26 | v5.0 refactor | 394 | **49 solved, ~603.6 score, budget=5s** | New baseline |
+| 2026-04-26 | Exp 0: Baseline | 25 conv tasks | 24/25 solved, score=253 | Baseline for conv |
+| 2026-04-26 | Exp 1: Skip ks=5,7,9 | 25 conv+30 unsolved | **HURTS 2 solved tasks (322@ks5, 299@ks9), helps 0 new** | **[-] REJECTED** |
+| 2026-04-26 | Exp 2: Best-of-N | 25 conv+30 unsolved | **No new solves on unsolved tasks** | **[~] NEUTRAL** — score opt only |
+| 2026-04-26 | Exp 3: Ridge reg | 4 victim tasks, 5 alphas | **0/4 pass arc-gen at any alpha** | **[-] REJECTED** |
+| 2026-04-26 | Exp 3: PCA/trunc-SVD | Task 129, thresh 0.5-0.99 | **0 pass, architecture mismatch** | **[-] REJECTED for lstsq** |
+
+### CRITICAL FINDING (2026-04-26)
+
+The "307→50 arc-gen survival gap" is **NOT primarily caused by lstsq overfitting**.
+
+**Evidence:**
+1. Only 18 of 345 unsolved tasks pass train-fit at ks≤9. Of these, 17 use ks=5,7,9.
+2. Ridge (L2) on 4 victim tasks × 5 alphas: **zero arc-gen passes**.
+3. PCA/truncated-SVD at thresholds 0.50-0.99: **zero arc-gen passes**.
+4. Inspecting victims reveals they require **global operations** (mode counting, flood fill) that NO local convolution can represent.
+
+**Root cause reclassified:** Architecture mismatch, not regularization. The literature predictions (Nakkiran 2019) were correct in theory but inapplicable — the tasks that fail arc-gen fail because conv is the wrong solver type, not because of bad regularization.
+
+**Impact on Phase 2:** Exps 1-5 are deprioritized. The fix must come from Phase 3 (new solver types) or new architectures. Best-of-N still useful for score optimization on existing solved tasks.
+
+---
+
+## Status Key
+
+| Symbol | Meaning |
+|--------|---------|
+| `⬜` / `[ ]` | Not started — designed, ready to implement |
+| `[~]` | In progress — experiment running |
+| `[x]` | Done — validated with arc-gen on ≥20 tasks, confirmed score increase |
+| `[!]` | Blocked — needs prerequisite or resource (e.g., GPU) |
+| `[-]` | Rejected — tested, did not improve arc-gen survival or score |
+
+## Research Queue (Papers Read ✅ / To Read)
+
+1. ✅ **Nakkiran et al. 2019** (`1912.02292`) — Double descent, interpolation threshold peak at p≈n
+2. ✅ **Segert 2023** (`2311.11093`) — Truncated SVD/PCA > Ridge for low-rank covariance
+3. ✅ **Zhou & Ge 2023** (`2302.00257`) — L1 near-minimax for sparse signals, L2 fails
+4. ✅ **Liu et al. 2023** (`2302.01088`) — More rows help only with regularization
+5. ✅ **Liao & Gu 2024** (`2512.06104`) — CompressARC: regularization enables ARC generalization
+6. ✅ **Ali et al. 2019** — GD early stopping ≡ Ridge (therefore suboptimal here)
+7. [ ] **ARC Prize 2025 Technical Report** (`2601.10904`) — competition landscape, top approaches
+
+> Loop: Research → Design → Experiment → Analyze → Research → ... until score increases.