Add LEARNING.md - full history, mistakes log, competitive intel, technical deep-dives

LEARNING.md

@@ -0,0 +1,159 @@
+# NeuroGolf Solver — Learning & History
+
+> This file accumulates everything learned across sessions. Read it to avoid repeating mistakes and to understand what techniques work. Newest entries first within each section.
+
+## Version History
+
+| Version | Date | Tasks (arc-gen validated) | Est LB | Key Changes |
+|---------|------|--------------------------|--------|-------------|
+| v4.1 | 2026-04-24 | 50 | ~670 | Color map Gather for permutations (+15 pts) |
+| v4.0 | 2026-04-24 | 50 | ~656 | ARC-GEN validation, new analytical solvers, s_flip fix, static profiler, submission.csv |
+| v3 | 2026-04-24 | 307 (local) / ~40 (LB) | 501 | Added concat_enhanced, varshape_spatial_gather, conv_var_diff |
+| v2 | prior | 294 (local) | unknown | Spatial_gather, variable-shape conv, diff-shape conv |
+| v1 | prior | 128 | unknown | Conv solver only |
+
+## Mistakes Log (DO NOT REPEAT)
+
+### 2026-04-24: CuPy/GPU for lstsq — DOES NOT HELP
+- **What**: Swapped numpy→cupy to GPU-accelerate lstsq conv fitting
+- **Result**: GPU hit 90%, crashed on task 4 (OOM), fell back to CPU, same speed
+- **Root cause**: lstsq is O(n³) — same algorithmic cost on any device. For ks=29 on 16 examples of 21×21: patch matrix is 7056×8410 = 59M elements, ~450MB float64. GPU memory fills and crashes.
+- **Rule**: NEVER try to GPU-accelerate lstsq. The bottleneck is algorithmic, not device. Use `--conv_budget` to cap time.
+
+### 2026-04-24: Channel Gather for non-permutation color maps — WRONG OUTPUT
+- **What**: Used `Gather(axis=1)` for all color maps
+- **Result**: Tasks 276, 309 produced double-active channels (ch2=1 AND ch6=1 simultaneously)
+- **Root cause**: Gather duplicates source channels. For map `{6→2}`, `gi[2]=6` copies ch6 to ch2, but ch6 also stays via `gi[6]=6`. Not valid one-hot.
+- **Rule**: Channel Gather ONLY works for **permutation** color maps (bijective, closed set). Non-permutations need Conv 1×1.
+
+### 2026-04-24: ARC-GEN not loaded — THE #1 SCORE KILLER (v3→v4 fix)
+- **What**: v3 `validate()` had `if 'arc-gen' in td` check, but arc-gen was never loaded into `td`
+- **Result**: 3267 local score → 501 LB. 85% of conv models fail on Kaggle's arc-gen validation
+- **Root cause**: `load_tasks_dir()` only loaded train+test from ARC-AGI files. Arc-gen data is in separate `ARC-GEN-100K/` files.
+- **Rule**: ALWAYS load arc-gen data. ALWAYS validate against it locally before submission.
+
+### 2026-04-24: s_flip used GatherElements — OPSET 11 BUG (v3→v4 fix)
+- **What**: `s_flip` solver used `GatherElements` with 4D indices
+- **Result**: Works on old ORT, fails on ORT 1.25+ which enforces opset correctly
+- **Rule**: NEVER use GatherElements with opset 10. Use `_build_gather_model()` (Gather on flattened spatial dim).
+
+### 2026-04-24: score_network fallback returned (0,0,0) — WRONG COSTS
+- **What**: When onnx_tool not installed, `score_network` returned zeros
+- **Result**: All costs appeared as 0, inflated estimated score
+- **Rule**: Use static profiler that counts params+nbytes+macs by walking the ONNX graph. Matches Kaggle's calculation.
+
+### 2026-04-24: Ignored EXCLUDED tasks
+- **What**: Tried to solve tasks {21, 55, 80, 184, 202, 366}
+- **Rule**: Skip these. Officially excluded, score 0 regardless.
+
+### Prior: GatherElements in v2 gather helpers
+- **What**: `_build_gather_model()` used GatherElements (opset 11)
+- **Fix**: Changed to Gather(opset 1) with 1D indices on flattened [1,10,900] spatial dim.
+
+## Competitive Intelligence
+
+### High-Scoring Notebook Architecture (2026-04-24 analysis)
+
+The top notebooks (4200+ points) are **BLENDERS**, not solvers:
+1. `neurogolf-2026-tiny-onnx-solver` (est 4197): Blends 12+ other notebooks' submission.zip files. Its own solver adds 0 new tasks.
+2. `4200-v5-neurogolf-fix` (est 5725): Same blend + 5 hand-crafted "LLM rescue" ONNX models for specific tasks.
+3. `the-2026-neurogolf-championship`: Own solver (288 tasks) + blend from other sources.
+
+**Key techniques competitors have that we still lack:**
+- PyTorch learned conv: multi-seed Adam (seeds 0,7,42), 3000 steps, ternary weight snapping
+- Two-layer conv: Conv→ReLU→Conv for nonlinear patterns
+- Channel reduction: reduce 10→N channels (fewer colors = cheaper)
+- Composition detectors: rotation+color, flip+color, transpose+color
+- Extract outline detector
+- Blending from multiple notebook outputs
+
+**Opset insight**: All top notebooks use opset 17 freely. It works on Kaggle.
+
+### Cost Benchmarks
+
+| Model Type | Typical Cost | Score |
+|-----------|-------------|-------|
+| Identity | 0 | 25.0 |
+| Transpose (perm only) | 0 | 25.0 |
+| Color map (Gather, permutation) | 50 | 21.1 |
+| Color map (Conv 1×1) | 90,500 | 13.6 |
+| Analytical (gather-based) | 12,663 | 15.5 |
+| Shift (gather) | 57,663 | 14.0 |
+| Conv ks=1 | 814,590–814,662 | 11.4 |
+| Conv ks=5 | 4,589,390 | 9.7 |
+| Conv ks=11 | 11,105,390 | 8.8 |
+| Conv ks=29 | 66,129,390 | 7.0 |
+
+### ARC-GEN Survival Rates
+
+From v4.0 full run (400 tasks):
+- **Analytical solvers**: 100% arc-gen survival (25/25 passed)
+- **conv_fixed (ks=1)**: ~80% survival (8/~10 passed)
+- **conv_var**: ~14% survival (17/~125 passed) — most fail with larger kernels
+- **conv_diff**: ~3% survival (1/~39 passed)
+- **spatial_gather**: ~25% survival (4/16 passed) — surprising failures
+
+Arc-gen fitting (same-size examples in lstsq) recovered ~10 additional conv tasks in v4.
+
+## Technical Deep-Dives
+
+### Why Conv Models Fail ARC-GEN
+
+Conv models fitted via lstsq on 6 train+test examples learn weights that perfectly separate those examples. But arc-gen has 250+ examples with:
+- Different pixel arrangements (same grid size but different content)
+- Edge cases the 6 training examples don't cover
+- The conv weights are a linear classifier — if the decision boundary isn't robust, new examples fall on the wrong side
+
+**What helps**: Including arc-gen examples in lstsq fitting (when grid sizes match). v4 adds up to 10 arc-gen examples, giving 16 total. This improved conv_var from 7→17 arc-gen validated tasks.
+
+**What doesn't help**: Including variable-size arc-gen examples in lstsq. The feature dimension changes with grid size for fixed-shape conv, and for variable-shape conv the 30×30 embedding creates too many zero-padded patches that dominate the lstsq solution.
+
+### lstsq Performance Characteristics
+
+For kernel size `ks` on `N` examples of size `H×W`:
+```
+Features = 10 × ks² (+ 1 if bias)
+Rows = N × H × W
+lstsq cost = O(rows × features²)  [for rows > features]
+           = O(rows² × features)  [for features > rows]
+```
+
+Practical timing (CPU, numpy):
+- ks=1, 6 examples of 10×10: ~0.001s
+- ks=5, 16 examples of 15×15: ~0.1s
+- ks=15, 16 examples of 20×20: ~5s
+- ks=29, 16 examples of 21×21: ~30s
+
+### The 113 Same-Size Fixed Tasks
+
+Analysis found 113 unsolved same-shape tasks where arc-gen uses IDENTICAL grid sizes to train/test. These are prime targets for arc-gen-enhanced lstsq fitting. v4 recovers ~10 of these; the rest need larger kernels or multi-layer networks.
+
+### Variable-Shape Tasks (77 unsolved)
+
+These tasks have input-dependent output shapes. No static ONNX graph can produce different-sized outputs. The only approach: conv learns to place content in the right 30×30 region, masked by `ReduceSum(input)`. But this fails when output extends beyond input bounds or when the spatial mapping depends on content.
+
+## Data Notes
+
+### ARC-GEN File Format
+```python
+# ARC-GEN-100K/{hex_id}.json — LIST of examples (not dict):
+[{"input": [[...]], "output": [[...]]}, ...]
+
+# On Kaggle, already embedded in task JSON:
+{"train": [...], "test": [...], "arc-gen": [...]}
+```
+
+### Task Numbering
+Tasks are numbered 1-400 based on alphabetical sort of hex filenames in `ARC-AGI/data/training/`. The hex ID → task number mapping is stable.
+
+### ARC-GEN Generator
+https://github.com/google/ARC-GEN — Can generate MORE examples per task for better fitting. Not yet explored.
+
+## Reference Notebooks (in repo as neurogolf-2026-solver-notebooks.zip)
+
+| Notebook | Est LB | Tasks Solved | Technique | Key Source Count |
+|----------|--------|-------------|-----------|-----------------|
+| neurogolf-2026-tiny-onnx-solver | ~4200 | 338 | Mega-blend 12+ zips | 203 from mega-agi-ensemble |
+| 4200-v5-neurogolf-fix | ~5725 | 341 | Same blend + 5 manual | 338 from zip_2 |
+| the-2026-neurogolf-championship | ~3200 est | 288 | Own solver + blend | gravity, outline, composition |
+| neurogolf-logic-driven-ensembling | — | 401 | Pure ensembling | — |