Add deep competitive analysis from notebook dissection (2026-04-25)

LEARNING.md

@@ -65,6 +65,175 @@
 
 ## Competitive Intelligence
 
+### Deep Notebook Dissection (2026-04-25) — THE DEFINITIVE ANALYSIS
+
+#### Why top notebooks score 4000+ and we score ~670
+
+The top notebooks are **BLENDERS**, not solvers. The entire leaderboard meta-game is about
+assembling the best portfolio of pre-solved ONNX models from public sources, not about
+building a better solver from scratch.
+
+#### Quantified Breakdown
+
+| Notebook | Own Solver Tasks | Blended from Others | Total Solved | Est Score |
+|---|---|---|---|---|
+| `neurogolf-2026-tiny-onnx-solver` | **0** from own solver | 338 from 12 ZIP + 5 dataset dirs | 338 | ~4200 |
+| `4200-v5-neurogolf-fix` | **5** manual LLM rescue | 341 from 5 ZIP sources | 346 | ~5700 |
+| `the-2026-neurogolf-championship` | ~20 from own solver | 288 from **24 Kaggle dataset** sources | 288 | ~3600 |
+| `neurogolf-4200-solver` (full solver) | ~20 analytical | 288 from 24 dataset sources | 288 | ~3600 |
+| **Our solver v4** | **~50** from solver | **0 blended** | 50 | ~670 |
+
+#### How the Blend Pipeline Works (from `neurogolf-2026-tiny-onnx-solver`)
+
+```
+Phase 1: ZIP Blend
+  - Auto-discovers ALL submission.zip files from attached Kaggle notebook outputs
+  - 12 sources: mega-agi-ensemble(203), the-2026-neurogolf-championship(105),
+    neurogolf-2026-starter(77), baseline-for-ensemble-1k(8), infinitesimals(4),
+    arc-nano-engine(2), + 6 more with 0 valid models
+  - Each model: strict_validate(raw, task_id) using neurogolf_utils
+    → verify_subset(session, train+test) + verify_subset(session, arc-gen)
+    → score_network(path) for official cost
+  - Keep cheapest valid model per task
+
+Phase 2: Dataset ONNX dirs
+  - Scans loose .onnx files from attached dataset directories
+  - Same strict validation
+
+Phase 3: Own solver (minimal)
+  - Only runs on unsolved tasks (62 remaining after blend)
+  - Detectors: identity, color_map, rotation, flip, transpose, tile, scale,
+    nonuniform_scale, mirror_h/v, quad_mirror, shift, fixed_crop,
+    rot+color, flip+color, transpose+color, gravity, extract_outline
+  - Learned conv: try_learned_conv(ks=1,3,5) with PyTorch + ternary snap
+  - Two-layer conv: Conv→ReLU→Conv(ks1=3,5, ks2=1)
+  - Result: +0 new tasks (all 62 remaining were too hard)
+```
+
+Result after all phases: 338/400 tasks, est 4197.5 points.
+
+#### How `the-2026-neurogolf-championship` Gets 288 Tasks (from `neurogolf-4200-solver`)
+
+This one has the richest **dataset source** collection — 24 Kaggle datasets:
+```
+Cross_Source: 227 ONNX       Task_Transformation: 266    Golf_Aura: 254
+ONNX_Solutions_v31: 252      Publi_Data: 206             Agent: 206
+Logic: 204                   Logic_for_ARC: 204          Yash_Submission: 172
+Yash_Submission_v1: 168      Claude_Golf: 160            Ashok_Submission: 160
+NeuroGolf1k_A: 158           NeuroGolf1k_B: 132
+TestGolf_S014-S203: 9× 207 each (task-specific strong models)
+Total: ~4632 pre-solved ONNX models across sources
+```
+
+After official validation: 288 unique tasks solved.
+Source breakdown: Cross_Source=169, Task_Transformation=55, ONNX_Solutions_v31=49, Golf_Aura=11.
+
+#### How `4200-v5-neurogolf-fix` Gets 341+ Tasks
+
+Blends from 5 ZIP sources:
+```
+SOURCE_ZIPS:
+  '1': neurogolf-2026-starter (335 models)
+  '2': neurogolf-2026-tiny-onnx-solver (338 models)  ← the blend notebook itself!
+  '5': infinitesimals (341 models)
+  '7': logic-decoder (338 models)
+  '8': neurogolf-2026-blended-341-tasks-lb-4215 (341 models)
+```
+
+Plus **5 hand-crafted "LLM Rescue" ONNX models** for tasks 076, 096, 118, 133, 264.
+Each is a "huge static graph" — a per-task ONNX network built by an LLM that embeds
+the entire set of known examples and builds a matching/dispatch circuit.
+
+#### The 6 Key Techniques They Have That We Lack
+
+**1. Opset 17 (NOT 10)**
+All top notebooks use `oh.make_opsetid('', 17)`. Opset 17 works fine on Kaggle.
+This enables:
+- `Slice` with negative steps (for flip/rotate — zero MACs, zero initializers)
+- `Pad` with dynamic pads
+- `ScatterND` for hash-based matchers
+- `Where` for conditional logic
+
+Their rot90 = `Crop → Transpose → Slice(reverse)` = **~0 cost**.
+Our rot90 = Gather with 900-element int64 index = **~12,663 cost**.
+**Switching to opset 17 alone would ~halve cost on all analytical solvers.**
+
+**2. Cheap Slice-based ONNX Builders (zero-cost transforms)**
+Instead of Gather-index models, they use:
+```python
+def make_rot90cw(h, w):
+    nodes = _crop('input', 'c', h, w)
+    nodes += [make_node('Transpose', ['c'], ['t'], perm=[0,1,3,2])]
+    nodes += _slice_reverse([3], [h], 't', 'output')  # Slice with step=-1
+    return _model(nodes, 'rot90cw')
+```
+No initializers, no Gather indices, no masks. Cost ≈ 0.
+
+**3. PyTorch Learned Conv with Ternary Snap**
+```python
+def try_learned_conv(train, all_pairs, kernel_size=1, steps=3000, lr=0.03, seeds=(0,7,42)):
+    for seed in seeds:
+        conv = nn.Conv2d(10, 10, ks, padding=ks//2, bias=False)
+        # Adam, 3000 steps, MSE loss
+        # Try both float weights AND ternary-snapped {-1, 0, 1}
+        for w_cand in [w_float, _ternary_snap(w_float)]:
+            model = make_conv_onnx(w_cand)
+            if verify_model(model, all_pairs):  # validates against train+test+arc-gen
+                candidates.append(model)
+```
+Key insight: ternary weights are much cheaper (fewer unique values = smaller model).
+
+**4. Two-Layer Conv (Conv→ReLU→Conv)**
+For nonlinear patterns that single-layer conv can't learn:
+```python
+net = Sequential(
+    Conv2d(10, hidden, ks1, padding=ks1//2, bias=False),
+    ReLU(),
+    Conv2d(hidden, 10, ks2, padding=ks2//2, bias=False),
+)
+```
+Tries ks1=3,5 with ks2=1, hidden=10. Both float and ternary-snapped versions tested.
+
+**5. Channel Reduction**
+When only 4-5 colors are used: `Conv1x1(10→N) → transform → Conv1x1(N→10)`.
+Fewer channels = smaller conv kernels = lower MACs = higher score per task.
+
+**6. LLM Rescue / Hash-Based Matchers**
+For tasks that no automated solver can handle, they build hand-crafted ONNX graphs:
+- **Task 118 (hash matcher)**: `MatMul(flatten(input), hash_weights) → Equal(hash, target_per_example) → ScatterND(delta)`. Hashes each input to a unique 2D vector, matches against all known examples, applies the stored diff.
+- **Task 096 (run-length + gap pattern detector)**: Builds a huge computation graph with depthwise convolutions to detect run lengths and gap patterns, then dispatches to the correct output.
+- **Task 076 (combinatorial matcher)**: Gathers non-zero positions, computes falling factorial polynomial to identify which known example matches, applies stored output template.
+- **Task 264 (3×3 shape detector)**: Uses 9 convolution kernels (3×3 shape masks) to detect which L/T/line shape is present, then dispatches to the correct pattern.
+
+These are the hardest tasks — the ones that need actual algorithmic reasoning encoded in ONNX.
+
+#### Can We Reach 4000+ WITHOUT Blending?
+
+**Short answer: Yes, but it's the hard path.**
+
+The 338 blended models were each independently solved by *someone's* solver. If we could
+make our own solver generate arc-gen-validated models for ~300 tasks, we'd match the blenders.
+
+**What's blocking us (breakdown of the ~250 tasks we solve locally but fail arc-gen):**
+
+| Category | Count | Why it Fails | Fix |
+|---|---|---|---|
+| lstsq overfitting (ks≥5) | ~170 | Underdetermined lstsq memorizes train, fails arc-gen | Train on arc-gen data (need GPU for PyTorch), or find smaller ks that generalizes |
+| lstsq overfitting (ks=1-3) | ~30 | Even small kernels can overfit with few examples | More arc-gen examples in fitting |
+| spatial_gather false positives | ~12 | Coincidental pixel alignments in train don't hold for arc-gen | Validate spatial_gather against arc-gen before accepting |
+| Variable diff-shape | ~40 | No static ONNX for input-dependent output shapes | Fundamentally unsolvable with static ONNX (need hash matchers) |
+
+**Realistic path to 3000+ without blending:**
+1. Switch to opset 17 → ~2x score per analytical task (~+200 pts)
+2. PyTorch learned conv on GPU with arc-gen fitting → ~+50-100 tasks
+3. Hash-based matchers for ~20 hard tasks → ~+300 pts
+4. Channel reduction → ~-20% cost across board (~+100 pts)
+5. Total estimate: ~150-200 validated tasks × ~12 avg score = ~2000-2500 pts
+
+**To actually reach 4000+, you'd need ~330+ validated tasks.** That requires either
+blending OR solving the hard algorithmic tasks (gravity, flood fill, counting, etc.)
+which need LLM-generated per-task ONNX graphs.
+
 ### High-Scoring Notebook Architecture (2026-04-24 analysis)
 
 The top notebooks (4200+ points) are **BLENDERS**, not solvers:
@@ -84,18 +253,17 @@ The top notebooks (4200+ points) are **BLENDERS**, not solvers:
 
 ### Cost Benchmarks
 
-| Model Type | Typical Cost | Score |
-|-----------|-------------|-------|
-| Identity | 0 | 25.0 |
-| Transpose (perm only) | 0 | 25.0 |
-| Color map (Gather, permutation) | 50 | 21.1 |
-| Color map (Conv 1×1) | 90,500 | 13.6 |
-| Analytical (gather-based) | 12,663 | 15.5 |
-| Shift (gather) | 57,663 | 14.0 |
-| Conv ks=1 | 814,590–814,662 | 11.4 |
-| Conv ks=5 | 4,589,390 | 9.7 |
-| Conv ks=11 | 11,105,390 | 8.8 |
-| Conv ks=29 | 66,129,390 | 7.0 |
+| Model Type | Typical Cost (ours, opset 10) | Their Cost (opset 17) | Score Diff |
+|-----------|------|------|------|
+| Identity | 0 | 0 | — |
+| Transpose | 36,000 (Gather-based) | ~0 (perm only) | +10 pts |
+| Rotation | ~165,663 (Gather+mask) | ~0 (Slice+Transpose) | +10 pts |
+| Flip | ~165,663 (Gather+mask) | ~0 (Slice reverse) | +10 pts |
+| Color map (Gather, permutation) | 50 | 50 | — |
+| Color map (Conv 1×1) | 90,500 | 90,500 | — |
+| Spatial gather | ~12,663 | ~12,663 | — |
+| Conv ks=1 | 814,590 | 814,590 | — |
+| Conv ks=5 | 4,589,390 | 4,589,390 | — |
 
 ### ARC-GEN Survival Rates
 
@@ -145,6 +313,30 @@ Analysis found 113 unsolved same-shape tasks where arc-gen uses IDENTICAL grid s
 
 These tasks have input-dependent output shapes. No static ONNX graph can produce different-sized outputs. The only approach: conv learns to place content in the right 30×30 region, masked by `ReduceSum(input)`. But this fails when output extends beyond input bounds or when the spatial mapping depends on content.
 
+### Hash-Based Matcher Architecture (from 4200-v5 notebook)
+
+For tasks that are impossible with conv/gather, the top notebooks build **per-task matcher networks**:
+
+```
+Architecture (task 118 example):
+  1. Flatten input: Reshape [1,10,30,30] → [1, 9000]
+  2. Hash: MatMul([1,9000], [9000,2]) → [1,2]  (random int weights [-7,+7])
+  3. For each known example i:
+     a. Equal(hash, target_hash_i) → bool match
+     b. Cast to float, ReduceSum → match_count
+     c. Equal(match_count, 2.0) → exact match
+     d. ScatterND(zero_grid, diff_indices_i, diff_values_i) → delta_i
+     e. Mul(delta_i, match_flag) → conditional_delta_i
+  4. Concat all conditional deltas → ReduceSum → total_delta
+  5. Add(input, total_delta) → output
+```
+
+This works because each input hashes to a unique 2D vector, so the network
+identifies which known example is present and applies the stored transformation.
+Cost is high but the model is guaranteed correct for all known examples.
+
+**Requirements**: opset 17 (ScatterND), all examples available at build time.
+
 ## Data Notes
 
 ### ARC-GEN File Format
@@ -162,11 +354,38 @@ Tasks are numbered 1-400 based on alphabetical sort of hex filenames in `ARC-AGI
 ### ARC-GEN Generator
 https://github.com/google/ARC-GEN — Can generate MORE examples per task for better fitting. Not yet explored.
 
+### Key Kaggle Public Datasets (from notebook analysis)
+
+These are the dataset sources that top solvers blend from:
+```
+limprog/neurogolf-blend/NeuroGolf_blend/Cross-Source      — 227 ONNX (biggest winner)
+karnakbaevarthur/neurogolf-2026-task-transformation-library — 269 ONNX
+sigmaborov/golf-aura                                       — 254 ONNX
+needless090/neurogolf-onnx-v31                             — 252 ONNX
+limprog/neurogolf-blend/NeuroGolf_blend/Publi_Data         — 206 ONNX
+sigmaborov/golf-solve-agent                                — 206 ONNX
+karnakbaevarthur/logic-for-each-arc-task                   — 204 ONNX
+yash9439/neurogolf-submission                              — 172 ONNX
+daphne4sg/claude-golf                                      — 160 ONNX
+hanifnoerrofiq/neurogolf1k                                 — 158+132 ONNX
+sigmaborov/test-golf (S_task014..S_task203)                — 9×207 ONNX (task-specific)
+```
+
+Key notebook submission.zip sources:
+```
+aliafzal9323/neurogolf-2026-tiny-onnx-solver  — 338 models (itself a mega-blend)
+sigmaborov/neurogolf-2026-starter             — 335 models
+jazivxt/infinitesimals                        — 341 models
+konbu17/neurogolf-2026-blended-341-tasks      — 341 models
+karnakbaevarthur/logic-decoder                — 338 models
+```
+
 ## Reference Notebooks (in repo as neurogolf-2026-solver-notebooks.zip)
 
 | Notebook | Est LB | Tasks Solved | Technique | Key Source Count |
 |----------|--------|-------------|-----------|-----------------|
 | neurogolf-2026-tiny-onnx-solver | ~4200 | 338 | Mega-blend 12+ zips | 203 from mega-agi-ensemble |
-| 4200-v5-neurogolf-fix | ~5725 | 341 | Same blend + 5 manual | 338 from zip_2 |
+| 4200-v5-neurogolf-fix | ~5725 | 341 | Same blend + 5 manual LLM rescue | 338 from zip_2 |
+| neurogolf-4200-solver | ~3600 | 288 | Own solver + 24 dataset sources | Cross_Source=169 |
 | the-2026-neurogolf-championship | ~3200 est | 288 | Own solver + blend | gravity, outline, composition |
-| neurogolf-logic-driven-ensembling | — | 401 | Pure ensembling | — |
+| neurogolf-logic-driven-ensembling | — | 352 | Pure ensembling (no solver) | 351 from 4275-submission |