LEARNING.md: log profiler.py silent fallback mistake + Kaggle rejection + fix plan

LEARNING.md

@@ -6,56 +6,99 @@
 
 | Version | Date | Tasks (arc-gen validated) | Est LB | Key Changes |
 |---------|------|--------------------------|--------|-------------|
-| **v5.1** | **2026-04-26** | **49** | **~603.6** | Exp 3: PCA/SVD tested on 400 tasks, 0 PCR solves. Refactored conv.py into composable primitives. PCR fallback added (deferred 2nd pass). No regressions. |
-| v5.0 | 2026-04-26 | 49 | ~603.6 | Refactored to 16-file package, opset 17 (IR 8), Slice-based flip/rotate (0 MACs), tensor-based Pad & ReduceSum, lstsq crash fix |
-| v4.3 | 2026-04-25 | 50 | ~670 | Updated TODO.md + SKILL.md + LEARNING.md with closed-loop methodology. NO code changes. |
-| v4.2 | 2026-04-24 | 50 | ~670 | Added PyTorch learned conv (single+two-layer, multi-seed, ternary snap). Needs GPU. |
+| **v5.2** | **2026-04-26** | **52 locally, REJECTED on Kaggle** | **~710 (local)** | gravity.py (Task 78), mode.py (Task 129), edge.py (0 matches). **Kaggle rejected submission — profiler/validation gap.** |
+| v5.1 | 2026-04-26 | 49 | ~604 | Exp 3: PCA/SVD 0 PCR solves. Refactored conv.py composable primitives. |
+| v5.0 | 2026-04-26 | 49 | ~604 | Refactored to 16-file package, opset 17 (IR 8), Slice-based flip/rotate, lstsq crash fix |
+| v4.3 | 2026-04-25 | 50 | ~670 | Updated docs. NO code changes. |
+| v4.2 | 2026-04-24 | 50 | ~670 | PyTorch learned conv. Needs GPU. |
 | v4.1 | 2026-04-24 | 50 | ~670 | Color map Gather for permutations (+15 pts) |
-| v4.0 | 2026-04-24 | 50 | ~656 | ARC-GEN validation, new analytical solvers, s_flip fix, static profiler, submission.csv |
-| v3 | 2026-04-24 | 307 (local) / ~40 (LB) | 501 | Added concat_enhanced, varshape_spatial_gather, conv_var_diff |
+| v4.0 | 2026-04-24 | 50 | ~656 | ARC-GEN validation, new analytical solvers, static profiler, submission.csv |
+| v3 | 2026-04-24 | 307 (local) / ~40 (LB) | 501 | concat_enhanced, varshape_spatial_gather, conv_var_diff |
 | v2 | prior | 294 (local) | unknown | Spatial_gather, variable-shape conv, diff-shape conv |
 | v1 | prior | 128 | unknown | Conv solver only |
 
 ## Mistakes Log (DO NOT REPEAT)
 
+### 2026-04-26: Agent replaced user's score_network (onnx_tool) with silent fallback — CAUSED KAGGLE REJECTION
+
+- **What**: The v5 refactor created `profiler.py` with a `_static_profile()` fallback that runs when `onnx_tool` is not installed. The fallback is wrapped in a bare `except: pass`, so if `onnx_tool` fails on a model (dynamic shapes, unsupported ops, opset 17 issues), the code **silently** falls through to a crude static profiler that returns fake scores instead of surfacing the error.
+- **Result**: User's v5.2 submission was **rejected by Kaggle**. The 49 previously-accepted tasks worked, but the 3 new models (gravity.py, edge.py, mode.py) likely failed `onnx_tool.loadmodel()` shape inference or profiling. The local static profiler returned numbers that looked valid, so the user had no warning before submitting.
+- **Root cause**: 
+  1. User originally coded `score_network` to call `neurogolf_utils.score_network()` directly — which uses `onnx_tool` and surfaces errors properly.
+  2. Agent's v5 refactor wrapped it in `try/except: pass` and added `_static_profile()` fallback.
+  3. `_static_profile()` only counts Conv MACs (misses ReduceSum, Where, MatMul, etc.), only counts initializer bytes, and does NOT verify static shapes or check `onnx_tool` compatibility.
+  4. The fallback **hides failures** — models that Kaggle's `score_network` would reject appear to score fine locally.
+- **The official validation pipeline** (from `neurogolf_utils.py`):
+  1. `check_network(filename)` — file size ≤ 1.44MB
+  2. `onnxruntime.InferenceSession(filename)` — model loads
+  3. `verify_subset(session, examples)` — correct outputs on all splits
+  4. `score_network(filename)` — uses `onnx_tool.loadmodel()` → `g.shape_infer()` → `g.profile()` → checks `g.valid_profile`, banned ops (UPPERCASE), negative memory. Returns `(None, None, None)` if ANY of these fail → model is NOT READY for submission.
+- **What the static profiler gets wrong**:
+  - Only counts Conv MACs — gravity model has Conv+ReduceSum+Where+Greater+And+Not per step, all uncounted
+  - Banned op check uses mixed-case `{'Loop', 'Scan', ...}` but Kaggle checks `op_type.upper()` against `["LOOP", "SCAN", ...]`
+  - No `onnx.checker.check_model()` call
+  - No static shape verification
+  - No `onnx_tool` compatibility check
+- **Rule**: NEVER silently fall back to a weaker validator. If the official scoring tool fails on a model, that model MUST be treated as unsolved. Surface the error, don't hide it.
+- **Rule**: NEVER change the user's validation pipeline without understanding what it does. The user's `score_network` call was correct — it used `onnx_tool` directly.
+
+### Fix Plan (must be done before next submission):
+
+1. **profiler.py**: Remove silent fallback. If `onnx_tool` is available, use it. If it returns `(None, None, None)`, the model is REJECTED (unsolved). If `onnx_tool` is not installed, print a loud WARNING that scores are approximate and may not match Kaggle.
+
+2. **validators.py**: Add `check_network()` equivalent — file size check (already done), `onnx.checker.check_model()`, banned op scan (UPPERCASE comparison), static shape verification on all tensors.
+
+3. **solver_registry.py**: After a model passes `validate()` (correct outputs), also run `score_network()` from profiler. If it returns `(None, None, None)` → treat model as failed, try next solver. This catches models that produce correct outputs but can't be scored by Kaggle.
+
+4. **main.py**: `--strict_size` already stops on oversized files. Add `--strict_score` (default True) — stop if any solved model returns `(None, None, None)` from `score_network()`.
+
+5. **Test on Kaggle notebook**: Before submitting, run `neurogolf_utils.verify_network()` on ALL solved models in a Kaggle notebook. This is the ONLY way to be sure — local testing without `onnx_tool` cannot catch all failure modes.
+
 ### 2026-04-26: Agent put entire 1400-line codebase into a single file, repeatedly overwrote user's code
 
 - **What**: When implementing v5 opset 17 changes, agent uploaded the entire solver as a single `neurogolf_solver.py` file — three times. Each upload overwrote the user's `run_tasks`, `main`, and W&B code that the agent couldn't read (the read tool truncates at ~1000 lines).
 - **Result**: User's W&B logging code was deleted. User's `run_tasks` function was deleted. User had to point agent to a specific commit (3f3d372) to recover.
-- **Root cause**: (1) Agent couldn't read the tail of the file due to tool truncation, so it rewrote the entire file from scratch instead of making surgical edits. (2) No Python best practice says "put all code in one file" — the opposite is true. (3) Agent prioritized "getting it done" over preserving existing working code.
-- **Rule**: NEVER rewrite an entire file when you can't read all of it. Use the `edit` tool for targeted string replacements. If the file is too large to read, split it into smaller files FIRST (which is what the user ultimately had to specify). NEVER destroy code you can't see.
+- **Root cause**: (1) Agent couldn't read the tail of the file due to tool truncation, so it rewrote the entire file from scratch instead of making surgical edits. (2) Agent prioritized "getting it done" over preserving existing working code.
+- **Rule**: NEVER rewrite an entire file when you can't read all of it. Make surgical edits. NEVER destroy code you can't see.
 
 ### 2026-04-26: lstsq SVD non-convergence crash on task 313
 
-- **What**: `np.linalg.lstsq(P, T_oh, rcond=None)` raised `LinAlgError: SVD did not converge` during `solve_conv_variable` for task 313.
-- **Result**: Entire solver crashed, no further tasks processed.
-- **Root cause**: The `_lstsq_conv` function had no try/except around the lstsq call. `solve_conv_var_diff` already had one, but `_lstsq_conv` (used by `solve_conv_fixed` and `solve_conv_variable`) did not.
-- **Fix**: Wrapped lstsq in `try/except (np.linalg.LinAlgError, ValueError): return None` in all three call sites (`_lstsq_conv`, `solve_conv_diffshape` inline lstsq).
-- **Rule**: EVERY lstsq call must be guarded. SVD non-convergence is rare but real, especially for ill-conditioned patch matrices from unusual grid patterns.
+- **What**: `np.linalg.lstsq(P, T_oh, rcond=None)` raised `LinAlgError: SVD did not converge`.
+- **Fix**: Wrapped lstsq in `try/except (LinAlgError, ValueError): return None` in all call sites.
+- **Rule**: EVERY lstsq call must be guarded.
 
 ### 2026-04-26: ReduceSum axes attribute invalid in opset 17
 
-- **What**: Code used `ReduceSum(['data'], ['output'], axes=[1,2,3], keepdims=1)` which puts axes as a node attribute. In opset 13+, axes must be a tensor input, not an attribute.
-- **Result**: Models would fail ONNX checker validation and potentially fail on Kaggle inference server.
-- **Fix**: Created `_build_reducesum()` helper that adds axes as an int64 initializer tensor and passes it as the 2nd input to ReduceSum. Applied to `s_constant` (axes=[1,2,3]), `solve_conv_variable` (axes=[1]), `solve_conv_var_diff` (axes=[1]).
-- **Rule**: When changing opset version, audit ALL operators for breaking API changes. Key opset 13 changes: ReduceSum, ReduceMean, ReduceMax all moved axes from attribute to tensor input. Pad moved pads from attribute to tensor input at opset 11. Slice added steps input at opset 13.
+- **What**: Code used axes as attribute instead of tensor input (opset 13+ requirement).
+- **Fix**: Created `_build_reducesum()` helper with axes as int64 initializer tensor.
+- **Rule**: Audit ALL operators for breaking API changes when changing opset.
+
+### 2026-04-26: Fake excluded tasks {21, 55, 80, 184, 202, 366}
+
+- **What**: Agent added 6 "excluded" tasks to constants.py. There are NO excluded tasks — all 400 count.
+- **Fix**: `EXCLUDED_TASKS = set()`
+- **Rule**: All 400 tasks must be attempted. Do not invent exclusions.
+
+### 2026-04-26: est_lb inflated by adding unsolved×1.0
+
+- **What**: `est_lb = total_score + unsolved_count * 1.0` double-counted unsolved task scores.
+- **Fix**: Report only solved score. Unsolved tasks get 1.0 on Kaggle automatically.
+- **Rule**: est_lb should reflect only what we earn from solved tasks.
 
 ### 2026-04-25: Agent wrote 1919 lines of v5 code WITHOUT running full 400-task arc-gen validation
-- **What**: Generated neurogolf_solver_v5.py with opset 17 Slice-based transforms, LOOCV Ridge tuning, stride_tricks, composition detectors, channel reduction wrapper — claimed all features were "working" in the docstring and README
-- **Result**: Uploaded to repo, overwrote neurogolf_solver.py. Tested only 10 individual tasks manually. 3/10 FAILED arc-gen validation (tasks 4, 6, 241 conv models). NEVER ran full 400 with arc-gen validation. LOOCV Ridge theory in code was never tested against actual data. Estimated LB score is UNKNOWN — cannot claim improvement over v4's proven ~670.
-- **Rule**: NEVER mark a feature as done until it is validated against full arc-gen data on a representative sample of tasks.
+- **Rule**: NEVER mark a feature as done until validated against full arc-gen.
 
-### 2026-04-25: Agent created version-named file (neurogolf_solver_v5.py) violating project convention
-- **Rule**: No version numbers in filenames. Use git commits for version tracking.
+### 2026-04-25: Agent created version-named file violating project convention
+- **Rule**: No version numbers in filenames.
 
-### 2026-04-25: Agent claimed LOOCV Ridge tuning would improve arc-gen survival without evidence
-- **Rule**: Theory from papers is NOT proof for our specific data. Run A/B experiments first.
+### 2026-04-25: Agent claimed LOOCV Ridge tuning would improve arc-gen without evidence
+- **Rule**: Theory from papers is NOT proof. Test first.
 
-### 2026-04-25: Agent misrepresented user's intent — BLENDING is NOT the user's strategy
-- **Rule**: LEARNING.md must reflect the USER'S strategy. Competitive intelligence goes in "What Others Do" section only.
+### 2026-04-25: Agent misrepresented user's intent — BLENDING is NOT the strategy
+- **Rule**: LEARNING.md reflects USER'S strategy.
 
 ### 2026-04-25: Composition detectors, channel reduction wrapper — untested dead code
-- **Rule**: Only add a solver if it demonstrably solves ≥1 task. Delete dead code.
+- **Rule**: Only add a solver if it demonstrably solves ≥1 task.
 
 ### 2026-04-25: Agent delivered untested code and asked user to validate it
 - **Rule**: VALIDATE FIRST, DELIVER SECOND.
@@ -98,56 +141,9 @@ Top notebooks are **BLENDERS** — they assemble pre-solved ONNX models from pub
 - `_solve_weights_pcr(P, T, T_oh, thresholds)` → WT via PCA regression
 - `_extract_weights(WT, ks, bias)` → Wconv, B for ONNX
 
-All 4 conv solvers use deferred 2-pass design:
-- Pass 1: raw lstsq at all ks (identical behavior to baseline)
-- Pass 2: PCR on ks values where lstsq fit train but failed arc-gen validation
-
-**PCR algorithm:**
-```python
-U, s, Vt = SVD(P)
-cumvar = cumsum(s²) / sum(s²)
-for thresh in [0.999, 0.99, 0.95]:
-    k = searchsorted(cumvar, thresh) + 1
-    k = max(k, 5)
-    P_red = U[:,:k] * s[:k]  # project to top-k components
-    w_red = lstsq(P_red, T_oh)
-    w_full = Vt[:k].T @ w_red  # map back to full p-dim
-```
-
-**Diagnostic results on 25 solved conv tasks:**
+**Full 400-task run:** 0 PCR solves, 0 regressions.
 
-| p/n regime | # Tasks | PCR train-fit? | Arc-gen impact |
-|------------|---------|----------------|----------------|
-| < 0.5 | 17 | Yes (0.99 thresh) | Already 100% — no improvement |
-| 0.5-1.0 | 0 | N/A | N/A |
-| > 1.0 | 8 | 4/8 fail at ALL thresholds | PCR removes signal-carrying dimensions |
-
-Key observation: at p/n > 1.0, the "noise" dimensions PCA removes actually carry part of the training signal. Truncation causes train_fail — the model can't even fit training data after dimensionality reduction.
-
-**Diagnostic results on 345 unsolved tasks (same-shape, ks≤9):**
-
-- Only **10 tasks** have any ks where lstsq fits training
-- PCR improves arc-gen on **4 tasks** but none reach 100%:
-  - Task 32: 87.5% → 94.9% (+7.4%)
-  - Task 389: 87.2% → 95.7% (+8.5%)  
-  - Task 129: 59.6% → 63.0% (+3.4%)
-  - Task 229: 57.0% → 60.0% (+3.0%)
-
-**Full 400-task run:** 0 PCR solves, 0 regressions, 49/49 baseline tasks preserved.
-
-**Why it failed:** Three distinct failure modes:
-1. **p/n < 0.5 (17/25 solved tasks):** lstsq already generalizes perfectly. PCR is unnecessary overhead.
-2. **p/n > 1.0 (8/25 solved tasks):** Signal requires ALL dimensions. PCA truncation destroys the training fit. The minimum-norm solution from lstsq distributes weight across ALL singular vectors, and removing any causes prediction errors.
-3. **335/345 unsolved tasks:** No ks fits training at all. The task requires non-local operations (flood fill, mode counting, conditional logic) that conv can't represent regardless of regularization.
-
-**Conclusion:** The "overfitting hypothesis" from Nakkiran 2019 was correct in theory but inapplicable. The tasks where conv fails arc-gen fail because conv is architecturally wrong, not because of bad regularization. Regularization experiments (Ridge, PCA, skip-ks) are exhausted.
-
-### lstsq Conv Research (2026-04-25)
-
-**Key Finding: Our overfitting is CATASTROPHIC, not benign.**
-- Bartlett et al. benign overfitting requires high effective rank of covariance. Our one-hot patches have LOW effective rank.
-- Double descent peak at ks=5,7,9 (p ≈ n).
-- Ridge predicted to fail; Lasso (ℓ₁) theoretically better for sparse signals.
+**Conclusion:** Architecture mismatch, not regularization. Regularization experiments exhausted.
 
 ### ONNX Opset 17 Migration Notes (2026-04-26)
 
@@ -158,61 +154,56 @@ Key observation: at p/n > 1.0, the "noise" dimensions PCA removes actually carry
 | ReduceMean | axes as **attribute** | axes as **tensor input** |
 | Pad | pads as **attribute** | pads as **tensor input** (since opset 11) |
 | Slice | no steps input | **steps** added as 5th tensor input |
-| Conv | pads as attribute | pads as attribute ✅ (unchanged) |
-| Transpose | perm as attribute | perm as attribute ✅ (unchanged) |
-| Gather | unchanged | unchanged ✅ |
 
-**IR version**: Opset 17 requires IR ≤ 8. We use IR=8.
+### Official Scoring Pipeline (from neurogolf_utils.py) — READ BEFORE CODING
 
-**Slice(step=-1) for reversing:**
-- `starts=[dim-1], ends=[INT64_MIN], axes=[ax], steps=[-1]` — reverses entire axis
-- INT64_MIN as end sentinel (not -1, which means dim-1 in ONNX)
-- Zero MACs, zero params, near-zero memory (just 4 int64 scalars)
+```python
+# This is what Kaggle runs. Our validator MUST match this.
+def check_network(filename):
+    # 1. File must exist
+    # 2. File size ≤ 1.44MB (1.44 * 1024 * 1024 bytes)
+    
+def score_network(filename):
+    # Uses onnx_tool.loadmodel() → shape_infer() → profile()
+    # Checks: g.valid_profile (static shapes required)
+    # Checks: op_type.upper() not in ["LOOP","SCAN","NONZERO","UNIQUE","SCRIPT","FUNCTION"]
+    # Checks: g.nodemap[key].memory >= 0
+    # Returns (macs, memory, params) or (None, None, None) on ANY failure
+    # (None, None, None) = "Your network performance could not be measured" = REJECTED
+
+def verify_network(network, task_num, examples):
+    # 1. onnx.save → check_network (size)
+    # 2. InferenceSession (loads ok?)
+    # 3. verify_subset on train+test (correct outputs?)
+    # 4. verify_subset on arc-gen (correct outputs?)
+    # 5. score_network (scoreable by onnx_tool?)
+    # ALL must pass for "IS READY for submission"
+```
 
 ## What Has NOT Worked
 
 | Technique | Result | Why |
 |-----------|--------|-----|
 | **PCA/Truncated SVD (Exp 3)** | **0/400 PCR solves** | **Signal in noise dims; unsolved tasks = architecture mismatch** |
+| **Silent profiler fallback** | **Kaggle rejection** | **Hides onnx_tool failures, returns fake scores** |
 | Ridge/LOOCV λ | Fails arc-gen | Catastrophic, not benign overfitting |
 | Skip ks=5,7,9 (Exp 1) | Hurts 2 tasks | Some tasks genuinely need interpolation-regime ks |
 | CuPy GPU lstsq | OOM + same speed | O(n³) SVD bottleneck |
 | PyTorch 2-layer (no arc-gen) | 0/30 arc-gen pass | Memorizes training |
-| Composition detectors | No tasks found | May not exist in dataset |
-| Channel reduction wrapper | Never executed | Disabled due to Gather incompatibility |
 
 ## Technical Notes
 
 ### ARC-AGI Task Statistics
-- 400 tasks total, 6 excluded: {21, 55, 80, 184, 202, 366}
-- ~25 analytical tasks, ~25 conv tasks that survive arc-gen, ~350 unsolved
+- 400 tasks total. NO excluded tasks — all 400 count.
+- ~25 analytical tasks, ~25 conv tasks survive arc-gen, ~350 unsolved
 
-### Score Calculation
+### Score Calculation (official, from neurogolf_utils.py)
 ```python
-score = max(1.0, 25.0 - math.log(macs + memory_bytes + params))
+# Uses onnx_tool for exact MACs/memory/params — NOT our static profiler
+macs, memory, params = score_network(filename)  # onnx_tool based
+points = max(1.0, 25.0 - math.log(macs + memory + params))
 ```
 
-### Conv Solver SVD Spectrum Analysis (Exp 3 data, 2026-04-26)
-
-Effective rank at 99% variance for solved conv tasks:
-| Task | ks | n patches | p features | p/n | eff_rank_99 | arc-gen acc |
-|------|----|-----------|-----------:|----:|------------:|------------:|
-| 171 | 3 | 799 | 90 | 0.11 | 5 | 100% |
-| 120 | 3 | 4103 | 90 | 0.02 | 22 | 100% |
-| 305 | 9 | 3584 | 810 | 0.23 | 416 | 100% |
-| 60 | 11 | 715 | 1210 | 1.69 | 245 | 98.5% |
-| 136 | 15 | 1400 | 2250 | 1.61 | 237 | 99.6% |
-| 322 | 5 | 126 | 250 | 1.98 | 100 | 97.0% |
-
-Key pattern: tasks with p/n < 0.5 → 100% arc-gen. Tasks with p/n > 1.0 → 97-99.6% arc-gen. The 0.4-3% error is the interpolation-regime overfitting, but it still passes validation.
-
-### Lstsq Matrix Sizes (for reference)
-| Grid | Examples | Patches (n) | ks=3 (p=90) | ks=7 (p=490) | ks=29 (p=8410) |
-|------|----------|-------------|-------------|--------------|----------------|
-| 7×7  | 4        | 196         | 196×90      | **196×490 (under!)** | 196×8410 |
-| 12×12| 6        | 576         | 576×90      | 576×490      | 576×8410 |
-| 21×21| 16       | 7056        | 7056×90     | 7056×490     | **7056×8410** |
-
 ## Session Notes for Future Agents
 
 **Before touching code:**
@@ -222,25 +213,25 @@ Key pattern: tasks with p/n < 0.5 → 100% arc-gen. Tasks with p/n > 1.0 → 97-
 4. Run the current solver on 20-50 tasks to establish baseline
 5. Only then: design experiment, implement, validate, compare
 
-**Code structure (v5.1):**
+**Code structure (v5.2):**
 - The solver is a Python package at `neurogolf_solver/`
 - Run with `python -m neurogolf_solver.main [args]`
-- **conv.py** now has composable primitives: `_build_patch_matrix` + `_solve_weights` + `_extract_weights`
-- To add new fitting methods: implement `_solve_weights_XXX(P, T, T_oh)` returning WT or None
+- Solvers in separate files: `analytical.py`, `geometric.py`, `tiling.py`, `conv.py`, `gravity.py`, `edge.py`, `mode.py`
 - Edit individual files surgically — NEVER rewrite the whole package
 - The legacy `neurogolf_solver.py` at root is v4, kept for reference — do NOT edit it
 
+**CRITICAL: Scoring & Validation:**
+- The ONLY reliable scoring is `neurogolf_utils.score_network()` which uses `onnx_tool`
+- `profiler.py`'s `_static_profile()` is a fallback that DOES NOT match Kaggle scoring
+- Before submitting: run `neurogolf_utils.verify_network()` on ALL solved models in a Kaggle notebook
+- If `score_network` returns `(None, None, None)`, the model is REJECTED — do not submit it
+
 **Before claiming a feature works:**
 - Must pass arc-gen on ≥20 tasks (or full 400 if cheap)
-- Must show >10% improvement in arc-gen survival rate OR total score
+- Must pass `neurogolf_utils.verify_network()` — not just our own validate()
 - Must include A/B comparison
 
 **Before uploading code:**
 - Must have run full 400-task arc-gen validation
 - Must confirm total score ≥ previous best
-
-**What to focus on next (post Exp 3):**
-1. **Phase 3: New solver types** — hash matchers, pattern detectors, LLM rescue
-2. **Phase 1a: Opset 17 analytical conversions** — reduce cost on existing 24 analytical tasks
-3. **Phase 4: ONNX optimizer** — reduce cost on all 49 solved tasks
-4. Lasso (Exp 5) is low priority — only 10 unsolved tasks even have lstsq fits, ceiling is very low
+- NEVER change the scoring/validation pipeline without understanding what it does