own-solver/LEARNING.md

NeuroGolf Solver — Learning & History

This file accumulates everything learned across sessions. Read it to avoid repeating mistakes and to understand what techniques work. Newest entries first within each section.

Version History

Version	Date	Tasks (arc-gen validated)	Est LB	Key Changes
v5.2	2026-04-26	52 locally, REJECTED on Kaggle	~710 (local)	gravity.py (Task 78), mode.py (Task 129), edge.py (0 matches). Kaggle rejected submission — profiler/validation gap.
v5.1	2026-04-26	49	~604	Exp 3: PCA/SVD 0 PCR solves. Refactored conv.py composable primitives.
v5.0	2026-04-26	49	~604	Refactored to 16-file package, opset 17 (IR 8), Slice-based flip/rotate, lstsq crash fix
v4.3	2026-04-25	50	~670	Updated docs. NO code changes.
v4.2	2026-04-24	50	~670	PyTorch learned conv. Needs GPU.
v4.1	2026-04-24	50	~670	Color map Gather for permutations (+15 pts)
v4.0	2026-04-24	50	~656	ARC-GEN validation, new analytical solvers, static profiler, submission.csv
v3	2026-04-24	307 (local) / ~40 (LB)	501	concat_enhanced, varshape_spatial_gather, conv_var_diff
v2	prior	294 (local)	unknown	Spatial_gather, variable-shape conv, diff-shape conv
v1	prior	128	unknown	Conv solver only

Mistakes Log (DO NOT REPEAT)

2026-04-26: Agent replaced user's score_network (onnx_tool) with silent fallback — CAUSED KAGGLE REJECTION

• What: The v5 refactor created profiler.py with a _static_profile() fallback that runs when onnx_tool is not installed. The fallback is wrapped in a bare except: pass, so if onnx_tool fails on a model (dynamic shapes, unsupported ops, opset 17 issues), the code silently falls through to a crude static profiler that returns fake scores instead of surfacing the error.
• Result: User's v5.2 submission was rejected by Kaggle. The 49 previously-accepted tasks worked, but the 3 new models (gravity.py, edge.py, mode.py) likely failed onnx_tool.loadmodel() shape inference or profiling. The local static profiler returned numbers that looked valid, so the user had no warning before submitting.
• Root cause:
  1. User originally coded score_network to call neurogolf_utils.score_network() directly — which uses onnx_tool and surfaces errors properly.
  2. Agent's v5 refactor wrapped it in try/except: pass and added _static_profile() fallback.
  3. _static_profile() only counts Conv MACs (misses ReduceSum, Where, MatMul, etc.), only counts initializer bytes, and does NOT verify static shapes or check onnx_tool compatibility.
  4. The fallback hides failures — models that Kaggle's score_network would reject appear to score fine locally.
• The official validation pipeline (from neurogolf_utils.py):
  1. check_network(filename) — file size ≤ 1.44MB
  2. onnxruntime.InferenceSession(filename) — model loads
  3. verify_subset(session, examples) — correct outputs on all splits
  4. score_network(filename) — uses onnx_tool.loadmodel() → g.shape_infer() → g.profile() → checks g.valid_profile, banned ops (UPPERCASE), negative memory. Returns (None, None, None) if ANY of these fail → model is NOT READY for submission.
• What the static profiler gets wrong:
  • Only counts Conv MACs — gravity model has Conv+ReduceSum+Where+Greater+And+Not per step, all uncounted
  • Banned op check uses mixed-case {'Loop', 'Scan', ...} but Kaggle checks op_type.upper() against ["LOOP", "SCAN", ...]
  • No onnx.checker.check_model() call
  • No static shape verification
  • No onnx_tool compatibility check
• Rule: NEVER silently fall back to a weaker validator. If the official scoring tool fails on a model, that model MUST be treated as unsolved. Surface the error, don't hide it.
• Rule: NEVER change the user's validation pipeline without understanding what it does. The user's score_network call was correct — it used onnx_tool directly.

Fix Plan (must be done before next submission):

1. profiler.py: Remove silent fallback. If onnx_tool is available, use it. If it returns (None, None, None), the model is REJECTED (unsolved). If onnx_tool is not installed, print a loud WARNING that scores are approximate and may not match Kaggle.

2. validators.py: Add check_network() equivalent — file size check (already done), onnx.checker.check_model(), banned op scan (UPPERCASE comparison), static shape verification on all tensors.

3. solver_registry.py: After a model passes validate() (correct outputs), also run score_network() from profiler. If it returns (None, None, None) → treat model as failed, try next solver. This catches models that produce correct outputs but can't be scored by Kaggle.

4. main.py: --strict_size already stops on oversized files. Add --strict_score (default True) — stop if any solved model returns (None, None, None) from score_network().

5. Test on Kaggle notebook: Before submitting, run neurogolf_utils.verify_network() on ALL solved models in a Kaggle notebook. This is the ONLY way to be sure — local testing without onnx_tool cannot catch all failure modes.

2026-04-26: Agent put entire 1400-line codebase into a single file, repeatedly overwrote user's code

• What: When implementing v5 opset 17 changes, agent uploaded the entire solver as a single neurogolf_solver.py file — three times. Each upload overwrote the user's run_tasks, main, and W&B code that the agent couldn't read (the read tool truncates at ~1000 lines).
• Result: User's W&B logging code was deleted. User's run_tasks function was deleted. User had to point agent to a specific commit (3f3d372) to recover.
• Root cause: (1) Agent couldn't read the tail of the file due to tool truncation, so it rewrote the entire file from scratch instead of making surgical edits. (2) Agent prioritized "getting it done" over preserving existing working code.
• Rule: NEVER rewrite an entire file when you can't read all of it. Make surgical edits. NEVER destroy code you can't see.

2026-04-26: lstsq SVD non-convergence crash on task 313

• What: np.linalg.lstsq(P, T_oh, rcond=None) raised LinAlgError: SVD did not converge.
• Fix: Wrapped lstsq in try/except (LinAlgError, ValueError): return None in all call sites.
• Rule: EVERY lstsq call must be guarded.

2026-04-26: ReduceSum axes attribute invalid in opset 17

• What: Code used axes as attribute instead of tensor input (opset 13+ requirement).
• Fix: Created _build_reducesum() helper with axes as int64 initializer tensor.
• Rule: Audit ALL operators for breaking API changes when changing opset.

2026-04-26: Fake excluded tasks {21, 55, 80, 184, 202, 366}

• What: Agent added 6 "excluded" tasks to constants.py. There are NO excluded tasks — all 400 count.
• Fix: EXCLUDED_TASKS = set()
• Rule: All 400 tasks must be attempted. Do not invent exclusions.

2026-04-26: est_lb inflated by adding unsolved×1.0

• What: est_lb = total_score + unsolved_count * 1.0 double-counted unsolved task scores.
• Fix: Report only solved score. Unsolved tasks get 1.0 on Kaggle automatically.
• Rule: est_lb should reflect only what we earn from solved tasks.

2026-04-25: Agent wrote 1919 lines of v5 code WITHOUT running full 400-task arc-gen validation

• Rule: NEVER mark a feature as done until validated against full arc-gen.

2026-04-25: Agent created version-named file violating project convention

• Rule: No version numbers in filenames.

2026-04-25: Agent claimed LOOCV Ridge tuning would improve arc-gen without evidence

• Rule: Theory from papers is NOT proof. Test first.

2026-04-25: Agent misrepresented user's intent — BLENDING is NOT the strategy

• Rule: LEARNING.md reflects USER'S strategy.

2026-04-25: Composition detectors, channel reduction wrapper — untested dead code

• Rule: Only add a solver if it demonstrably solves ≥1 task.

2026-04-25: Agent delivered untested code and asked user to validate it

• Rule: VALIDATE FIRST, DELIVER SECOND.

2026-04-24: PyTorch 2-layer conv — fits training but doesn't generalize to arc-gen

2026-04-24: Arc-gen in lstsq fitting exposes overfitting

2026-04-24: CuPy/GPU for lstsq — DOES NOT HELP

2026-04-24: Channel Gather for non-permutation color maps — WRONG OUTPUT

2026-04-24: ARC-GEN not loaded — THE #1 SCORE KILLER (v3→v4 fix)

2026-04-24: s_flip used GatherElements — OPSET 11 BUG

2026-04-24: score_network fallback returned (0,0,0)

2026-04-24: Ignored EXCLUDED tasks

Competitive Intelligence

What Others Do (For Awareness Only — We Do NOT Blend)

Why top notebooks score 4000+ and we score ~670

Top notebooks are BLENDERS — they assemble pre-solved ONNX models from public sources.

Our strategy: Build our own solver. No blending. No public datasets.

The 6 Key Techniques They Have That We Lack

1. Opset 17 — ✅ DONE in v5. Slice+Transpose for near-zero cost transforms.
2. Channel Reduction Wrapper — 🔲 Not yet. Conv1x1(10→N) → transform → Conv1x1(N→10).
3. Composition Detectors — 🔲 Not yet. Need to scan 400 tasks to find actual instances first.
4. Best-of-N Model Selection — 🔲 Not yet. Generate 20+ candidates, keep cheapest valid.
5. ONNX Optimizer Pass — 🔲 Not yet. onnxoptimizer.optimize() for dead-code elimination.
6. LLM Rescue — 🔲 Not yet. Per-task ONNX graphs for algorithmic tasks (gravity, outline, etc.)

Deep Research Findings

Exp 3: PCA/Truncated SVD Before lstsq — FULL RESULTS (2026-04-26)

Implementation: Refactored conv.py into composable primitives:

• _build_patch_matrix(exs, ks, bias, full_30) → P, T, T_oh
• _solve_weights(P, T, T_oh) → WT via raw lstsq
• _solve_weights_pcr(P, T, T_oh, thresholds) → WT via PCA regression
• _extract_weights(WT, ks, bias) → Wconv, B for ONNX

Full 400-task run: 0 PCR solves, 0 regressions.

Conclusion: Architecture mismatch, not regularization. Regularization experiments exhausted.

ONNX Opset 17 Migration Notes (2026-04-26)

Breaking changes from opset 10:

Operator	Opset 10	Opset 13+ (incl. 17)
ReduceSum	axes as attribute	axes as tensor input
ReduceMean	axes as attribute	axes as tensor input
Pad	pads as attribute	pads as tensor input (since opset 11)
Slice	no steps input	steps added as 5th tensor input

Official Scoring Pipeline (from neurogolf_utils.py) — READ BEFORE CODING

# This is what Kaggle runs. Our validator MUST match this.
def check_network(filename):
    # 1. File must exist
    # 2. File size ≤ 1.44MB (1.44 * 1024 * 1024 bytes)
    
def score_network(filename):
    # Uses onnx_tool.loadmodel() → shape_infer() → profile()
    # Checks: g.valid_profile (static shapes required)
    # Checks: op_type.upper() not in ["LOOP","SCAN","NONZERO","UNIQUE","SCRIPT","FUNCTION"]
    # Checks: g.nodemap[key].memory >= 0
    # Returns (macs, memory, params) or (None, None, None) on ANY failure
    # (None, None, None) = "Your network performance could not be measured" = REJECTED

def verify_network(network, task_num, examples):
    # 1. onnx.save → check_network (size)
    # 2. InferenceSession (loads ok?)
    # 3. verify_subset on train+test (correct outputs?)
    # 4. verify_subset on arc-gen (correct outputs?)
    # 5. score_network (scoreable by onnx_tool?)
    # ALL must pass for "IS READY for submission"

What Has NOT Worked

Technique	Result	Why
PCA/Truncated SVD (Exp 3)	0/400 PCR solves	Signal in noise dims; unsolved tasks = architecture mismatch
Silent profiler fallback	Kaggle rejection	Hides onnx_tool failures, returns fake scores
Ridge/LOOCV λ	Fails arc-gen	Catastrophic, not benign overfitting
Skip ks=5,7,9 (Exp 1)	Hurts 2 tasks	Some tasks genuinely need interpolation-regime ks
CuPy GPU lstsq	OOM + same speed	O(n³) SVD bottleneck
PyTorch 2-layer (no arc-gen)	0/30 arc-gen pass	Memorizes training

Technical Notes

ARC-AGI Task Statistics

• 400 tasks total. NO excluded tasks — all 400 count.
• ~25 analytical tasks, ~25 conv tasks survive arc-gen, ~350 unsolved

Score Calculation (official, from neurogolf_utils.py)

# Uses onnx_tool for exact MACs/memory/params — NOT our static profiler
macs, memory, params = score_network(filename)  # onnx_tool based
points = max(1.0, 25.0 - math.log(macs + memory + params))

Session Notes for Future Agents

Before touching code:

1. Read this file (LEARNING.md) — all the way through
2. Read SKILL.md — especially "Development Methodology" and "Submission Checklist"
3. Read TODO.md — check experiment log and research queue
4. Run the current solver on 20-50 tasks to establish baseline
5. Only then: design experiment, implement, validate, compare

Code structure (v5.2):

• The solver is a Python package at neurogolf_solver/
• Run with python -m neurogolf_solver.main [args]
• Solvers in separate files: analytical.py, geometric.py, tiling.py, conv.py, gravity.py, edge.py, mode.py
• Edit individual files surgically — NEVER rewrite the whole package
• The legacy neurogolf_solver.py at root is v4, kept for reference — do NOT edit it

CRITICAL: Scoring & Validation:

• The ONLY reliable scoring is neurogolf_utils.score_network() which uses onnx_tool
• profiler.py's _static_profile() is a fallback that DOES NOT match Kaggle scoring
• Before submitting: run neurogolf_utils.verify_network() on ALL solved models in a Kaggle notebook
• If score_network returns (None, None, None), the model is REJECTED — do not submit it

Before claiming a feature works:

• Must pass arc-gen on ≥20 tasks (or full 400 if cheap)
• Must pass neurogolf_utils.verify_network() — not just our own validate()
• Must include A/B comparison

Before uploading code:

• Must have run full 400-task arc-gen validation
• Must confirm total score ≥ previous best
• NEVER change the scoring/validation pipeline without understanding what it does