Upload 47 files

LEARNING.md

@@ -0,0 +1,20 @@
+# Lessons Learned (Critical Notes)
+
+## Core lessons
+- **Log the candidate arrays early.** Without the actual candidate field, debugging accepted candidates is guesswork. Always include a small, quantized snapshot in logs for reproducibility.
+- **Avoid side effects inside the solver run.** External logging (W&B) must be optional and executed outside the core run to avoid recursion and hidden failures.
+- **Transforms must produce visible edits.** If transforms are no‑ops for typical inputs, the beam cannot reduce residue. Unit‑test transforms with small inputs.
+- **Coerce logged gate values to booleans.** Strings like `"True"` break automated gate analysis and mislead debugging.
+- **Quantize and resize consistently.** Residue must be computed on the same quantized and tiled array the beam evaluates (use `np.rint` and `tile_transform`).
+
+## Debugging checklist
+1. Confirm `experiments/` contains `*_phi_best.npy` and `*_logs.json`.  
+2. Run `experiments/postprocess_logs.py` to coerce gates and attach candidate snapshot.  
+3. Inspect `logs[0]` for `candidate_array` and recomputed residue.  
+4. Run `itt_solver.tests.run_atomic_effects()` to verify transforms change the input.  
+5. Run a relaxed smoke beam (lock_coeff=0, max_fraction=1.0, beam_width≥6) and inspect `sigmas`.
+
+## Practical tips
+- After editing package files, **clear Python module cache** or restart the interpreter.  
+- Keep `candidate_array` optional in logs to avoid huge files; include it for debugging runs.  
+- Use small, deterministic transforms for initial debugging (Rotate, Reflect, ShiftedTile).

README.md

@@ -0,0 +1,21 @@
+## Pre‑Emergence Mechanics Framework (PEMF) — ARC‑AGI
+
+Short summary  
+The Pre‑Emergence Mechanics Framework (PEMF) frames ARC tasks as a boundary‑constrained field problem solved by minimizing irreducible residue (o) under writability gates. PEMF implements four core primitives — **Scalar Potential (+)**, **Gradient Ordering (V)**, **Residue (o)**, and **Boundary Charge (p_q)** — and composes atomic transforms (tile, shifted tile, fill_enclosed, rotate, reflect, etc.) in a beam search to drain residue and produce stable outputs.
+
+Quick verification
+- Run the PEMF example to verify primitives and a tiny compositional loop:
+```bash
+python SKILLS/pre_emergence_mechanics_framework/references/examples/verify_pemf.py
+```
+- Run a single experiment (example):
+```bash
+python scripts/entrypoint.py --task example1 --out_dir experiments
+```
+- Postprocess logs to attach candidate snapshot and coerce gates:
+```bash
+python experiments/postprocess_logs.py
+```
+
+Where to find skill artifacts
+- `SKILLS/pre_emergence_mechanics_framework/` — howto, runnable example `references/examples/verify_pemf.py`, and README for the skill.

README_PEMF.md

@@ -0,0 +1,40 @@
+# Pre‑Emergence Mechanics Framework (PEMF) — ARC‑AGI
+
+Short summary  
+The Pre‑Emergence Mechanics Framework (PEMF) frames ARC tasks as a boundary‑constrained field problem solved by minimizing irreducible residue (o) under writability gates. PEMF implements four core primitives — **Scalar Potential (+)**, **Gradient Ordering (V)**, **Residue (o)**, and **Boundary Charge (p_q)** — and composes atomic transforms (tile, shifted tile, fill_enclosed, rotate, reflect, etc.) in a beam search to drain residue and produce stable outputs.
+
+Why this matters  
+PEMF shows how ARC tasks can be solved mechanically (o‑minimization + gates) rather than by symbolic heuristics. The approach maps CTS/ITT primitives to executable operators (potential fields, gradients, Dirichlet masks, complex projections) and yields a reproducible solver recipe.
+
+Key concepts (one line each)
+- **Scalar Potential (+):** represent grid as numeric potential field (initialize_potential).  
+- **Gradient Ordering (V):** discrete gradients direct admissible edits.  
+- **Residue (o):** L1 misalignment after quantize+tile; objective to minimize.  
+- **Boundary Charge (p_q):** Dirichlet boundary mask that enforces writability gates.  
+- **Layer‑1 diagnostics:** complex projection (FFT imag component) to find latent edit zones when real signal is weak.
+
+Files and examples
+- **Skill artifacts:** `SKILLS/pre_emergence_mechanics_framework/` — howto, runnable example `references/examples/verify_pemf.py`, and README for the skill.  
+- **Postprocess logs:** `experiments/postprocess_logs.py` — coerce gate booleans and attach candidate snapshots for offline inspection.  
+- **Headless entry:** `scripts/entrypoint.py` — run experiments from CLI; `--use_wandb` flag is optional and defaults to off.
+
+Quick verification (headless)
+1. Run the PEMF example to verify primitives and a tiny compositional loop:
+```bash
+python SKILLS/pre_emergence_mechanics_framework/references/examples/verify_pemf.py
+```
+2. Run a single experiment (example):
+```bash
+python scripts/entrypoint.py --task example1 --out_dir experiments
+```
+3. Postprocess logs to attach candidate snapshot and coerce gates:
+```bash
+python experiments/postprocess_logs.py
+```
+
+Acceptance checks
+- `verify_pemf.py` prints a residue trace and reports at least one admissible edit zone from the complex projection.  
+- `experiments/*_phi_best.npy` and `experiments/*_logs.fixed.json` exist after a run and contain candidate snapshot and boolean gates for inspection.
+
+References and provenance
+This README summarizes the executable PEMF recipe derived from the ARC‑AGI exposition (PEMF / CTS / ITT). See `SKILLS/pre_emergence_mechanics_framework/references/` for runnable examples and a step‑by‑step how‑to.

SKILLS/README_SKILLS.md

@@ -0,0 +1,2 @@
+SKILLS directory: central place for skill documentation and runnable examples.
+Use SKILL.md to find the skill you need, then open the corresponding references/examples.

SKILLS/SKILL.md

@@ -0,0 +1,23 @@
+# SKILL Index (project root)
+
+Purpose
+- Central index of skills and reference artifacts for developers and agents.
+- Each skill has a short description here; larger skills include a `references/` folder with howto and runnable examples.
+
+Top-level skills (descriptive names)
+- code_reviewer — automated review heuristics and safe-change checklist.
+- unit_tester — unit test design, fixtures, and templates.
+- arc_agi_project — domain knowledge and reproducible checklist for ARC‑AGI.
+- transform_designer — design, test, and verify transforms.
+- logging_observability — logging, postprocessing, and headless diagnostics.
+- experiment_runner — headless entrypoint patterns and safe external logging.
+
+How to use
+- Small skills: keep the full recipe in SKILL.md under the skill title.
+- Larger skills: open `SKILLS/<skill>/references/howto.md` and `SKILLS/<skill>/references/examples/*` for runnable examples.
+- Add new skills by creating `SKILLS/<skill>/references/` and adding `howto.md` + examples.
+
+Verification principle
+- Each skill must include at least one deterministic example and one verification step.
+
+- pre_emergence_mechanics_framework — PEMF skill (ARC‑AGI primitives and runnable examples).

SKILLS/arc_agi_project/references/examples/minimal_runner.py

@@ -0,0 +1,25 @@
+#!/usr/bin/env python3
+import numpy as np
+
+def tile_transform(phi, out_shape):
+    a = np.array(phi)
+    h_out, w_out = out_shape
+    h_in, w_in = a.shape
+    reps_h = (h_out + h_in - 1) // h_in
+    reps_w = (w_out + w_in - 1) // w_in
+    tiled = np.tile(a, (reps_h, reps_w))
+    return tiled[:h_out, :w_out]
+
+def quantize(phi):
+    return np.rint(phi).astype(int)
+
+INPUT = np.array([[0,7,7],[7,7,7],[0,7,7]])
+TARGET = np.zeros((9,9), dtype=int)
+
+def rotate90(phi):
+    return np.rot90(phi, 1)
+
+if __name__ == '__main__':
+    cand = rotate90(INPUT)
+    cand_for_target = tile_transform(cand, TARGET.shape)
+    print("Residue:", int((quantize(cand_for_target) != TARGET).sum()))

SKILLS/arc_agi_project/references/howto.md

@@ -0,0 +1,8 @@
+# ARC-AGI Project — How To
+
+Purpose
+- Domain checklist and minimal reproducible runner for ARC tasks.
+
+Key points
+- Tasks are small grids; scoring is L1 after quantize+tile.
+- Always attach candidate snapshots for debugging.

SKILLS/code_reviewer/references/examples/review_checks.py

@@ -0,0 +1,32 @@
+#!/usr/bin/env python3
+# Lightweight static check: find suspicious top-level calls
+import ast, os, json, sys
+
+def suspicious_top_level_calls(path):
+    with open(path, 'r', encoding='utf-8') as fh:
+        src = fh.read()
+    tree = ast.parse(src)
+    for node in tree.body:
+        if isinstance(node, ast.Expr) and isinstance(node.value, ast.Call):
+            # ignore docstring
+            if not (isinstance(node.value.func, ast.Name) and node.value.func.id == '__doc__'):
+                return True
+    return False
+
+def run_check(root='.'):
+    issues = []
+    for dirpath, _, files in os.walk(root):
+        if '.git' in dirpath or 'venv' in dirpath or 'SKILLS' in dirpath:
+            continue
+        for f in files:
+            if f.endswith('.py'):
+                p = os.path.join(dirpath, f)
+                try:
+                    if suspicious_top_level_calls(p):
+                        issues.append(p)
+                except Exception:
+                    pass
+    print(json.dumps({'suspicious_files': issues}, indent=2))
+
+if __name__ == '__main__':
+    run_check()

SKILLS/code_reviewer/references/howto.md

@@ -0,0 +1,13 @@
+# Code Reviewer — How To
+
+Purpose
+- Checklist and lightweight automated checks to ensure safe, minimal changes.
+
+Quick checklist
+- No side effects at import time.
+- Avoid circular imports; prefer lazy imports.
+- Logging must be JSON serializable; gate values must be booleans.
+- Add tests for any change.
+
+Verification
+- Run the example script in references/examples/review_checks.py.

SKILLS/experiment_runner/references/examples/entrypoint_example.py

@@ -0,0 +1,18 @@
+#!/usr/bin/env python3
+import argparse, json, os
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--task', required=True)
+    parser.add_argument('--out_dir', default='experiments')
+    parser.add_argument('--use_wandb', action='store_true')
+    args = parser.parse_args()
+    os.makedirs(args.out_dir, exist_ok=True)
+    import numpy as np
+    phi = np.zeros((9,9))
+    np.save(os.path.join(args.out_dir, f"{args.task}_phi_best.npy"), phi)
+    logs = [[{'accepted': True, 'atomic': '<Demo>', 'residue': 0, 'gates': {'A': True}}]]
+    with open(os.path.join(args.out_dir, f"{args.task}_logs.json"), 'w') as fh:
+        json.dump(logs, fh)
+    print("Wrote artifacts to", args.out_dir)
+if __name__ == '__main__':
+    main()

SKILLS/experiment_runner/references/howto.md

@@ -0,0 +1,8 @@
+# Experiment Runner — How To
+
+Purpose
+- Headless entrypoint patterns and safe external logging.
+
+Guidelines
+- Keep W&B optional and external to core run.
+- Provide CLI entrypoint with --use_wandb default false.

SKILLS/logging_observability/references/examples/postprocess_logs.py

@@ -0,0 +1,27 @@
+#!/usr/bin/env python3
+import glob, json, numpy as np
+
+def coerce_gates(g):
+    out = {}
+    for k,v in (g or {}).items():
+        if isinstance(v, str):
+            lv = v.strip().lower()
+            out[k] = lv in ('true','1','yes')
+        else:
+            out[k] = bool(v)
+    return out
+
+logs_path = sorted(glob.glob('experiments/*_logs.json'))[-1]
+phi_path = sorted(glob.glob('experiments/*_phi_best.npy'))[-1]
+logs = json.load(open(logs_path))
+phi = np.load(phi_path)
+
+for entry in logs[0]:
+    entry['gates'] = coerce_gates(entry.get('gates'))
+    if entry.get('accepted') and 'candidate_array' not in entry:
+        entry['candidate_array'] = phi.tolist()
+
+fixed = logs_path.replace('_logs.json','_logs.fixed.json')
+with open(fixed,'w') as fh:
+    json.dump(logs, fh, indent=2)
+print("Wrote", fixed)

SKILLS/logging_observability/references/howto.md

@@ -0,0 +1,8 @@
+# Logging & Observability — How To
+
+Purpose
+- How to log experiments for offline inspection and headless operation.
+
+Guidelines
+- Logs must be JSON serializable and compact.
+- Candidate snapshots optional; include when debugging.

SKILLS/transform_designer/references/examples/transforms_demo.py

@@ -0,0 +1,27 @@
+#!/usr/bin/env python3
+import numpy as np
+
+def tile_transform(phi, out_shape):
+    a = np.array(phi)
+    h_out, w_out = out_shape
+    h_in, w_in = a.shape
+    reps_h = (h_out + h_in - 1) // h_in
+    reps_w = (w_out + w_in - 1) // w_in
+    tiled = np.tile(a, (reps_h, reps_w))
+    return tiled[:h_out, :w_out]
+
+class ShiftedTile:
+    def __init__(self, shift=(1,1), factor=3):
+        self.shift = shift
+        self.factor = factor
+    def apply(self, phi):
+        h,w = phi.shape
+        tiled = tile_transform(phi, (h*self.factor, w*self.factor))
+        return np.roll(tiled, shift=self.shift, axis=(0,1))
+
+if __name__ == '__main__':
+    inp = np.array([[0,7,7],[7,7,7],[0,7,7]])
+    T = ShiftedTile()
+    out = T.apply(inp)
+    out_resized = tile_transform(out, inp.shape) if out.shape != inp.shape else out
+    print("Changed cells:", int((out_resized != inp).sum()))

SKILLS/transform_designer/references/howto.md

@@ -0,0 +1,8 @@
+# Transform Designer — How To
+
+Purpose
+- Design deterministic, testable transforms useful to the beam.
+
+Guidelines
+- Transforms must be pure functions with documented shape semantics.
+- Add a small shift for tiled transforms to avoid no-ops.

SKILLS/unit_tester/references/examples/test_templates.py

@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+import numpy as np
+
+def tile_transform(phi, out_shape):
+    a = np.array(phi)
+    h_out, w_out = out_shape
+    h_in, w_in = a.shape
+    reps_h = (h_out + h_in - 1) // h_in
+    reps_w = (w_out + w_in - 1) // w_in
+    tiled = np.tile(a, (reps_h, reps_w))
+    return tiled[:h_out, :w_out]
+
+def test_tile_transform():
+    inp = np.array([[1,2],[3,4]])
+    out = tile_transform(inp, (3,3))
+    assert out.shape == (3,3)
+    assert out[0,0] == 1
+
+if __name__ == '__main__':
+    test_tile_transform()
+    print("unit_tester example passed")

SKILLS/unit_tester/references/howto.md

@@ -0,0 +1,9 @@
+# Unit Tester — How To
+
+Purpose
+- Design deterministic, fast unit tests for core primitives.
+
+Recipe
+- Use small canonical fixtures.
+- Assert exact arrays or counts.
+- Mock external services.

TODO.md

@@ -0,0 +1,26 @@
+---
+
+#### `TODO.md`
+```markdown
+# TODO (Prioritised)
+
+## Immediate (blockers)
+- **Add candidate snapshot to beam logs** (if not already present) so each accepted candidate can be inspected offline.  
+- **Ensure gate values are booleans** in logs (coerce string values).  
+- **Make tile transform nontrivial** (shifted tiling) so transforms can change the field.  
+- **Implement robust fill_enclosed** (dependency‑free BFS hole filler).
+
+## Short term
+- Add CLI entrypoint with `--use_wandb` flag (default: false).  
+- Add unit tests for `tile_transform`, `fill_enclosed`, and transform `.apply()` semantics.  
+- Add small visualization notebook for `phi_best`, diff maps, and Layer‑1 masks.
+
+## Medium term
+- Improve Layer‑1 mask generation (percentile + min_abs + dilation).  
+- Add a toggle to include/exclude `candidate_array` in logs to control log size.  
+- Create a reproducible benchmark harness for parameter sweeps and CSV aggregation.
+
+## Long term
+- Integrate a safe external W&B uploader (runs after experiments finish).  
+- Add more ARC tasks and automated evaluation harness.  
+- Document reproducibility steps and expected outputs for each example task.

experiments/example1_20260428T172250Z_logs.json

@@ -0,0 +1 @@
+[[{"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Rotate_90>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Reflect_h>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}], [{"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Rotate_90>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Reflect_h>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Rotate_90>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Reflect_h>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Rotate_90>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Reflect_h>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Rotate_90>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Reflect_h>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}], [{"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Rotate_90>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Reflect_h>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Rotate_90>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Reflect_h>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Rotate_90>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Reflect_h>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Rotate_90>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Reflect_h>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Rotate_90>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Reflect_h>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Rotate_90>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform Reflect_h>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}]]

experiments/example1_20260428T172250Z_phi_best.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:660ada98c4dfce4cdf016cac4f3432f7e589a0c758e0a74a97f5719f4972caee
+size 776

experiments/example1_20260428T172250Z_result.json

@@ -0,0 +1,23 @@
+{
+  "task_name": "example1",
+  "params": {
+    "beam_width": 6,
+    "max_depth": 3,
+    "lock_coeff": 0.0,
+    "max_fraction": 1.0,
+    "enable_layer_minus_one": true,
+    "boundary_source": "target",
+    "wandb_project": "itt_solver",
+    "wandb_anonymous": "allow"
+  },
+  "final_sigma": 98.0,
+  "sigma_trace": [
+    98.0,
+    98.0,
+    98.0,
+    98.0
+  ],
+  "time_s": 0.008741617202758789,
+  "transform": "<Transform Id\u2218tile_to_target\u2218tile_to_target\u2218tile_to_target>",
+  "states_count": 4
+}

experiments/example1_20260428T172311Z_logs.json

@@ -0,0 +1 @@
+[[{"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}], [{"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform tile_to_target>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}, {"atomic": "<Transform FillEnclosedHarmonic>", "score": 98.0, "residue": 98.0, "energy": 2352.0, "gates": {"A_boundary": true, "B_localization": "True", "C_quantization": "True", "passed": "True"}, "accepted": true, "shape": [9, 9]}]]

experiments/example1_20260428T172311Z_phi_best.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:660ada98c4dfce4cdf016cac4f3432f7e589a0c758e0a74a97f5719f4972caee
+size 776

experiments/example1_20260428T172311Z_result.json

@@ -0,0 +1,21 @@
+{
+  "task_name": "example1",
+  "params": {
+    "beam_width": 4,
+    "max_depth": 2,
+    "lock_coeff": 0.0,
+    "max_fraction": 0.5,
+    "enable_layer_minus_one": true,
+    "boundary_source": "target",
+    "use_symmetry": false
+  },
+  "final_sigma": 98.0,
+  "sigma_trace": [
+    98.0,
+    98.0,
+    98.0
+  ],
+  "time_s": 0.0020961761474609375,
+  "transform": "<Transform Id\u2218tile_to_target\u2218tile_to_target>",
+  "states_count": 3
+}

experiments/results.csv

@@ -0,0 +1,5 @@
+task_name,params,final_sigma,time_s,transform,sigma_trace
+example1,"{""beam_width"": 4, ""max_depth"": 2, ""lock_coeff"": 0.0, ""max_fraction"": 0.5, ""enable_layer_minus_one"": false, ""boundary_source"": ""target"", ""use_symmetry"": true}",98.0,0.003506183624267578,<Transform Id∘tile_to_target∘tile_to_target>,"[98.0, 98.0, 98.0]"
+example1,"{""beam_width"": 4, ""max_depth"": 2, ""lock_coeff"": 0.0, ""max_fraction"": 0.5, ""enable_layer_minus_one"": false, ""boundary_source"": ""target"", ""use_symmetry"": false}",98.0,0.0017173290252685547,<Transform Id∘tile_to_target∘tile_to_target>,"[98.0, 98.0, 98.0]"
+example1,"{""beam_width"": 4, ""max_depth"": 2, ""lock_coeff"": 0.0, ""max_fraction"": 0.5, ""enable_layer_minus_one"": true, ""boundary_source"": ""target"", ""use_symmetry"": true}",98.0,0.0046575069427490234,<Transform Id∘tile_to_target∘tile_to_target>,"[98.0, 98.0, 98.0]"
+example1,"{""beam_width"": 4, ""max_depth"": 2, ""lock_coeff"": 0.0, ""max_fraction"": 0.5, ""enable_layer_minus_one"": true, ""boundary_source"": ""target"", ""use_symmetry"": false}",98.0,0.0020961761474609375,<Transform Id∘tile_to_target∘tile_to_target>,"[98.0, 98.0, 98.0]"

experiments_analysis.py

@@ -0,0 +1,159 @@
+"""
+Quick diagnostics for itt_solver experiments.
+
+Usage (from notebook or shell):
+  python experiments_analysis.py
+
+It will:
+ - list recent files in experiments/
+ - print the latest result.json
+ - print depth-0 logs (candidates, gates, residues)
+ - load the latest phi_best and compute L1 vs a provided target (if you set TARGET_GRID below)
+ - test atomic transforms from default_atomic_factory to see if they change the input
+"""
+
+import os
+import glob
+import json
+import numpy as np
+from pprint import pprint
+
+# === Configure your example target here if you want an automatic L1 check ===
+# Replace TARGET_GRID with your task's target grid (9x9 in your example).
+TARGET_GRID = [
+    [0,0,0,0,7,7,0,7,7],
+    [0,0,0,7,7,7,7,7,7],
+    [0,0,0,0,7,7,0,7,7],
+    [0,7,7,0,7,7,0,7,7],
+    [7,7,7,7,7,7,7,7,7],
+    [0,7,7,0,7,7,0,7,7],
+    [0,0,0,0,7,7,0,7,7],
+    [0,0,0,7,7,7,7,7,7],
+    [0,0,0,0,7,7,0,7,7],
+]
+
+EXPERIMENTS_DIR = "experiments"
+
+def list_recent_files(n=20):
+    files = sorted(glob.glob(os.path.join(EXPERIMENTS_DIR, "*")))
+    print(f"Recent files (last {n}):")
+    for f in files[-n:]:
+        print(" ", f)
+    return files
+
+def load_latest_result():
+    res_files = sorted(glob.glob(os.path.join(EXPERIMENTS_DIR, "*_result.json")))
+    if not res_files:
+        print("No result.json files found in experiments/")
+        return None, None
+    latest = res_files[-1]
+    print("\nLatest result file:", latest)
+    with open(latest) as fh:
+        data = json.load(fh)
+    pprint(data)
+    return latest, data
+
+def load_latest_logs():
+    logs_files = sorted(glob.glob(os.path.join(EXPERIMENTS_DIR, "*_logs.json")))
+    if not logs_files:
+        print("No logs.json files found in experiments/")
+        return None, None
+    latest = logs_files[-1]
+    print("\nLatest logs file:", latest)
+    with open(latest) as fh:
+        logs = json.load(fh)
+    # pretty-print depth 0 entries if present
+    if logs and isinstance(logs, list) and len(logs) > 0:
+        print("\nDepth 0 log entries (summary):")
+        for i, entry in enumerate(logs[0]):
+            atomic = entry.get('atomic')
+            accepted = entry.get('accepted')
+            residue = entry.get('residue')
+            energy = entry.get('energy')
+            gates = entry.get('gates')
+            print(f"{i}: {atomic} | accepted={accepted} | residue={residue} | energy={energy} | gates={gates}")
+    else:
+        print("Logs format unexpected or empty.")
+    return latest, logs
+
+def load_latest_phi():
+    phi_files = sorted(glob.glob(os.path.join(EXPERIMENTS_DIR, "*_phi_best.npy")))
+    if not phi_files:
+        print("No phi_best.npy files found in experiments/")
+        return None, None
+    latest = phi_files[-1]
+    print("\nLatest phi_best file:", latest)
+    phi = np.load(latest)
+    print("phi_best shape:", phi.shape, "unique values:", np.unique(phi))
+    return latest, phi
+
+def l1_residue_check(phi, target_grid):
+    if phi is None:
+        print("No phi provided for residue check.")
+        return
+    target = np.array(target_grid, dtype=phi.dtype)
+    if phi.shape != target.shape:
+        print("phi and target shapes differ:", phi.shape, target.shape)
+        # try to tile/resize target to phi shape if phi is a tiled version
+        try:
+            from itt_solver.solver_core import tile_transform
+            target_resized = tile_transform(target, phi.shape)
+            print("Resized target to phi shape for comparison.")
+        except Exception:
+            print("Could not resize target automatically.")
+            return
+    else:
+        target_resized = target
+    l1 = float(np.sum(np.abs(phi - target_resized)))
+    print("L1 residue between phi_best and target:", l1)
+    return l1
+
+def test_atomic_effects():
+    print("\nTesting atomic transforms from default_atomic_factory...")
+    try:
+        from itt_solver.experiment_driver import default_atomic_factory
+        from itt_solver.solver_core import initialize_potential, tile_transform
+    except Exception as e:
+        print("Could not import default_atomic_factory or solver_core:", e)
+        return
+    params = {'beam_width':6,'max_depth':3,'lock_coeff':0.0,'max_fraction':1.0,'enable_layer_minus_one':True,'boundary_source':'target'}
+    task_stub = {'target_shape': (9,9)}
+    atomic_library = default_atomic_factory(params, task_stub)
+    phi_in = initialize_potential([[0,7,7],[7,7,7],[0,7,7]])
+    print("Input shape:", phi_in.shape, "unique:", np.unique(phi_in))
+    for T in atomic_library:
+        try:
+            out = T.apply(phi_in.copy())
+        except Exception as e:
+            print(repr(T), "apply() raised:", e)
+            continue
+        # if shapes differ, try to tile output back to input shape for comparison
+        out_resized = out
+        if out.shape != phi_in.shape:
+            try:
+                out_resized = tile_transform(out, phi_in.shape)
+            except Exception:
+                # fallback: broadcast if possible
+                try:
+                    out_resized = np.broadcast_to(out, phi_in.shape)
+                except Exception:
+                    out_resized = None
+        if out_resized is None:
+            changed = None
+        else:
+            changed = int(np.sum(out_resized != phi_in))
+        print(repr(T), "-> out shape", out.shape, "changed cells (compared to input):", changed)
+
+def main():
+    print("=== experiments_analysis.py diagnostics ===")
+    list_recent_files()
+    load_latest_result()
+    load_latest_logs()
+    _, phi = load_latest_phi()
+    if phi is not None:
+        l1_residue_check(phi, TARGET_GRID)
+    test_atomic_effects()
+    print("\nDone.")
+
+if __name__ == "__main__":
+    main()

itt_solver/README.md.md

@@ -0,0 +1,19 @@
+# ARC-AGI Reproduction Project
+
+**Project goal**  
+Recreate the system described in *A Pre-Emergence Mechanics Framework for Solving the Abstraction and Reasoning Corpus ARC-AGI* by Achilles. The codebase implements a boundary constrained field solver (Intent Tensor Theory substrate) and a beam search driver to propose transforms that reduce L1 residue against ARC tasks.
+
+**Current status**  
+- Core driver `itt_solver/experiment_driver.py` runs experiments and saves artifacts to `experiments/`.  
+- Runs complete and save `*_phi_best.npy`, `*_logs.json`, `*_result.json`.  
+- Immediate blockers discovered and addressed: W&B recursion removed; logging lacked candidate arrays; some transforms were no-ops. Diagnostics and small patches are in `experiments/` and `itt_solver/` to inspect and fix behavior.  
+- Smoke tests show the beam accepts candidates but residue has not decreased for the example task. See `experiments/` for logs.
+
+**Quick start minimal run**
+1. Install dependencies (if not present): `numpy`, `matplotlib`.  
+2. From project root run a single job in a notebook cell:
+```python
+from itt_solver.experiment_driver import default_atomic_factory, run_single
+# build task and params (example provided in repo)
+res = run_single(task, default_atomic_factory(params, task), params, out_dir="experiments")
+print(res)

itt_solver/__init__.py

@@ -0,0 +1,10 @@
+from .solver_core import (
+    initialize_potential,
+    discrete_gradient,
+    dirichlet_energy,
+    sigma_l1,
+    tile_transform,
+    fill_enclosed,
+    Transform,
+    beam_minimize,
+)

itt_solver/beam_logging.py

@@ -0,0 +1,142 @@
+import numpy as np
+from .solver_core import (
+    sigma_l1,
+    dirichlet_energy,
+    Transform,
+    tile_transform,
+)
+from .gates import validate_gates
+from .layer_minus_one import admissible_edit_mask
+
+def _resize_to_target(phi, target):
+    if phi.shape == target.shape:
+        return phi
+    return tile_transform(phi, target.shape)
+
+def _compute_boundary_mask(phi_in, phi_target, target_shape, boundary_source='target'):
+    """
+    boundary_source: 'input' | 'resized_input' | 'target'
+    - 'input' uses phi_in != 0 then tiles inside validate_gates (less efficient)
+    - 'resized_input' tiles phi_in != 0 to target_shape and returns that mask
+    - 'target' uses phi_target != 0 (already target-shaped)
+    """
+    if boundary_source == 'target':
+        return (phi_target != 0)
+    if boundary_source == 'resized_input':
+        return _resize_to_target((phi_in != 0).astype(int), phi_target).astype(bool)
+    # fallback: 'input' -> return original small mask (validate_gates will tile if needed)
+    return (phi_in != 0)
+
+def beam_minimize_with_log(phi_in, phi_target, atomic_library,
+                           beam_width=4, max_depth=4, lock_coeff=0.1,
+                           max_fraction=0.5, allowed_symbols=None,
+                           enable_layer_minus_one=False,
+                           boundary_source='target'):
+    """
+    Beam search with gate validation and optional Layer-1 admissible mask.
+    boundary_source controls where ρ_q is derived from: 'target' recommended for expansion tasks.
+    Returns best transform, best field, states list, sigmas list, and logs per depth.
+    """
+    phi_in = np.array(phi_in, dtype=float)
+    phi_target = np.array(phi_target, dtype=float)
+
+    # Resize initial phi to target shape
+    phi0 = _resize_to_target(phi_in, phi_target)
+
+    identity = Transform(lambda p: p, "Id")
+    beam = [(identity, phi0, 0.0, [phi0], [sigma_l1(phi0, phi_target)])]
+    best = None
+
+    # Precompute Layer-1 mask if enabled
+    layer_mask = None
+    if enable_layer_minus_one:
+        try:
+            layer_mask, _ = admissible_edit_mask(phi0)
+        except Exception:
+            layer_mask = None
+
+    # Precompute boundary mask according to boundary_source
+    boundary_mask_resized = _compute_boundary_mask(phi_in, phi_target, phi_target.shape, boundary_source=boundary_source)
+
+    logs = []
+
+    for depth in range(max_depth):
+        candidates = []
+        depth_log = []
+        for T_cur, cur_field_resized, _, path_states, path_sigmas in beam:
+            base_field_for_apply = path_states[-1]
+
+            for idx, T_atomic in enumerate(atomic_library):
+                # Apply atomic transform
+                try:
+                    phi_after_atomic = T_atomic.apply(base_field_for_apply)
+                except TypeError:
+                    phi_after_atomic = T_atomic.apply(phi_in)
+
+                # Resize to target shape before scoring
+                phi_new_resized = _resize_to_target(phi_after_atomic, phi_target)
+
+                # If Layer-1 enabled and mask exists, allow edits only inside mask by zeroing changes outside
+                if enable_layer_minus_one and layer_mask is not None:
+                    masked = cur_field_resized.copy()
+                    masked[layer_mask] = phi_new_resized[layer_mask]
+                    phi_candidate = masked
+                else:
+                    phi_candidate = phi_new_resized
+
+                residue = sigma_l1(phi_candidate, phi_target)
+                energy = dirichlet_energy(phi_candidate)
+                score = residue + lock_coeff * energy
+
+                # Validate gates using precomputed boundary mask
+                gates_info = validate_gates(phi_candidate, phi_in, phi_target,
+                                            boundary_mask=boundary_mask_resized,
+                                            max_fraction=max_fraction,
+                                            allowed_symbols=allowed_symbols)
+
+                # Only accept candidate if gates pass
+                if not gates_info.get('passed', False):
+                    depth_log.append({
+                        'atomic': repr(T_atomic),
+                        'score': score,
+                        'residue': residue,
+                        'energy': energy,
+                        'gates': gates_info,
+                        'accepted': False,
+                        'shape': phi_candidate.shape,
+                    })
+                    continue
+
+                new_states = path_states + [phi_candidate]
+                new_sigmas = path_sigmas + [residue]
+                T_new = T_cur.compose(T_atomic)
+
+                candidates.append((T_new, phi_candidate, score, new_states, new_sigmas))
+
+                depth_log.append({
+                    'atomic': repr(T_atomic),
+                    'score': score,
+                    'residue': residue,
+                    'energy': energy,
+                    'gates': gates_info,
+                    'accepted': True,
+                    'shape': phi_candidate.shape,
+                })
+
+        logs.append(depth_log)
+
+        if not candidates:
+            break
+
+        candidates.sort(key=lambda x: x[2])
+        beam = candidates[:beam_width]
+        best = beam[0]
+
+        if sigma_l1(best[1], phi_target) == 0:
+            break
+
+    if best is None:
+        return identity, phi0, [phi0], [sigma_l1(phi0, phi_target)], logs
+
+    T_best, phi_best, _, states_best, sigmas_best = best
+    return T_best, phi_best, states_best, sigmas_best, logs

itt_solver/experiment_driver.py

@@ -0,0 +1,112 @@
+import csv
+import os
+import time
+import json
+import importlib
+import numpy as np
+from itertools import product
+from datetime import datetime
+from .solver_core import initialize_potential
+from .beam_logging import beam_minimize_with_log
+
+def reload_modules():
+    import itt_solver.solver_core as sc
+    import itt_solver.beam_logging as bl
+    import itt_solver.transforms as tr
+    import itt_solver.gates as gates
+    import itt_solver.layer_minus_one as l1
+    importlib.reload(sc); importlib.reload(bl); importlib.reload(tr); importlib.reload(gates); importlib.reload(l1)
+
+def param_grid(grid_dict):
+    keys = list(grid_dict.keys())
+    vals = [grid_dict[k] for k in keys]
+    for combo in product(*vals):
+        yield dict(zip(keys, combo))
+
+def run_single(task, atomic_library, params, out_dir):
+    """
+    Run one experiment and save artifacts to out_dir.
+    This function does NOT call W&B itself to avoid recursion; call the external
+    uploader (itt_solver.wandb_runner.run_and_log_wandb) after this returns if desired.
+    """
+    os.makedirs(out_dir, exist_ok=True)
+
+    phi_in = initialize_potential(task['input'])
+    phi_target = initialize_potential(task['target'])
+    start = time.time()
+    T_best, phi_best, states, sigmas, logs = beam_minimize_with_log(
+        phi_in, phi_target, atomic_library,
+        beam_width=params.get('beam_width',4),
+        max_depth=params.get('max_depth',3),
+        lock_coeff=params.get('lock_coeff',0.01),
+        max_fraction=params.get('max_fraction',0.5),
+        allowed_symbols=params.get('allowed_symbols', list(range(10))),
+        enable_layer_minus_one=params.get('enable_layer_minus_one', False),
+        boundary_source=params.get('boundary_source','target'),
+    )
+    elapsed = time.time() - start
+    result = {
+        'task_name': task.get('name','task'),
+        'params': params,
+        'final_sigma': float(sigmas[-1]) if sigmas else None,
+        'sigma_trace': [float(s) for s in sigmas],
+        'time_s': elapsed,
+        'transform': repr(T_best),
+        'states_count': len(states),
+    }
+    # save best field and logs
+    ts = datetime.utcnow().strftime("%Y%m%dT%H%M%SZ")
+    base = f"{task.get('name','task')}_{ts}"
+    np.save(os.path.join(out_dir, base + "_phi_best.npy"), phi_best)
+    with open(os.path.join(out_dir, base + "_result.json"), "w") as f:
+        json.dump(result, f, indent=2)
+    with open(os.path.join(out_dir, base + "_logs.json"), "w") as f:
+        json.dump(logs, f, default=str)
+
+    # IMPORTANT: do not call run_and_log_wandb here to avoid recursion.
+    # If you want W&B logging, call itt_solver.wandb_runner.run_and_log_wandb(...) externally.
+
+    return result
+
+def sweep(tasks, atomic_library_factory, grid, out_dir="experiments", max_runs=None):
+    os.makedirs(out_dir, exist_ok=True)
+    reload_modules()
+    csv_path = os.path.join(out_dir, "results.csv")
+    header_written = os.path.exists(csv_path)
+    runs = 0
+    with open(csv_path, "a", newline="") as csvfile:
+        writer = csv.DictWriter(csvfile, fieldnames=["task_name","params","final_sigma","time_s","transform","sigma_trace"])
+        if not header_written:
+            writer.writeheader()
+        for params in param_grid(grid):
+            for task in tasks:
+                if max_runs and runs >= max_runs:
+                    return
+                atomic_library = atomic_library_factory(params, task)
+                res = run_single(task, atomic_library, params, out_dir)
+                writer.writerow({
+                    "task_name": res['task_name'],
+                    "params": json.dumps(res['params']),
+                    "final_sigma": res['final_sigma'],
+                    "time_s": res['time_s'],
+                    "transform": res['transform'],
+                    "sigma_trace": json.dumps(res['sigma_trace']),
+                })
+                csvfile.flush()
+                runs += 1
+    return
+
+# Example atomic library factory for common ARC families
+def default_atomic_factory(params, task):
+    import itt_solver.transforms as tr
+    from itt_solver.solver_core import tile_transform  # ensure tile_transform is available here
+    libs = []
+    # always include tile and fill
+    libs.append(tr.Transform(lambda p: tile_transform(p, (task['target_shape'][0], task['target_shape'][1])), "tile_to_target"))
+    libs.append(tr.FillEnclosedHarmonic())
+    # optional rotations/reflections
+    if params.get('use_symmetry', True):
+        libs.append(tr.Rotate(1))
+        libs.append(tr.Reflect('h'))
+    return libs
+

itt_solver/gates.py

@@ -0,0 +1,51 @@
+from .solver_core import tile_transform
+import numpy as np
+
+def gate_boundary_respect(phi_candidate, phi_in, phi_target, boundary_mask=None):
+    """Gate A: candidate must not change locked boundary cells.
+    If boundary_mask is None, infer from phi_in != 0.
+    If mask shape differs from candidate, resize by tiling.
+    """
+    if boundary_mask is None:
+        boundary_mask = (phi_in != 0)
+
+    # If mask shape differs, resize it to candidate shape using tiling
+    if boundary_mask.shape != phi_candidate.shape:
+        # convert to int for tile_transform then back to bool
+        try:
+            resized = tile_transform(boundary_mask.astype(int), phi_candidate.shape).astype(bool)
+        except Exception:
+            # fallback: broadcast along axes if possible
+            resized = np.broadcast_to(boundary_mask, phi_candidate.shape)
+    else:
+        resized = boundary_mask
+
+    # If any boundary cell in phi_target differs from candidate where resized True, reject
+    changed = np.any((phi_candidate[resized] != phi_target[resized]))
+    return not changed
+
+
+def gate_sigma_localization(phi_candidate, phi_target, max_fraction=0.5):
+    """Gate B: ensure residue is localized. Compute fraction of cells that differ; must be <= max_fraction."""
+    diff = (phi_candidate != phi_target)
+    frac = np.sum(diff) / diff.size
+    return frac <= max_fraction
+
+def gate_quantization(phi_candidate, allowed_symbols=None, tol=1e-6):
+    """Gate C: candidate values must quantize to allowed_symbols (integers 0-9 by default).
+    Allow small numerical tolerance.
+    """
+    if allowed_symbols is None:
+        allowed_symbols = list(range(10))
+    # Round candidate to nearest integer and check membership
+    rounded = np.rint(phi_candidate).astype(int)
+    mask = np.isin(rounded, allowed_symbols)
+    return np.all(mask)
+
+def validate_gates(phi_candidate, phi_in, phi_target, boundary_mask=None, max_fraction=0.5, allowed_symbols=None):
+    """Run all gates and return a dict of booleans and an overall pass boolean."""
+    a = gate_boundary_respect(phi_candidate, phi_in, phi_target, boundary_mask)
+    b = gate_sigma_localization(phi_candidate, phi_target, max_fraction=max_fraction)
+    c = gate_quantization(phi_candidate, allowed_symbols=allowed_symbols)
+    passed = a and b and c
+    return {"A_boundary": a, "B_localization": b, "C_quantization": c, "passed": passed}

itt_solver/layer_minus_one.py

@@ -0,0 +1,33 @@
+import numpy as np
+from scipy.ndimage import binary_dilation
+
+def compute_complex_phi(phi):
+    F = np.fft.fft2(phi)
+    phi_c = np.fft.ifft2(F)
+    return phi_c
+
+def admissible_edit_mask(phi, imag_grad_threshold=None, percentile=95, min_abs=1e-3, dilate_iters=1):
+    """
+    Robust mask:
+    - If imag_grad_threshold provided, use it.
+    - Else compute threshold as max(percentile value, min_abs).
+    - Optionally dilate mask to give edit zones some area.
+    Returns (mask, mag).
+    """
+    phi_c = compute_complex_phi(phi)
+    imag = np.imag(phi_c)
+    gx, gy = np.gradient(imag)
+    mag = np.sqrt(gx**2 + gy**2)
+
+    if imag_grad_threshold is None:
+        pval = np.percentile(mag, percentile)
+        thresh = max(pval, min_abs)
+    else:
+        thresh = imag_grad_threshold
+
+    mask = mag > thresh
+
+    if dilate_iters > 0:
+        mask = binary_dilation(mask, iterations=dilate_iters)
+
+    return mask, mag

itt_solver/solver_core.py

@@ -0,0 +1,136 @@
+import numpy as np
+
+def initialize_potential(grid):
+    return np.array(grid, dtype=float)
+
+def discrete_gradient(phi):
+    gx = np.zeros_like(phi)
+    gy = np.zeros_like(phi)
+    gx[:-1, :] = phi[1:, :] - phi[:-1, :]
+    gy[:, :-1] = phi[:, 1:] - phi[:, :-1]
+    mag = np.sqrt(gx**2 + gy**2)
+    return gx, gy, mag
+
+def dirichlet_energy(phi):
+    _, _, mag = discrete_gradient(phi)
+    return np.sum(mag**2)
+
+def sigma_l1(phi_current, phi_target):
+    return np.sum(np.abs(phi_target - phi_current))
+
+def tile_transform(phi_in, out_shape):
+    h_in, w_in = phi_in.shape
+    h_out, w_out = out_shape
+    out = np.zeros(out_shape, dtype=float)
+    for i in range(h_out):
+        for j in range(w_out):
+            out[i, j] = phi_in[i % h_in, j % w_in]
+    return out
+# paste into itt_solver/solver_core.py replacing old fill_enclosed
+import numpy as np
+from collections import Counter
+from collections import deque
+
+import numpy as _np
+from collections import Counter, deque
+
+def fill_enclosed(phi, boundary_mask=None):
+    """
+    Fill zero regions that are fully enclosed by non-zero boundary.
+    Returns a new array with enclosed holes filled with modal neighbor color.
+    """
+    arr = _np.array(phi, dtype=int)
+    h, w = arr.shape
+    if boundary_mask is None:
+        boundary = (arr != 0)
+    else:
+        boundary = _np.array(boundary_mask, dtype=bool)
+
+    visited = _np.zeros((h, w), dtype=bool)
+    filled = arr.copy()
+
+    def flood(start_r, start_c):
+        q = deque()
+        q.append((start_r, start_c))
+        comp = []
+        touches_border = False
+        visited[start_r, start_c] = True
+        while q:
+            r, c = q.popleft()
+            comp.append((r, c))
+            if r == 0 or c == 0 or r == h-1 or c == w-1:
+                touches_border = True
+            for dr, dc in ((1,0),(-1,0),(0,1),(0,-1)):
+                nr, nc = r+dr, c+dc
+                if 0 <= nr < h and 0 <= nc < w and not visited[nr, nc]:
+                    if arr[nr, nc] == 0:
+                        visited[nr, nc] = True
+                        q.append((nr, nc))
+        return comp, touches_border
+
+    for i in range(h):
+        for j in range(w):
+            if arr[i, j] == 0 and not visited[i, j]:
+                comp, touches_border = flood(i, j)
+                if not touches_border:
+                    neighbor_vals = []
+                    for (r, c) in comp:
+                        for dr, dc in ((1,0),(-1,0),(0,1),(0,-1)):
+                            nr, nc = r+dr, c+dc
+                            if 0 <= nr < h and 0 <= nc < w:
+                                v = arr[nr, nc]
+                                if v != 0:
+                                    neighbor_vals.append(int(v))
+                    if neighbor_vals:
+                        mode_color = Counter(neighbor_vals).most_common(1)[0][0]
+                    else:
+                        mode_color = 1
+                    for (r, c) in comp:
+                        filled[r, c] = mode_color
+    return filled
+
+
+class Transform:
+    def __init__(self, func, name, params=None):
+        self.func = func
+        self.name = name
+        self.params = params or {}
+
+    def apply(self, phi):
+        return self.func(phi, **self.params)
+
+    def compose(self, other):
+        def composed(phi):
+            return other.apply(self.apply(phi))
+        return Transform(lambda p: composed(p), f"{self.name}∘{other.name}")
+
+    def __repr__(self):
+        return f"<Transform {self.name}>"
+
+def beam_minimize(phi_in, phi_target, atomic_library, beam_width=4, max_depth=4, lock_coeff=0.1):
+    identity = Transform(lambda p: p, "Id")
+    beam = [(identity, phi_in, 0.0)]
+    best = None
+
+    for depth in range(max_depth):
+        candidates = []
+        for T_cur, _, _ in beam:
+            for T_atomic in atomic_library:
+                T_new = T_cur.compose(T_atomic)
+                phi_new = T_new.apply(phi_in)
+                residue = sigma_l1(phi_new, phi_target)
+                energy = dirichlet_energy(phi_new)
+                score = residue + lock_coeff * energy
+                candidates.append((T_new, phi_new, score))
+
+        if not candidates:
+            break
+
+        candidates.sort(key=lambda x: x[2])
+        beam = candidates[:beam_width]
+        best = beam[0]
+
+        if sigma_l1(best[1], phi_target) == 0:
+            break
+
+    return best[0], best[1]

itt_solver/tests.py

@@ -0,0 +1,297 @@
+"""
+Utility tests and debug helpers for the itt_solver beam runs.
+
+Save this file into the `itt_solver/` package and import the helpers from your notebook.
+Example usage (in a notebook cell after a run that produced `states, sigmas, logs, phi_target`):
+
+from itt_solver import tests
+tests.print_depth0_logs(logs)
+tests.check_first_accepted_score(logs, lock_coeff=0.01)
+tests.gate_failure_summary(logs)
+tests.plot_layer1_mask(states[0], l1_module)   # pass the layer_minus_one module you imported as l1
+
+Added helpers:
+- transform_effect_test(transform, phi): returns number of changed cells and a small diff map.
+- sigma_decrease_smoke_test(beam_func, phi_in, phi_target, atomic_library): runs a relaxed beam and reports sigma trace and whether sigma decreased.
+- run_all_quick_checks(...) convenience runner for quick local verification.
+"""
+
+from pprint import pprint
+from collections import Counter
+import matplotlib.pyplot as plt
+import numpy as np
+
+def print_depth0_logs(logs):
+    """Pretty-print depth-0 logs and accepted candidate summary."""
+    if not logs:
+        print("No logs available.")
+        return
+    if len(logs) <= 0:
+        print("Logs list is empty.")
+        return
+
+    depth0 = logs[0]
+    print("Depth 0 log entries:", len(depth0))
+    pprint(depth0)
+
+    accepted = [r for r in depth0 if r.get('accepted')]
+    print("Accepted count at depth 0:", len(accepted))
+    for i, r in enumerate(accepted):
+        print(i, r.get('atomic'), "score", r.get('score'), "gates", r.get('gates'))
+
+
+def check_first_accepted_score(logs, lock_coeff=0.01, tolerance=1e-8):
+    """
+    Find the first accepted candidate in depth 0 and assert score == residue + lock_coeff * energy.
+    Returns True if check passes, False otherwise.
+    """
+    if not logs or len(logs) == 0:
+        print("No logs to check.")
+        return False
+
+    first_accepted = next((r for r in logs[0] if r.get('accepted')), None)
+    if first_accepted is None:
+        print("No accepted candidates at depth 0. See logs for gate failures.")
+        return False
+
+    res = first_accepted.get('residue')
+    E = first_accepted.get('energy')
+    score = first_accepted.get('score')
+    if res is None or E is None or score is None:
+        print("Missing numeric fields in the candidate log.")
+        return False
+
+    ok = abs(score - (res + lock_coeff * E)) < tolerance
+    if ok:
+        print("Score check passed for first accepted candidate.")
+    else:
+        print("Score check FAILED.")
+        print(f"Logged score: {score}")
+        print(f"Computed  : {res + lock_coeff * E}")
+    return ok
+
+
+def gate_failure_summary(logs):
+    """
+    Count gate failures in depth 0 and print a summary.
+    Returns a Counter with failure counts.
+    """
+    if not logs or len(logs) == 0:
+        print("No logs to summarize.")
+        return Counter()
+
+    c = Counter()
+    for r in logs[0]:
+        g = r.get('gates')
+        if not g:
+            c['no_gate_info'] += 1
+            continue
+        if not g.get('A_boundary', True):
+            c['A_boundary_failed'] += 1
+        if not g.get('B_localization', True):
+            c['B_localization_failed'] += 1
+        if not g.get('C_quantization', True):
+            c['C_quantization_failed'] += 1
+        if g.get('passed') is False:
+            c['total_rejected'] += 1
+        else:
+            c['total_accepted'] += 1
+    print("Gate failure summary (depth 0):", dict(c))
+    return c
+
+
+def plot_layer1_mask(state, l1_module, imag_grad_threshold=None, figsize=(4,4)):
+    """
+    Compute and plot the Layer-1 admissible edit mask and magnitude.
+    - state: a NumPy array (resized candidate / phi field).
+    - l1_module: the imported layer_minus_one module (e.g., import itt_solver.layer_minus_one as l1).
+    - imag_grad_threshold: optional threshold to pass through to admissible_edit_mask.
+    """
+    if state is None:
+        print("No state provided.")
+        return
+
+    try:
+        mask, mag = l1_module.admissible_edit_mask(state, imag_grad_threshold)
+    except Exception as e:
+        print("Error computing Layer-1 mask:", e)
+        return
+
+    plt.figure(figsize=figsize)
+    plt.imshow(mask, cmap='gray')
+    plt.title('Layer-1 admissible edit mask')
+    plt.axis('off')
+    plt.show()
+
+    plt.figure(figsize=figsize)
+    plt.imshow(mag, cmap='magma')
+    plt.title('||∇Im(Φ_c)|| magnitude')
+    plt.colorbar()
+    plt.axis('off')
+    plt.show()
+
+
+def assert_states_shape(states, phi_target):
+    """Assert all states have the same shape as phi_target. Returns True if OK, False otherwise."""
+    if not states:
+        print("No states provided.")
+        return False
+    target_shape = tuple(phi_target.shape)
+    for i, s in enumerate(states):
+        if tuple(s.shape) != target_shape:
+            print(f"State {i} shape mismatch: {s.shape} != {target_shape}")
+            return False
+    print("All states match target shape:", target_shape)
+    return True
+
+
+# --- New tests added below ---------------------------------------------------
+
+def transform_effect_test(transform, phi, show_diff=False):
+    """
+    Apply a Transform-like object (must have .apply(phi)) to phi and return:
+      - changed_count: number of cells that differ after transform
+      - diff_map: boolean array where True indicates changed cells
+    If show_diff True, also plot the diff map.
+    """
+    if not hasattr(transform, 'apply'):
+        raise ValueError("transform must have an apply(phi) method")
+    phi = np.array(phi, dtype=float)
+    phi_after = transform.apply(phi.copy())
+    if phi_after.shape != phi.shape:
+        # try to resize phi_after to phi shape by simple tiling if shapes differ
+        from .solver_core import tile_transform
+        try:
+            phi_after = tile_transform(phi_after, phi.shape)
+        except Exception:
+            # fallback: broadcast if possible
+            phi_after = np.broadcast_to(phi_after, phi.shape)
+    diff_map = (phi_after != phi)
+    changed_count = int(np.sum(diff_map))
+    if show_diff:
+        plt.figure(figsize=(4,4))
+        plt.imshow(diff_map, cmap='gray')
+        plt.title(f'Transform effect diff (changed={changed_count})')
+        plt.axis('off')
+        plt.show()
+    return changed_count, diff_map
+
+
+def sigma_decrease_smoke_test(beam_func, phi_in, phi_target, atomic_library,
+                              beam_kwargs=None):
+    """
+    Run a relaxed beam (lock_coeff=0, max_fraction=1.0) to check whether sigma can decrease.
+    beam_func: callable with signature beam_func(phi_in, phi_target, atomic_library, **kwargs)
+    Returns a dict with keys:
+      - 'sigmas': sigma trace list
+      - 'decreased': True if final sigma < initial sigma
+      - 'result': tuple returned by beam_func
+    """
+    beam_kwargs = dict(beam_kwargs or {})
+    # enforce relaxed settings for smoke test
+    beam_kwargs.setdefault('lock_coeff', 0.0)
+    beam_kwargs.setdefault('max_fraction', 1.0)
+    beam_kwargs.setdefault('enable_layer_minus_one', True)
+    beam_kwargs.setdefault('boundary_source', 'target')
+    # allow all quantized symbols for this test
+    beam_kwargs.setdefault('allowed_symbols', list(range(10)))
+
+    result = beam_func(phi_in, phi_target, atomic_library, **beam_kwargs)
+    # beam_func expected to return (T_best, phi_best, states, sigmas, logs)
+    if not result or len(result) < 4:
+        return {'sigmas': None, 'decreased': False, 'result': result}
+
+    sigmas = result[3]
+    decreased = False
+    if sigmas and len(sigmas) >= 2:
+        decreased = float(sigmas[-1]) < float(sigmas[0])
+    return {'sigmas': sigmas, 'decreased': decreased, 'result': result}
+
+
+def run_all_quick_checks(states, logs, phi_target, l1_module=None, lock_coeff=0.01,
+                         beam_smoke_runner=None, phi_in=None, atomic_library=None):
+    """
+    Convenience runner that executes the basic checks and the smoke sigma test (if beam_smoke_runner provided).
+    Returns a dict of results.
+    """
+    results = {}
+    results['shape_ok'] = assert_states_shape(states, phi_target)
+    try:
+        print_depth0_logs(logs)
+        results['print_logs'] = True
+    except Exception:
+        results['print_logs'] = False
+
+    results['first_accepted_score_ok'] = check_first_accepted_score(logs, lock_coeff=lock_coeff)
+    results['gate_summary'] = gate_failure_summary(logs)
+    if l1_module is not None:
+        try:
+            print("Plotting Layer-1 mask for states[0]...")
+            plot_layer1_mask(states[0], l1_module)
+            results['layer1_plotted'] = True
+        except Exception:
+            results['layer1_plotted'] = False
+
+    if beam_smoke_runner is not None and phi_in is not None and atomic_library is not None:
+        print("Running sigma decrease smoke test (relaxed beam)...")
+        smoke = sigma_decrease_smoke_test(beam_smoke_runner, phi_in, phi_target, atomic_library)
+        results['smoke_sigmas'] = smoke.get('sigmas')
+        results['smoke_decreased'] = smoke.get('decreased')
+    return results
+
+def run_atomic_effects(task_input=None, params=None, target_shape=(9,9)):
+    """
+    Build the default atomic library and report whether each transform changes the provided task_input.
+    Prints shape and changed-cell counts.
+
+    - task_input: either a NumPy array or a small grid (list of lists). If None, a default 3x3 example is used.
+    - params: dict of parameters passed to default_atomic_factory (optional).
+    - target_shape: tuple used to construct the atomic library via default_atomic_factory.
+    """
+    # lazy imports to avoid top-level dependency issues
+    from .experiment_driver import default_atomic_factory
+    from .solver_core import initialize_potential, tile_transform
+    import numpy as _np
+
+    if task_input is None:
+        task_input = [[0,7,7],[7,7,7],[0,7,7]]
+    phi_in = initialize_potential(task_input)
+
+    if params is None:
+        params = {'beam_width':6,'max_depth':3,'lock_coeff':0.0,'max_fraction':1.0,'enable_layer_minus_one':True,'boundary_source':'target'}
+
+    task_stub = {'target_shape': target_shape}
+    atomic_library = default_atomic_factory(params, task_stub)
+
+    print("Testing atomic library transforms on input shape", phi_in.shape)
+    results = []
+    for T in atomic_library:
+        try:
+            phi_after = T.apply(phi_in.copy())
+        except Exception as e:
+            print(f"{repr(T)} raised exception during apply(): {e}")
+            results.append({'transform': repr(T), 'error': str(e)})
+            continue
+
+        # If shapes differ, try to tile phi_after to phi_in shape for comparison
+        if phi_after.shape != phi_in.shape:
+            try:
+                phi_after_resized = tile_transform(phi_after, phi_in.shape)
+            except Exception:
+                try:
+                    phi_after_resized = _np.broadcast_to(phi_after, phi_in.shape)
+                except Exception:
+                    phi_after_resized = None
+        else:
+            phi_after_resized = phi_after
+
+        if phi_after_resized is None:
+            changed = None
+        else:
+            diff_map = (phi_after_resized != phi_in)
+            changed = int(_np.sum(diff_map))
+
+        print(repr(T), "-> out shape", None if phi_after is None else phi_after.shape, "changed cells:", changed)
+        results.append({'transform': repr(T), 'out_shape': None if phi_after is None else phi_after.shape, 'changed_cells': changed})
+
+    return results

itt_solver/transforms.py

@@ -0,0 +1,50 @@
+import numpy as np
+from .solver_core import tile_transform, fill_enclosed
+
+class Transform:
+    def __init__(self, func, name):
+        self.func = func
+        self.name = name
+    def apply(self, phi):
+        return self.func(phi)
+    def __repr__(self):
+        return f"<Transform {self.name}>"
+
+def tile_to_target_shifted(shift=(1,1), tile_factor=3):
+    """
+    Tile the input to a canvas tile_factor x tile_factor times, apply a small roll,
+    and return the tiled canvas. This guarantees the transform output differs from input.
+    """
+    def fn(phi):
+        h_in, w_in = phi.shape
+        out_shape = (h_in * tile_factor, w_in * tile_factor)
+        tiled = tile_transform(phi, out_shape)
+        tiled = np.roll(tiled, shift=shift, axis=(0,1))
+        return tiled
+    return Transform(fn, f"tile_to_target_shift{shift}")
+
+def FillEnclosedHarmonic(boundary_mask=None):
+    def fn(phi):
+        bm = (phi != 0) if boundary_mask is None else boundary_mask
+        return fill_enclosed(phi, bm)
+    return Transform(fn, "FillEnclosedHarmonic")
+
+def Rotate(k=1):
+    def fn(phi):
+        return np.rot90(phi, k)
+    return Transform(fn, f"Rotate_{90*k}")
+
+def Reflect(axis='h'):
+    def fn(phi):
+        if axis == 'h':
+            return np.flipud(phi)
+        return np.fliplr(phi)
+    return Transform(fn, f"Reflect_{axis}")
+
+def ColorMap(mapping):
+    def fn(phi):
+        out = phi.copy()
+        for k,v in mapping.items():
+            out[phi==k] = v
+        return out
+    return Transform(fn, f"ColorMap_{mapping}")

itt_solver/viz.py

@@ -0,0 +1,30 @@
+import matplotlib.pyplot as plt
+
+def show_grid(grid, title=None, cmap='tab10', vmin=0, vmax=9):
+    plt.figure(figsize=(3, 3))
+    plt.imshow(grid, cmap=cmap, vmin=vmin, vmax=vmax)
+    plt.colorbar(fraction=0.046, pad=0.04)
+    if title:
+        plt.title(title)
+    plt.axis('off')
+    plt.show()
+
+def show_side_by_side(grids, titles, cmap='tab10', vmin=0, vmax=9):
+    n = len(grids)
+    plt.figure(figsize=(3 * n, 3))
+    for i, (g, t) in enumerate(zip(grids, titles)):
+        plt.subplot(1, n, i + 1)
+        plt.imshow(g, cmap=cmap, vmin=vmin, vmax=vmax)
+        plt.title(t)
+        plt.axis('off')
+    plt.tight_layout()
+    plt.show()
+
+def plot_sigma_trace(sigmas, title="σ trace"):
+    plt.figure(figsize=(4, 3))
+    plt.plot(range(len(sigmas)), sigmas, marker='o')
+    plt.xlabel("Step")
+    plt.ylabel("σ (L1 residue)")
+    plt.title(title)
+    plt.grid(True, alpha=0.3)
+    plt.show()

itt_solver/wandb_runner.py

@@ -0,0 +1,117 @@
+import os
+import json
+import numpy as np
+import matplotlib.pyplot as plt
+import wandb
+
+from typing import Any, Dict
+from itt_solver import experiment_driver as ed
+
+def _save_phi_image(phi: np.ndarray, path: str):
+    """Save a small visualization of phi to path (PNG)."""
+    fig, ax = plt.subplots(figsize=(4,4))
+    ax.imshow(phi, cmap='tab20')
+    ax.axis('off')
+    fig.savefig(path, bbox_inches='tight', dpi=150)
+    plt.close(fig)
+
+def run_and_log_wandb(task: Dict[str, Any],
+                      atomic_library,
+                      params: Dict[str, Any],
+                      out_dir: str = "experiments",
+                      wandb_project: str = "itt_solver",
+                      wandb_entity: str | None = None,
+                      resume: bool | str = "allow"):
+    """
+    Run a single experiment via experiment_driver.run_single and log results to W&B
+    using a context manager (with wandb.init(...) as run: ...).
+
+    - task: dict with keys 'name','input','target','target_shape' (same format used by experiment_driver).
+    - atomic_library: list of Transform objects for the run.
+    - params: hyperparameter dict passed to run_single (will also be logged to W&B).
+    - out_dir: local directory where run_single writes artifacts (result.json, logs.json, phi_best.npy).
+    - wandb_project: W&B project name.
+    - wandb_entity: optional W&B entity (team/user).
+    - anonymous: "allow" to log anonymously if no API key is set, or False/None to require login.
+    """
+    # Run the experiment locally first (this writes files under out_dir)
+    result = ed.run_single(task, atomic_library, params, out_dir)
+
+    # Compose artifact base name used by run_single (it uses timestamped base)
+    # We expect run_single saved files: <base>_phi_best.npy, <base>_result.json, <base>_logs.json
+    # result dict contains 'transform' and 'final_sigma' and 'sigma_trace' etc.
+    # Find the most recent files for this task in out_dir
+    base_prefix = task.get('name', 'task')
+    # find candidate files saved by run_single (match base prefix and timestamp)
+    files = sorted([f for f in os.listdir(out_dir) if f.startswith(base_prefix) and ("_result.json" in f or "_phi_best.npy" in f or "_logs.json" in f)])
+    # group by base (strip suffix)
+    bases = sorted({f.rsplit("_result",1)[0].rsplit("_phi_best",1)[0].rsplit("_logs",1)[0] for f in files})
+    # choose last base if multiple
+    base = bases[-1] if bases else None
+
+    # Prepare artifact file paths if available
+    phi_path = os.path.join(out_dir, base + "_phi_best.npy") if base else None
+    result_path = os.path.join(out_dir, base + "_result.json") if base else None
+    logs_path = os.path.join(out_dir, base + "_logs.json") if base else None
+
+    # Start W&B run using context manager
+    with wandb.init(project=wandb_project,
+                    entity=wandb_entity,
+                    config=params,
+                    name=f"{task.get('name','task')}_{int(wandb.util.generate_id(), 36)}",
+                    reinit=True,
+                    resume=resume) as run:
+
+        # Log scalar metrics
+        try:
+            run.log({
+                "final_sigma": result.get("final_sigma"),
+                "time_s": result.get("time_s"),
+                "states_count": result.get("states_count")
+            })
+        except Exception:
+            pass
+
+        # Log sigma trace as a series
+        try:
+            run.log({"sigma_trace": result.get("sigma_trace", [])})
+        except Exception:
+            pass
+
+        # Attach artifacts (phi_best, result.json, logs.json) if present
+        try:
+            art = wandb.Artifact(f"{task.get('name','task')}_run", type="itt_run")
+            if phi_path and os.path.exists(phi_path):
+                art.add_file(phi_path, name="phi_best.npy")
+            if result_path and os.path.exists(result_path):
+                art.add_file(result_path, name="result.json")
+            if logs_path and os.path.exists(logs_path):
+                art.add_file(logs_path, name="logs.json")
+            run.log_artifact(art)
+        except Exception:
+            pass
+
+        # Log a small image preview of phi_best
+        try:
+            if phi_path and os.path.exists(phi_path):
+                phi = np.load(phi_path)
+                tmp_png = os.path.join(out_dir, base + "_phi_preview.png")
+                _save_phi_image(phi, tmp_png)
+                run.log({"phi_best_image": wandb.Image(tmp_png)})
+                # optionally remove tmp_png
+                try:
+                    os.remove(tmp_png)
+                except Exception:
+                    pass
+        except Exception:
+            pass
+
+    # Return the local result dict for further local analysis
+    return result
+
+
+# Example usage (call from a notebook cell):
+# from itt_solver.wandb_runner import run_and_log_wandb
+# res = run_and_log_wandb(task, atomic_library, params, out_dir="experiments",
+#                         wandb_project="itt_solver", anonymous="allow")
+# print("W&B logged run, final sigma:", res.get("final_sigma"))

scripts/fix_and_inspect_logs.py

@@ -0,0 +1,107 @@
+import glob, json, numpy as np, os
+from pprint import pprint
+
+def load_latest(pattern):
+    files = sorted(glob.glob(pattern))
+    return files[-1] if files else None
+
+logs_path = load_latest("experiments/*_logs.json")
+phi_path = load_latest("experiments/*_phi_best.npy")
+res_path = load_latest("experiments/*_result.json")
+
+print("logs:", logs_path)
+print("phi_best:", phi_path)
+print("result:", res_path)
+
+if not logs_path:
+    raise SystemExit("No logs file found")
+
+logs = json.load(open(logs_path))
+res = json.load(open(res_path)) if res_path else {}
+
+# coerce gate values to booleans for all depth entries
+def coerce_gates(g):
+    if not isinstance(g, dict):
+        return g
+    out = {}
+    for k,v in g.items():
+        if isinstance(v, str):
+            lv = v.strip().lower()
+            if lv in ("true","1","yes"):
+                out[k] = True
+            elif lv in ("false","0","no"):
+                out[k] = False
+            else:
+                # try numeric
+                try:
+                    out[k] = bool(int(v))
+                except Exception:
+                    out[k] = v
+        else:
+            out[k] = v
+    return out
+
+for depth_idx, depth in enumerate(logs):
+    for entry in depth:
+        if 'gates' in entry:
+            entry['gates'] = coerce_gates(entry['gates'])
+
+# attach phi_best into the first accepted entry (if not present)
+accepted_entry = None
+for entry in logs[0]:
+    if entry.get('accepted'):
+        accepted_entry = entry
+        break
+
+phi = np.load(phi_path) if phi_path else None
+if accepted_entry is not None:
+    if 'candidate_array' not in accepted_entry:
+        accepted_entry['candidate_array'] = phi.tolist() if phi is not None else None
+
+# recompute residue for that candidate vs a target (use target from your task if available)
+# If result does not contain the target, you can edit TARGET_GRID below to match your task.
+TARGET_GRID = [
+  [0,0,0,0,7,7,0,7,7],
+  [0,0,0,7,7,7,7,7,7],
+  [0,0,0,0,7,7,0,7,7],
+  [0,7,7,0,7,7,0,7,7],
+  [7,7,7,7,7,7,7,7,7],
+  [0,7,7,0,7,7,0,7,7],
+  [0,0,0,0,7,7,0,7,7],
+  [0,7,7,0,7,7,0,7,7],
+  [0,0,0,0,7,7,0,7,7],
+]
+TARGET = np.array(TARGET_GRID, dtype=int)
+
+def tile_transform(phi, out_shape):
+    # minimal tile_transform fallback: tile or crop to out_shape
+    a = np.array(phi)
+    h_out, w_out = out_shape
+    h_in, w_in = a.shape
+    reps_h = (h_out + h_in - 1) // h_in
+    reps_w = (w_out + w_in - 1) // w_in
+    tiled = np.tile(a, (reps_h, reps_w))
+    return tiled[:h_out, :w_out]
+
+if accepted_entry is not None and accepted_entry.get('candidate_array') is not None:
+    cand = np.array(accepted_entry['candidate_array'], dtype=float)
+    if cand.shape != TARGET.shape:
+        cand_resized = tile_transform(cand, TARGET.shape)
+    else:
+        cand_resized = cand
+    cand_q = np.rint(cand_resized).astype(int)
+    l1 = float(np.sum(np.abs(cand_q - TARGET)))
+    print("Recomputed L1 residue for first accepted candidate:", l1)
+    print("Candidate unique values:", np.unique(cand_q))
+    diff = (cand_q != TARGET).astype(int)
+    print("Changed cells count:", int(diff.sum()))
+    print("Diff map (1=diff):")
+    print(diff)
+else:
+    print("No candidate array available in logs or phi_best missing.")
+
+# write fixed logs copy
+fixed_path = logs_path.replace("_logs.json", "_logs.fixed.json")
+with open(fixed_path, "w") as fh:
+    json.dump(logs, fh, indent=2)
+print("Wrote fixed logs to", fixed_path)