Phase 5 release: 7 TopK SAEs + specificity / null-steering JSON artifacts

.gitattributes

@@ -1,35 +1,3 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
 *.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,74 @@
+---
+license: mit
+library_name: sae_lens
+tags:
+  - interpretability
+  - sparse-autoencoder
+  - sae
+  - mechanistic-interpretability
+  - topk-sae
+---
+
+# InterpGPT — Phase 5 TopK SAEs
+
+Seven sparse autoencoders trained on the residual stream
+(`hook_resid_post`) of the two Phase 1 InterpGPT models
+([`interpgpt-standard-23M`](https://huggingface.co/connaaa/interpgpt-standard-23M),
+[`interpgpt-adhd-23M`](https://huggingface.co/connaaa/interpgpt-adhd-23M)).
+
+| Model | Layer | Hook | Subdir |
+|---|---|---|---|
+| standard | 0 | hook_resid_post | `standard_L0_hook_resid_post/` |
+| standard | 1 | hook_resid_post | `standard_L1_hook_resid_post/` |
+| standard | 2 | hook_resid_post | `standard_L2_hook_resid_post/` |
+| standard | 3 | hook_resid_post | `standard_L3_hook_resid_post/` |
+| adhd     | 1 | hook_resid_post | `adhd_L1_hook_resid_post/` |
+| adhd     | 2 | hook_resid_post | `adhd_L2_hook_resid_post/` |
+| adhd     | 3 | hook_resid_post | `adhd_L3_hook_resid_post/` |
+
+## Training setup
+
+- Library: [`sae_lens`](https://github.com/jbloomAus/SAELens) TopK training SAE
+- `k = 40`, `d_sae = 4096`
+- All 7 SAEs pass quality gates: FVE 0.87–0.92, dead features < 2%
+
+## Phase 1 result artifacts (included)
+
+- `feature_diff.json` — 312 ADHD-L2 features firing at step-onset that the
+  standard model lacks. Feature 2504 highlighted (2000× cross-model asymmetry).
+- `causal_nulls_per_seed.json` — 5-seed causal ablation nulls for the L3 swap.
+- `deepdive_steering.json` — feature 2504 four-panel steering results (all four
+  interventions Δ within ±0.025 of null, below 2 SEM).
+- `three_probes.json` — three-probe causal-check outputs.
+
+## Loading
+
+### Minimal
+
+```python
+from huggingface_hub import snapshot_download
+from sae_lens import SAE
+
+repo = "connaaa/interpgpt-sae-phase5"
+local = snapshot_download(repo_id=repo, allow_patterns=["adhd_L2_hook_resid_post/*"])
+sae = SAE.load_from_disk(f"{local}/adhd_L2_hook_resid_post")
+print(sae)
+```
+
+### Pull everything
+
+```python
+from huggingface_hub import snapshot_download
+local = snapshot_download(repo_id="connaaa/interpgpt-sae-phase5")
+```
+
+## Reproducibility
+
+Training script: `phase5_sae.py` in
+[github.com/cwklurks/interpgpt](https://github.com/cwklurks/interpgpt).
+Production driver: `phase5_production.py`. Four-panel steering harness:
+`phase5_steering_ci.py`.
+
+## License
+
+MIT.

adhd_L1_hook_resid_post/config.json

@@ -0,0 +1,19 @@
+{
+  "model": "adhd",
+  "layer": 1,
+  "d_sae": 4096,
+  "k": 40,
+  "normalize_activations": "expected_average_only_in",
+  "n_tokens": 10000000,
+  "batch_size": 4096,
+  "lr": 0.0003,
+  "final_training": {
+    "step": 2441,
+    "loss": 3.830482244491577,
+    "l0": 40.0,
+    "fve": 0.9020028367114843,
+    "n_dead": 5,
+    "mse": 0.0073381345719099045
+  },
+  "dead_pct_heldout": 1.3671875
+}

adhd_L1_hook_resid_post/history.json

@@ -0,0 +1,106 @@
+[
+  {
+    "step": 200,
+    "loss": 7.214208602905273,
+    "l0": 40.0,
+    "fve": 0.8173658289454013,
+    "n_dead": 0,
+    "mse": 0.014090251177549362
+  },
+  {
+    "step": 400,
+    "loss": 11.098737716674805,
+    "l0": 40.0,
+    "fve": 0.8590923953729561,
+    "n_dead": 488,
+    "mse": 0.011645233258605003
+  },
+  {
+    "step": 600,
+    "loss": 7.969749450683594,
+    "l0": 40.0,
+    "fve": 0.8684529110409115,
+    "n_dead": 148,
+    "mse": 0.010252815671265125
+  },
+  {
+    "step": 800,
+    "loss": 5.30736780166626,
+    "l0": 40.0,
+    "fve": 0.8735112871918154,
+    "n_dead": 18,
+    "mse": 0.00969421211630106
+  },
+  {
+    "step": 1000,
+    "loss": 4.6798553466796875,
+    "l0": 40.0,
+    "fve": 0.8860032616097528,
+    "n_dead": 4,
+    "mse": 0.0090001430362463
+  },
+  {
+    "step": 1200,
+    "loss": 4.3763346672058105,
+    "l0": 40.0,
+    "fve": 0.8946085205816868,
+    "n_dead": 1,
+    "mse": 0.008514291606843472
+  },
+  {
+    "step": 1400,
+    "loss": 4.231931209564209,
+    "l0": 40.0,
+    "fve": 0.8952190554998088,
+    "n_dead": 0,
+    "mse": 0.008265490643680096
+  },
+  {
+    "step": 1600,
+    "loss": 4.067999839782715,
+    "l0": 40.0,
+    "fve": 0.8984853600784243,
+    "n_dead": 0,
+    "mse": 0.007945312187075615
+  },
+  {
+    "step": 1800,
+    "loss": 3.9912521839141846,
+    "l0": 40.0,
+    "fve": 0.897026286041148,
+    "n_dead": 0,
+    "mse": 0.007795413956046104
+  },
+  {
+    "step": 2000,
+    "loss": 3.987309694290161,
+    "l0": 40.0,
+    "fve": 0.9006864810283836,
+    "n_dead": 1,
+    "mse": 0.007757413201034069
+  },
+  {
+    "step": 2200,
+    "loss": 3.8046491146087646,
+    "l0": 40.0,
+    "fve": 0.9039961047080753,
+    "n_dead": 1,
+    "mse": 0.007402045652270317
+  },
+  {
+    "step": 2400,
+    "loss": 3.7656211853027344,
+    "l0": 40.0,
+    "fve": 0.9085856363510846,
+    "n_dead": 4,
+    "mse": 0.00724159087985754
+  },
+  {
+    "step": 2441,
+    "loss": 3.830482244491577,
+    "l0": 40.0,
+    "fve": 0.9020028367114843,
+    "n_dead": 5,
+    "mse": 0.0073381345719099045
+  }
+]

adhd_L1_hook_resid_post/sae.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1bfe52fb2ee7e5992b148d7aecd779fa71dfe2ebfd701619c93b3ebe01c7db11
+size 16798005

adhd_L2_hook_resid_post/config.json

@@ -0,0 +1,19 @@
+{
+  "model": "adhd",
+  "layer": 2,
+  "d_sae": 4096,
+  "k": 40,
+  "normalize_activations": "expected_average_only_in",
+  "n_tokens": 10000000,
+  "batch_size": 4096,
+  "lr": 0.0003,
+  "final_training": {
+    "step": 2441,
+    "loss": 7.990331649780273,
+    "l0": 40.0,
+    "fve": 0.8937416797380295,
+    "n_dead": 1,
+    "mse": 0.015545391477644444
+  },
+  "dead_pct_heldout": 1.46484375
+}

adhd_L2_hook_resid_post/history.json

@@ -0,0 +1,106 @@
+[
+  {
+    "step": 200,
+    "loss": 14.795553207397461,
+    "l0": 40.0,
+    "fve": 0.8293616575146343,
+    "n_dead": 0,
+    "mse": 0.028897564858198166
+  },
+  {
+    "step": 400,
+    "loss": 22.638526916503906,
+    "l0": 40.0,
+    "fve": 0.8359017185027313,
+    "n_dead": 663,
+    "mse": 0.023937705904245377
+  },
+  {
+    "step": 600,
+    "loss": 19.389955520629883,
+    "l0": 40.0,
+    "fve": 0.8645987532704418,
+    "n_dead": 222,
+    "mse": 0.021326521411538124
+  },
+  {
+    "step": 800,
+    "loss": 11.63227653503418,
+    "l0": 40.0,
+    "fve": 0.8760963178028377,
+    "n_dead": 35,
+    "mse": 0.02005489356815815
+  },
+  {
+    "step": 1000,
+    "loss": 9.950961112976074,
+    "l0": 40.0,
+    "fve": 0.8784041379676274,
+    "n_dead": 8,
+    "mse": 0.018850065767765045
+  },
+  {
+    "step": 1200,
+    "loss": 9.30925178527832,
+    "l0": 40.0,
+    "fve": 0.8774098117815495,
+    "n_dead": 2,
+    "mse": 0.018041376024484634
+  },
+  {
+    "step": 1400,
+    "loss": 8.932659149169922,
+    "l0": 40.0,
+    "fve": 0.8964732688104713,
+    "n_dead": 1,
+    "mse": 0.01737871766090393
+  },
+  {
+    "step": 1600,
+    "loss": 8.559926986694336,
+    "l0": 40.0,
+    "fve": 0.8907664732630304,
+    "n_dead": 0,
+    "mse": 0.016718603670597076
+  },
+  {
+    "step": 1800,
+    "loss": 8.413354873657227,
+    "l0": 40.0,
+    "fve": 0.8881362105445537,
+    "n_dead": 0,
+    "mse": 0.01643233373761177
+  },
+  {
+    "step": 2000,
+    "loss": 8.195343017578125,
+    "l0": 40.0,
+    "fve": 0.8944998626527212,
+    "n_dead": 1,
+    "mse": 0.015944253653287888
+  },
+  {
+    "step": 2200,
+    "loss": 8.069483757019043,
+    "l0": 40.0,
+    "fve": 0.9000171435610246,
+    "n_dead": 1,
+    "mse": 0.01569938287138939
+  },
+  {
+    "step": 2400,
+    "loss": 7.750567436218262,
+    "l0": 40.0,
+    "fve": 0.9054223585460653,
+    "n_dead": 0,
+    "mse": 0.015137827023863792
+  },
+  {
+    "step": 2441,
+    "loss": 7.990331649780273,
+    "l0": 40.0,
+    "fve": 0.8937416797380295,
+    "n_dead": 1,
+    "mse": 0.015545391477644444
+  }
+]

adhd_L2_hook_resid_post/sae.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:725267a3fca5866bcf9c8dcab2fd77cddd7b27ecc42cb631250395d36cbcf62a
+size 16798005

adhd_L3_hook_resid_post/config.json

@@ -0,0 +1,19 @@
+{
+  "model": "adhd",
+  "layer": 3,
+  "d_sae": 4096,
+  "k": 40,
+  "normalize_activations": "expected_average_only_in",
+  "n_tokens": 10000000,
+  "batch_size": 4096,
+  "lr": 0.0003,
+  "final_training": {
+    "step": 2441,
+    "loss": 16.80815315246582,
+    "l0": 40.0,
+    "fve": 0.8691270860953982,
+    "n_dead": 0,
+    "mse": 0.032828424125909805
+  },
+  "dead_pct_heldout": 2.2216796875
+}

adhd_L3_hook_resid_post/history.json

@@ -0,0 +1,106 @@
+[
+  {
+    "step": 200,
+    "loss": 29.05353546142578,
+    "l0": 40.0,
+    "fve": 0.7888412205173247,
+    "n_dead": 0,
+    "mse": 0.05674518644809723
+  },
+  {
+    "step": 400,
+    "loss": 44.51063537597656,
+    "l0": 40.0,
+    "fve": 0.828121429866145,
+    "n_dead": 769,
+    "mse": 0.04685475304722786
+  },
+  {
+    "step": 600,
+    "loss": 40.60480880737305,
+    "l0": 40.0,
+    "fve": 0.8353118397308505,
+    "n_dead": 239,
+    "mse": 0.04323723167181015
+  },
+  {
+    "step": 800,
+    "loss": 22.681713104248047,
+    "l0": 40.0,
+    "fve": 0.8478801929634973,
+    "n_dead": 29,
+    "mse": 0.03988751769065857
+  },
+  {
+    "step": 1000,
+    "loss": 19.90431785583496,
+    "l0": 40.0,
+    "fve": 0.8681096795485392,
+    "n_dead": 4,
+    "mse": 0.03827929496765137
+  },
+  {
+    "step": 1200,
+    "loss": 19.122100830078125,
+    "l0": 40.0,
+    "fve": 0.8613335049168376,
+    "n_dead": 2,
+    "mse": 0.037058740854263306
+  },
+  {
+    "step": 1400,
+    "loss": 18.61587142944336,
+    "l0": 40.0,
+    "fve": 0.854929133645014,
+    "n_dead": 2,
+    "mse": 0.03607767075300217
+  },
+  {
+    "step": 1600,
+    "loss": 17.96778106689453,
+    "l0": 40.0,
+    "fve": 0.8535506236950299,
+    "n_dead": 1,
+    "mse": 0.03495676815509796
+  },
+  {
+    "step": 1800,
+    "loss": 17.5817928314209,
+    "l0": 40.0,
+    "fve": 0.8671142899286272,
+    "n_dead": 1,
+    "mse": 0.03420582413673401
+  },
+  {
+    "step": 2000,
+    "loss": 17.37525749206543,
+    "l0": 40.0,
+    "fve": 0.8686602042934782,
+    "n_dead": 2,
+    "mse": 0.03367304056882858
+  },
+  {
+    "step": 2200,
+    "loss": 16.68439483642578,
+    "l0": 40.0,
+    "fve": 0.877142112507239,
+    "n_dead": 1,
+    "mse": 0.03245990723371506
+  },
+  {
+    "step": 2400,
+    "loss": 16.43060874938965,
+    "l0": 40.0,
+    "fve": 0.8754777867952352,
+    "n_dead": 0,
+    "mse": 0.03209103271365166
+  },
+  {
+    "step": 2441,
+    "loss": 16.80815315246582,
+    "l0": 40.0,
+    "fve": 0.8691270860953982,
+    "n_dead": 0,
+    "mse": 0.032828424125909805
+  }
+]

adhd_L3_hook_resid_post/sae.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f314c6a8d95b68b24e83107e45a2a41003088be5556ec13d5b3ed019f68f7ad5
+size 16798005

causal_nulls_per_seed.json

@@ -0,0 +1,104 @@
+{
+  "n_seeds": 4,
+  "n_prompts": 30,
+  "coefs": [
+    0.0,
+    2.0,
+    5.0,
+    10.0
+  ],
+  "results": {
+    "L2_up_std": {
+      "0.0": [
+        0.08018867924528301,
+        0.08653846153846154,
+        0.09090909090909091,
+        0.06862745098039216
+      ],
+      "2.0": [
+        0.08133971291866028,
+        0.08695652173913043,
+        0.08530805687203792,
+        0.07352941176470588
+      ],
+      "5.0": [
+        0.07655502392344497,
+        0.08695652173913043,
+        0.09268292682926829,
+        0.07881773399014778
+      ],
+      "10.0": [
+        0.08018867924528301,
+        0.08490566037735849,
+        0.11682242990654206,
+        0.08866995073891626
+      ]
+    },
+    "L2_down_adhd": {
+      "0.0": [
+        0.4488888888888889,
+        0.3949771689497717,
+        0.44794188861985473,
+        0.42403628117913833
+      ],
+      "2.0": [
+        0.43935926773455375,
+        0.382830626450116,
+        0.44364508393285373,
+        0.432183908045977
+      ],
+      "5.0": [
+        0.4387990762124711,
+        0.39902676399026765,
+        0.42857142857142855,
+        0.42755344418052255
+      ],
+      "10.0": [
+        0.39172749391727496,
+        0.3949771689497717,
+        0.40648379052369077,
+        0.4230769230769231
+      ]
+    },
+    "L2_zero_adhd": {
+      "baseline": [
+        0.4488888888888889,
+        0.3949771689497717,
+        0.44794188861985473,
+        0.42403628117913833
+      ],
+      "zero_step_onset": [
+        0.44742729306487694,
+        0.38672768878718533,
+        0.4348894348894349,
+        0.4217687074829932
+      ]
+    },
+    "L1_up_std": {
+      "0.0": [
+        0.08018867924528301,
+        0.08653846153846154,
+        0.09090909090909091,
+        0.06862745098039216
+      ],
+      "2.0": [
+        0.08095238095238096,
+        0.08695652173913043,
+        0.0861244019138756,
+        0.06862745098039216
+      ],
+      "5.0": [
+        0.0761904761904762,
+        0.07766990291262135,
+        0.08695652173913043,
+        0.07352941176470588
+      ],
+      "10.0": [
+        0.07692307692307693,
+        0.08163265306122448,
+        0.0784313725490196,
+        0.07881773399014778
+      ]
+    }
+  }
+}

deepdive_steering.json

@@ -0,0 +1,238 @@
+{
+  "feat_2504_deep_dive": {
+    "top_contexts": [
+      {
+        "act": 2.2637319564819336,
+        "pos_class": "step_onset",
+        "context": "for 2 minutes at easy pace *<|sep|>* sip water <|sep|> fist pump \u2014",
+        "variant": "adhd"
+      },
+      {
+        "act": 2.1833174228668213,
+        "pos_class": "step_onset",
+        "context": "acy resources to prevent recur rence *<|sep|>* deep breath <|sep|> close eyes briefly",
+        "variant": "adhd"
+      },
+      {
+        "act": 2.110867500305176,
+        "pos_class": "step_onset",
+        "context": "excess grout while wet using sponge *<|sep|>* close eyes briefly <|sep|> apply grout",
+        "variant": "adhd"
+      },
+      {
+        "act": 2.1062774658203125,
+        "pos_class": "step_onset",
+        "context": "donate remaining items to charity organizations *<|sep|>* quick stretch <|sep|> pause and breathe",
+        "variant": "adhd"
+      },
+      {
+        "act": 2.099095106124878,
+        "pos_class": "step_onset",
+        "context": "e red details and architectural elements *<|sep|>* quick stretch <|sep|> pause and breathe",
+        "variant": "adhd"
+      },
+      {
+        "act": 2.096402645111084,
+        "pos_class": "step_onset",
+        "context": "new skills to build competen ce *<|sep|>* roll shoulders <|sep|> seek feedback from",
+        "variant": "adhd"
+      },
+      {
+        "act": 2.071779251098633,
+        "pos_class": "step_onset",
+        "context": "up monthly and weekly spread s *<|sep|>* roll shoulders <|sep|> begin adding daily",
+        "variant": "adhd"
+      },
+      {
+        "act": 2.044790029525757,
+        "pos_class": "step_onset",
+        "context": "methodology and divide data collection tasks *<|sep|>* close eyes briefly <|sep|> quick focus",
+        "variant": "adhd"
+      },
+      {
+        "act": 2.0143749713897705,
+        "pos_class": "step_onset",
+        "context": "hydration products on longer training rides *<|sep|>* close eyes briefly <|sep|> complete a",
+        "variant": "adhd"
+      },
+      {
+        "act": 2.014024257659912,
+        "pos_class": "step_onset",
+        "context": "briefly <|sep|> refin ed training intensity *<|sep|>* shake out hands <|sep|> complete weeks",
+        "variant": "adhd"
+      },
+      {
+        "act": 2.0133330821990967,
+        "pos_class": "step_onset",
+        "context": "<|sep|> rinse all scrub bed surfaces *<|sep|>* roll shoulders <|sep|> scrub the lav",
+        "variant": "adhd"
+      },
+      {
+        "act": 2.0048322677612305,
+        "pos_class": "step_onset",
+        "context": "<|sep|> write results and discussion sections *<|sep|>* present findings in final report or",
+        "variant": "adhd"
+      },
+      {
+        "act": 1.988505244255066,
+        "pos_class": "step_onset",
+        "context": "roll shoulders <|sep|> increased cardio sessions *<|sep|>* deep breath <|sep|> 30 - second",
+        "variant": "adhd"
+      },
+      {
+        "act": 1.963397741317749,
+        "pos_class": "step_onset",
+        "context": "increased cardio sessions <|sep|> deep breath *<|sep|>* 30 - second stretch <|sep|> execute",
+        "variant": "adhd"
+      },
+      {
+        "act": 1.9471516609191895,
+        "pos_class": "step_onset",
+        "context": "with proper organization and archi ving *<|sep|>* wiggle fingers <|sep|> research family gen",
+        "variant": "adhd"
+      },
+      {
+        "act": 1.9434456825256348,
+        "pos_class": "step_onset",
+        "context": "until you see no soap residue *<|sep|>* sip water <|sep|> wipe down with",
+        "variant": "adhd"
+      },
+      {
+        "act": 1.936253547668457,
+        "pos_class": "step_onset",
+        "context": "for 5 minutes at easy pace *<|sep|>* quick stretch <|sep|> hydrate and refuel",
+        "variant": "adhd"
+      },
+      {
+        "act": 1.9354044198989868,
+        "pos_class": "step_onset",
+        "context": "online and wait for approval decision *<|sep|>* wiggle fingers <|sep|> fist pump \u2014",
+        "variant": "adhd"
+      },
+      {
+        "act": 1.9305871725082397,
+        "pos_class": "step_onset",
+        "context": "for user interface and user experience *<|sep|>* sip water <|sep|> choose development framework",
+        "variant": "adhd"
+      },
+      {
+        "act": 1.9305052757263184,
+        "pos_class": "step_onset",
+        "context": "<|sep|> add color and shading details *<|sep|>* deep breath <|sep|> pause and breathe",
+        "variant": "adhd"
+      }
+    ],
+    "top_cofire_partners": [
+      2418,
+      653,
+      1216,
+      225,
+      1131,
+      2959,
+      1650,
+      3953,
+      702,
+      352
+    ]
+  },
+  "symmetry_control_substantive_count": 5,
+  "symmetry_control_top_15": [
+    1406,
+    1258,
+    2097,
+    156,
+    697,
+    120,
+    1539,
+    1385,
+    3795,
+    1065,
+    531,
+    3280,
+    3927,
+    3967,
+    3846
+  ],
+  "steering_results": {
+    "std_baseline": {
+      "sep_rate": 0.08425135764158262,
+      "mean_step_count": 6.8125,
+      "spearman_rho": 0.5305203306664215,
+      "spearman_pval": 4.155845216990538e-07,
+      "regulation_rate": 0.09357798165137615,
+      "n_tokens_total": 6445,
+      "n_steps_total": 545,
+      "reached_end_rate": 0.975
+    },
+    "std_coef_+2.0": {
+      "sep_rate": 0.08416458852867831,
+      "mean_step_count": 6.775,
+      "spearman_rho": 0.530739309117657,
+      "spearman_pval": 4.102184726171915e-07,
+      "regulation_rate": 0.0940959409594096,
+      "n_tokens_total": 6416,
+      "n_steps_total": 542,
+      "reached_end_rate": 0.975
+    },
+    "std_coef_+5.0": {
+      "sep_rate": 0.08429777916796997,
+      "mean_step_count": 6.7625,
+      "spearman_rho": 0.5327086311005121,
+      "spearman_pval": 3.64818647432245e-07,
+      "regulation_rate": 0.09057301293900184,
+      "n_tokens_total": 6394,
+      "n_steps_total": 541,
+      "reached_end_rate": 0.975
+    },
+    "std_coef_+10.0": {
+      "sep_rate": 0.08608971454463073,
+      "mean_step_count": 7.1375,
+      "spearman_rho": 0.4614572824649014,
+      "spearman_pval": 1.6459708212616308e-05,
+      "regulation_rate": 0.08756567425569177,
+      "n_tokens_total": 6621,
+      "n_steps_total": 571,
+      "reached_end_rate": 0.975
+    },
+    "adhd_baseline": {
+      "sep_rate": 0.12945795615661468,
+      "mean_step_count": 15.0625,
+      "spearman_rho": 0.8158904455366707,
+      "spearman_pval": 3.038475633405349e-20,
+      "regulation_rate": 0.41327800829875516,
+      "n_tokens_total": 9169,
+      "n_steps_total": 1205,
+      "reached_end_rate": 0.7625
+    },
+    "adhd_coef_-2.0": {
+      "sep_rate": 0.12845057880676758,
+      "mean_step_count": 14.625,
+      "spearman_rho": 0.7957172265026485,
+      "spearman_pval": 1.1613642559127553e-18,
+      "regulation_rate": 0.4008547008547009,
+      "n_tokens_total": 8984,
+      "n_steps_total": 1170,
+      "reached_end_rate": 0.7875
+    },
+    "adhd_coef_-5.0": {
+      "sep_rate": 0.1263146117699709,
+      "mean_step_count": 14.3375,
+      "spearman_rho": 0.780151209846579,
+      "spearman_pval": 1.4775933869589044e-17,
+      "regulation_rate": 0.4167393199651264,
+      "n_tokens_total": 8938,
+      "n_steps_total": 1147,
+      "reached_end_rate": 0.7625
+    },
+    "adhd_coef_-10.0": {
+      "sep_rate": 0.12151652624756967,
+      "mean_step_count": 14.225,
+      "spearman_rho": 0.7471782967641961,
+      "spearman_pval": 1.7289858832058174e-15,
+      "regulation_rate": 0.3945518453427065,
+      "n_tokens_total": 9258,
+      "n_steps_total": 1138,
+      "reached_end_rate": 0.725
+    }
+  }
+}

feature_diff.json

@@ -0,0 +1,120 @@
+{
+  "sae_results": {
+    "standard_L1": {
+      "final_fve": 0.9156635482984559,
+      "final_l0": 40.0,
+      "dead_pct_train": 0.0244140625,
+      "dead_pct_heldout": 1.5380859375
+    },
+    "standard_L2": {
+      "final_fve": 0.9112300092129357,
+      "final_l0": 40.0,
+      "dead_pct_train": 0.0244140625,
+      "dead_pct_heldout": 1.220703125
+    },
+    "standard_L3": {
+      "final_fve": 0.8956661837197387,
+      "final_l0": 40.0,
+      "dead_pct_train": 0.0244140625,
+      "dead_pct_heldout": 1.025390625
+    },
+    "adhd_L1": {
+      "final_fve": 0.9020028367114843,
+      "final_l0": 40.0,
+      "dead_pct_train": 0.1220703125,
+      "dead_pct_heldout": 1.3671875
+    },
+    "adhd_L2": {
+      "final_fve": 0.8937416797380295,
+      "final_l0": 40.0,
+      "dead_pct_train": 0.0244140625,
+      "dead_pct_heldout": 1.46484375
+    },
+    "adhd_L3": {
+      "final_fve": 0.8691270860953982,
+      "final_l0": 40.0,
+      "dead_pct_train": 0.0,
+      "dead_pct_heldout": 2.2216796875
+    }
+  },
+  "primary_count": 312,
+  "symmetry_count": 139,
+  "layer_control_count": 194,
+  "primary_top_features": [
+    {
+      "feat_id": 2418,
+      "adhd_rate": 0.8974166512489319,
+      "std_rate": 0.0010833332780748606
+    },
+    {
+      "feat_id": 653,
+      "adhd_rate": 0.8386666774749756,
+      "std_rate": 0.0
+    },
+    {
+      "feat_id": 1216,
+      "adhd_rate": 0.6754166483879089,
+      "std_rate": 0.009583333507180214
+    },
+    {
+      "feat_id": 225,
+      "adhd_rate": 0.6380833387374878,
+      "std_rate": 0.0
+    },
+    {
+      "feat_id": 1131,
+      "adhd_rate": 0.6314166784286499,
+      "std_rate": 0.0
+    },
+    {
+      "feat_id": 2504,
+      "adhd_rate": 0.546999990940094,
+      "std_rate": 0.0005000000237487257
+    },
+    {
+      "feat_id": 1650,
+      "adhd_rate": 0.5131666660308838,
+      "std_rate": 8.333333244081587e-05
+    },
+    {
+      "feat_id": 2959,
+      "adhd_rate": 0.4754999876022339,
+      "std_rate": 0.00016666666488163173
+    },
+    {
+      "feat_id": 3953,
+      "adhd_rate": 0.4663333296775818,
+      "std_rate": 0.00016666666488163173
+    },
+    {
+      "feat_id": 352,
+      "adhd_rate": 0.42516666650772095,
+      "std_rate": 0.0016666667070239782
+    },
+    {
+      "feat_id": 702,
+      "adhd_rate": 0.4244999885559082,
+      "std_rate": 0.0
+    },
+    {
+      "feat_id": 2505,
+      "adhd_rate": 0.4099166691303253,
+      "std_rate": 0.0
+    },
+    {
+      "feat_id": 1156,
+      "adhd_rate": 0.3932499885559082,
+      "std_rate": 0.0
+    },
+    {
+      "feat_id": 2512,
+      "adhd_rate": 0.38883334398269653,
+      "std_rate": 0.0
+    },
+    {
+      "feat_id": 1835,
+      "adhd_rate": 0.3605000078678131,
+      "std_rate": 0.00016666666488163173
+    }
+  ]
+}

loading_example.py

@@ -0,0 +1,10 @@
+"""
+Minimal example: load one of the InterpGPT Phase 5 TopK SAEs from HuggingFace.
+"""
+from huggingface_hub import snapshot_download
+from sae_lens import SAE
+
+repo_id = "connaaa/interpgpt-sae-phase5"
+local = snapshot_download(repo_id=repo_id, allow_patterns=["adhd_L2_hook_resid_post/*"])
+sae = SAE.load_from_disk(f"{local}/adhd_L2_hook_resid_post")
+print(sae)

standard_L0_hook_resid_post/analysis.json

@@ -0,0 +1,444 @@
+{
+  "act_rate_distribution": {
+    "mean": 0.009765625,
+    "dead_count": 768,
+    "high_count": 0
+  },
+  "sample_features": [
+    {
+      "feature_id": 159,
+      "act_rate": 0.1606599986553192,
+      "top_examples": [
+        {
+          "activation": 0.5935809016227722,
+          "position": 10,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5935809016227722,
+          "position": 10,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5628541707992554,
+          "position": 22,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5628541707992554,
+          "position": 22,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5378819108009338,
+          "position": 8,
+          "variant": "standard"
+        }
+      ]
+    },
+    {
+      "feature_id": 836,
+      "act_rate": 0.12184999883174896,
+      "top_examples": [
+        {
+          "activation": 0.6280968189239502,
+          "position": 15,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.6280968189239502,
+          "position": 15,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5908636450767517,
+          "position": 1,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5908636450767517,
+          "position": 1,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5908636450767517,
+          "position": 1,
+          "variant": "standard"
+        }
+      ]
+    },
+    {
+      "feature_id": 2203,
+      "act_rate": 0.1174900010228157,
+      "top_examples": [
+        {
+          "activation": 0.6077227592468262,
+          "position": 7,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.6077227592468262,
+          "position": 7,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.6077227592468262,
+          "position": 7,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.6077227592468262,
+          "position": 7,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.6077227592468262,
+          "position": 7,
+          "variant": "standard"
+        }
+      ]
+    },
+    {
+      "feature_id": 3106,
+      "act_rate": 0.11097999662160873,
+      "top_examples": [
+        {
+          "activation": 0.458252489566803,
+          "position": 3,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.458252489566803,
+          "position": 3,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.45644882321357727,
+          "position": 47,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.45644882321357727,
+          "position": 47,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.4534014165401459,
+          "position": 46,
+          "variant": "standard"
+        }
+      ]
+    },
+    {
+      "feature_id": 2939,
+      "act_rate": 0.10080999881029129,
+      "top_examples": [
+        {
+          "activation": 0.9360920190811157,
+          "position": 12,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.9360920190811157,
+          "position": 12,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.9333191514015198,
+          "position": 5,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.9333191514015198,
+          "position": 5,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.8574860692024231,
+          "position": 20,
+          "variant": "standard"
+        }
+      ]
+    },
+    {
+      "feature_id": 61,
+      "act_rate": 0.0965299978852272,
+      "top_examples": [
+        {
+          "activation": 0.49229905009269714,
+          "position": 4,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.49229905009269714,
+          "position": 4,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.4829713702201843,
+          "position": 8,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.4829713702201843,
+          "position": 8,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.47879478335380554,
+          "position": 10,
+          "variant": "standard"
+        }
+      ]
+    },
+    {
+      "feature_id": 4034,
+      "act_rate": 0.09401000291109085,
+      "top_examples": [
+        {
+          "activation": 0.5372604131698608,
+          "position": 52,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5372604131698608,
+          "position": 52,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5347657203674316,
+          "position": 108,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5347657203674316,
+          "position": 108,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5321744680404663,
+          "position": 60,
+          "variant": "standard"
+        }
+      ]
+    },
+    {
+      "feature_id": 3787,
+      "act_rate": 0.091279998421669,
+      "top_examples": [
+        {
+          "activation": 0.5871065258979797,
+          "position": 22,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5871065258979797,
+          "position": 22,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5592474937438965,
+          "position": 55,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5592474937438965,
+          "position": 55,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5409792065620422,
+          "position": 91,
+          "variant": "standard"
+        }
+      ]
+    },
+    {
+      "feature_id": 280,
+      "act_rate": 0.09025000035762787,
+      "top_examples": [
+        {
+          "activation": 0.534440279006958,
+          "position": 7,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.534440279006958,
+          "position": 7,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5156826376914978,
+          "position": 9,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5156826376914978,
+          "position": 9,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5135127305984497,
+          "position": 9,
+          "variant": "standard"
+        }
+      ]
+    },
+    {
+      "feature_id": 2159,
+      "act_rate": 0.0877000018954277,
+      "top_examples": [
+        {
+          "activation": 0.603478729724884,
+          "position": 9,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.603478729724884,
+          "position": 9,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.603478729724884,
+          "position": 9,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5963488817214966,
+          "position": 9,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.5963488817214966,
+          "position": 9,
+          "variant": "standard"
+        }
+      ]
+    },
+    {
+      "feature_id": 3645,
+      "act_rate": 0.003819999983534217,
+      "top_examples": [
+        {
+          "activation": 0.21240393817424774,
+          "position": 202,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.21240393817424774,
+          "position": 202,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.20894747972488403,
+          "position": 217,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.20894747972488403,
+          "position": 217,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.20496299862861633,
+          "position": 40,
+          "variant": "standard"
+        }
+      ]
+    },
+    {
+      "feature_id": 2682,
+      "act_rate": 0.003809999907389283,
+      "top_examples": [
+        {
+          "activation": 0.2780674993991852,
+          "position": 3,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.2780674993991852,
+          "position": 3,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.2724977433681488,
+          "position": 3,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.2724977433681488,
+          "position": 3,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.263476699590683,
+          "position": 3,
+          "variant": "standard"
+        }
+      ]
+    },
+    {
+      "feature_id": 2786,
+      "act_rate": 0.003800000064074993,
+      "top_examples": [
+        {
+          "activation": 0.3066694736480713,
+          "position": 2,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.3066694736480713,
+          "position": 2,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.3066694438457489,
+          "position": 2,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.3066694438457489,
+          "position": 2,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.3066694438457489,
+          "position": 2,
+          "variant": "standard"
+        }
+      ]
+    },
+    {
+      "feature_id": 1681,
+      "act_rate": 0.003800000064074993,
+      "top_examples": [
+        {
+          "activation": 0.21353764832019806,
+          "position": 40,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.21353764832019806,
+          "position": 40,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.2118273675441742,
+          "position": 217,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.2118273675441742,
+          "position": 217,
+          "variant": "standard"
+        },
+        {
+          "activation": 0.21169668436050415,
+          "position": 202,
+          "variant": "standard"
+        }
+      ]
+    }
+  ],
+  "n_total_features": 4096
+}

standard_L0_hook_resid_post/config.json

@@ -0,0 +1,22 @@
+{
+  "model": "standard",
+  "layer": 0,
+  "hook": "hook_resid_post",
+  "d_sae": 4096,
+  "k": 40,
+  "n_tokens": 500000,
+  "batch_size": 4096,
+  "lr": 0.0003,
+  "final": {
+    "step": 122,
+    "loss": 3.7668752670288086,
+    "losses": {
+      "mse_loss": 3.7668752670288086,
+      "auxiliary_reconstruction_loss": 0.0
+    },
+    "l0": 40.0,
+    "mse": 0.007357178255915642,
+    "fve": 0.7880726447111882,
+    "n_dead": 0
+  }
+}

standard_L0_hook_resid_post/history.json

@@ -0,0 +1,38 @@
+[
+  {
+    "step": 50,
+    "loss": 6.100651741027832,
+    "losses": {
+      "mse_loss": 6.100651741027832,
+      "auxiliary_reconstruction_loss": 0.0
+    },
+    "l0": 40.0,
+    "mse": 0.011915335431694984,
+    "fve": 0.6626028674075588,
+    "n_dead": 0
+  },
+  {
+    "step": 100,
+    "loss": 4.176002502441406,
+    "losses": {
+      "mse_loss": 4.176002502441406,
+      "auxiliary_reconstruction_loss": 0.0
+    },
+    "l0": 40.0,
+    "mse": 0.008156255818903446,
+    "fve": 0.7637865209658066,
+    "n_dead": 0
+  },
+  {
+    "step": 122,
+    "loss": 3.7668752670288086,
+    "losses": {
+      "mse_loss": 3.7668752670288086,
+      "auxiliary_reconstruction_loss": 0.0
+    },
+    "l0": 40.0,
+    "mse": 0.007357178255915642,
+    "fve": 0.7880726447111882,
+    "n_dead": 0
+  }
+]

standard_L0_hook_resid_post/sae.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d41284ab9bceb803122433b0ecdfbdbe5027bacb38b6fa2d54727dd395c26cc1
+size 16798005

standard_L1_hook_resid_post/config.json

@@ -0,0 +1,19 @@
+{
+  "model": "standard",
+  "layer": 1,
+  "d_sae": 4096,
+  "k": 40,
+  "normalize_activations": "expected_average_only_in",
+  "n_tokens": 10000000,
+  "batch_size": 4096,
+  "lr": 0.0003,
+  "final_training": {
+    "step": 2441,
+    "loss": 4.313589572906494,
+    "l0": 40.0,
+    "fve": 0.9156635482984559,
+    "n_dead": 1,
+    "mse": 0.008392197079956532
+  },
+  "dead_pct_heldout": 1.5380859375
+}

standard_L1_hook_resid_post/history.json

@@ -0,0 +1,106 @@
+[
+  {
+    "step": 200,
+    "loss": 8.472447395324707,
+    "l0": 40.0,
+    "fve": 0.8475441618823183,
+    "n_dead": 0,
+    "mse": 0.016547750681638718
+  },
+  {
+    "step": 400,
+    "loss": 13.104448318481445,
+    "l0": 40.0,
+    "fve": 0.8741473969735887,
+    "n_dead": 602,
+    "mse": 0.013663655146956444
+  },
+  {
+    "step": 600,
+    "loss": 11.666557312011719,
+    "l0": 40.0,
+    "fve": 0.897244935256183,
+    "n_dead": 294,
+    "mse": 0.012026979587972164
+  },
+  {
+    "step": 800,
+    "loss": 8.557323455810547,
+    "l0": 40.0,
+    "fve": 0.9050136010213505,
+    "n_dead": 141,
+    "mse": 0.011152522638440132
+  },
+  {
+    "step": 1000,
+    "loss": 5.929265975952148,
+    "l0": 40.0,
+    "fve": 0.8892356006901958,
+    "n_dead": 26,
+    "mse": 0.010530667379498482
+  },
+  {
+    "step": 1200,
+    "loss": 5.310263633728027,
+    "l0": 40.0,
+    "fve": 0.9071743648944192,
+    "n_dead": 6,
+    "mse": 0.010134927928447723
+  },
+  {
+    "step": 1400,
+    "loss": 5.067918300628662,
+    "l0": 40.0,
+    "fve": 0.9030405773914236,
+    "n_dead": 2,
+    "mse": 0.00982158724218607
+  },
+  {
+    "step": 1600,
+    "loss": 4.7904052734375,
+    "l0": 40.0,
+    "fve": 0.9108113011155183,
+    "n_dead": 2,
+    "mse": 0.009283771738409996
+  },
+  {
+    "step": 1800,
+    "loss": 4.771254062652588,
+    "l0": 40.0,
+    "fve": 0.9153215139625523,
+    "n_dead": 6,
+    "mse": 0.009105676785111427
+  },
+  {
+    "step": 2000,
+    "loss": 4.466042518615723,
+    "l0": 40.0,
+    "fve": 0.9176716499875504,
+    "n_dead": 1,
+    "mse": 0.008688799105584621
+  },
+  {
+    "step": 2200,
+    "loss": 4.474887371063232,
+    "l0": 40.0,
+    "fve": 0.9006545558548696,
+    "n_dead": 2,
+    "mse": 0.00867227278649807
+  },
+  {
+    "step": 2400,
+    "loss": 4.392549991607666,
+    "l0": 40.0,
+    "fve": 0.9094376632806204,
+    "n_dead": 1,
+    "mse": 0.00854581780731678
+  },
+  {
+    "step": 2441,
+    "loss": 4.313589572906494,
+    "l0": 40.0,
+    "fve": 0.9156635482984559,
+    "n_dead": 1,
+    "mse": 0.008392197079956532
+  }
+]

standard_L1_hook_resid_post/sae.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1099ad67ddea1fb3b4ef9752f9c57b6dba97c3cbfc88d595316e5c52c54704e1
+size 16798005

standard_L2_hook_resid_post/config.json

@@ -0,0 +1,19 @@
+{
+  "model": "standard",
+  "layer": 2,
+  "d_sae": 4096,
+  "k": 40,
+  "normalize_activations": "expected_average_only_in",
+  "n_tokens": 10000000,
+  "batch_size": 4096,
+  "lr": 0.0003,
+  "final_training": {
+    "step": 2441,
+    "loss": 9.37691879272461,
+    "l0": 40.0,
+    "fve": 0.9112300092129357,
+    "n_dead": 1,
+    "mse": 0.018243035301566124
+  },
+  "dead_pct_heldout": 1.220703125
+}

standard_L2_hook_resid_post/history.json

@@ -0,0 +1,106 @@
+[
+  {
+    "step": 200,
+    "loss": 17.65301513671875,
+    "l0": 40.0,
+    "fve": 0.8277783719921248,
+    "n_dead": 0,
+    "mse": 0.03447854518890381
+  },
+  {
+    "step": 400,
+    "loss": 26.82003402709961,
+    "l0": 40.0,
+    "fve": 0.8783504287003407,
+    "n_dead": 885,
+    "mse": 0.02819967456161976
+  },
+  {
+    "step": 600,
+    "loss": 24.623018264770508,
+    "l0": 40.0,
+    "fve": 0.8645155231541486,
+    "n_dead": 466,
+    "mse": 0.025606488808989525
+  },
+  {
+    "step": 800,
+    "loss": 20.887020111083984,
+    "l0": 40.0,
+    "fve": 0.8902317708880264,
+    "n_dead": 211,
+    "mse": 0.02367589809000492
+  },
+  {
+    "step": 1000,
+    "loss": 13.424010276794434,
+    "l0": 40.0,
+    "fve": 0.8956536633832358,
+    "n_dead": 45,
+    "mse": 0.02241015061736107
+  },
+  {
+    "step": 1200,
+    "loss": 11.20499324798584,
+    "l0": 40.0,
+    "fve": 0.9072051860620202,
+    "n_dead": 9,
+    "mse": 0.021145537495613098
+  },
+  {
+    "step": 1400,
+    "loss": 10.573310852050781,
+    "l0": 40.0,
+    "fve": 0.9040789272639144,
+    "n_dead": 3,
+    "mse": 0.02041182667016983
+  },
+  {
+    "step": 1600,
+    "loss": 10.361445426940918,
+    "l0": 40.0,
+    "fve": 0.9104271378826995,
+    "n_dead": 1,
+    "mse": 0.020158452913165092
+  },
+  {
+    "step": 1800,
+    "loss": 9.941110610961914,
+    "l0": 40.0,
+    "fve": 0.9068027304791877,
+    "n_dead": 1,
+    "mse": 0.01934068836271763
+  },
+  {
+    "step": 2000,
+    "loss": 9.595905303955078,
+    "l0": 40.0,
+    "fve": 0.9189567435127948,
+    "n_dead": 1,
+    "mse": 0.01866907998919487
+  },
+  {
+    "step": 2200,
+    "loss": 9.470767974853516,
+    "l0": 40.0,
+    "fve": 0.9136617571485228,
+    "n_dead": 1,
+    "mse": 0.01842562109231949
+  },
+  {
+    "step": 2400,
+    "loss": 9.251761436462402,
+    "l0": 40.0,
+    "fve": 0.9031395392007785,
+    "n_dead": 1,
+    "mse": 0.01799953728914261
+  },
+  {
+    "step": 2441,
+    "loss": 9.37691879272461,
+    "l0": 40.0,
+    "fve": 0.9112300092129357,
+    "n_dead": 1,
+    "mse": 0.018243035301566124
+  }
+]

standard_L2_hook_resid_post/sae.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c36a71a6c67b951c929ca6a65422b6a1490dd39bcb78f79480fd3576b95e341
+size 16798005

standard_L3_hook_resid_post/config.json

@@ -0,0 +1,19 @@
+{
+  "model": "standard",
+  "layer": 3,
+  "d_sae": 4096,
+  "k": 40,
+  "normalize_activations": "expected_average_only_in",
+  "n_tokens": 10000000,
+  "batch_size": 4096,
+  "lr": 0.0003,
+  "final_training": {
+    "step": 2441,
+    "loss": 19.480852127075195,
+    "l0": 40.0,
+    "fve": 0.8956661837197387,
+    "n_dead": 1,
+    "mse": 0.03790052607655525
+  },
+  "dead_pct_heldout": 1.025390625
+}

standard_L3_hook_resid_post/history.json

@@ -0,0 +1,106 @@
+[
+  {
+    "step": 200,
+    "loss": 34.482032775878906,
+    "l0": 40.0,
+    "fve": 0.8283586159312837,
+    "n_dead": 0,
+    "mse": 0.06734772026538849
+  },
+  {
+    "step": 400,
+    "loss": 53.01472473144531,
+    "l0": 40.0,
+    "fve": 0.8312126634199706,
+    "n_dead": 1054,
+    "mse": 0.05626294016838074
+  },
+  {
+    "step": 600,
+    "loss": 48.47029495239258,
+    "l0": 40.0,
+    "fve": 0.884058701037366,
+    "n_dead": 593,
+    "mse": 0.05088154226541519
+  },
+  {
+    "step": 800,
+    "loss": 42.332054138183594,
+    "l0": 40.0,
+    "fve": 0.8804909955406929,
+    "n_dead": 205,
+    "mse": 0.04835565760731697
+  },
+  {
+    "step": 1000,
+    "loss": 28.461124420166016,
+    "l0": 40.0,
+    "fve": 0.888121025084119,
+    "n_dead": 58,
+    "mse": 0.04564930498600006
+  },
+  {
+    "step": 1200,
+    "loss": 24.47071647644043,
+    "l0": 40.0,
+    "fve": 0.8850747737111373,
+    "n_dead": 21,
+    "mse": 0.044211357831954956
+  },
+  {
+    "step": 1400,
+    "loss": 23.19893455505371,
+    "l0": 40.0,
+    "fve": 0.8942765037624966,
+    "n_dead": 15,
+    "mse": 0.04282252490520477
+  },
+  {
+    "step": 1600,
+    "loss": 21.993581771850586,
+    "l0": 40.0,
+    "fve": 0.8978253015716876,
+    "n_dead": 10,
+    "mse": 0.04135049134492874
+  },
+  {
+    "step": 1800,
+    "loss": 21.293956756591797,
+    "l0": 40.0,
+    "fve": 0.8922899854658137,
+    "n_dead": 8,
+    "mse": 0.040335513651371
+  },
+  {
+    "step": 2000,
+    "loss": 20.39199447631836,
+    "l0": 40.0,
+    "fve": 0.9074694820636785,
+    "n_dead": 4,
+    "mse": 0.03921680152416229
+  },
+  {
+    "step": 2200,
+    "loss": 20.141448974609375,
+    "l0": 40.0,
+    "fve": 0.8943716286803387,
+    "n_dead": 3,
+    "mse": 0.038883697241544724
+  },
+  {
+    "step": 2400,
+    "loss": 19.7071475982666,
+    "l0": 40.0,
+    "fve": 0.8941642474910445,
+    "n_dead": 1,
+    "mse": 0.03834077715873718
+  },
+  {
+    "step": 2441,
+    "loss": 19.480852127075195,
+    "l0": 40.0,
+    "fve": 0.8956661837197387,
+    "n_dead": 1,
+    "mse": 0.03790052607655525
+  }
+]

standard_L3_hook_resid_post/sae.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b9dc2de61616a842a8ac0c32f8d1da82606b2fa17f57a91f7252d3d9dcaf215f
+size 16798005

three_probes.json

@@ -0,0 +1,62 @@
+{
+  "probe_1_variant_split": {
+    "onsets_all": 12000,
+    "onsets_adhd": 8770,
+    "onsets_std": 3230,
+    "primary_mixed": 312,
+    "symmetry_mixed": 139,
+    "primary_adhd_only": 312,
+    "symmetry_std_only": 216,
+    "feat2504_rate_adhd_only_in_adhd_L2": 0.5907639861106873,
+    "feat2504_rate_adhd_only_in_std_L2": 0.00034207524731755257
+  },
+  "probe_2_zero_ablation": {
+    "adhd_baseline_reg_rate": 0.41327800829875516,
+    "adhd_zero_ablate_L2_step_onset_reg_rate": 0.4056761268781302,
+    "adhd_zero_ablate_L2_all_positions_reg_rate": 0.404344193817878,
+    "delta_step_onset": -0.007601881420624956,
+    "delta_all_positions": -0.008933814480877156
+  },
+  "probe_3_L1_steering": {
+    "baseline": {
+      "sep_rate": 0.08425135764158262,
+      "mean_step_count": 6.8125,
+      "spearman_rho": 0.5305203306664215,
+      "spearman_pval": 4.155845216990538e-07,
+      "regulation_rate": 0.09357798165137615,
+      "n_tokens_total": 6445,
+      "n_steps_total": 545,
+      "reached_end_rate": 0.975
+    },
+    "coef_+2.0": {
+      "sep_rate": 0.08416458852867831,
+      "mean_step_count": 6.775,
+      "spearman_rho": 0.5369945327780861,
+      "spearman_pval": 2.8189998198585957e-07,
+      "regulation_rate": 0.0940959409594096,
+      "n_tokens_total": 6416,
+      "n_steps_total": 542,
+      "reached_end_rate": 0.975
+    },
+    "coef_+5.0": {
+      "sep_rate": 0.08443188951663529,
+      "mean_step_count": 6.75,
+      "spearman_rho": 0.5421974917403188,
+      "spearman_pval": 2.051407247246999e-07,
+      "regulation_rate": 0.08703703703703704,
+      "n_tokens_total": 6372,
+      "n_steps_total": 540,
+      "reached_end_rate": 0.975
+    },
+    "coef_+10.0": {
+      "sep_rate": 0.08677685950413223,
+      "mean_step_count": 6.8125,
+      "spearman_rho": 0.33306527760157806,
+      "spearman_pval": 0.002537468671163372,
+      "regulation_rate": 0.08440366972477065,
+      "n_tokens_total": 6292,
+      "n_steps_total": 545,
+      "reached_end_rate": 0.975
+    }
+  }
+}