Add model card describing Phase 5 + 5.7 ckpts

README.md

@@ -0,0 +1,143 @@
+---
+language: en
+license: mit
+library_name: pytorch
+tags:
+  - sparse-autoencoder
+  - temporal-sae
+  - crosscoder
+  - interpretability
+  - temporal-crosscoder
+pipeline_tag: feature-extraction
+---
+
+# Temporal Crosscoder checkpoints (Phase 5 + Phase 5.7)
+
+Trained-checkpoint artifact for the [temp_xc](https://github.com/chainik1125/temp_xc)
+research repository. These are **Sparse Autoencoder / Crosscoder**
+checkpoints trained on `google/gemma-2-2b-it` layer 13 residual stream
+activations over 24 000 FineWeb sequences.
+
+See `summary.md` and the autoresearch plan in the code repo for the
+full story. This HF repo only hosts the binary `.pt` checkpoints;
+all training / probing / plotting code lives in the github repo.
+
+## Quick start
+
+Download all ckpts:
+
+```bash
+pip install huggingface_hub
+huggingface-cli download han1823123123/txcdr --local-dir ./ckpts
+```
+
+Or a single file:
+
+```bash
+huggingface-cli download han1823123123/txcdr \
+  ckpts/txcdr_contrastive_t5__seed42.pt \
+  --local-dir ./ckpts
+```
+
+## Loading a checkpoint
+
+Each `.pt` is saved as a dict with keys
+`{"state_dict", "arch", "meta", "state_dict_dtype"}`. The
+`state_dict` is in **fp16** to halve disk cost; the loader in
+`run_probing.py` casts it back to fp32 when building the model. Example
+(from `experiments/phase5_downstream_utility/probing/run_probing.py`):
+
+```python
+import torch
+from src.architectures.crosscoder import TemporalCrosscoder
+
+state = torch.load("ckpts/txcdr_t5__seed42.pt",
+                   map_location="cuda", weights_only=False)
+meta = state["meta"]
+T = meta["T"]                # e.g. 5
+k_eff = meta["k_win"] or (meta["k_pos"] * T)
+model = TemporalCrosscoder(d_in=2304, d_sae=18432, T=T, k=k_eff).cuda()
+cast = {k: v.float() if v.dtype == torch.float16 else v
+        for k, v in state["state_dict"].items()}
+model.load_state_dict(cast)
+model.eval()
+```
+
+For other architectures, the corresponding class lives under
+`src/architectures/` in the github repo. The `arch` field in each
+ckpt's state tells you which class to instantiate; the `meta` dict
+carries the constructor arguments.
+
+## Contents
+
+### Phase 5 canonical benchmark (seed 42, 25 archs)
+
+Trained on the full 6 000-sequence GPU-preloaded subset, plateau-stop
+at <2 %/1k-step loss drop, Adam lr=3e-4, batch=1024, max 25 000 steps.
+Probed on 36 binary tasks at `last_position` and `mean_pool`
+aggregations — see `docs/han/research_logs/phase5_downstream_utility/summary.md`
+in the github repo for the leaderboard.
+
+| family | arch(s) |
+|---|---|
+| Token SAE | `topk_sae` |
+| Layer crosscoder | `mlc`, `mlc_contrastive` |
+| Temporal crosscoder (T-sweep) | `txcdr_t{2,3,5,8,10,15,20}` |
+| Stacked per-position | `stacked_t{5,20}` |
+| Matryoshka (position-nested) | `matryoshka_t5` |
+| Weight-sharing ablation | `txcdr_shared_dec_t5`, `txcdr_shared_enc_t5`, `txcdr_tied_t5`, `txcdr_pos_t5`, `txcdr_causal_t5` |
+| Sparse-structure variant | `txcdr_block_sparse_t5` |
+| Decoder rank variant | `txcdr_lowrank_dec_t5`, `txcdr_rank_k_dec_t5` |
+| Time-contrastive (Ye et al. 2025) | `temporal_contrastive` (single-token) |
+| Time × Layer (novel) | `time_layer_crosscoder_t5` (d_sae=8192) |
+| TFA | `tfa_small`, `tfa_pos_small` (d_sae=4096, seq_len=32) |
+
+### Phase 5.7 autoresearch archs (seed 42)
+
+Novel architectures explored after the 25-arch benchmark. See
+`docs/han/research_logs/phase5_downstream_utility/2026-04-21-phase5_7-architectures.md`
+for design details.
+
+**Tier-1 / Tier-2 candidates:**
+
+| arch | role | status |
+|---|---|---|
+| `txcdr_contrastive_t5` | A2: TXCDR + Matryoshka H/L + InfoNCE on adjacent windows | FINALIST |
+| `matryoshka_txcdr_contrastive_t5` | A3: position-nested Matryoshka + InfoNCE on scale-1 prefix | FINALIST |
+| `txcdr_rotational_t5` | A1: rank-K Lie-group decoder | DISCARD |
+| `txcdr_basis_expansion_t5` | A5: decoder as K=3 basis combination | DISCARD |
+| `mlc_temporal_t3` | A4: MLC with shared-across-time encoder | DISCARD |
+| `time_layer_contrastive_t5` | A10: time_layer_crosscoder + InfoNCE on (T,L)-mean prefix | AMBIGUOUS |
+| `txcdr_dynamics_t5` | A8: recurrent sparse latent with per-feature gate | DISCARD |
+
+**Part B α-sweep variants (seed 42):**
+
+| arch | α | role |
+|---|---|---|
+| `txcdr_contrastive_t5_alpha003` | 0.03 | A2 under-weight |
+| `txcdr_contrastive_t5_alpha100` | 1.00 | A2 paper-default |
+| `txcdr_contrastive_t5_k2x` | 0.10 | A2 at k_win=1000 |
+| `matryoshka_txcdr_contrastive_t5_alpha003` | 0.03 | A3 under-weight |
+| `matryoshka_txcdr_contrastive_t5_alpha100` | 1.00 | A3 paper-default |
+| `matryoshka_txcdr_contrastive_t5_k2x` | 0.10 | A3 at k_win=1000 |
+
+**T17 seed-variance checkpoints (in-progress fragment):**
+a few additional `__seed{1,2,3}` variants exist on some archs from an
+early seed-variance experiment that was scrapped mid-flight in favour
+of Phase 5.7 autoresearch. See the autoresearch plan for context.
+
+## Reproduction
+
+For a full from-scratch reproduction (tokenise FineWeb → build 5-layer
+activation cache → build probe cache → train 25 archs → probe →
+headline plots → T-sweep plot), see
+`docs/han/research_logs/phase5_downstream_utility/2026-04-21-reproduction-brief.md`
+in the github repo. ~120 GB disk + ~12-15 h compute on an A40-class GPU.
+
+## Citation
+
+No paper yet; work-in-progress for NeurIPS submission.
+
+## License
+
+MIT. See LICENSE in the github repo.