---
license: mit
library_name: pytorch
tags:
  - test-time-training
  - conformal-prediction
  - reasoning
  - early-stopping
  - llm
datasets:
  - wzekai99/ORCA
---

# ORCA TTT-Probes

Trained Test-Time Training probes for *Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning* ([arXiv:2604.01170](https://arxiv.org/abs/2604.01170)).

## Layout (17 probes)

```
qwen2.5-32b/supervised/{no_kq, qk_dh128,
                        qk_dh32, qk_dh64, qk_dh256, qk_dh512,
                        qk_dh128_ln, qk_dh128_ln_res, qk_dh128_share_kq,
                        qk_dh128_eta_learn, qk_dh128_mlp}/
qwen2.5-32b/consistent/{no_kq, qk_dh128}/
qwq-32b/supervised/{no_kq, qk_dh128}/
llama-3.3-70b/supervised/{no_kq, qk_dh128}/
```

Per probe directory:

| File              | Contents                                                       |
|-------------------|----------------------------------------------------------------|
| `probe.pt`        | State dict: W0, b0, log_eta; QK variants also include theta_K, theta_Q |
| `config.json`     | Training hyperparameters (d_hidden, base_lr, epochs, ...)      |
| `lambdas.json`    | LTT thresholds, keyed by delta                                 |
| `metrics.json`    | Step-level savings and error rate per delta                    |
| `ood_*.json`      | Per-OOD-benchmark metrics (Qwen2.5-32B probes only)            |

## Use

Probes are loaded by the `TTTProbe` class in https://github.com/wzekai99/ORCA. Quick example:

```bash
hf download wzekai99/ORCA --local-dir probes
hf download wzekai99/ORCA --repo-type dataset --local-dir data
python code/test.py \
    --method ttt --no_kq \
    --dataset_path data/qwen2.5-32b/s1k.pkl \
                   data/qwen2.5-32b/openr1_2k.pkl \
                   data/qwen2.5-32b/deepmath_2k.pkl \
    --probe_path probes/qwen2.5-32b/supervised/no_kq/probe.pt \
    --label_mode supervised --delta 0.1 --epsilon 0.05
```

## License

MIT.

## Citation

```bibtex
@article{zhou2026online,
  title={Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning},
  author={Zhou, Cai and Wang, Zekai and Wu, Menghua and Zhu, Qianyu Julie and Shi, Flora C and Wang, Chenyu and Wilson, Ashia and Jaakkola, Tommi and Bates, Stephen},
  journal={arXiv preprint arXiv:2604.01170},
  year={2026}
}
```