Model loading
Hi
Thanks for training it and sharing!
Have an issue loading it, wanted to make sure I don't miss anything:
I load with
from circuit_tracer import ReplacementModel
model = ReplacementModel.from_pretrained(
"Qwen/Qwen2.5-7B-Instruct",
"KokosDev/qwen2p5vl-7b-plt",
dtype=torch.bfloat16,
trust_remote_code=True,
device=torch.device("cuda"),
)
First question - is it correct to load the text only 7B-Instruct model? According to your description the transcoders are for the VLM, but I assume it can work on the text variant the VLM was created from.
2nd question is the layers for which transcoders exist: Qwen/Qwen2.5-7B-Instruct has 28 layers, but the transcoders set is a ModuleList with 27 layers.
Alternatively if the right way is to load the VL model to ReplacementModel, I'm not sure how this can work, as TransformerLens don't seem to support it.
Will appreciate your input here!
I trained these transcoders on the VL (image + text) model, so please load the VL base:
from circuit_tracer import ReplacementModel
model = ReplacementModel.from_pretrained(
"Qwen/Qwen2.5-VL-7B-Instruct", # VL base (matches training)
"KokosDev/qwen2p5vl-7b-plt",
dtype=torch.bfloat16,
trust_remote_code=True,
device="cuda",
)
Q1: Using the text‑only base (
Qwen/Qwen2.5-7B-Instruct) isn’t recommended. The transcoders were trained on the VL architecture’s language backbone, and the hook/module names differ between VL and text‑only. Loading on the text‑only variant can lead to partial or mismatched attachment.Q2: The 27 vs 28 discrepancy is expected—the last layer was left out on purpose (PLT). Reasons:
- The final block feeds directly into final norm +
lm_head; even tiny reconstruction errors at that point tend to amplify into logit drift and worse perplexity. - Many replacement/eval pipelines implicitly assume a “next” block after the replaced one; the last block has no downstream block, which makes replacement logic and calibration brittle.
- Limited upside vs higher risk; earlier blocks provide most of the utility while keeping outputs more stable.
- The final block feeds directly into final norm +
If your workflow requires TransformerLens: TL doesn’t support VL models. In that case, please use the VL base directly via Hugging Face (as above). If you absolutely need to run on the text‑only base, you’d need a careful remap of hook names and accept that the checkpoint wasn’t trained on that architecture, so behavior may diverge.
Happy to help further. if you hit an error, sharing the exact stack trace will let me diagnose quickly.
Got it, thanks for the help!