bsanch52/qwen3-adapter-correction

Qwen3 Adapter Correction

Post-transformer adapters that correct suppressed log-probabilities on ideology-sensitive topics in the Qwen3 model family.

Paper: Correcting Suppressed Log-Probabilities in Language Models with Post-Transformer Adapters

Usage

import mlx.core as mx
import mlx.nn as nn
import numpy as np
from mlx_lm import load as mlx_load

# Load model
model, tokenizer = mlx_load("Qwen/Qwen3-8B-Base")

# Load adapter
class SwiGLUAdapter(nn.Module):
    def __init__(self, d_model, d_inner):
        super().__init__()
        self.gate_proj = nn.Linear(d_model, d_inner, bias=False)
        self.up_proj = nn.Linear(d_model, d_inner, bias=False)
        self.down_proj = nn.Linear(d_inner, d_model, bias=False)
    def __call__(self, h):
        return self.down_proj(nn.sigmoid(self.gate_proj(h)) * self.up_proj(h))

adapter = SwiGLUAdapter(4096, 64)  # d_model=4096 for 8B
weights = dict(np.load("ideology_8b_swiglu.npz"))
adapter.load_weights([(k, mx.array(v)) for k, v in weights.items()])

# Apply: hidden state -> adapter correction -> logits
h = model.model(tokens)
h = model.model.norm(h)
h = h + adapter(mx.stop_gradient(h))
logits = h @ model.model.embed_tokens.weight.T

Files

• ideology_8b_swiglu.npz - SwiGLU adapter for Qwen3-8B-Base (786K params)
• ideology_4b_swiglu.npz - SwiGLU adapter for Qwen3-4B-Base (491K params)
• ideology_14b_swiglu.npz - SwiGLU adapter for Qwen3-14B-Base (983K params)

License

MIT