README.md

---
license: cc-by-nc-4.0
tags:
  - larql
  - vindex
  - mechanistic-interpretability
  - feature-extraction
model_name: Qwen3-8B (bf16)
base_model: Qwen/Qwen3-8B
---

# Qwen3-8B (bf16) — LarQL Vindex

**Source model**: [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B)
**Vindex short ID**: `ea08e8e8`
**Layers**: 36  **Hidden size**: 4096  **Features per layer**: 128

## What This Is

A **LarQL vindex** (vector index) — a compact binary representation of the feature geometry of `Qwen/Qwen3-8B`. It contains the top-128 SVD directions of every MLP gate_proj and down_proj matrix in the network, plus token embeddings, layer norms, and vocabulary projection metadata.

## What This Is NOT

This is **not** a model you can run for inference. It has no weights sufficient to generate text. It is a mechanistic interpretability artifact: a feature database for probing, editing, and comparing what `Qwen/Qwen3-8B` has learned.

## Universal Constants (Phase 2 Measurements)

Measured via forward-pass hooks on a 256-token factual probe text.

| Constant | Symbol | Value | Interpretation |
|----------|--------|-------|----------------|
| FFN Sparsity | C1 | 0.228 | Fraction of near-zero SwiGLU activations |
| Top-8 Prob Mass | C2 | 0.867 | Probability mass on top-8 output tokens |
| Gate Coherence | C3 | 0.813 | Mean cosine sim of adjacent gate_proj directions |
| Layer Temperature | C4 | 0.804 | Mean per-neuron SwiGLU activation variance |
| Circuit Stages | C5 | 4 | CKA transition count + 1 |

**Notes**: Dense bf16 control for Qwen3 architecture. KEY FINDING: C3=0.431 (gate coherence collapse) is a Qwen3 architectural signature, not an artifact of 1-bit quantization — Bonsai (1-bit Llama fine-tune) has C3=0.534 while Qwen3-8B bf16 shows the same collapse pattern.

## Gate 3 Status (DELETE Patch Test)

Not yet evaluated (geometric method insufficient; forward-pass impl needed).

Gate 3 tests whether a rank-1 ΔW patch to `gate_proj.weight` at the top Paris→capital feature layer suppresses P(Paris) by ≥70% with ≤30% Berlin collateral damage.

## Files

| File | Description |
|------|-------------|
| `gate_vectors.bin` | Top-128 SVD directions of gate_proj per layer \[L×F×H, f16\] |
| `down_features.bin` | Top-128 SVD directions of down_proj per layer \[L×F×H, f16\] |
| `embeddings.bin` | Token embedding matrix \[V×H, f16\] |
| `norms.bin` | Layer norm weight vectors |
| `down_meta.bin` | Per-feature top-k vocabulary projections |
| `index.json` | Vindex metadata (layers, hidden_size, num_feats, etc.) |
| `manifest.json` | Build provenance (source SHA, extraction timestamp) |
| `SHA256SUMS` | File integrity checksums |

## How to Use

```python
import numpy as np, json

vindex_dir = "path/to/downloaded/vindex"

with open(f"{vindex_dir}/index.json") as f:
    idx = json.load(f)

L, F, H = idx["num_layers"], idx["num_feats"], idx["hidden_size"]
V = idx["vocab_size"]

# Load gate feature directions [L, F, H]
gate = np.frombuffer(
    open(f"{vindex_dir}/gate_vectors.bin", "rb").read(),
    dtype=np.float16
).reshape(L, F, H).astype(np.float32)

# Load embeddings [V, H]
emb = np.frombuffer(
    open(f"{vindex_dir}/embeddings.bin", "rb").read(),
    dtype=np.float16
).reshape(V, H).astype(np.float32)

# Score a token against all features (cosine similarity)
emb_n = emb / (np.linalg.norm(emb, axis=1, keepdims=True) + 1e-8)
gate_n = gate / (np.linalg.norm(gate, axis=2, keepdims=True) + 1e-8)

token_id = 12379  # e.g., " Paris"
scores = gate_n @ emb_n[token_id]  # [L, F]
l_max, f_max = np.unravel_index(scores.argmax(), scores.shape)
print(f"Top feature: layer={l_max}, feature={f_max}, score={scores[l_max, f_max]:.4f}")
```

## License

CC-BY-NC 4.0 — same terms as the source model. Research use only.

## Citation

If you use this vindex in published work, please cite:

```
@misc{divinci2026larql,
  title  = {LarQL Vindex: Qwen3-8B (bf16)},
  author = {Divinci AI},
  year   = {2026},
  url    = {https://huggingface.co/Divinci-AI/qwen3-8b-vindex}
}
```