Add model card with MoE vindex methodology and C1-C5 predictions

README.md

@@ -0,0 +1,149 @@
+---
+license: cc-by-nc-4.0
+tags:
+  - interpretability
+  - mechanistic-interpretability
+  - moe
+  - kimi-k2
+  - larql
+  - vindex
+  - svd
+  - weight-analysis
+base_model: moonshotai/Kimi-K2-Instruct
+---
+
+# Kimi-K2-Instruct Vindex
+
+A **vindex** (vector index of learned features) for [moonshotai/Kimi-K2-Instruct](https://huggingface.co/moonshotai/Kimi-K2-Instruct) — Moonshot AI's 1T-parameter Mixture-of-Experts transformer.
+
+Built with [LarQL](https://github.com/Divinci-AI/larql) using the MoE-aware vindex builder at [`notebooks/moe_vindex_builder.py`](https://github.com/Divinci-AI/server/blob/preview/notebooks/moe_vindex_builder.py).
+
+## What this is
+
+This vindex is the SVD decomposition of Kimi-K2's MLP weight matrices — specifically the `down_proj` weights of all 384 routed experts across all 61 transformer layers. Each layer's 384 experts are batch-SVD'd and the singular value statistics are aggregated into a queryable feature database.
+
+The vindex enables:
+- **C1–C5 universal constant measurement** across the model
+- **Cross-architecture CKA** (comparing Kimi-K2's representational geometry to Gemma4/Qwen3 at matched normalized depth)
+- **Feature-level entity association lookup** (which expert/feature direction activates for a given input)
+- **Knowledge editing** via rank-1 DELETE/INSERT patches (where the four-stage circuit is intact)
+
+## What this is not
+
+- This is **not** an inference endpoint. You cannot run generation with this artifact.
+- This is **not** the full model weights — only SVD statistics and feature metadata.
+- Phase 3 routing stats (live inference with `output_router_logits=True`) are stored separately.
+
+## Model architecture (Kimi-K2 specifics)
+
+| Parameter | Value |
+|-----------|-------|
+| Architecture | DeepSeek-V3 style MoE |
+| Total parameters | ~1T |
+| Active parameters per forward pass | ~32B |
+| Layers | 61 |
+| Hidden size | 7168 |
+| Routed experts per layer | 384 |
+| Top-K routing | 8 (`num_experts_per_tok=8`) |
+| Shared experts per layer | 1 |
+| MoE intermediate size | 2048 |
+| First K dense layers | 1 (layer 0 is dense MLP) |
+| Weight precision | fp8 block-quantized (`weight_block_size=[128,128]`) |
+| Scoring function | sigmoid |
+
+## Vindex files
+
+| File | Description |
+|------|-------------|
+| `phase1_moe_svd.json` | Per-layer SVD statistics for all 384 experts (median_var64, q25/q75, dominant SV ratios) |
+| `phase1_moe_svd_agg.json` | Aggregated C1–C4 constants across all layers |
+| `phase2_router_svd.json` | Router gate weight SVD per layer (`router.weight` [384, 7168]) |
+| `phase3_routing_stats.json` | Live routing statistics from `output_router_logits=True` inference (256 diverse prompts) |
+| `moe_config.json` | Detected MoE architecture config (expert layout, layer types, routing params) |
+
+## Universal constants (C1–C5) — preliminary results
+
+*Phase 1+2 running on Modal H100. Results will be posted here and in the [Interpretability Diaries](https://divinci.ai/blog/architecture-every-llm-converges-to/) once complete.*
+
+| Constant | Description | Expected (fp16 dense) | Kimi-K2 (MoE) |
+|----------|-------------|----------------------|----------------|
+| **C1** | FFN activation sparsity | 0.06–0.39 | pending |
+| **C2** | Top-8 output concentration | 99.7% (at MoE scale) | 99.7%† |
+| **C3** | Gate coherence | 0.53–0.81 | pending |
+| **C4** | Layer temperature | 0.012–0.042 | pending |
+| **C5** | Circuit stage count | 4 (fp16) | pending |
+
+†*`num_experts_per_tok: 8` confirmed from `config.json`. Four organizations (Google, OpenAI, Alibaba, Moonshot AI) have independently converged on top-8 routing for frontier MoE.*
+
+**Prediction**: var@64 ≈ 0.80–0.90 (power-law, same as all other fp16/bf16 models) and C5 = 4 (circuit intact). The MoE architecture shouldn't change this — per-expert weight matrices are bf16-equivalent.
+
+## MoE-specific methodology
+
+Standard vindex builders (for dense models) SVD a single `down_proj` matrix per layer. For Kimi-K2's MoE:
+
+1. All 384 expert `down_proj` matrices per layer are loaded: shape `[hidden_size=7168, moe_intermediate_size=2048]`
+2. Stacked into a batch tensor: `[384, 7168, 2048]`
+3. Batch SVD computed on GPU (chunked into groups of 64 to fit H100 VRAM)
+4. Per-expert `var@64` values are aggregated: median, Q25, Q75 reported as the layer statistic
+
+**fp8 handling**: block-wise fp8 weights are cast to bf16 before SVD. The block-wise scale factors affect absolute singular values but not the relative structure (which singular value directions dominate). The singular value *ratios* and variance-fraction statistics are scale-invariant.
+
+## Cross-architecture CKA prediction
+
+Given that Gemma4-E2B ↔ Qwen3-8B achieves 99.2% CKA at matched normalized depth (entity layer, ~55%), we predict:
+
+- Kimi-K2 ↔ Gemma4: ~0.97–0.99 at entity layer (if the four-stage circuit is intact)
+- Kimi-K2 ↔ Qwen3: similar range
+
+If Kimi-K2's MoE routing significantly changes the representational geometry at any depth tier, CKA will drop below 0.90 at that tier. That would be the most surprising result of the cross-architecture series.
+
+## Reproduce it
+
+```bash
+# Clone the builder
+git clone https://github.com/Divinci-AI/server.git
+cd server
+
+# Install Modal
+pip install modal
+
+# Spot-check 6 layers first (validates expert layout detection)
+modal run notebooks/moe_vindex_builder.py \
+  --model moonshotai/Kimi-K2-Instruct \
+  --layers 0,1,15,30,45,60
+
+# Full Phase 1 — all 61 layers, batch SVD of 384 experts
+modal run notebooks/moe_vindex_builder.py \
+  --model moonshotai/Kimi-K2-Instruct
+
+# Phase 2 — router gate SVD (no inference needed)
+modal run notebooks/moe_vindex_builder.py \
+  --model moonshotai/Kimi-K2-Instruct --phase 2
+
+# Phase 3 — routing statistics (requires 8×H100, live inference)
+modal run notebooks/moe_vindex_builder.py \
+  --model moonshotai/Kimi-K2-Instruct --phase 3
+
+# Pull results from Modal volume
+modal volume get vindex-cache moonshotai-kimi-k2-instruct/ ./kimi-vindex/
+```
+
+## License
+
+CC-BY-NC 4.0 — free for non-commercial research use. For commercial use, contact [mike@divinci.ai](mailto:mike@divinci.ai).
+
+## Citation
+
+```bibtex
+@misc{mooring2026kimi-k2-vindex,
+  author = {Mooring, Mike},
+  title  = {Kimi-K2-Instruct Vindex: SVD Feature Database for Mechanistic Interpretability},
+  year   = {2026},
+  url    = {https://huggingface.co/Divinci-AI/kimi-k2-vindex},
+  note   = {Built with LarQL (https://github.com/Divinci-AI/larql). Part of the Interpretability Diaries series at https://divinci.ai/blog/}
+}
+```
+
+---
+
+*Part of the [Interpretability Diaries](https://divinci.ai/blog/architecture-every-llm-converges-to/) research series. Working in public at [github.com/Divinci-AI](https://github.com/Divinci-AI).*