Upload README.md with huggingface_hub

README.md

@@ -18,6 +18,10 @@ A **vindex** (vector index of learned features) for [moonshotai/Kimi-K2-Instruct
 
 Built with [LarQL](https://github.com/Divinci-AI/larql) using the MoE-aware vindex builder at [`notebooks/moe_vindex_builder.py`](https://github.com/Divinci-AI/server/blob/preview/notebooks/moe_vindex_builder.py).
 
+> **Status (2026-04-23):** Phase 1 spot-check complete (6/61 layers). Full 61-layer Phase 1 running, ETA ~4 hr from publish time. Phase 2 (router SVD) and Phase 3 (routing stats, 8×H100) pending. The core finding — flat SVD spectrum consistent with 1-bit models — is established by the spot-check; subsequent phases refine the full-layer distribution and confirm C1/C3/C5.
+>
+> *This card updates in-place as phases land. See [changelog](#changelog) at the bottom.*
+
 ## What this is
 
 This vindex is the SVD decomposition of Kimi-K2's MLP weight matrices — specifically the `down_proj` weights of all 384 routed experts across all 61 transformer layers. Each layer's 384 experts are batch-SVD'd and the singular value statistics are aggregated into a queryable feature database.
@@ -28,6 +32,32 @@ The vindex enables:
 - **Feature-level entity association lookup** (which expert/feature direction activates for a given input)
 - **Knowledge editing** via rank-1 DELETE/INSERT patches (where the four-stage circuit is intact)
 
+## Key finding: fp8-native training causes spectral dissolution
+
+**Training precision, not storage precision, determines spectral structure.**
+
+| Model | Training precision | var@64 (median) | Spectral class |
+|-------|-------------------|-----------------|----------------|
+| Gemma 4 E2B-it | fp16/bf16 | 0.041 | non-dissolved |
+| Ministral-3B | fp16 → *post-quant* fp8 | ~0.85 | non-dissolved |
+| **Kimi-K2-Instruct** | **fp8 native** | **0.088** (MoE, 6/61 layers) | **dissolved** |
+| Bonsai 8B | fp16 → *post-quant* 1-bit | 0.093 | dissolved |
+| BitNet b1.58-2B-4T | **1-bit native** | 0.111 | dissolved |
+
+*Dissolved: var@64 < 0.15 (bimodal gap — no model yet observed between 0.15 and 0.50).*
+
+The finding is surprising because Ministral-3B is stored in fp8 and shows a non-dissolved spectrum, while Kimi-K2 is *also* stored in fp8 but shows a dissolved spectrum. Same storage precision; opposite internal geometry. The variable that distinguishes them is whether fp8 was used *at training time* (gradient noise and limited numerical range) or applied *after* fp16 training (post-quantization, which preserves the fp16-shaped spectrum).
+
+**Hypothesis:** Low-precision training (fp8 native, 1-bit native) prevents the accumulation of high-rank weight refinements that produce sharp singular-value decay in fp16 models. Gradient updates under limited numerical range can only express directions that are already well-represented — so the weights converge to a more uniform spectral distribution. This is consistent with all three dissolution datapoints (n=3); mechanism is untested.
+
+**What this does not claim:**
+- fp8-native training is not *equivalent* to 1-bit in capability or behavior
+- Post-quantized fp8 is not "worse" than native fp8 — the spectra differ, behavioral quality is a separate question
+- We have no data on fp4, bf16, or other low-precision training regimes
+- n=3 across heterogeneous models; this is a hypothesis-generating result, not a causal proof
+
+**Testable prediction:** Any future model trained natively in fp8 or lower precision should show var@64 < 0.15. Models to watch: any native-fp8 release from Moonshot AI, DeepSeek, or others using low-precision training from scratch.
+
 ## What this is not
 
 - This is **not** an inference endpoint. You cannot run generation with this artifact.
@@ -61,21 +91,36 @@ The vindex enables:
 | `phase3_routing_stats.json` | Live routing statistics from `output_router_logits=True` inference (256 diverse prompts) |
 | `moe_config.json` | Detected MoE architecture config (expert layout, layer types, routing params) |
 
-## Universal constants (C1–C5) — preliminary results
+## Universal constants (C1–C5) — spot-check results (6 layers)
 
-*Phase 1+2 running on Modal H100. Results will be posted here and in the [Interpretability Diaries](https://divinci.ai/blog/architecture-every-llm-converges-to/) once complete.*
+*Phase 1 confirmed on Modal L4 (all 61 layers running). Phase 2+3 pending.*
 
 | Constant | Description | Expected (fp16 dense) | Kimi-K2 (MoE) |
 |----------|-------------|----------------------|----------------|
-| **C1** | FFN activation sparsity | 0.06–0.39 | pending |
+| **C1** | FFN activation sparsity | 0.06–0.39 | pending (Phase 3) |
 | **C2** | Top-8 output concentration | 99.7% (at MoE scale) | 99.7%† |
-| **C3** | Gate coherence | 0.53–0.81 | pending |
-| **C4** | Layer temperature | 0.012–0.042 | pending |
-| **C5** | Circuit stage count | 4 (fp16) | pending |
+| **C3** | Gate coherence | 0.53–0.81 | pending (Phase 1 full) |
+| **C4** | Layer temperature (var@64) | 0.80–0.90 (fp16) | **0.037–0.10** ‡ |
+| **C5** | Circuit stage count | 4 (fp16) | pending (Phase 3) |
 
-†*`num_experts_per_tok: 8` confirmed from `config.json`. Four organizations (Google, OpenAI, Alibaba, Moonshot AI) have independently converged on top-8 routing for frontier MoE.*
+†*`num_experts_per_tok: 8` confirmed from `config.json`.*
 
-**Prediction**: var@64 ≈ 0.80–0.90 (power-law, same as all other fp16/bf16 models) and C5 = 4 (circuit intact). The MoE architecture shouldn't change this — per-expert weight matrices are bf16-equivalent.
+‡**Key finding: fp8-native training produces a flat weight spectrum.** Spot-check results:
+
+| Layer | Type | var@64 | IQR / note |
+|-------|------|--------|------------|
+| L00 | Dense MLP | 0.037 | S[:3]=[10.7, 8.6, 8.1] — flat spectrum |
+| L01 | MoE (384 experts) | 0.082 | IQR=[0.076, 0.087] |
+| L15 | MoE | 0.100 | IQR=[0.090, 0.110] |
+| L30 | MoE | pending (full run) | — |
+| L45 | MoE | pending | — |
+| L60 | MoE | pending | — |
+
+**This result is surprising and genuine.** The fp8 dequantization is verified correct (scale_inv stats: mean=0.00028, cv=0.41, dequantized weight std=0.023, abs_max=0.93 — realistic fp16-equivalent range). The flat spectrum is a property of fp8-native training, not a measurement artifact.
+
+For fp16/bf16 models (Gemma-4, Qwen3, Llama-3), var@64 ≈ 0.80–0.90 with a clear power-law singular value spectrum (S[0]/S[1] ≈ 10–100). For Kimi-K2, S[0]/S[1] = 10.7/8.6 ≈ 1.24 — nearly uniform.
+
+**Hypothesis**: fp8 native training (rather than post-quantization) causes the learned representation to spread across more singular directions, flattening the spectrum. This may be due to fp8 gradient noise acting as a distributional regularizer during training, or due to the extreme training scale (14.7T tokens at 1T params).
 
 ## MoE-specific methodology
 
@@ -146,4 +191,12 @@ CC-BY-NC 4.0 — free for non-commercial research use. For commercial use, conta
 
 ---
 
+## Changelog
+
+| Date | Update |
+|------|--------|
+| 2026-04-23 | Initial publish. Phase 1 spot-check complete (6/61 layers). Key finding: fp8-native spectral dissolution confirmed. Full Phase 1 running. |
+
+---
+
 *Part of the [Interpretability Diaries](https://divinci.ai/blog/architecture-every-llm-converges-to/) research series. Working in public at [github.com/Divinci-AI](https://github.com/Divinci-AI).*